An image processing method, apparatus and medium

Abstract

This disclosure provides one or more embodiments of an image processing method, apparatus, and medium. The method includes: performing convolution processing on a target image to be processed to obtain a feature map; inputting the feature map into a ViT (Vision Transformer), and sequentially extracting features from the feature map through coding blocks in the ViT until a feature map output by the last coding block is obtained; the coding block extracts features from the input feature map, including: performing convolution operations on the feature map input to the coding block to obtain local features; performing downsampling processing on the feature map input to the coding block, and performing global feature extraction on the downsampling feature map through a self-attention mechanism module to obtain global features; fusing the local features and the global features to obtain a first fused feature; and obtaining a feature map output by the coding block based on the first fused feature, wherein the output feature map represents the image features of the feature map input to the coding block.

Description

Technical Field

[0001] This specification relates to computer vision technology in one or more embodiments, and more particularly to an image processing method, apparatus and medium. Background Technology

[0002] The Transformer framework was first proposed for natural language processing. Due to its use of self-attention (rather than local convolution) to capture global contextual information, it has proven to be a powerful feature extraction paradigm, exhibiting excellent performance in machine translation tasks. Vision Transformer (ViT) is a pioneering work applying Transformer to computer vision tasks. It applies the coded blocks from the standard Transformer to vision tasks, dividing the input image into blocks and analogizing these image blocks to a word in a natural language processing task.

[0003] ViT shows great promise for achieving higher performance in various vision tasks, demonstrating high performance in image processing. However, ViT has high computational complexity and low computational efficiency, which results in significant resource consumption and low processing efficiency when applying ViT in engineering practice. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide an image processing method, apparatus, and medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, an image processing method is provided, the method comprising:

[0007] The target image to be processed is convolved to obtain a feature map;

[0008] The feature map is input into ViT, and features are extracted from the feature map by sequentially passing through the coding blocks in ViT until the feature map output by the last coding block is obtained.

[0009] Specifically, the coding block in ViT performs feature extraction on the input feature map, including:

[0010] Perform convolution operation on the feature map of the input coded block to obtain local features;

[0011] The feature map of the input coding block is downsampled, and the downsampled feature map is then used by a self-attention mechanism module to extract global features to obtain global features.

[0012] The local features and the global features are fused to obtain the first fused feature;

[0013] Based on the first fusion feature, a feature map output by the coding block is obtained, and the output feature map is used to represent the image features of the feature map input to the coding block.

[0014] According to a second aspect of one or more embodiments of this specification, an image processing apparatus is provided, the apparatus comprising:

[0015] The image preprocessing unit is used to perform convolution processing on the target image to be processed to obtain a feature map;

[0016] ViT image coding unit, the ViT image coding unit includes at least two coding blocks, and the at least two coding blocks are concatenated;

[0017] The ViT image coding unit is used to receive the feature map and extract features from the feature map sequentially through each coding block included in the ViT image coding unit until the feature map output by the last coding block is obtained.

[0018] The coding block in the ViT image coding unit performs feature extraction on the input feature map, specifically including: performing convolution operation on the input feature map of the coding block to obtain local features; performing downsampling processing on the input feature map of the coding block, and performing global feature extraction on the downsampled feature map through a self-attention mechanism module to obtain global features; fusing the local features and the global features to obtain a first fused feature; and obtaining the output feature map of the coding block based on the first fused feature, wherein the output feature map is used to represent the image features of the input feature map of the coding block.

[0019] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0020] processor;

[0021] Memory used to store processor-executable instructions;

[0022] The processor implements the method of any embodiment of this specification by running the executable instructions.

[0023] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the methods of any embodiment of this specification.

[0024] The image processing method, apparatus, electronic device, and medium of this disclosure reduce the computational complexity of ViT and improve its processing efficiency by downsampling the feature map before inputting it into the self-attention mechanism module of the coding block of ViT. Furthermore, by extracting local and global features from the feature map and obtaining the feature map based on the fused features, the coding block retains both local and global features during the processing of the feature map, which helps to ensure the accuracy and high performance of ViT. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in one or more embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in one or more embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an exemplary embodiment of a ViT framework architecture diagram.

[0027] Figure 2 This is a schematic diagram of the structure of an coded block provided in an exemplary embodiment.

[0028] Figure 3 This is a flowchart of an image processing method provided in an exemplary embodiment.

[0029] Figure 4 This is a schematic diagram of the structure of an coded block provided in an exemplary embodiment.

[0030] Figure 5 This is an exemplary embodiment of the output flowchart of a global feature.

[0031] Figure 6 This is a schematic diagram of the structure of another coded block provided in an exemplary embodiment.

[0032] Figure 7 This is a flowchart of another image processing method provided in an exemplary embodiment.

[0033] Figure 8 This is a schematic diagram of a feature fusion method in an coded block provided in an exemplary embodiment.

[0034] Figure 9 This is a schematic diagram of a weight matrix provided in an exemplary embodiment.

[0035] Figure 10a This is a schematic diagram of a feature fusion method in an coded block provided in an exemplary embodiment.

[0036] Figure 10b This is a schematic diagram of another feature fusion method in an coded block provided by an exemplary embodiment.

[0037] Figure 10c This is a schematic diagram of another feature fusion method in an coded block provided by an exemplary embodiment.

[0038] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0040] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0041] The image processing method of this disclosure is to process images using the ViT (Vision Transformer, Transformer for vision) framework. Figure 1 This is an exemplary embodiment providing a framework architecture diagram of ViT, such as... Figure 1 As shown, when ViT is applied to basic image classification, it can include multiple stages, for example, Figure 1 The examples in the text refer to stages 1, 2, ..., n, where n is a natural number. For example, n could be 4. These multiple stages can represent multi-level processing of the input ViT image.

[0042] like Figure 1As shown, each stage may include at least one sequentially connected coding block (ViT block), the structure of which will be described later. For example, layer normalization (LN) and fully connected layers (FC) may be set between two adjacent stages.

[0043] Combination Figure 1 The network structure shown allows for feature extraction of the image from left to right, sequentially passing through each coding block of each stage. After a stage outputs a feature map, this map is processed by LN and FC before being input into the coding block of the next stage. This process continues until the final stage (stage n) outputs its feature map, which is then input into the classifier to obtain the image classification result.

[0044] The image processing method of this disclosure improves the network structure of ViT, mainly by improving the structure of the coding blocks in ViT. The improved structure of ViT and the process of image processing using the improved ViT will be described below.

[0045] Figure 2 This is a schematic diagram of the structure of an coded block provided in an exemplary embodiment. Figure 3 This is a flowchart of an exemplary embodiment of an image processing method, combined with... Figure 2 and Figure 3 As shown, the method may include:

[0046] First, the target image to be processed is convolved to obtain a feature map, which includes multiple image blocks.

[0047] In this embodiment, the image to be processed can be referred to as the "target image". For example, the target image can be an H*W*3 RGB image. In this embodiment, the target image can be encoded into multiple image patches through two consecutive convolutions (for example, a convolution kernel size of 3*3). These image patches can be regarded as words (tokens) in natural language processing tasks.

[0048] For example, after the two consecutive convolutions described above, a feature map can be obtained, which may include multiple image patches. For instance, each pixel in the feature map can be referred to as an image patch. Of course, image patches of other sizes can also be defined; this embodiment does not impose any limitations.

[0049] Next, the feature map obtained by the above convolution process can be input into the ViT of this embodiment of the present disclosure, and multiple image blocks in the feature map can be extracted sequentially through multiple coding blocks in the ViT until the feature map output by the last coding block is obtained. The feature map output by the last coding block is the image feature of the target image finally extracted.

[0050] This image feature can be used for subsequent processing. Depending on the specific image processing requirements, the image feature can be input into different processing networks. For example, for image classification, the final image feature can be input into a classifier (which may include average pooling layers and fully connected layers) to determine the category of the target image. Alternatively, if the goal is to detect a specific object within an image, the image feature can be input into the object detection network to identify the target object within the image.

[0051] In addition, combined Figure 1 As shown, during the process of extracting features from multiple image blocks in the feature map through multiple coding blocks in ViT, as the stage processing progresses, for example, stage 1, stage 2... stage n, the number of image blocks gradually decreases and the number of channels gradually increases in later processing stages.

[0052] For example, taking ViT, which includes four stages, as an example, the initial target image size is H*W*3. After the initial two convolutions, the size of the feature map input to ViT is... That is, the number of channels in the feature map is C, and the number of image patches in each channel is In stage 2 processing, the size of the feature map becomes That is, the number of channels is 2C, and the number of image blocks per channel is In stage 3 processing, the size of the feature map is That is, the number of channels is 4C, and the number of image blocks per channel is In stage 4 processing, the size of the feature map is It is evident that as the ViT processing layer deepens, the number of image blocks decreases while the number of channels increases.

[0053] The processing performed on each coded block in ViT can be combined with... Figure 3 As shown, it should be noted that the step numbers below do not indicate any restriction on the execution order of the steps. For example, steps 300 and 302 can be performed in parallel.

[0054] In step 300, a convolution operation is performed on the feature map of the input coding block to obtain local features (also known as spatial features).

[0055] This step involves performing a convolution operation on the feature map of the input coded block to obtain local features. In the embodiments of this specification, these local features are referred to as the first feature.

[0056] Combination Figure 2 As illustrated, the coding block (ViT block) may include a spatial module 21 (SM) and a semantic module 22 (CM).

[0057] The spatial module 21 can be used to perform convolution operations on the input feature map to obtain local features. For example, it can be encoded using a depthwise separable convolution with a kernel size of 3*3; or it can be encoded using standard convolution, dilated convolution, or other methods.

[0058] In step 302, the feature map of the input coding block is downsampled, and the feature map after downsampling is used to extract global features (also known as semantic features) through a self-attention mechanism module.

[0059] In this embodiment, global features can also be extracted from the feature map through the semantic module 22. In the description of this embodiment, the global feature can be referred to as the second feature.

[0060] like Figure 2 As shown, the semantic module 22 may include a self-attention mechanism module 221. For example, this self-attention mechanism module 221 may be a window-based self-attention mechanism module. In this embodiment, a downsampling module 222 is added before the feature map is input into the self-attention mechanism module 221. The feature map is first downsampled by this downsampling module 222, and then the downsampled feature map is input into the self-attention mechanism module 221. The downsampling process may be convolutional downsampling, pooling, or other similar processing methods.

[0061] In one exemplary manner, the downsampling module 222 may include a convolutional block for fast downsampling, which performs three consecutive convolutions on the feature map to obtain high-level semantic information. This convolutional block may include a 1x1 convolution, a 3x3 separable convolution with a stride of 2 for downsampling, and a 1x1 projective convolution for integrating channel information. Subsequently, the self-attention mechanism module 221 may employ N (N is a natural number) window-based self-attention mechanism modules to further extract intermediate and high-level (i.e., long-range) features, i.e., global features.

[0062] By downsampling the feature map before inputting it into the self-attention mechanism module 221, computational and memory costs can be reduced, and the receptive field can be expanded. ViT has high computational complexity and low computational efficiency. Its bottleneck lies in the fact that the size of the self-attention mechanism module is related to the size of the input image; the larger the image input to the self-attention mechanism module, the greater the computational load. Based on this, this embodiment reduces the size of the feature map by downsampling it before inputting it into the self-attention mechanism module, thereby improving the processing efficiency of ViT and reducing memory costs. For example, if the feature map input to the window-based self-attention mechanism module is downsampled to 1 / 4 of its original size, the computational complexity of ViT will also be significantly reduced.

[0063] In step 304, the local features and global features are fused to obtain a first fused feature, and the feature map output by the coding block is obtained based on the first fused feature.

[0064] like Figure 2 As shown, the second feature extracted by semantic module 22 and the first feature extracted by spatial module 21 are complementary. Specifically, during the extraction of global features, semantic module 22 may result in the loss of fine-grained details. However, when using ViT for image recognition tasks, preserving fine-grained spatial information is crucial for high performance. Spatial module 21 can precisely compensate for the lack of detailed local features in the image; it can extract local features from the feature map through convolution.

[0065] In this step, by fusing the first and second features, both global and local features can be preserved during feature extraction. Preserving these features facilitates better extraction of semantic information from the feature map, ensuring the performance of the image processing task performed using ViT. Furthermore, the spatial module 21 in this embodiment has high convolution efficiency and processes in parallel with the semantic module 22, without affecting the overall efficiency of ViT.

[0066] Furthermore, the step of obtaining the feature map output by the coding block based on the first fusion feature can include various implementation methods. For example, the coding block in ViT, in addition to... Figure 2 In addition to the self-attention mechanism module shown, a feedforward network (FFN) is also included, which is located after the self-attention mechanism module. The fusion of the first feature and the second feature can be performed after the self-attention mechanism module and before the feedforward network, or it can be performed after the feedforward network. This embodiment does not limit the location of the fusion.

[0067] The feature map output from the coded block in this embodiment can be fed into the next coded block. Combined with... Figure 1 As shown, if the encoded block is the last encoded block in a stage, it can be processed by LN and FC before being input into the first encoded block of the next stage for further processing. It is understandable that if the encoded block is the last encoded block in the ViT framework, its output feature map can be fed into subsequent processing networks such as classifiers.

[0068] The image processing method in this embodiment reduces the computational complexity of ViT and improves its processing efficiency by downsampling the feature map before inputting it into the self-attention mechanism module of the coding block of ViT. Furthermore, by extracting local and global features from the feature map and obtaining the feature map based on the fused features, the coding block retains both local and global features during the processing of the feature map, which helps to ensure the accuracy and high performance of ViT.

[0069] In another embodiment, this disclosure further improves the ViT block by optimizing the structure of the semantic modules in the coding block and compensating for information loss caused by downsampling of the semantic modules. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the structure of an coded block provided in an exemplary embodiment.

[0070] Figure 5 This is an exemplary embodiment provided according to Figure 4 The output flowchart of global features executed by the coding block structure, i.e. Figure 5 The process by which the semantic module obtains global features is described, combined with Figure 4 and Figure 5 The method may include the following processes, wherein the steps that are the same as those in the foregoing embodiments will be briefly described, and for details, please refer to other embodiments.

[0071] In step 500, the feature map of the input coded block is downsampled.

[0072] For example, this step can use a convolutional block for fast downsampling to perform three consecutive convolutions on the feature map to obtain high-level semantic information.

[0073] In step 502, feature compensation processing is performed on the feature map of the input coded block to obtain compensated features.

[0074] The downsampling of the feature map in step 500 results in the loss of some information, particularly salient features. This step addresses this by using feature compensation to recover the lost salient features. For example, this feature compensation can be achieved using max pooling, a parameter-free method that is simple to implement and does not increase the number of parameters in the ViT network.

[0075] In this embodiment, the features obtained through feature compensation processing can be called compensated features. For example, compensated features can be obtained by performing maxpooling processing on the feature map of the input coded block.

[0076] In step 504, the compensated features are fused with the feature map after downsampling to obtain the second fused features.

[0077] In this step, since the fusion of the compensated features with the downsampled feature map occurs before the Self-Attention (SA) module 221, it can be called Pre-SA fusion. Figure 4 As shown, a second fusion feature can be obtained after Pre-SA fusion, and the second fusion feature is input into the self-attention mechanism module 221 for further processing.

[0078] In addition, such as Figure 4 As shown, correspondingly, the first feature and the second feature mentioned earlier are fused to obtain the first fused feature. The fusion position is after the processing of the self-attention mechanism module (Self-Attention, SA)221, so it can be called Post-SA fusion.

[0079] In step 506, the second fused feature is processed by the self-attention mechanism module to obtain the attention output feature.

[0080] For example, the self-attention mechanism module 221 in this embodiment can perform layer normalization (LN) processing before processing the second fused feature using the self-attention mechanism, and then perform window-based self-attention mechanism processing. In this embodiment, the features output by the self-attention mechanism module 221 can be referred to as attention output features.

[0081] In step 508, the attention output features and the second fused features are fused to obtain global features.

[0082] like Figure 4 As shown, the attention output features output by the self-attention mechanism module 221 can be fused with the second fusion features to obtain the global features output by the semantic module 22, i.e., the second features.

[0083] The following formula (1) illustrates how global features are obtained:

[0084]

[0085] As above, Represents global features. This represents the feature map in the input coded block; This indicates that the feature map is subjected to max pooling, meaning that the feature compensation process is based on max pooling. The downsampling module 222 in semantic module 22 represents the processing of the feature map. In one example, this could be achieved by performing three consecutive convolutions on the feature map using a convolutional block for fast downsampling.

[0086] That is, the second fusion feature. That is, the attention output feature output by the self-attention mechanism module 221. This attention output feature can be obtained by first performing a layer normalization operation LN on the second fusion feature and then processing it with the self-attention mechanism.

[0087] The method in this embodiment adds feature compensation processing to the semantic module of the coding block and fuses the compensated features with the feature map after downsampling. This can compensate for the information loss caused by downsampling, thereby enabling better extraction of semantic information and improving the accuracy of ViT.

[0088] Furthermore, as mentioned in the above embodiments, this disclosure does not limit the fusion position of the first feature and the second feature. In yet another embodiment, this disclosure illustrates a coding block structure in which the fusion position of the first feature and the second feature is set after the self-attention mechanism module and before the feedforward network.

[0089] Figure 6 This is a schematic diagram of the structure of an coded block provided in an exemplary embodiment, such as... Figure 6 As shown, in addition to spatial module 21 and semantic module 22, the coding block also includes a feedforward network 23. For example, the feedforward network 23 may include an LN and an MLP (Multilayer Perceptron). Figure 7 This is an exemplary embodiment provided according to Figure 6 The image processing method performed by the coded block structure may include the following processing, wherein the steps that are the same as those in the foregoing embodiments will be briefly described, and for details, please refer to other embodiments.

[0090] In step 700, a convolution operation is performed on the feature map of the input coding block to obtain local features, namely the first feature.

[0091] In step 702, global features are extracted from the feature map of the input coding block to obtain the global features, i.e., the second features.

[0092] In step 704, the first feature and the second feature are fused to obtain the first fused feature.

[0093] In step 706, the feedforward network in the first fused feature input coding block is subjected to feature processing to obtain the feature map output by the coding block.

[0094] In this step, the first fused feature can be input into the feedforward network in the coding block. For example, in the feedforward network, it can first undergo LN layer normalization processing, followed by MLP processing. Please refer to... Figure 6 As shown, the features output by the feedforward network can be fused with the first fusion feature to finally obtain the feature map output by the coding block.

[0095] For example, the following formula (2) represents the above processing procedure:

[0096]

[0097] As above, This represents the first fusion feature. This means that the first fused feature is processed sequentially through the LN and MLP in the feedforward network. This represents the feature map of the final output of the coded block.

[0098] The method in this embodiment performs the fusion of local and global features after the self-attention mechanism module and before the feedforward network. This allows the local and global features to complement and combine better. Furthermore, since the fusion of local and global features occurs before the feedforward network in the coding block, the complexity of the spatial module is lower, which reduces the complexity of the ViT network, decreases the consumption of computing and memory resources, and thus improves the processing efficiency of the ViT network.

[0099] Figure 8 This is a schematic diagram of a feature fusion method in an coded block provided by an exemplary embodiment. In this embodiment, it mainly provides an exemplary method for fusing local and global features to better combine these two types of features. Figure 8 The fusion module shown receives a first feature output by the spatial module and a second feature output by the semantic module, and fuses these two different types of features to obtain a first fused feature.

[0100] For example, the first and second features both have a size of L*C, meaning the feature map includes C channels, and each channel contains L image blocks. Since the features extracted by the semantic module lack inter-channel interaction, this embodiment employs a channel attention mechanism in the fusion module to calculate the second feature, thereby enhancing the useful features in the channel dimension of the second feature output by the semantic module and suppressing noise information.

[0101] For example, the channel attention mechanism module in this embodiment can employ a Squeeze-and-Excitation (SE) network. This might involve first performing global average pooling, followed by two fully connected layers and ReLU. Ultimately, a second fusion weight is obtained, consisting of C weights. This means the channel attention module outputs weights corresponding to the C channels, learning attention weights along the channel dimension to represent the importance of each channel in the feature map.

[0102] The following formula (3) represents the method for obtaining the second fusion weight:

[0103]

[0104] in, Indicates the second characteristic, This indicates that the second feature underwent SE processing in the channel attention mechanism module. The second fusion weight is obtained. Since the feature map output by the semantic module has C channels, a total of C second fusion weights can be obtained.

[0105] For the first feature output by the spatial module, the feature map contains rich fine-grained local features but lacks global semantic information. Therefore, in this embodiment, the first feature is processed by the spatial attention mechanism module in the fusion module to learn the weight of the spatial dimension, enhance the useful spatial regions in the feature map, and suppress irrelevant spatial regions.

[0106] For example, the spatial attention mechanism module in this embodiment can be processed using a fully connected layer, and the final weights can be called the first fusion weights. Since the feature map output by the spatial module has L image patches in each channel, the number of learned first fusion weights is L. That is, the spatial attention mechanism module outputs weights corresponding to L spatial regions in each channel, and learns the importance of each feature in the spatial dimension.

[0107] The following formula (4) shows how the first fusion weight is obtained:

[0108]

[0109] in, Indicates the first feature, This indicates that the first feature was processed by a fully connected layer in the spatial attention mechanism module. This represents the first fusion weight obtained.

[0110] Please combine Figure 8 As shown, for the L first fusion weights calculated above... and C second fusion weights It can be expanded. For example, by combining... Figure 9 The illustration assumes C=4 and L=6, meaning the feature map has 4 channels, and each channel has 6 image patches. The second fusion weight mentioned above... It can include: L first fusion weights It can include:

[0111] like Figure 9 As shown, each image patch in each channel of the feature map can be set as the second fusion weight corresponding to that channel. For example, Figure 9 In the left-hand weight matrix, the second fusion weights of each image patch in the first channel are... (that is, the weight corresponding to the first channel is) The second fusion weights for each image patch in the second channel are... And so on. Similarly, Figure 9 In the weight matrix on the right, the first fusion weights of each image patch in the first channel are respectively (That is, the weights corresponding to each spatial region respectively). Similarly, the first fusion weights of the image patches on other channels are also respectively. to

[0112] Next, you can first... Figure 9 The two weight matrices on the left and right sides of the matrix are added together, that is, the two weights at corresponding positions in the matrix are added together. Figure 9 For the image patch in the upper left corner, the first fusion weight corresponding to this image patch is The corresponding second fusion weight is These two weights can be added together. Then add the two weights together. By using the Sigmoid function transformation, it is transformed to the (0,1) interval, i.e. The first and second fusion weights corresponding to each image block are processed in the same way as described above.

[0113] Then, if the values ​​of each image patch after being transformed by the Sigmoid function are represented by W, the fusion of the first and second features can be described by the following formula (5):

[0114]

[0115] in, Indicates the first fusion feature; Indicates the first feature, This indicates the second characteristic.

[0116] Alternatively, the first and second features can be fused using the following formula (6):

[0117]

[0118] The method in this embodiment learns the weights of the spatial dimension by using a spatial attention module on the features output by the spatial module and learns the weights of the channel dimension by using a channel attention module on the features output by the semantic module. This can better enhance the important features in both the channel dimension and the spatial dimension, and make the features output by the semantic module and the features output by the spatial module complement each other. Each image block in the feature map can simultaneously obtain high-level global features and low-level local features. The fusion of these different levels of information and the enhancement of important features through the attention mechanism all contribute to improving the performance of ViT.

[0119] Furthermore, the method of fusing the first feature and the second feature in the embodiments of this disclosure is not limited to... Figure 8 The fusion method shown in the example. For example, Figure 10a , Figure 10b and Figure 10c Several other fusion methods are illustrated. All three methods process the first feature and the second feature through the attention mechanism module to obtain the first fusion weight corresponding to the first feature and the second fusion weight corresponding to the second feature. Based on the first fusion weight and the second fusion weight, the first feature and the second feature are fused to obtain the first fused feature.

[0120] Among them, Figure 10a In the fusion method shown, both the first and second features are processed through the channel attention mechanism module; for example, both undergo SE processing. Figure 10b In the fusion method shown, both the first and second features are processed through a spatial attention mechanism module. Figure 10c In the fusion method shown, the first feature is processed through the channel attention mechanism module, and the second feature is processed through the spatial attention mechanism module.

[0121] In other implementations, the fusion of the first and second features can also be achieved without using an attention mechanism module to obtain fusion weights. For example, the first and second features can be directly summed (SUM), that is, the feature values ​​at corresponding positions in the two feature maps are added together. Alternatively, the first and second features can be multiplied to obtain a multiplicative result (MUL), the first and second features can be added together to obtain an additive result (SUM), and the additive result and the multiplicative result can be added together again, i.e., SUM+MUL.

[0122] The ViT architecture provided in this disclosure can be applied to various scenarios. For example, it can be used for basic image classification, where the feature map output from the last coding block of ViT can be input into a classifier to obtain the category to which the target image belongs. Alternatively, the ViT architecture can also be applied to semantic segmentation, object detection, and other purposes; in this case, the classifier can be replaced with the corresponding network structure for semantic segmentation or object detection.

[0123] For example, taking a map application scenario, this ViT architecture can be applied to crowdsourced map updates and change detection. For instance, it can acquire a target image to be processed, which may be a road image. The road image can be uploaded by a user or collected by a data collection vehicle, etc.

[0124] After obtaining the road image, the road image can be input into the ViT network of this embodiment of the disclosure, and then... Figure 1 The multi-stage feature extraction process, as shown, yields the image features, specifically the feature map output by the last coding block. This feature map is then input into the corresponding network structure for object detection, identifying the road elements included in the road image. These road elements may include, but are not limited to, traffic poles, lane lines, and road signs. By comparing the identified road elements with existing road elements in the map, changes in road elements can be detected, such as the addition of a traffic pole, allowing for map updates.

[0125] Because the ViT network provided in this embodiment has high processing efficiency and superior accuracy, it can more quickly identify road elements in road images and achieve better recognition results, thereby improving the efficiency and accuracy of business processing.

[0126] The ViT network of this disclosure embodiment can be trained in a supervised manner. For example, it can be trained using a sample image set used to train the network. Figure 1The ViT network shown can be used to classify each sample image in the sample image set. The network parameters of ViT are adjusted according to the difference between the predicted class output by ViT and the class label.

[0127] Furthermore, the ViT network provided in this disclosure achieves good efficiency and performance across different model sizes. For example, the feature maps input to the ViT network can have different numbers of channels; regardless of the number of channels, the ViT network can complete the processing with high efficiency and high accuracy.

[0128] To implement the method of any embodiment of this disclosure, this disclosure also provides an image processing apparatus. The structure of the apparatus is briefly described below, wherein the processing of each module can be combined with the method embodiments described. Figure 11 This is an exemplary embodiment of an image processing apparatus, such as... Figure 8 As shown, the device may include an image preprocessing unit 81 and a ViT image encoding unit 82.

[0129] Image preprocessing unit 81 is used to perform convolution processing on the target image to be processed to obtain a feature map.

[0130] The ViT image coding unit 82 includes at least two coding blocks, and the at least two coding blocks are concatenated. The structure of the ViT image coding unit 82 can be seen in conjunction with [reference needed]. Figure 1 This is an illustration. The ViT image coding unit 82 is used to receive the feature map and sequentially extract features from the feature map through each coding block included in the ViT image coding unit until the feature map output by the last coding block is obtained.

[0131] The encoding block in the ViT image encoding unit 82 performs feature extraction on the input feature map, specifically including: performing convolution operation on the feature map of the input encoding block to obtain local features; performing downsampling processing on the feature map of the input encoding block, and performing global feature extraction on the downsampled feature map through a self-attention mechanism module to obtain global features; fusing the local features and the global features to obtain a first fused feature; and obtaining the feature map output by the encoding block based on the first fused feature, wherein the output feature map is used to represent the image features of the feature map of the input encoding block.

[0132] In one example, the ViT image coding unit 82, when performing feature extraction on the input feature map, further includes: performing feature compensation processing on the feature map of the input coding block to obtain compensated features. When the coding block in the ViT image coding unit 82 performs global feature extraction on the downsampled feature map through a self-attention mechanism module to obtain global features, it includes: fusing the compensated features with the downsampled feature map to obtain a second fused feature; processing the second fused feature through the self-attention mechanism module to obtain an attention output feature; and fusing the attention output feature and the second fused feature to obtain the global feature.

[0133] In one example, the ViT image coding unit 82, when obtaining the feature map output by the coding block based on the first fusion feature, includes: further processing the first fusion feature through a feedforward network in the coding block to obtain the feature map output by the coding block.

[0134] In one example, the ViT image coding unit 82, when fusing the local features and global features to obtain a first fused feature, includes: performing feature processing on the local features and global features respectively through an attention mechanism module to obtain a first fusion weight corresponding to the local features and a second fusion weight corresponding to the global features; and fusing the local features and global features according to the first fusion weight and the second fusion weight to obtain the first fused feature.

[0135] In one example, the ViT image coding unit 82, when performing feature processing on the local features and global features through an attention module to obtain a first fusion weight corresponding to the local features and a second fusion weight corresponding to the global features, includes: performing feature processing on the local features through a spatial attention mechanism module to obtain the first fusion weight; and performing feature processing on the global features through a channel attention mechanism module to obtain the second fusion weight.

[0136] In one example, the target image to be processed is a road image; the ViT image encoding unit 82 is also used to perform target detection processing based on the feature map output by the last encoding block, and to identify the road elements included in the road image.

[0137] This disclosure also provides an electronic device. At the hardware level, the device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as by the processor reading the corresponding computer program from non-volatile memory into memory and then running it. The processor implements the method of any embodiment of this specification by executing the executable instructions. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0138] This disclosure also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the methods of any embodiment of this specification.

[0139] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0140] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] This disclosure also provides a computer program product for implementing the image processing method described in any embodiment of this disclosure.

[0144] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0146] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0147] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0148] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. An image processing method, characterized by, The method includes: A feature map is obtained by performing convolution on the target image to be processed, and the feature map includes multiple image blocks; The feature map is input into ViT, and features are extracted from the feature map by sequentially passing through the coding blocks in ViT until the feature map output by the last coding block is obtained. Specifically, the coding block in ViT performs feature extraction on the input feature map, including: Perform convolution operation on the feature map of the input coded block to obtain local features; The feature map of the input coding block is downsampled to obtain semantic information. The downsampled feature map is then subjected to global feature extraction through a self-attention mechanism module to obtain global features. The local features and the global features are fused to obtain the first fused feature; Based on the first fusion feature, a feature map output by the coding block is obtained, and the output feature map is used to represent the image features of the feature map input to the coding block.

2. The method of claim 1, wherein, The coding block in ViT extracts features from the input feature map, and also includes: The feature map of the input coded block is subjected to feature compensation processing to obtain compensated features; The feature map after downsampling is subjected to global feature extraction through a self-attention mechanism module to obtain global features, including: The compensated feature is fused with the feature map after downsampling to obtain the second fused feature; The second fused feature is processed by the self-attention mechanism module to obtain the attention output feature; The global feature is obtained by fusing the attention output feature and the second fusion feature.

3. The method of claim 2, wherein, The step of performing feature compensation processing on the feature map of the input coded block to obtain compensated features includes: The feature map is subjected to max pooling to obtain the compensated features.

4. The method of claim 1, wherein, The step of obtaining the feature map output by the coding block based on the first fusion feature includes: The first fused feature is processed by the feedforward network in the coding block to obtain the feature map output by the coding block.

5. The method according to any one of claims 1 to 4, characterized in that, The step of fusing the local features and the global features to obtain the first fused feature includes: The local and global features are processed by the attention mechanism module to obtain the first fusion weight corresponding to the local features and the second fusion weight corresponding to the global features. Based on the first fusion weight and the second fusion weight, the local features and global features are fused to obtain the first fused feature.

6. The method of claim 5, wherein, The step of processing the local and global features using an attention mechanism module to obtain a first fusion weight for the local features and a second fusion weight for the global features includes: The local features are processed by the spatial attention mechanism module to obtain the first fusion weight; The global features are processed by the channel attention mechanism module to obtain the second fusion weight.

7. The method according to claim 1, characterized in that, The target image to be processed is: a road image; The method further includes: performing target detection processing based on the feature map output by the last coding block to identify the road elements included in the road image.

8. An image processing apparatus characterized by comprising: The device includes: An image preprocessing unit is used to perform convolution processing on the target image to be processed to obtain a feature map, wherein the feature map includes multiple image blocks; ViT image coding unit, the ViT image coding unit includes at least two coding blocks, and the at least two coding blocks are concatenated; The ViT image coding unit is used to receive the feature map and extract features from the feature map sequentially through each coding block included in the ViT image coding unit until the feature map output by the last coding block is obtained. The coding block in the ViT image coding unit performs feature extraction on the input feature map, specifically including: performing convolution operation on the feature map of the input coding block to obtain local features; performing downsampling processing on the feature map of the input coding block to obtain semantic information; performing global feature extraction on the downsampled feature map through a self-attention mechanism module to obtain global features; fusing the local features and the global features to obtain a first fused feature; and obtaining the feature map output by the coding block based on the first fused feature, wherein the output feature map is used to represent the image features of the feature map of the input coding block.

9. The apparatus according to claim 8, characterized in that, The coding block in the ViT image coding unit, when used for feature extraction of the input feature map, further includes: performing feature compensation processing on the feature map of the input coding block to obtain compensated features; The coding block in the ViT image coding unit, when used to extract global features from the feature map after downsampling through the self-attention mechanism module to obtain global features, includes: The compensated feature is fused with the feature map after downsampling to obtain the second fused feature; The second fused feature is processed by the self-attention mechanism module to obtain the attention output feature; The global feature is obtained by fusing the attention output feature and the second fusion feature.

10. A computer readable storage medium having stored thereon computer instructions, wherein, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images