Image processing method and system based on transformer

Abstract

The application provides a Transformer-based image processing method and a system thereof, wherein the image processing method comprises the following steps: acquiring an image sample; performing local feature extraction on the image sample through a convolutional neural network to obtain a two-dimensional feature map; converting the two-dimensional feature map into a first one-dimensional feature sequence; adding a feature vector in the first one-dimensional feature sequence to a first position encoding to obtain a first one-dimensional encoded feature sequence; performing global feature extraction on the first one-dimensional encoded feature sequence through a Transformer to obtain a one-dimensional to-be-processed feature sequence; and performing at least one superposition operation on the one-dimensional to-be-processed feature sequence as an input, so that the one-dimensional to-be-processed feature sequence is subjected to global feature extraction of the Transformer at least once to obtain a one-dimensional target feature sequence. The application can strengthen the feature extraction capability of the Transformer.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image processing method and system based on Transformer. Background Technology

[0002] With the continuous development of deep learning, image classification technology has been greatly improved. Classification methods based on Transformer neural networks have demonstrated high performance in various fields.

[0003] Among them, Transformer uses self-attention mechanism as the core to calculate feature similarity. Due to the calculation method of self-attention mechanism, it pays more attention to global features and has a huge computational cost. ViT (Vision Transformer) is the first neural network constructed solely by transformer for visual classification tasks, which verifies the feasibility of using transformer in the vision field. However, ViT uses a single invariant feature size from input to output, which is not suitable for downstream detection tasks. Researchers began to study the construction of multi-scale transformer networks.

[0004] Subsequent visual Transformer networks, such as Swin-Transformer (Shifted windows-Transformer) and PVT (Pyramid Vision Transformer), only modified the transformer network to adapt to the traditional convolutional neural network in form. However, they could not fully achieve the extraction of local features through sliding windows like convolutional neural networks, and thus could not achieve a perfect fusion of local and global features.

[0005] However, for image recognition tasks, some local details are very important. For example, shallow features such as object contours, edges, or object shapes are very important for classification and detection tasks.

[0006] Therefore, how to construct a more general multi-scale feature extraction network, improve the feature extraction capability of the model, reduce the computational load, further expand the application of Transformer in downstream tasks in the vision field, and strengthen the fusion of the initial local features and deep global features of the network has become an urgent problem to be solved. Summary of the Invention

[0007] To address the aforementioned issues, the present invention provides an image processing method and system based on Transformer. By combining convolutional neural networks with Transformer, the feature extraction capability of the combined neural network model can be improved, the computational load reduced, and the fusion of initial local features and deep global features in the neural network strengthened.

[0008] In a first aspect, the present invention provides an image processing method based on Transformer, comprising:

[0009] Obtain image samples;

[0010] A two-dimensional feature map is obtained by extracting local features from image samples using a convolutional neural network.

[0011] Transform the two-dimensional feature map into a first one-dimensional feature sequence;

[0012] Add the feature vector in the first one-dimensional feature sequence to the first position code to obtain the first one-dimensional coded feature sequence;

[0013] The first one-dimensional encoded feature sequence is globally extracted using a Transformer to obtain a one-dimensional feature sequence to be processed.

[0014] The one-dimensional feature sequence to be processed is used as input for at least one superposition operation, so that the one-dimensional feature sequence to be processed is subjected to at least one global feature extraction by Transformer to obtain a one-dimensional target feature sequence.

[0015] Optionally, the overlay operation includes:

[0016] The input one-dimensional feature sequence is downsampled to obtain a second one-dimensional feature sequence.

[0017] The feature vector in the second one-dimensional feature sequence is added to the second position code to obtain the second one-dimensional coded feature sequence;

[0018] The second one-dimensional encoded feature sequence is input into the Transformer for global feature extraction to obtain a one-dimensional global feature sequence.

[0019] The input one-dimensional feature sequence includes: a one-dimensional feature sequence to be processed, or includes: a one-dimensional feature sequence to be processed and a one-dimensional global feature sequence; the one-dimensional global feature sequence obtained by the last superposition operation is the one-dimensional target feature sequence.

[0020] Optionally, the step of downsampling the input one-dimensional feature sequence includes:

[0021] One-dimensional convolution is used to perform a convolution operation on the input one-dimensional feature sequence to obtain a one-dimensional convolutional feature sequence.

[0022] A one-dimensional projected feature sequence is obtained by multiplying the one-dimensional convolutional feature sequence with a preset linear projection matrix.

[0023] The one-dimensional projected feature sequence is normalized to obtain the second one-dimensional feature sequence.

[0024] Optionally, before the step of performing global feature extraction on the first one-dimensional encoded feature sequence using a Transformer, the image processing method further includes:

[0025] The first one-dimensional encoded feature sequence is processed by the first activation function;

[0026] Before the step of inputting the second one-dimensional encoded feature sequence into the Transformer for global feature extraction, the image processing method also includes:

[0027] The second one-dimensional encoded feature sequence is processed by the second activation function.

[0028] Optionally, after the step of performing at least one superposition operation on the one-dimensional feature sequence to be processed as input, the image processing method further includes:

[0029] The elements representing the size of features in a one-dimensional target feature sequence are averaged to obtain a multi-scale one-dimensional mean feature sequence.

[0030] By processing the multi-scale one-dimensional mean feature sequence through a fully connected layer, a one-dimensional class feature sequence is obtained, which includes multiple classes.

[0031] The one-dimensional categorical feature sequence is processed by a normalized exponential function to obtain the probability corresponding to each class.

[0032] Optionally, after the step of performing at least one superposition operation on the one-dimensional feature sequence to be processed as input, the image processing method further includes:

[0033] The first one-dimensional feature sequence, the one-dimensional feature sequence to be processed, and the one-dimensional global feature sequence obtained after each superposition operation are all transformed into a two-dimensional target feature map.

[0034] Secondly, the present invention provides an image processing system based on Transformer, comprising:

[0035] The acquisition module is configured to acquire image samples;

[0036] The local processing module is configured to extract local features from image samples using a convolutional neural network to obtain a two-dimensional feature map.

[0037] The first transformation module is configured to transform the two-dimensional feature map into a first one-dimensional feature sequence;

[0038] The encoding module is configured to add the feature vector in the first one-dimensional feature sequence to the first position code to obtain the first one-dimensional encoded feature sequence;

[0039] The global processing module is configured to perform global feature extraction on the first one-dimensional encoded feature sequence through Transformer to obtain a one-dimensional feature sequence to be processed.

[0040] The overlay operation module is configured to take the one-dimensional feature sequence to be processed as input and perform at least one overlay operation, so that the one-dimensional feature sequence to be processed is subjected to at least one global feature extraction by Transformer to obtain a one-dimensional target feature sequence.

[0041] Optionally, the overlay operation module includes:

[0042] The processing submodule is configured to downsample the input one-dimensional feature sequence to obtain a second one-dimensional feature sequence.

[0043] The encoding submodule is configured to add the feature vector in the second one-dimensional feature sequence to the second position code to obtain the second one-dimensional encoded feature sequence;

[0044] The global processing submodule is configured to input the second one-dimensional encoded feature sequence into the Transformer for global feature extraction, thereby obtaining a one-dimensional global feature sequence.

[0045] The input one-dimensional feature sequence includes: a one-dimensional feature sequence to be processed, or includes: a one-dimensional feature sequence to be processed and a one-dimensional global feature sequence; the one-dimensional global feature sequence obtained by the global processing submodule at the last time is the one-dimensional target feature sequence.

[0046] Optionally, the processing submodule includes:

[0047] The convolutional unit is configured to perform a one-dimensional convolution operation on the input one-dimensional feature sequence to obtain a one-dimensional convolutional feature sequence.

[0048] The multiplication unit is configured to multiply a one-dimensional convolutional feature sequence with a preset linear projection matrix to obtain a one-dimensional projected feature sequence.

[0049] The normalization unit is configured to normalize the one-dimensional projected feature sequence to obtain a second one-dimensional feature sequence.

[0050] Optionally, the image processing system also includes:

[0051] The mean-taking module is configured to take the mean of the elements in the one-dimensional target feature sequence that represent the size of the feature, so as to obtain a multi-scale one-dimensional mean feature sequence.

[0052] The fully connected module is configured to process the multi-scale one-dimensional mean feature sequence through a fully connected layer to obtain a one-dimensional class feature sequence, which includes multiple classes;

[0053] The classification module is configured to process the one-dimensional class feature sequence using a normalized exponential function to obtain the probability corresponding to each class.

[0054] The image processing method and system based on Transformer provided in this invention combine convolutional neural networks with Transformer, making full use of Transformer's global feature extraction capability and deeply integrating it with the local feature aggregation capability of convolutional neural networks. Based on Transformer, the accuracy of the system in recognizing local features is improved, thereby enhancing the feature extraction capability of the neural network model formed by the combination, reducing the amount of computation, and strengthening the fusion of initial local features and deep global features in the neural network. This provides a novel multi-scale feature network architecture for classification tasks and downstream dense prediction tasks in the field of computer vision. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic flowchart illustrating an embodiment of the Transformer-based image processing method of this application;

[0057] Figure 2 This is a schematic diagram illustrating the implementation process of a Transformer-based image processing method according to an embodiment of this application;

[0058] Figure 3 This is a schematic structural diagram of a Transformer-based image processing system according to an embodiment of this application. Detailed Implementation

[0059] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0061] Spatial relation terms such as “below,” “under,” “below,” “under,” “above,” “above,” etc., are used herein to describe the relationship between one element or feature shown in the figure and other elements or features. It should be understood that, in addition to the orientation shown in the figure, spatial relation terms also include different orientations of the device in use and operation. For example, if the device in the figure is flipped, the element or feature described as “below,” “under,” or “below” will be oriented “above” the other element or feature. Therefore, the exemplary terms “below” and “under” can include both above and below orientations. Furthermore, the device may also include other orientations (e.g., rotated 90 degrees or other orientations), and the spatial descriptive terms used herein will be interpreted accordingly.

[0062] It should be noted that when an element is referred to as "fixedly connected" to another element, it can be directly on the other element or there may be an intervening element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intervening element. Conversely, when an element is referred to as being "directly on" another element, there is no intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0063] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0064] Firstly, this embodiment provides an image processing method based on Transformer, see [link to previous document]. Figure 1 and Figure 2The image processing method includes steps S101 to S106:

[0065] Step S101: Obtain image samples.

[0066] The image sample consists of multiple images to be processed, and the number of images to be processed is the batch size, which is determined by system settings. For example, during neural network model training, if there are 1000 image samples, these image samples are divided into 10 batches. The size of each batch is 100, that is, batch size = 100.

[0067] In this embodiment, the image sample serves as the input feature map, with a feature size of HW (H×W) and a feature dimension of C. Here, H and W represent the height and width of the input feature map, and C = 3.

[0068] Step S102: Extract local features from image samples using a convolutional neural network to obtain a two-dimensional feature map.

[0069] The convolutional neural network includes: two-dimensional convolutional (Conv2D) layers, normalization (Norm) layers, and pooling (Pool) layers. This convolutional neural network can be implemented using any existing specific convolutional neural network; this embodiment does not specifically limit the calculation process of the two-dimensional feature maps. It is understood that when multiple images to be processed are convolved, multiple sets of two-dimensional feature maps will be obtained.

[0070] The computational complexity is reduced by using a convolutional neural network to compute a two-dimensional feature map compared to simply using a Transformer.

[0071] Step S103: Convert the two-dimensional feature map into a first one-dimensional feature sequence.

[0072] In this embodiment, a two-dimensional feature map can be converted into a first one-dimensional feature sequence through standard patch embedding operations.

[0073] It should be noted that in this invention, the term "feature sequence" is preceded by "one-dimensional" to indicate that the feature sequence is a one-dimensional feature sequence. For example, "first one-dimensional feature sequence" means that the first one-dimensional feature sequence is a one-dimensional feature sequence.

[0074] Step S104: Add the feature vector in the first one-dimensional feature sequence to the first position code to obtain the first one-dimensional coded feature sequence.

[0075] The first position code can be either an absolute position code or a relative position code; this embodiment does not specifically limit it.

[0076] Step S105: Use Transformer to perform global feature extraction on the first one-dimensional encoded feature sequence to obtain a one-dimensional feature sequence to be processed.

[0077] Step S106: Take the one-dimensional feature sequence to be processed as input and perform at least one superposition operation so that the one-dimensional feature sequence to be processed can be subjected to at least one global feature extraction by Transformer to obtain a one-dimensional target feature sequence.

[0078] The superposition operation includes: downsampling the input one-dimensional feature sequence to obtain a second one-dimensional feature sequence; adding the feature vector in the second one-dimensional feature sequence to the second positional encoding to obtain a second one-dimensional encoded feature sequence; and inputting the second one-dimensional encoded feature sequence into the Transformer for global feature extraction to obtain a one-dimensional global feature sequence.

[0079] Furthermore, the aforementioned downsampling processing method can be implemented using a movable window transformation neural network or a pyramid visual transformation neural network, etc. The input one-dimensional feature sequence includes: a one-dimensional feature sequence to be processed, or includes: a one-dimensional feature sequence to be processed and a one-dimensional global feature sequence, which can be determined based on the number of times the superposition operation is executed. Specifically, if the number of times the superposition operation is executed is 1, then the input one-dimensional feature sequence is the one-dimensional feature sequence to be processed; if the number of times the superposition operation is executed is greater than 1, then the input one-dimensional feature sequence is the one-dimensional feature sequence to be processed and a one-dimensional global feature sequence. In this case, after the first superposition operation is executed, the one-dimensional global feature sequence obtained from the previous superposition operation is used as the input one-dimensional feature sequence for calculation.

[0080] It should be noted that the one-dimensional global feature sequence obtained from the last superposition operation is a one-dimensional target feature sequence.

[0081] In one optional embodiment, the step of downsampling the input one-dimensional feature sequence includes: performing a one-dimensional convolution operation on the input one-dimensional feature sequence to obtain a one-dimensional convolutional feature sequence; multiplying the one-dimensional convolutional feature sequence with a preset linear projection matrix to obtain a one-dimensional projection feature sequence; and normalizing the one-dimensional projection feature sequence to obtain a second one-dimensional feature sequence.

[0082] The convolution operation using one-dimensional convolution is similar to that of two-dimensional convolution, and it can be implemented using existing calculation methods. The preset linear projection matrix can be an existing matrix used to calculate one-dimensional feature sequences, which will not be elaborated further in this embodiment.

[0083] In this optional embodiment, the downsampling effect equivalent to that of a movable window transformation neural network or a pyramid vision transformer is achieved by processing the input one-dimensional feature order through one-dimensional convolution and linear projection matrix. Furthermore, convolution operation is introduced to achieve deep integration of convolutional neural network and Transformer, which further enhances the generalization ability of the model after combining convolutional neural network and Transformer.

[0084] In an optional embodiment, before the step of performing global feature extraction on the first one-dimensional encoded feature sequence using a Transformer, the image processing method further includes: processing the first one-dimensional encoded feature sequence using a first activation function. Before the step of inputting the second one-dimensional encoded feature sequence into a Transformer for global feature extraction, the image processing method further includes: processing the second one-dimensional encoded feature sequence using a second activation function.

[0085] The first activation function and the second activation function may be the same or different. This embodiment uses the same activation function as an example. By processing the first and second one-dimensional encoded feature sequences using activation functions, nonlinear transformations can be performed on the first and second one-dimensional encoded feature sequences, thereby enhancing the expressive power of the model.

[0086] In an optional embodiment, after the step of performing at least one superposition operation on the one-dimensional feature sequence to be processed as input, the image processing method further includes: averaging the elements in the one-dimensional target feature sequence that represent the feature size to obtain a multi-scale one-dimensional mean feature sequence; processing the multi-scale one-dimensional mean feature sequence through a fully connected layer to obtain a one-dimensional class feature sequence, wherein the one-dimensional class feature sequence includes multiple classes; and processing the one-dimensional class feature sequence through a normalization exponent (softmax) function to obtain the probability corresponding to each class.

[0087] In this optional embodiment, averaging the elements within the feature size makes the numerous one-dimensional target feature sequences more standardized, facilitating subsequent classification of image samples in the same batch. Before processing the multi-scale one-dimensional mean feature sequences through the fully connected layer, the activation function operation is considered as mapping the original data to the hidden feature space. The fully connected layer then acts as a "classifier," mapping the learned "distributed feature representation" to the sample label space. For example, 100 images to be processed can be classified into 30 classes using the above method.

[0088] In an optional embodiment, after the step of performing at least one superposition operation on the one-dimensional feature sequence to be processed as input, the image processing method further includes: converting the first one-dimensional feature sequence, the one-dimensional feature sequence to be processed, and the one-dimensional global feature sequence obtained after each superposition operation into a two-dimensional target feature map.

[0089] By transforming the first one-dimensional feature sequence, the one-dimensional feature sequence to be processed, and the one-dimensional global feature sequence obtained after each superposition operation into a two-dimensional target feature map, the two-dimensional target feature map can be used as multi-scale two-dimensional features of image samples. This allows the formation of a backbone network of multi-scale two-dimensional features, which can then be used to complete image detection and downstream dense prediction tasks. For example, in 100 images to be processed, the above method can be used to label objects of different categories in each image.

[0090] Secondly, this embodiment provides an image processing method based on CNN and Transformer, which includes five stages. The first stage includes step S101 and step S102 from the first aspect; the second stage includes step S103 and step S105 from the first aspect; the third stage includes step S106 from the first aspect; the fourth stage is the same as the third stage; and the last stage includes an embodiment from the first aspect involving "averaging the elements representing the feature size in a one-dimensional target feature sequence".

[0091] It should be noted that the third and fourth stages represent two superposition operations performed in this embodiment. The third stage process can be repeated for different numbers of superposition operations.

[0092] For details, see Figure 2 In the first stage, the H×W×3 of an image sample from the same batch of inputs is processed to produce a two-dimensional feature map output, i.e., a two-dimensional feature map.

[0093] In the above, 3 and C1 represent the feature dimensions corresponding to the features of the respective processes.

[0094] For example, if an image sample has a scale of 224*224, after performing convolution operations with a kernel size of 7*7 and a stride of 2, and then performing max pooling operations with a kernel size of 3*3 and a stride of 2, a two-dimensional feature map of size 56×56×64 is obtained.

[0095] In the second stage, the image obtained in the first stage is processed using standard Patch Embedding (dividing the image into a sequence of blocks) operations. Two-dimensional feature patches form a one-dimensional sequence That is, the first one-dimensional feature sequence; then... The result is added to an absolute positional code of the same length and input into the encoder module of a standard Transformer for global feature extraction to obtain a one-dimensional feature sequence to be processed.

[0096] For example, if the previous stage output is a 56×56×64 two-dimensional feature map, dividing it using a 2-patch configuration with a feature output dimension of 128, the corresponding one-dimensional feature sequence will be 784×128, which is... That is, the first one-dimensional feature sequence; then, Add it to the absolute position code of the same length, and input it into the standard Transformer encoder module, i.e. Figure 2 The Transformer Encoder in the code extracts global features to obtain a one-dimensional feature sequence to be processed.

[0097] In the third stage, the one-dimensional feature sequence obtained in the previous stage is... Perform downsampling operation, that is, after Figure 2 The feature sequence is processed by a Down Sample layer, followed by a normalization layer to normalize the features, resulting in a second one-dimensional feature sequence. Since the feature sequence length has changed, a positional encoding of the same length is added. Then, it passes through an activation function layer and is input into a standard Transformer for computation, performing global feature extraction on the feature sequence input to the Transformer to obtain a one-dimensional global feature sequence.

[0098] The downsampling operation, based on the idea of ​​depthwise separable convolution, is implemented as follows: First, a one-dimensional convolution (kernel size 1*1, stride 1) is performed on the input to obtain... That is, a one-dimensional convolutional feature sequence, which is then multiplied by a linear projection matrix to obtain... This involves a one-dimensional projected feature sequence to achieve downsampling. The convolution operation includes passing the one-dimensional feature sequence through a one-dimensional convolutional (Conv1D) layer, a fully connected (Linear) layer, and a normalization (Norm) layer. The fourth stage is the same as the third stage, except that the one-dimensional global feature sequence output from the third stage is used as the input to the fourth stage for secondary computation, and then outputs...

[0099] In the final stage, the feature map size of the output from the fourth stage is... The one-dimensional feature sequence can be averaged in the Class Head to obtain 1×C4, and then passed through a fully connected layer to obtain 1×Class_nums (the number of classes classified by the last fully connected layer). This refers to the feature size.

[0100] For example, using the mini_Imagenet dataset, the output consists of 100 classes. The fourth stage output is 49×512, and after calculation, the final output is 1×100. The probability of each class can be obtained by using the softmax function.

[0101] It should be noted that, in Figure 2 In this model, m×, n×, and L× represent the depth of the transformer module, i.e., the number of transformers used in each stage. The number of transformers used in each stage can be modified according to the actual situation, and different sizes of transformers can be selected based on specific circumstances. This embodiment does not impose specific limitations on this. In addition, the Patch Embed (a layer structure used to divide the input feature sequence into block sequences) in the second stage also has a similar downsampling effect. It can convert the two-dimensional feature map into a one-dimensional feature sequence for subsequent stage operations.

[0102] This embodiment uses multiple layers of transformers for global feature extraction, which yields deeper features from image samples. Specifically, in this embodiment, this means obtaining multi-scale one-dimensional feature sequences through global feature extraction using m, n, and L transformers in the second, third, and fourth stages. By employing a downsampling method, the input image sequence length varies in each stage, thus constructing a multi-scale feature extraction network. The first stage uses a CNN to perform convolutional feature extraction on the input image, reducing the computational cost of the transformer modules in subsequent stages. Furthermore, subsequent stages build upon the feature maps extracted by convolution, effectively fusing the local features extracted by convolution within the transformer computation.

[0103] Thirdly, this embodiment provides an image processing method based on CNN and Transformer. This method differs from the method in the second aspect in that, in the final stage, the first one-dimensional feature sequence, the one-dimensional feature sequence to be processed, and the one-dimensional global feature sequence obtained after each superposition operation are all converted into two-dimensional target feature maps. All the converted two-dimensional target features are then used as multi-scale two-dimensional features for image sample detection or prediction tasks. This embodiment does not limit the specific process of performing image sample detection or prediction tasks using multi-scale two-dimensional features.

[0104] Fourthly, this embodiment provides a Transformer-based image processing system 300, see [link to documentation]. Figure 3 The Transformer-based image processing system 300 includes:

[0105] The acquisition module 301 is configured to acquire image samples;

[0106] The local processing module 302 is configured to extract local features from image samples using a convolutional neural network to obtain a two-dimensional feature map.

[0107] The first conversion module 303 is configured to convert a two-dimensional feature map into a first one-dimensional feature sequence.

[0108] The encoding module 304 is configured to add the feature vector in the first one-dimensional feature sequence to the first position code to obtain the first one-dimensional encoded feature sequence;

[0109] The global processing module 305 is configured to perform global feature extraction on the first one-dimensional encoded feature sequence through Transformer to obtain a one-dimensional feature sequence to be processed.

[0110] The overlay operation module 306 is configured to take the one-dimensional feature sequence to be processed as input and perform at least one overlay operation, so that the one-dimensional feature sequence to be processed is subjected to at least one global feature extraction by Transformer to obtain a one-dimensional target feature sequence.

[0111] In one optional embodiment, the overlay operation module 306 includes:

[0112] The processing submodule is configured to downsample the input one-dimensional feature sequence to obtain a second one-dimensional feature sequence.

[0113] The encoding submodule is configured to add the feature vector in the second one-dimensional feature sequence to the second position code to obtain the second one-dimensional encoded feature sequence;

[0114] The global processing submodule is configured to input the second one-dimensional encoded feature sequence into the Transformer for global feature extraction, thereby obtaining a one-dimensional global feature sequence.

[0115] The input one-dimensional feature sequence includes: a one-dimensional feature sequence to be processed, or includes: a one-dimensional feature sequence to be processed and a one-dimensional global feature sequence; the one-dimensional global feature sequence obtained by the global processing submodule at the last time is the one-dimensional target feature sequence.

[0116] In one optional embodiment, the processing submodule includes:

[0117] The convolutional unit is configured to perform a one-dimensional convolution operation on the input one-dimensional feature sequence to obtain a one-dimensional convolutional feature sequence.

[0118] The multiplication unit is configured to multiply a one-dimensional convolutional feature sequence with a preset linear projection matrix to obtain a one-dimensional projected feature sequence.

[0119] The normalization unit is configured to normalize the one-dimensional projected feature sequence to obtain a second one-dimensional feature sequence.

[0120] In an optional embodiment, the Transformer-based image processing system 300 further includes:

[0121] The mean-taking module is configured to take the mean of the elements in the one-dimensional target feature sequence that represent the size of the feature, so as to obtain a multi-scale one-dimensional mean feature sequence.

[0122] The fully connected module is configured to process the multi-scale one-dimensional mean feature sequence through a fully connected layer to obtain a one-dimensional class feature sequence, which includes multiple classes;

[0123] The classification module is configured to process the one-dimensional class feature sequence using a normalized exponential function to obtain the probability corresponding to each class.

[0124] In an optional embodiment, the Transformer-based image processing system 300 further includes:

[0125] The first activation module is configured to process the first one-dimensional encoded feature sequence through the first activation function before executing the global processing module;

[0126] The second activation module is configured to process the second one-dimensional encoded feature sequence through the second activation function before executing the global processing submodule.

[0127] In an optional embodiment, the Transformer-based image processing system 300 further includes:

[0128] The second conversion module is configured to convert the first one-dimensional feature sequence, the one-dimensional feature sequence to be processed, and the one-dimensional global feature sequence obtained after each superposition operation into a two-dimensional target feature map.

[0129] In the description of this specification, the references to terms such as "some embodiments," "other embodiments," "ideal embodiments," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method based on Transformer, characterized in that, include: Obtain image samples; Local feature extraction is performed on the image samples using a convolutional neural network to obtain a two-dimensional feature map. The two-dimensional feature map is converted into a first one-dimensional feature sequence; Add the feature vector in the first one-dimensional feature sequence to the first position code to obtain the first one-dimensional coded feature sequence; The first one-dimensional encoded feature sequence is subjected to global feature extraction by Transformer to obtain a one-dimensional feature sequence to be processed. The one-dimensional feature sequence to be processed is used as input for at least one superposition operation, so that the one-dimensional feature sequence to be processed is subjected to at least one global feature extraction by Transformer to obtain a one-dimensional target feature sequence. The superposition operation includes: downsampling the input one-dimensional feature sequence to obtain a second one-dimensional feature sequence; adding the feature vector in the second one-dimensional feature sequence to the second positional encoding to obtain a second one-dimensional encoded feature sequence; and inputting the second one-dimensional encoded feature sequence into a Transformer for global feature extraction to obtain a one-dimensional global feature sequence. The input one-dimensional feature sequence includes: the one-dimensional feature sequence to be processed, or includes: the one-dimensional feature sequence to be processed and the one-dimensional global feature sequence; the one-dimensional global feature sequence obtained by the last superposition operation is the one-dimensional target feature sequence; The step of downsampling the input one-dimensional feature sequence includes: performing a one-dimensional convolution operation on the input one-dimensional feature sequence to obtain a one-dimensional convolutional feature sequence; multiplying the one-dimensional convolutional feature sequence with a preset linear projection matrix to obtain a one-dimensional projection feature sequence; and normalizing the one-dimensional projection feature sequence to obtain a second one-dimensional feature sequence.

2. The image processing method according to claim 1, characterized in that, Before the step of performing global feature extraction on the first one-dimensional encoded feature sequence using a Transformer, the image processing method further includes: The first one-dimensional encoded feature sequence is processed by the first activation function; Before the step of inputting the second one-dimensional encoded feature sequence into the Transformer for global feature extraction, the image processing method further includes: The second one-dimensional encoded feature sequence is processed by the second activation function.

3. The image processing method according to any one of claims 1 to 2, characterized in that, After the step of performing at least one superposition operation on the one-dimensional feature sequence to be processed as input, the image processing method further includes: The elements representing the feature size in the one-dimensional target feature sequence are averaged to obtain a multi-scale one-dimensional mean feature sequence. The multi-scale one-dimensional mean feature sequence is processed by a fully connected layer to obtain a one-dimensional category feature sequence, which includes multiple classes. The one-dimensional category feature sequence is processed by a normalized exponential function to obtain the probability corresponding to each class.

4. The image processing method according to any one of claims 1 to 2, characterized in that, After the step of performing at least one superposition operation on the one-dimensional feature sequence to be processed as input, the image processing method further includes: The first one-dimensional feature sequence, the one-dimensional feature sequence to be processed, and the one-dimensional global feature sequence obtained after each superposition operation are all converted into a two-dimensional target feature map.

5. An image processing system based on Transformer, characterized in that, include: The acquisition module is configured to acquire image samples; The local processing module is configured to extract local features from the image samples using a convolutional neural network to obtain a two-dimensional feature map. The first conversion module is configured to convert the two-dimensional feature map into a first one-dimensional feature sequence; The encoding module is configured to add the feature vector in the first one-dimensional feature sequence to the first position code to obtain the first one-dimensional encoded feature sequence; The global processing module is configured to perform global feature extraction on the first one-dimensional encoded feature sequence through Transformer to obtain a one-dimensional feature sequence to be processed. The overlay operation module is configured to take the one-dimensional feature sequence to be processed as input and perform at least one overlay operation, so that the one-dimensional feature sequence to be processed is subjected to at least one global feature extraction by Transformer to obtain a one-dimensional target feature sequence. The overlay operation module includes: The processing submodule is configured to downsample the input one-dimensional feature sequence to obtain a second one-dimensional feature sequence. The encoding submodule is configured to add the feature vector in the second one-dimensional feature sequence to the second position code to obtain the second one-dimensional encoded feature sequence; The global processing submodule is configured to input the second one-dimensional encoded feature sequence into the Transformer for global feature extraction to obtain a one-dimensional global feature sequence. The input one-dimensional feature sequence includes: the one-dimensional feature sequence to be processed, or includes: the one-dimensional feature sequence to be processed and the one-dimensional global feature sequence; the one-dimensional global feature sequence obtained by the global processing submodule last time is the one-dimensional target feature sequence; The processing submodule includes: The convolutional unit is configured to perform a one-dimensional convolution operation on the input one-dimensional feature sequence to obtain a one-dimensional convolutional feature sequence. The multiplication unit is configured to multiply the one-dimensional convolutional feature sequence with a preset linear projection matrix to obtain a one-dimensional projection feature sequence; The normalization unit is configured to normalize the one-dimensional projected feature sequence to obtain a second one-dimensional feature sequence.

6. The image processing system according to claim 5, characterized in that, The image processing system also includes: The averaging module is configured to average the elements representing the feature size in the one-dimensional target feature sequence to obtain a multi-scale one-dimensional mean feature sequence. The fully connected module is configured to process the multi-scale one-dimensional mean feature sequence through a fully connected layer to obtain a one-dimensional class feature sequence, wherein the one-dimensional class feature sequence includes multiple classes; The classification module is configured to process the one-dimensional category feature sequence using a normalized exponential function to obtain the probability corresponding to each class.

Images