An image processing method, apparatus, electronic device, and storage medium
By using a lightweight Transformer model for information exchange and communication between inside and outside the window, the problems of high computational cost and limited deployment in existing technologies are solved, achieving efficient image processing and convenient deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2022-03-24
- Publication Date
- 2026-05-26
Smart Images
Figure CN115953654B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer vision technology, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology
[0002] The Transformer model is a deep neural network based on a self-attention mechanism. This mechanism is not limited by local interactions, can uncover long-range dependencies, and performs parallel computation. Furthermore, it can learn appropriate inductive biases based on different task objectives. In recent years, the outstanding performance of the Transformer model in natural language processing has prompted visual researchers to begin interdisciplinary research on visual transformers.
[0003] Existing methods for image processing based on window-based Transformer models typically use two Transformer models sequentially to implement intra-window information exchange and inter-window information exchange, respectively. The shortcomings of these existing methods include at least the following: networks containing two Transformer models have a large number of layers and complex structures, resulting in high computational costs, high inference latency, and deployment limitations imposed by platform resources. Summary of the Invention
[0004] This disclosure provides an image processing method, apparatus, electronic device, and storage medium that can perform image processing based on a lightweight Transformer model. While ensuring processing quality, it can reduce computational load, achieve efficient and fast visual image processing, and facilitate deployment on different platforms.
[0005] In a first aspect, embodiments of this disclosure provide an image processing method, including:
[0006] Perform the following steps using the Transformer model:
[0007] The window images of the image to be processed, and the window feature data to be learned corresponding to each window image are combined to obtain the combined data.
[0008] Each combination of data is subjected to pixel-dimensional self-attention transformation to obtain information exchange images within each window and the learned window feature data.
[0009] Based on the learned window feature data, determine the influence weights between the information exchange images within each window;
[0010] Based on the influence weights, a window-dimensional self-attention transformation is performed on the information exchange images within each window to obtain the information fusion image between the windows.
[0011] Secondly, embodiments of this disclosure also provide an image processing apparatus, comprising:
[0012] The Transformer model is used to perform the following steps through the following modules:
[0013] The combination module is used to combine the window images of the image to be processed and the window feature data to be learned corresponding to each window image to obtain the combined data.
[0014] The first Transformer model is used to perform self-attention transformation on each of the combined data in the pixel dimension to obtain the information exchange image within each window and the learned window feature data.
[0015] The weight determination module is used to determine the influence weights between the information exchange images within each window based on the window feature data that has been learned.
[0016] The second Transformer model is used to perform window-dimensional self-attention transformation on the information exchange images within each window according to the influence weights, so as to obtain the information fusion image between each window.
[0017] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:
[0018] One or more processors;
[0019] Storage device for storing one or more programs.
[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any of the embodiments of this disclosure.
[0021] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any of the embodiments of this disclosure.
[0022] The technical solution of this disclosure embodiment performs the following steps through the Transformer model: combining the window images of the image to be processed and the window feature data to be learned corresponding to each window image to obtain combined data; performing self-attention transformation on each combined data at the pixel dimension to obtain information exchange images within each window and learned window feature data; determining the influence weights between information exchange images within each window based on the learned window feature data; and performing self-attention transformation at the window dimension on the information exchange images within each window based on each influence weight to obtain information fusion images between each window.
[0023] Information exchange within a window can be achieved through pixel-level self-attention transformation. Furthermore, by embedding the window feature data to be learned into the window image during the pixel-level self-attention transformation process, the window feature data can learn the information of each pixel within the window, ensuring that the learned window feature data fully represents the information of the corresponding window. Subsequently, by modeling the influence weights between information exchange images within each window based on the window feature data, attention relationships between windows can be modeled with minimal computational overhead. Finally, based on the mutual influence weights, information exchange images from other windows can be integrated into the information exchange images within each window, thereby achieving global information exchange.
[0024] Compared to the traditional method of using two Transformer models to separately implement intra-window information exchange and inter-window information exchange, the technical solution of this disclosure can achieve the same processing effect based on a single Transformer model, greatly simplifying the network structure and making the Transformer model more lightweight. This reduces computational load, achieves efficient and fast visual image processing, and facilitates deployment on different platforms. Attached Figure Description
[0025] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0026] Figure 1 This is a schematic flowchart of an image processing method provided in Embodiment 1 of this disclosure;
[0027] Figure 2 This is a structural block diagram of the Transformer model in an image processing method provided in Embodiment 1 of this disclosure;
[0028] Figure 3 This is a schematic diagram of a grouping and merging window in an image processing method provided in Embodiment 1 of this disclosure;
[0029] Figure 4 This is a schematic flowchart of an image processing method provided in Embodiment 2 of this disclosure;
[0030] Figure 5 This is a structural block diagram of the image processing network in an image processing method provided in Embodiment 2 of this disclosure;
[0031] Figure 6This is a schematic diagram of the structure of an image processing apparatus provided in Embodiment 3 of this disclosure;
[0032] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this disclosure. Detailed Implementation
[0033] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0034] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0035] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0036] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0037] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0038] Example 1
[0039] Figure 1 This is a schematic flowchart illustrating an image processing method provided in Embodiment 1 of this disclosure. This embodiment is applicable to situations where information exchange occurs within and between windows of an image to be processed using a window-based Transformer model. The method can be executed by an image processing device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer.
[0040] like Figure 1 As shown, the image processing method provided in this embodiment may include performing the following steps through a Transformer model:
[0041] S110. Combine the window images of the image to be processed and the window feature data to be learned corresponding to each window image to obtain the combined data.
[0042] The Transformer model applied to image processing can also be called the visual Transformer model, and can include Transformer-like models as well as models derived from the Transformer model.
[0043] In the field of visual image processing, a window-based self-attention mechanism has been proposed to reduce the computational complexity of self-attention models. This mechanism divides the image into several non-overlapping local window images of a fixed window size and restricts attention computation to these local window images. However, this mechanism cannot establish information exchange between windows.
[0044] Based on the above mechanism, this embodiment of the invention, after obtaining the window images of the image to be processed, can set corresponding window feature data to be learned for each window image. It is called "window feature data to be learned" because the window feature data set at this time is based on empirical or experimental values, and cannot fully represent the information contained in each window.
[0045] Combining each window image with its corresponding window feature data to be learned can be considered as embedding the window feature data into the window image to be learned. The number of channels in the window feature data can be equal to the number of channels in the window image, and the height × width of the window feature data can be 1×1 or other sizes. For example, when the height × width of the window feature data is 1×1, the combined data can be considered as embedding one pixel's information into the original window image.
[0046] S120. Perform self-attention transformation on each combination of data at the pixel dimension to obtain the information exchange image within each window and the learned window feature data.
[0047] In this embodiment, each set of combined data can undergo local pixel-level self-attention transformation separately. These pixel-level self-attention transformations of each set of combined data can be considered to be executed in parallel and independent of each other. Specifically, for each set of combined data, a self-attention transformation mechanism can be used to determine the pixel weights of different pixels in the window image. A larger pixel weight indicates a greater impact on the execution result of subsequent visual tasks. Then, each pixel in the combined image can be multiplied by its corresponding weight to achieve pixel-level self-attention transformation.
[0048] In the combined data after pixel-level self-attention transformation, the data corresponding to the original window image can constitute an information exchange image within the window, and the data corresponding to the original window feature data to be learned can be called the learned window feature data. By embedding the window feature data to be learned into the corresponding window image and using a self-attention mechanism to exchange information between the window feature data and all pixel information in its local window, the window feature data can learn the global feature representation of the corresponding local window, and the information of each channel within the local window image can also be exchanged.
[0049] S130. Based on the window feature data learned, determine the influence weights between information exchange images within each window.
[0050] Since the feature data of each window that has completed learning can fully represent the information in the corresponding window, the self-attention transformation mechanism can be used to model the attention relationship between different windows based on the feature data of each window that has completed learning, that is, to determine the influence weight between the information exchange images in each window.
[0051] Following traditional window-based self-attention mechanisms, a new model is needed to establish information exchange between windows. Compared to traditional methods, the embodiments of this disclosure can model the attention relationships between windows based on the window feature data learned during the pixel-dimensional self-attention transformation process. This eliminates the need for the deployment and computation of another model in traditional solutions, enabling the modeling of attention relationships between windows with minimal overhead.
[0052] S140. Based on the influence weights, perform self-attention transformation on the information exchange images within each window at the window dimension to obtain the information fusion image between each window.
[0053] The influence weights between information exchange images within each window can be asymmetric. For example, the influence weight of window A on window B might be 0.7, while the influence weight of window B on window A might be 0.4. For each information exchange image within a window, the influence weights of other windows on that window can be determined from the influence weights.
[0054] The subsequent process of performing window-dimensional self-attention transformation on the information exchange image within each window may include: according to the influence weight of other windows on this window, incorporating information from other windows into the information exchange image within this window to obtain the corresponding inter-window information fusion image, thereby obtaining the inter-window information fusion image and realizing global information exchange between windows.
[0055] In this embodiment, a single Transformer model can be used to perform two consecutive self-attention transformations on a window image: a pixel-level self-attention transformation and a window-level self-attention transformation. The pixel-level self-attention transformation enables information exchange within a local window, while the window-level self-attention transformation enables information exchange between global windows. Compared to the traditional method of using two separate Transformer models for intra-window and inter-window information exchange, this embodiment achieves the same processing effect with a single Transformer model, significantly simplifying the network structure and making the Transformer model more lightweight. This reduces computational load, enables efficient and fast visual image processing, and facilitates deployment on different platforms.
[0056] For example, Figure 2 This is a structural block diagram of the Transformer model in an image processing method provided in Embodiment 1 of this disclosure. See also... Figure 2 The Transformer model can sequentially contain layer normalization (LN), separable attention (Sep-Attn) modules, LN layers, and multi-layer perceptron (MLP) network layers, and non-contiguous layers can have connectivity relationships (e.g., Figure 2 Lines with arrows can connect non-continuous layers, and the connection point can be indicated by a plus sign inside a circle to signify data fusion.
[0057] The input to the Transformer model can include the image to be processed. The Transformer model can be configured to learn multiple window feature data sets (represented by win_tokens in the diagram), and each window feature data set corresponds to a window image of the image to be processed. For example, Figure 2The image to be processed can be 9×9×C in size, and can be divided into 4 window images, each with a size of 3×3×C. Each window image in the image to be processed corresponds to a corresponding window feature data (i.e., win_tokens) in win_tokens. For example, the window image at the top right corner of the image to be processed corresponds to the win_token at the top right corner of win_tokens, and each win_token can have a size of 1×1×C.
[0058] Each window image and its corresponding feature data to be learned, after being normalized by the LN layer, can enter the Sep-Attn module. Based on the Sep-Attn module, steps such as pixel-dimensional self-attention transformation (represented by Depthwise self-attention in the figure) and window-dimensional self-attention transformation (represented by Pointwise self-attention in the figure) in this embodiment of the disclosure can be performed.
[0059] In some optional implementations, the combined data are subjected to pixel-level self-attention transformation, including: performing matrix transformation on each combined data to obtain a first query vector, a first key vector, and a first value vector corresponding to each combined data; determining a first attention map based on each first query vector and the corresponding first key vector; and multiplying the first attention map with the corresponding first value vector.
[0060] See you again Figure 2 The diagram contains four combined data sets, each consisting of a window image and its corresponding win_token. The window data contains 9 pixels, and the win_token is equivalent to 1 pixel, meaning the combined data contains a total of 10 pixels. Each of the four combined data sets can undergo pixel-level self-attention transformation. The diagram only shows the pixel-level self-attention transformation process for one of the combined data sets; the transformation processes for the other combined data sets are similar, indicated by "...".
[0061] Figure 2 The pixel-dimensional self-attention transformation process shown in the figure may include: performing matrix transformation on the combined data based on the self-attention mechanism (i.e., W in the figure). Q1 W K1 W V1 This yields the corresponding first query vector Q1, first key vector K1, and first value vector V1. Multiplying the first query vector Q1 and the first key vector K1 (represented by a multiplication sign inside a circle in the diagram, and the same notation can be used for the multiplication process described below) determines the first attention map. Multiplying the corresponding pixels in the first attention map and the first value vector V1 changes the pixel weights, resulting in combined data after pixel-level self-attention transformation.
[0062] In the combined data after pixel-dimensional self-attention transformation, the data corresponding to the original window image can constitute an information exchange image within the window, and the data corresponding to the original window feature data to be learned can be called the learned window feature data.
[0063] In some optional implementations, the influence weights between information exchange images within each window are determined based on the learned window feature data. This includes: concatenating the learned window feature data to obtain concatenated data; performing matrix transformation on the concatenated data after layer normalization and activation processing to obtain a second query vector and a second key vector; determining a second attention map based on the second query vector and the second key vector; and determining the influence weights between information exchange images within each window based on the second attention map.
[0064] See you again Figure 2 The learned win_tokens can be concatenated into concatenated data (referred to as win_tokens again in the diagram, but here they can be used to represent the global information of the corresponding window). The win_tokens can be normalized and activated through LN layers and activation layers (Activation, Act) to adjust their validity. After normalization and activation, the win_tokens can undergo matrix transformation based on a self-attention mechanism (i.e., W in the diagram). Q2 W K2 This yields the second query vector Q2 and the second key vector K2. Multiplying the second query vector Q2 and the second key vector K2 determines the second attention map.
[0065] For example, Figure 2 The `win_tokens` array contains four `win_tokens`, each representing information about a corresponding window. The corresponding second attention map can have a height × width of 4 × 4, and each element in the second attention map represents the influence weight of the window containing the column on the window containing the row. It can be considered that the influence weights between information exchange images within each window can be found based on the second attention map.
[0066] In some optional implementations, a window-dimensional self-attention transformation is performed on the information exchange images within each window based on each influence weight. This includes: determining the target influence weight of information exchange images in other windows on the information exchange images in this window based on each influence weight; multiplying the information exchange images in other windows with their corresponding target influence weights; and integrating the multiplication results into the information exchange images in this window.
[0067] See you again Figure 2The diagram contains four information exchange images within windows. Each information exchange image within a window can be referred to as the information exchange image within that window, and the images other than itself within a window can be referred to as information exchange images within other windows. Figure 2 The diagram only shows the process of self-attention transformation of the information exchange image within one window at the window dimension. The transformation process of the information exchange images within other windows is similar, and is indicated by dashed arrows in the diagram.
[0068] Figure 2 The self-attention transformation process of the window dimension shown in the figure may include: selecting row elements from the second attention map that correspond to the information exchange image in this window, where each element in the row can represent the influence weight of the window in which the column is located on the target of the current window.
[0069] To facilitate the fusion of information exchange images from other windows with those within the current window, each element in the row can be used as a target influence weight. This target influence weight includes not only the influence weights of other windows on the current window but also the influence weight of the current window on itself. Multiplying the information exchange images from other windows by their corresponding target influence weights and then integrating the results into the information exchange image within the current window can be achieved by multiplying each information exchange image in each window by its corresponding target influence weight and then summing the results. By performing the aforementioned window dimension transformation on all information exchange images within each window, a fused information image between the windows can be obtained.
[0070] In some optional implementations, after obtaining the combined data, the method further includes: grouping and merging the combined data to obtain merged window data; correspondingly, performing pixel-level self-attention transformation on each combined data, including: performing pixel-level self-attention transformation on the merged window data.
[0071] See you again Figure 2 If each combination of data is treated as a single channel, then the combination data from adjacent channels can be merged to obtain a merged window of data. Furthermore, Figure 3 This is a schematic diagram of a grouping and merging window in an image processing method provided in Embodiment 1 of this disclosure. See also... Figure 3 Assuming there are 16 window images before grouping, these windows can be grouped and merged. For example, four adjacent window images can be grouped together and merged into one large window, resulting in four large window images. Furthermore, these large window images can be combined with the corresponding window feature data to be learned to obtain merged window data.
[0072] Both of the above merging methods can yield merged window data. Then, the merged window data can be subjected to pixel-level self-attention transformation to obtain the information exchange images within each merged window and the window feature data of the learned merged window. See [link to documentation]. Figure 3 During the self-attention transformation process at the pixel level, the pixel containing the dot within the window / large window image can exchange information with other pixels within the window / large window.
[0073] Furthermore, based on the window feature data of the merged windows after learning, the influence weights between information exchange images within each merged window can be determined. See also Figure 3 If the feature data of the window where the dot is located is used as the feature data of the current window, then the target influence weight of each window on the current window can be determined.
[0074] Finally, based on the influence weights, a window-dimensional self-attention transformation can be performed on the information exchange images within each merged window to obtain the information fusion image between each merged window. Figure 3 The pixels containing the circles within the medium / large window images, after undergoing self-attention transformations at both the pixel and window dimensions, yield corresponding self-attention transformed pixels. By grouping and merging the combined data, the self-attention transformed pixels can capture long-distance visual dependencies between multiple windows, thereby improving model performance to some extent.
[0075] The technical solution of this disclosure embodiment combines the window images of the image to be processed and the window feature data to be learned corresponding to each window image to obtain combined data; performs self-attention transformation on each combined data at the pixel dimension to obtain information exchange images within each window and learned window feature data; determines the influence weight between information exchange images within each window based on the learned window feature data; and performs self-attention transformation at the window dimension on the information exchange images within each window based on each influence weight to obtain information fusion images between each window.
[0076] Information exchange within a window can be achieved through pixel-level self-attention transformation. Furthermore, by embedding the window feature data to be learned into the window image during the pixel-level self-attention transformation process, the window feature data can learn the information of each pixel within the window, ensuring that the learned window feature data fully represents the information of the corresponding window. Subsequently, by modeling the influence weights between information exchange images within each window based on the window feature data, attention relationships between windows can be modeled with minimal computational overhead. Finally, based on the mutual influence weights, information exchange images from other windows can be integrated into the information exchange images within each window, thereby achieving global information exchange.
[0077] Compared to the traditional method of using two separate Transformer models to achieve intra-window and inter-window information exchange, the technical solution of this disclosure can achieve the same processing effect based on a single Transformer model, greatly simplifying the network structure and making the Transformer model more lightweight. This reduces computational load, enables efficient and fast visual image processing, and facilitates deployment on different platforms.
[0078] Example 2
[0079] The embodiments disclosed herein can be combined with the various optional schemes in the image processing methods provided in the above embodiments. The image processing method provided in this embodiment can be applied to at least one of the following image processing networks: image classification network, image segmentation network, and image detection network. It can achieve a relatively advanced level in various visual tasks such as image classification, semantic segmentation, and object detection, while making the model more lightweight.
[0080] Furthermore, the image processing network can adopt a multi-stage hierarchical structure, with each stage including a downsampling layer and the Transformer model provided in this embodiment, thereby extracting target feature images from the original image. In addition, the Transformer model can configure corresponding window feature data to be learned for each window image of the currently processed original image based on window feature data corresponding to previously processed images, thereby achieving iterative learning and optimization of the window feature data.
[0081] Figure 4 This is a schematic flowchart of an image processing method provided in Embodiment 2 of this disclosure. Figure 4 As shown, the image processing method provided in this embodiment is applied to at least one of the following image processing networks: image classification network, image segmentation network, and image detection network, and may include:
[0082] S410. Downsample the input original image to obtain the image to be processed.
[0083] In this embodiment, the original image can be an image that requires image classification, semantic segmentation, or object detection. This original image can be a real-time acquired image or a pre-stored image. Using existing downsampling methods, a feature image with higher semantic meaning can be obtained from the original image, and this feature image can be used as the image to be processed.
[0084] Perform the following steps S420-S480 using the Transformer model:
[0085] S420. Use a sliding window to slide on the image to be processed to obtain non-overlapping window images.
[0086] S430. Based on the learned window feature data corresponding to the processed images, determine the window feature data to be learned for each window image.
[0087] The Transformer model can set the window feature data to be learned for each window in the image to be processed, based on the learned window feature data corresponding to the processed images. This allows the window feature data to be continuously learned and updated. The optimized and updated window feature data is more reasonable, thereby accelerating the efficiency of self-attention transformation at the pixel level.
[0088] S440. Combine the window images of the image to be processed and the window feature data to be learned corresponding to each window image to obtain the combined data.
[0089] S450. Perform self-attention transformation on each combination of data at the pixel dimension to obtain the information exchange image within each window and the learned window feature data.
[0090] S460. Based on the learned window feature data, determine the influence weights between information exchange images within each window.
[0091] S470. Based on the influence weights, perform self-attention transformation on the information exchange images within each window at the window dimension to obtain the information fusion image between each window.
[0092] S480. Determine the attention-transformed image based on the image fusion of information between windows.
[0093] See you again Figure 2 After obtaining the fused image between each window, the fused image can be stitched together according to the corresponding positions of the original window images to obtain the stitched image. This stitched image can be further processed by other network layers of the Transformer model (such as LN layers and MLP layers), and the processed result output by the Transformer model can be called the attention-transformed image.
[0094] S490. Downsample the attention-transformed image to obtain a new image to be processed, until the attention-transformed image is the target feature image.
[0095] The image after attention transformation can be considered as the downsampled feature image after undergoing self-attention transformation. By performing self-attention transformation on the feature image, it is possible to improve the performance accuracy of visual tasks.
[0096] In performing visual tasks, it is typically necessary to determine multi-level feature images to synthesize the prediction results of feature images from different levels. Therefore, in this embodiment, after obtaining the attention-transformed image, a downsampling operation can be performed to obtain a higher-level feature image. This higher-level feature image can then be used as a new image to be processed, and steps S420-S480 can be executed again. This process is repeated until the feature image of the required level is determined. The feature image of the required level can then undergo attention transformation, and the corresponding attention-transformed image can be used as the target feature image.
[0097] For example, Figure 5 This is a structural block diagram of the image processing network in an image processing method provided in Embodiment 2 of this disclosure. See also... Figure 5 Image processing networks can employ a multi-stage hierarchical structure, as shown in the figure with four hierarchical structures. Each stage of the hierarchical structure may include a downsampling layer (represented by Patch Merging in the figure) and the Transformer model provided in this embodiment (represented by Sep-ViT Block in the figure). The original image (represented by Image in the figure) will change its size after each hierarchical processing stage. For example, the original image size is H×W×3; the size of the downsampled image in the first hierarchical structure is... The size of the downsampled image in the second layer structure is The size of the downsampled image in the third layer structure is The size of the downsampled image in the fourth layer structure is The image output from the fourth layer structure will continue to be processed to obtain the target feature image, and the corresponding visual task will be performed based on the target feature image.
[0098] The technical solutions of this disclosure can be applied to at least one of the following image processing networks: image classification network, image segmentation network, and image detection network. They can achieve a relatively advanced level in various visual tasks such as image classification, semantic segmentation, and object detection, while making the model more lightweight.
[0099] Furthermore, the image processing network can adopt a multi-stage hierarchical structure, with each stage including a downsampling layer and the Transformer model provided in this embodiment, thereby extracting target feature images from the original image. In addition, the Transformer model can configure corresponding window feature data to be learned for each window image of the currently processed original image based on window feature data corresponding to previously processed images, thereby achieving iterative learning and optimization of the window feature data.
[0100] The image processing method provided in this embodiment belongs to the same concept as the image processing method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.
[0101] Example 3
[0102] Figure 6 This is a schematic diagram of an image processing apparatus provided in Embodiment 3 of this disclosure. The image processing apparatus provided in this embodiment is suitable for situations where information exchange within a window and information exchange between windows is performed on the image to be processed using a window-based Transformer model.
[0103] like Figure 6 As shown, the image processing apparatus provided in this embodiment may include:
[0104] The Transformer model is used to perform the following steps through the following modules:
[0105] The combination module 610 is used to combine the window images of the image to be processed and the window feature data to be learned corresponding to each window image to obtain the combined data.
[0106] The first self-attention module 620 is used to perform pixel-dimensional self-attention transformation on each combination of data to obtain information exchange images within each window and learned window feature data.
[0107] The weight determination module 630 is used to determine the influence weights between information exchange images within each window based on the learned window feature data.
[0108] The second self-attention module 640 is used to perform window-dimensional self-attention transformation on the information exchange images within each window according to each influence weight, so as to obtain the information fusion image between each window.
[0109] In some alternative implementations, the first self-attention module can be used for:
[0110] Perform matrix transformations on each combination of data to obtain the first query vector, the first key vector, and the first value vector corresponding to each combination of data.
[0111] The first attention map is determined based on each first query vector and the corresponding first key vector, and the first attention map is multiplied by the corresponding first value vector.
[0112] In some alternative implementations, the weight determination module can be used for:
[0113] The learned window feature data is concatenated to obtain concatenated data;
[0114] After layer normalization and activation processing, the concatenated data is transformed into a matrix to obtain the second query vector and the second key vector.
[0115] The second attention map is determined based on the second query vector and the second key vector, and the influence weights between the information exchange images in each window are determined based on the second attention map.
[0116] In some alternative implementations, the second self-attention module can be used for:
[0117] Based on the influence weights, determine the target influence weights of information exchange images in other windows on the information exchange images in this window;
[0118] Multiply the information exchange images in other windows with their corresponding target influence weights, and then integrate the multiplication results into the information exchange images in this window.
[0119] In some alternative implementations, the Transformer model may also include:
[0120] The merging module is used to group and merge the combined data after obtaining each combination of data to obtain the merged window data;
[0121] Correspondingly, the first self-attention module can be used to perform pixel-level self-attention transformation on the merged window data.
[0122] In some alternative implementations, the image processing apparatus may be applied to at least one of the following image processing networks: image classification network, image segmentation network, and image detection network.
[0123] In some alternative implementations, the image processing apparatus may further include:
[0124] The downsampling layer is used to downsample the input original image before combining the window images of the image to be processed and the window feature data to be learned corresponding to each window image through the Transformer model, so as to obtain the image to be processed.
[0125] The Transformer model is also used to slide a sliding window across the image to be processed to obtain non-overlapping window images.
[0126] Based on the learned window feature data corresponding to the processed images, determine the window feature data to be learned for each window image.
[0127] In some optional implementations, the Transformer model is also used to determine the attention-transformed image based on the information fusion image between windows after obtaining the information fusion image between windows through the Transformer model;
[0128] The attention-transformed image is downsampled using a downsampling layer to obtain a new image to be processed, until the attention-transformed image becomes the target feature image.
[0129] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.
[0130] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.
[0131] Example 4
[0132] The following is for reference. Figure 7 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 7 The diagram below shows the structure of the terminal device or server 700. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0133] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0134] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the image processing method of embodiments of this disclosure.
[0136] The electronic device provided in this embodiment and the image processing method provided in the above embodiments belong to the same disclosed concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.
[0137] Example 5
[0138] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the image processing method provided in the above embodiments.
[0139] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0140] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.
[0141] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0142] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:
[0143] The Transformer model performs the following steps: Combine the window images of the image to be processed with the corresponding window feature data to be learned to obtain combined data; perform pixel-level self-attention transformation on each combined data to obtain the information exchange image within each window and the learned window feature data; determine the influence weights between the information exchange images within each window based on the learned window feature data; and perform window-level self-attention transformation on the information exchange images within each window based on the influence weights to obtain the information fusion image between windows.
[0144] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0146] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.
[0147] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.
[0148] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0149] According to one or more embodiments of this disclosure, [Example 1] provides an image processing method, the method comprising:
[0150] Perform the following steps using the Transformer model:
[0151] The window images of the image to be processed, and the window feature data to be learned corresponding to each window image are combined to obtain the combined data.
[0152] Each combination of data is subjected to pixel-dimensional self-attention transformation to obtain information exchange images within each window and the learned window feature data.
[0153] Based on the learned window feature data, determine the influence weights between the information exchange images within each window;
[0154] Based on the influence weights, a window-dimensional self-attention transformation is performed on the information exchange images within each window to obtain the information fusion image between the windows.
[0155] According to one or more embodiments of this disclosure, [Example 2] provides an image processing method, further comprising:
[0156] In some optional implementations, performing pixel-level self-attention transformation on each of the combined data includes:
[0157] Each combination of data is subjected to matrix transformation to obtain a first query vector, a first key vector, and a first value vector corresponding to each combination of data.
[0158] A first attention map is determined based on each first query vector and its corresponding first key vector, and the first attention map is multiplied by its corresponding first value vector.
[0159] According to one or more embodiments of this disclosure, [Example 3] provides an image processing method, further comprising:
[0160] In some optional implementations, determining the influence weights between the information exchange images within each window based on the learned window feature data includes:
[0161] The learned window feature data are concatenated to obtain concatenated data;
[0162] The concatenated data is subjected to layer normalization and activation processing, followed by matrix transformation to obtain the second query vector and the second key vector.
[0163] A second attention map is determined based on the second query vector and the second key vector, and the influence weights between the information exchange images within each window are determined based on the second attention map.
[0164] According to one or more embodiments of this disclosure, [Example 4] provides an image processing method, further comprising:
[0165] In some optional implementations, the step of performing window-dimensional self-attention transformation on the information exchange images within each window according to the respective influence weights includes:
[0166] Based on the influence weights described above, determine the target influence weights of information exchange images in other windows on information exchange images in this window;
[0167] The information exchange images in the other windows are multiplied with the corresponding target influence weights, and the multiplication results are integrated into the information exchange images in this window.
[0168] According to one or more embodiments of this disclosure, [Example 5] provides an image processing method, further comprising:
[0169] In some optional implementations, after obtaining each combination of data, the method further includes: grouping and merging the combination of data to obtain merged window data;
[0170] Accordingly, performing pixel-level self-attention transformation on each of the combined data includes performing pixel-level self-attention transformation on the merged window data.
[0171] According to one or more embodiments of this disclosure, [Example Six] provides an image processing method, further comprising:
[0172] In some alternative implementations, it is applied to at least one of the following image processing networks: image classification network, image segmentation network, and image detection network.
[0173] According to one or more embodiments of this disclosure, [Example Seven] provides an image processing method, further comprising:
[0174] In some optional implementations, before combining the window images of the image to be processed and the window feature data to be learned corresponding to each window image using the Transformer model, the following steps are also included:
[0175] The input original image is downsampled to obtain the image to be processed;
[0176] Using the Transformer model, a sliding window is used to slide on the image to be processed to obtain non-overlapping window images;
[0177] Based on the learned window feature data corresponding to the processed images, determine the window feature data to be learned corresponding to each window image.
[0178] According to one or more embodiments of this disclosure, [Example Eight] provides an image processing method, further comprising:
[0179] In some alternative implementations, after obtaining the fused image of information between windows through the Transformer model, the following steps are also included:
[0180] The attention-transformed image is determined by fusing information from the windows using the Transformer model.
[0181] The attention-transformed image is downsampled to obtain a new image to be processed, until the attention-transformed image is the target feature image.
[0182] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.
[0183] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0184] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. An image processing method, characterized in that, include: Perform the following steps using the Transformer model: The window images of the image to be processed and the window feature data to be learned corresponding to each window image are combined to obtain each combined data, wherein each combined data is obtained by embedding the window feature data to be learned into their respective corresponding window images. Each combination of data is subjected to matrix transformation to obtain a first query vector, a first key vector, and a first value vector corresponding to each combination of data. A first attention map is determined based on each first query vector and the corresponding first key vector, and the first attention map is multiplied by the corresponding first value vector to obtain the information exchange image in each window and the learned window feature data, wherein the learned window feature data includes the global feature representation of the corresponding window image. The learned window feature data are concatenated to obtain concatenated data; The concatenated data is subjected to layer normalization and activation processing, followed by matrix transformation to obtain the second query vector and the second key vector. A second attention map is determined based on the second query vector and the second key vector, and the influence weights between the information exchange images within each window are determined based on the second attention map. Based on the influence weights, a window-dimensional self-attention transformation is performed on the information exchange images within each window to obtain the information fusion image between the windows.
2. The method according to claim 1, characterized in that, The step of performing window-dimensional self-attention transformation on the information exchange images within each window according to the respective influence weights includes: Based on the influence weights described above, determine the target influence weights of information exchange images in other windows on information exchange images in this window; The information exchange images in the other windows are multiplied with the corresponding target influence weights, and the multiplication results are integrated into the information exchange images in this window.
3. The method according to claim 1, characterized in that, After obtaining each combination of data, the method further includes: grouping and merging the combination of data to obtain merged window data; Accordingly, performing pixel-level self-attention transformation on each of the combined data includes performing pixel-level self-attention transformation on the merged window data.
4. The method according to claim 1, characterized in that, It can be applied to at least one of the following image processing networks: image classification network, image segmentation network, and image detection network.
5. The method according to claim 4, characterized in that, Before combining the window images of the image to be processed and the window feature data to be learned corresponding to each window image using the Transformer model, the process further includes: The input original image is downsampled to obtain the image to be processed; Using the Transformer model, a sliding window is used to slide on the image to be processed to obtain non-overlapping window images; Based on the learned window feature data corresponding to the processed images, determine the window feature data to be learned corresponding to each window image.
6. The method according to claim 4, characterized in that, After obtaining the fused image of information between windows using the Transformer model, the process also includes: The attention-transformed image is determined by fusing information from the windows using the Transformer model. The attention-transformed image is downsampled to obtain a new image to be processed, until the attention-transformed image is the target feature image.
7. An image processing apparatus, characterized in that, include: The Transformer model is used to perform the following steps through the following modules: The combination module is used to combine the window images of the image to be processed and the window feature data to be learned corresponding to the window images to obtain the combined data. The combined data is obtained by embedding the window feature data to be learned into their respective window images. The first self-attention module is used to perform pixel-dimensional self-attention transformation on each of the combined data to obtain information exchange images within each window and the learned window feature data, wherein the learned window feature data includes the global feature representation of the corresponding window image. The weight determination module is used to determine the influence weights between the information exchange images within each window based on the window feature data that has been learned. The second self-attention module is used to perform window-dimensional self-attention transformation on the information exchange images within each window according to the influence weights, so as to obtain the information fusion image between each window. The first self-attention module is used for: Perform matrix transformations on each combination of data to obtain the first query vector, the first key vector, and the first value vector corresponding to each combination of data. The first attention map is determined based on each first query vector and the corresponding first key vector, and the first attention map is multiplied by the corresponding first value vector; The weight determination module is used for: The learned window feature data is concatenated to obtain concatenated data; After layer normalization and activation processing, the concatenated data is transformed into a matrix to obtain the second query vector and the second key vector. The second attention map is determined based on the second query vector and the second key vector, and the influence weights between the information exchange images in each window are determined based on the second attention map.
8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-6.
9. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any one of claims 1-6.