Image processing method and device and computer readable storage medium

Through the combination of the dual-branch structure and multi-head convolution and attention network, local and global information of the image is extracted and fused, the problem of mismatch between local and global information in image processing is solved, and the concerns of local and global information in image processing is improved. The accuracy and performance of image processing.

CN118941913APending Publication Date: 2024-11-12BEIJING JINGDONG TUOXIAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410612520.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-16
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, in image processing, convolutional neural networks are difficult to capture global structures when paying attention to local information, while the Transformer model ignores local information, resulting in performance degradation, especially in tasks that need to emphasize local information.

Method used

A two-branch structure is adopted, combining a multi-head convolutional network and a multi-head attention network to extract local and global information of the image respectively, and feature alignment and fusion are performed through the processing network, adaptively amplifying or suppressing local and global information.

Benefits of technology

It improves the accuracy of image processing, can better integrate local and global information, and improves the model's understanding of images and generalization performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941913A_ABST
    Figure CN118941913A_ABST
Patent Text Reader

Abstract

The invention relates to an image processing method and device and a computer readable storage medium, and relates to the field of artificial intelligence. The image processing method comprises the steps that local information of an image is determined through a local information extraction network, the local information represents the dependency relationship between local neighborhoods in the image, and the local information extraction network comprises a multi-head convolutional network; global information of the image is determined by using a global information extraction network, the global information extraction network comprises a multi-head attention network, and the global information represents a global dependency relationship of the image; and generating an image processing result by using a processing network according to the local information of the image and the global information of the image. According to the invention, the accuracy of image processing is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to an image processing method and device, and a computer-readable storage medium. Background Art

[0002] In image processing tasks, Convolutional Neural Networks (CNN) and Transformer models are the two main deep learning model structures.

[0003] Convolutional neural networks use a series of convolutional layers to extract local information from images, and downsample through pooling layers to reduce computation and improve the model's translation invariance. Unlike the local receptive field of convolutional neural networks, Transformer captures global contextual information through a self-attention mechanism, which enables Transformer to better understand the overall structure and long-distance dependencies of images. Summary of the invention

[0004] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: using a local information extraction network to determine local information of an image, wherein the local information represents dependencies between local neighborhoods in the image, and the local information extraction network comprises a multi-head convolutional network; using a global information extraction network to determine global information of the image, wherein the global information extraction network comprises a multi-head attention network, and the global information represents global dependencies of the image; and generating an image processing result using a processing network based on the local information of the image and the global information of the image.

[0005] In some embodiments, the multi-head convolutional network includes multiple convolution kernels, and the method of using the local information extraction network to determine the local information of the image includes: using multiple convolution kernels of the multi-head convolutional network to extract the local information of the image from the image in parallel.

[0006] In some embodiments, the use of multiple convolution kernels of a multi-head convolutional network to extract local information of an image from an image in parallel includes: obtaining a first feature map having multiple channels; using each of the multiple convolution kernels to convolve some of the multiple channels of the first feature map to obtain a convolution result, wherein the some channels corresponding to different convolution kernels are not repeated; and determining the local information of the image based on the convolution results of the multiple convolution kernels.

[0007] In some embodiments, determining the local information of the image based on the convolution results of multiple convolution kernels includes: splicing the convolution results of multiple convolution kernels along the channel dimension to obtain the local information of the image.

[0008] In some embodiments, obtaining a first feature map having multiple channels includes: generating a convolutional position code of an image; and generating the first feature map according to the image and the convolutional position code.

[0009] In some embodiments, the method generates an image processing result based on the local information of the image and the global information of the image using a processing network, including: using a convolution kernel of a first size to convolve the local information of the image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, the first size exceeds the size of multiple convolution kernels in the multi-head convolution network, and the first size does not exceed the second size; generating an image processing result based on the local convolution result and the global information of the image.

[0010] In some embodiments, the processing network includes a gating unit, and the generating of an image processing result based on the local information of the image and the global information of the image using the processing network includes: using the processing network to fuse the local information of the image and the global information of the image to obtain fused information; during the fusion process, using the gating unit to adaptively amplify at least one of the local information and the global information of the image, or adaptively suppress at least one of the local information and the global information of the image; generating an image processing result based on the fused information.

[0011] In some embodiments, generating an image processing result based on local information of the image and global information of the image includes: performing point-by-point convolution on the global information of the image to obtain a global convolution result; generating an image processing result based on the local information of the image and the global convolution result.

[0012] In some embodiments, the global information extraction network also includes a dynamic convolutional network, and the use of the global information extraction network to determine the global information of the image includes: using the dynamic convolutional network to downsample the image to obtain first input information; based on the first input information, using a multi-head attention network to determine the global information of the image.

[0013] In some embodiments, the local information extraction network also includes a dynamic convolutional network, and the method of using the local information extraction network to determine the local information of the image includes: using the dynamic convolutional network to downsample the image to obtain second input information; and based on the second input information, using a multi-head convolutional network to determine the local information of the image.

[0014] In some embodiments, the local information extraction network also includes a first feedforward neural network; the global information extraction network also includes a second feedforward neural network and a residual attention network.

[0015] In some embodiments, at least one of the first feedforward neural network and the second feedforward neural network includes, in sequence, a first linear layer, an activation layer, and a second linear layer, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer.

[0016] According to a second aspect of the present disclosure, a training method for an image processing model is provided, wherein the image processing model includes a local information extraction network, a global information extraction network and a processing network, and the training method for the image processing model includes: obtaining image training samples, wherein the image training samples include images and labels; using the local information extraction network to determine local information of the image, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information represents the dependency relationship between local neighborhoods in the image; using the global information extraction network to determine the global information of the image, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency relationship of the image; using the processing network to generate an image processing result based on the local information of the image and the global information of the image; and training the image processing model based on the image processing result and the label.

[0017] According to a third aspect of the present disclosure, there is provided an image processing device, comprising: a local determination module, configured to determine local information of an image using a local information extraction network, wherein the local information represents dependencies between local neighborhoods in the image, and the local information extraction network comprises a multi-head convolutional network; a global determination module, configured to determine global information of the image using a global information extraction network, wherein the global information extraction network comprises a multi-head attention network, and the global information represents global dependencies of the image; and a generation module, configured to generate an image processing result using a processing network based on the local information of the image and the global information of the image.

[0018] According to a fourth aspect of the present disclosure, a training device for an image processing model is provided, wherein the image processing model includes a local information extraction network, a global information extraction network and a processing network, and the training method of the image processing model includes: an acquisition module, configured to acquire image training samples, wherein the image training samples include images and labels; a local determination module, configured to determine local information of the image using the local information extraction network, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information represents the dependency relationship between local neighborhoods in the image; a global determination module, configured to determine global information of the image using the global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency relationship of the image; a generation module, configured to generate an image processing result based on the local information of the image and the global information of the image using the processing network; and a training module, configured to train the image processing model based on the image processing result and the label.

[0019] According to a fifth aspect of the present disclosure, an image processing device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the image processing method described in any embodiment of the present disclosure or the training method of the image processing model described in any embodiment of the present disclosure based on instructions stored in the memory.

[0020] According to the sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the instructions are executed by a processor, the image processing method described in any embodiment of the present disclosure or the training method of the image processing model described in any embodiment of the present disclosure is implemented.

[0021] According to the seventh aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, implement the image processing method according to any embodiment of the present disclosure or the training method of the image processing model according to any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0024] Figure 1 A flowchart showing an image processing method according to some embodiments of the present disclosure;

[0025] Figure 2 Schematic diagram showing a multi-head convolutional network according to some embodiments of the present disclosure

[0026] Figure 3 A schematic diagram showing a local information extraction network according to some embodiments of the present disclosure;

[0027] Figure 4 A schematic diagram showing a dynamic convolutional network according to some embodiments of the present disclosure;

[0028] Figure 5 A schematic diagram showing a fusion network according to some embodiments of the present disclosure;

[0029] Figure 6 A schematic diagram showing an image processing model according to some embodiments of the present disclosure;

[0030] Figure 7 A schematic diagram showing sMRI preprocessing according to some embodiments of the present disclosure;

[0031] Figure 8 A flowchart showing a method for training an image processing model according to some embodiments of the present disclosure;

[0032] Fig. 9 A block diagram showing an image processing apparatus according to some embodiments of the present disclosure;

[0033] Fig.10 A block diagram showing a training apparatus for an image processing model according to some embodiments of the present disclosure;

[0034] Fig.11 A block diagram showing an image processing apparatus according to some other embodiments of the present disclosure; Fig.12 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0036] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0037] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0038] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0039] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] The Transformer model is an architecture based entirely on the attention mechanism and is more suitable for processing sequence data, such as natural language processing tasks. The Transformer model establishes global dependencies in the sequence through the self-attention mechanism, but in image data, this global dependency may not be able to effectively capture the connection between local information. If the Transformer model is directly embedded in the convolutional neural network architecture, due to the Transformer's characteristic of focusing on global dependencies, it will ignore or weaken the focus on local information. This will cause the model's performance to deteriorate when processing image data, especially for tasks that need to emphasize local information.

[0042] Convolutional neural networks mainly focus on local information. Although sometimes convolutional neural networks can also construct global representations by integrating local information, in the convolution layer, the features extracted at different levels and scales are still local and cannot directly provide global structural and semantic information about the entire input, lacking global consistency.

[0043] The present disclosure proposes an image processing method and device, and a computer-readable storage medium, which can better integrate local information and global information of an image and improve the accuracy of image processing.

[0044] Figure 1 A flowchart of an image processing method according to some embodiments of the present disclosure is shown.

[0045] like Figure 1 As shown, the image processing method includes steps S11 to S13. In some embodiments, the image processing method is executed by an image processing device.

[0046] In step S11, local information of the image is determined using a local information extraction network, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information extraction network includes a multi-head convolutional network.

[0047] "Dependency between local neighborhoods in an image" means that the pixel values ​​in each local area of ​​the image do not exist independently, but are related to each other. This correlation is reflected by calculating the convolution output in the local area.

[0048] Convolutional neural networks exploit this local dependency to build a more complex and abstract understanding of image content and are very effective in capturing local contextual relationships, while attention mechanisms are good at capturing global dependencies.

[0049] Therefore, the present disclosure proposes a dual-branch structure to extract local and global context features at the same time, in which one branch uses convolution operation and the other branch calculates attention distribution. In addition, by designing the convolutional network as a "multi-head" structure unified with the attention mechanism, the features extracted by the two branches (i.e., the local information extracted by the local information extraction network and the global information extracted by the attention mechanism) can be aligned before entering the processing network, thus promoting better fusion of subsequent local information and global information.

[0050] For example, the multi-head convolutional network is a pure convolutional structure without introducing attention operations. The convolution result of each convolution kernel is similar to a score in the multi-head attention network. In this way, the multi-head convolutional network can extract local discriminative features in the image within a limited receptive field, while retaining the spatial structure of the feature map and aligning the features extracted by the multi-head convolutional network and the multi-head attention network.

[0051] In some embodiments, the multi-head convolution network includes multiple convolution kernels, and uses a local information extraction network to determine the local information of the image, including: using multiple convolution kernels of the multi-head convolution network to extract the local information of the image from the image in parallel. For example, the multi-head convolution network adopts a style similar to the multi-head attention network, and the multi-head convolution network extracts features from each "head" in parallel through multiple convolution kernels.

[0052] In some embodiments, multiple convolution kernels of a multi-head convolutional network are used to extract local information of an image from an image in parallel, including: obtaining a first feature map having multiple channels; using each of the multiple convolution kernels to convolve some of the multiple channels of the first feature map to obtain a convolution result, wherein some of the channels corresponding to different convolution kernels are not repeated; and determining the local information of the image based on the convolution results of the multiple convolution kernels.

[0053] Figure 2 A schematic diagram of a multi-head convolutional network according to some embodiments of the present disclosure is shown.

[0054] like Figure 2 As shown, given a set of anchor tags {H i} 1:N , where N represents the number of anchor mark, which is also the number of convolution kernels, and i represents the anchor mark H i The position of i is less than or equal to N. The anchor point markers correspond to the convolution kernels one by one. Through a small neighborhood Ω on the first feature map d×h×w Local convolution operation is performed on to learn the correlation between features. The calculation formula is as follows.

[0055]

[0056] in, is the learnable convolution kernel parameter matrix, (ij) represents the anchor mark H i and H j The relative position between them, d×h×w is the size of the learnable convolution kernel parameter matrix, d is the number of channels in this small neighborhood, h is the height of the neighborhood, and w is the width of the neighborhood. ★ indicates a convolution operation. (H j ,H i ) represents the convolution from the jth convolution kernel to the ith convolution kernel. V n Represents the convolution result of the nth convolution kernel.

[0057] Each head of the multi-head convolutional network corresponds to some channels of the feature map, and the channels corresponding to any two heads are not repeated. For example, all channels of the feature map are divided into multiple groups, each convolution kernel corresponds to a group of channels, each convolution kernel slides on each channel of the corresponding group of channels, and convolves with the area where each channel slides, and then the convolution results of this group of channels in the area are weighted summed to obtain the convolution results of this group of channels in the area. The local information of the image is obtained by combining the convolution results of multiple groups of channels.

[0058] In some embodiments, determining local information of an image based on the convolution results of multiple convolution kernels includes: splicing the convolution results of multiple convolution kernels along the channel dimension to obtain local information of the image. For example, multiple convolution kernels are each convolved on a corresponding channel, and after extracting features using multiple convolution kernels, multiple features are aggregated in a splicing manner in the channel dimension.

[0059] Figure 3 A schematic diagram of a local information extraction network according to some embodiments of the present disclosure is shown.

[0060] The following will be combined Figure 3 , a local information extraction network according to some embodiments of the present disclosure is introduced in detail.

[0061] In some embodiments, obtaining a first feature map having multiple channels includes: generating a convolutional position code of an image; and generating the first feature map according to the image and the convolutional position code.

[0062] For example, in the local information extraction network, there is a convolution layer before the multi-head convolution network. The convolution layer is used to generate convolution position codes for pixels in the image. The convolution position codes and the features input to the local information extraction network are fused in an additive manner, for example, to obtain local image information containing position information. The local image information containing position information passes through a normalization layer and is input into the multi-head convolution network. Compared with absolute position codes and relative position codes, convolution position codes can flexibly embed positional relationships, reduce computational burden, and improve efficiency.

[0063] In some embodiments, the local information extraction network further comprises a first feedforward neural network. Figure 3 As shown, a feedforward network, such as a fully-connected feedforward network (FFN), is introduced behind each multi-head convolutional network. The output of the multi-head convolutional network, for example, passes through a normalization layer and then enters the first feedforward neural network.

[0064] In some embodiments, the first feedforward neural network includes a first linear layer, an activation layer, and a second linear layer in sequence, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer. For example, the first linear layer expands the channel dimension by 4 times, while the second linear layer reduces it back to the original level. This operation helps to further filter features and improve the nonlinear expression ability of the model. The activation layer is, for example, a nonlinear activation function, and the nonlinear activation function is, for example, a Gaussian Error Linear Unit (GELU).

[0065] Expanding the channel dimension by 4 times can increase the capacity of the model and allow the model to learn more complex feature representations. More channels mean that the network can process more types of features at the same time, which enhances the model's expressiveness. Then, the expanded features are transformed by the activation function, which can introduce the complex nonlinear relationships required by the model, so that the model can learn the originally linearly inseparable patterns in the input data. The second linear layer restores the number of channels to the original level, which is similar to a feature screening or compression process. When the number of channels is large, the model selects and combines the most valuable features by learning the weight matrix, discarding redundant or unimportant features, thereby achieving effective feature screening and extraction. Through such a structural design, the model can enhance the nonlinear expression of the input data while maintaining a low computational cost, and improve the generalization performance of the model through feature screening and combination. At the same time, this "wide-narrow-wide" structure can also alleviate the gradient vanishing and explosion problems, which helps to train deeper and more complex networks.

[0066] In some embodiments, the local information extraction network also includes a dynamic convolutional network, and the local information extraction network is used to determine the local information of the image, including: using the dynamic convolutional network to downsample the image to obtain second input information; based on the second input information, using a multi-head convolutional network to determine the local information of the image.

[0067] For example, in the local information extraction network, a dynamic convolution network is also set before the multi-head convolution network. The dynamic convolution network can adjust the size of the convolution kernel according to the different sizes of the input data, effectively reducing the number of parameters of the model, reducing the amount of calculation of the model, and improving the robustness and generalization of the convolutional neural network. The second input information can be the same as or different from the first input information.

[0068] Figure 4 A schematic diagram of a dynamic convolutional network according to some embodiments of the present disclosure is shown.

[0069] Dynamic convolutional network, also known as dynamic embedding block, is a combination of traditional convolution and dynamic convolution. Figure 4 As shown, the dynamic convolution network consists of two dynamic convolution layers and a normal convolution layer with a stride of 2 or 4 for downsampling.

[0070] Before performing multi-head convolution at each stage, the dynamic convolutional network uses convolution operations to split the feature map of the original input into many blocks in a non-overlapping manner, where the stride and block size are the same. Before and after this convolution operation, the dynamic convolutional network introduces dynamic convolution layers to guide subsequent multi-head modules to focus on more meaningful features. The dynamic convolution layer is, for example, a neural network layer that performs convolution operations on three-dimensional data.

[0071] Through the dynamic convolution network, all the tags are integrated before being input into the multi-head convolution network. This design combined with dynamic convolution has the following advantages: First, the dynamic convolution operation is more adaptable to different input resolutions. Second, dynamic convolution is lightweight and can greatly alleviate the overfitting problem when the amount of data is small.

[0072] Compared with traditional convolution, the introduction of dynamic convolution can enhance the robustness of the model because the model can focus on distinctive features faster. In traditional convolutional neural networks, the convolution kernel is fixed, and the same filter is used for all input samples throughout the training process. Dynamic convolution dynamically generates convolution kernels for each input position. The dynamic convolution network adjusts the size of the convolution kernel according to the different sizes of the input data, and adaptively adjusts the size of the parameters, thereby effectively reducing the number of parameters of the model. The dynamic convolution layer allows the convolution kernel to adaptively adjust the weights according to the input data. This adaptability allows the network to focus on important feature areas more flexibly when processing different inputs. When combined with a multi-head convolutional network, the dynamic convolution layer can pre-process the input feature map to highlight local information that is more critical or meaningful to the current task. For example, dynamic convolution learns the importance of certain specific patterns and enhances the feature responses corresponding to these patterns.

[0073] Subsequently, these features enhanced by dynamic convolution enter the multi-head convolution network. Since multi-head convolution can process information in parallel from multiple perspectives, each head can focus on different subspaces of the input features. When the input features have been screened by dynamic convolution and their importance has been emphasized, the multi-head convolution network can capture these enhanced features more effectively, thereby improving the entire model's attention to and utilization efficiency of meaningful features. The combination of dynamic convolution and multi-head convolution enables the network to have greater adaptability and sensitivity to valuable features when performing complex sequence modeling and visual recognition tasks.

[0074] In addition to the convolution layer, the dynamic convolution network may also include a normalization and activation layer, a flattening layer, a normalization layer, a shape transformation layer, etc. Among them, the activation layer is, for example, a GELU layer.

[0075] In step S12, the global information of the image is determined using a global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency of the image.

[0076] An example of a multi-head attention network is a multi-head self-attention network. Self-attention is a method to capture long-range dependencies between features and construct a discriminative global representation. The multi-head attention structure is used to process sequence information. Each "head" learns an independent feature map and obtains a global view through linear aggregation. The multi-head self-attention network uses multiple parallel attention heads, each of which independently calculates an attention weight distribution and performs weighted summation on the input to generate the corresponding output vector, and finally obtains the global information of the image. The formula for global information extraction is as follows.

[0077]

[0078] The above formula is a Softmax operator. n () and K n () For example, H is a fully connected transformation. n Indicates the features extracted by the nth "head". D, H, W represent the number of channels, height, and width of the feature map. j'∈Ω D×H×W , indicating that j' belongs to the global scope and represents the attention of the multi-head attention network to global dependencies, j represents n The corresponding channel.

[0079] In some embodiments, the multi-head attention network includes a residual attention network. For example, residual connections are introduced in the multi-head attention network. Residual connections may not be applied to the local information extraction network, but only to the multi-head attention network. This design is because residual connections help deep networks to quickly backpropagate shallow features. The convolution module has fewer network layers and residual connections may not be introduced because gradients can quickly propagate to lower layers without residual connections. However, for global self-attention operations based on fully connected layers, residual connections play a key role in promoting better information propagation. Residual connections help alleviate the problem of gradient decay and ensure that key global information is effectively retained and propagated during the global modeling process.

[0080] In some embodiments, the global information extraction network also includes a dynamic convolutional network, which uses the global information extraction network to determine the global information of the image: using the dynamic convolutional network, downsampling the image to obtain the first input information; based on the first input information, using the multi-head attention network, the global information of the image is determined. For example, in the processing model, a dynamic convolutional network is also provided before the multi-head attention network. The dynamic convolutional network can adjust the size of the convolution kernel according to the different sizes of the input data, effectively reducing the number of parameters of the model. And improve the robustness of the convolutional neural network.

[0081] The global information extraction network is similar to the dynamic embedding block in the local information extraction network, and will be referred to below. Figure 4 Make an introduction.

[0082] The dynamic convolutional network is a single dynamic embedding block, which is a combination of traditional convolution and dynamic convolution. The dynamic convolutional network consists of two dynamic convolutional layers and a normal convolutional layer with, for example, a stride of 2 or 4 for downsampling.

[0083] like Figure 4 As shown, before performing multi-head attention at each stage, the dynamic convolutional network uses a convolution operation to split the feature map of the original input into many blocks in a non-overlapping manner, where the stride and block size are the same. Before and after this convolution operation, the dynamic convolutional network introduces a dynamic convolution layer to guide the subsequent multi-head attention network to focus on more meaningful features. The dynamic convolution layer is, for example, a neural network layer that performs convolution operations on three-dimensional data.

[0084] Through the dynamic convolutional network, all the labels are integrated before being input into the multi-head attention network. The dynamic convolution layer allows the convolution kernel to adaptively adjust its weights according to the input data. This adaptability allows the network to focus on important feature areas more flexibly when processing different inputs. The dynamic convolution network adjusts the size of the convolution kernel according to the different sizes of the input data and adaptively adjusts the size of the parameters, thereby effectively reducing the number of parameters of the model. When combined with the multi-head attention network, the dynamic convolution layer can pre-process the input feature map to highlight those features that are more critical or meaningful to the current task. For example, the dynamic convolution will learn the importance of certain specific patterns and enhance the feature responses corresponding to these patterns.

[0085] Subsequently, these features enhanced by dynamic convolution enter the multi-head attention network. Since multi-head attention can process information from multiple perspectives in parallel, each attention head focuses on a different subspace of the input features. When the input features have been screened and emphasized by dynamic convolution, the multi-head attention network can more effectively capture these enhanced features, thereby improving the attention and utilization efficiency of the entire model to meaningful features. The combination of dynamic convolution and multi-head attention enables the network to have stronger adaptability and sensitivity to valuable features when performing complex sequence modeling and visual recognition tasks.

[0086] Compared with traditional convolution, introducing dynamic convolution before the multi-head attention network can enhance the robustness of the model because the model can focus on discriminative features faster.

[0087] In some embodiments, the multi-head attention network includes a second feed-forward neural network.

[0088] In some embodiments, the second feedforward neural network includes a first linear layer, an activation layer, and a second linear layer in sequence, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer. For example, the first linear layer expands the channel dimension by 4 times, and the second linear layer reduces it back to the original level. This operation helps to further filter features and improve the nonlinear expression ability of the model. The activation layer is, for example, a nonlinear activation function, and the nonlinear activation function is, for example, a Gaussian error linear unit.

[0089] In step S13, an image processing result is generated using a processing network according to local information of the image and global information of the image.

[0090] Image processing tasks include classification tasks, segmentation tasks, recognition tasks, etc. The following takes the classification task as an example for introduction.

[0091] For classification tasks, the processing network includes, for example, a fusion network and a classification layer. The fusion network fuses local information of the image with global information of the image, and then inputs the information into the classification layer for classification.

[0092] In some embodiments, an image processing result is generated using a processing network based on the local information of the image and the global information of the image, including: using a convolution kernel of a first size to convolve the local information of the image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, the first size exceeds the size of multiple convolution kernels in the multi-head convolution network, and the first size does not exceed the second size; and an image processing result is generated based on the local convolution result and the global information of the image. For example, the first size is 5×5×5. For local information, the size of its feature map is 5×6×5, and a convolution kernel larger than the size of the convolution kernel of the previous multi-head convolution network is used to focus on more global features. Through a larger convolution kernel, local information can perceive global information, which is conducive to subsequent information interaction between local information and global information.

[0093] In some embodiments, the image processing result is generated according to the local information of the image and the global information of the image, including: performing point-by-point convolution on the global information of the image to obtain a global convolution result; and generating the image processing result according to the local information of the image and the global convolution result. For example, for the global information (i.e., the global feature map), point-by-point convolution is used to effectively retain the global information focused on by the self-attention, thereby improving the accuracy of subsequent processing.

[0094] In some embodiments, a convolution kernel of a first size is used to convolve local information of an image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, the first size exceeds the size of multiple convolution kernels in the multi-head convolution network, and the first size does not exceed the second size; the global information of the image is convolved point by point to obtain a global convolution result; and the image processing result is generated based on the local convolution result and the global convolution result of the image. For example, the local information and the global information are convolved with convolution kernels of different sizes respectively, and the convolution results are fused.

[0095] In some embodiments, the processing network includes a gating unit, and the processing network is used to generate an image processing result based on the local information of the image and the global information of the image, including: using the processing network to fuse the local information of the image and the global information of the image to obtain fused information; during the fusion process, using the gating unit to adaptively amplify at least one of the local information and the global information of the image, or adaptively suppress at least one of the local information and the global information of the image; generating an image processing result based on the fused information.

[0096] The gating unit is, for example, a Sigmoid function, thereby realizing adaptive feature selection. For example, the value range of the Sigmoid function is a decimal between 0 and 1. If the value obtained after the sigmoid function is 0.4, that is, the final feature must obtain 40% from the global information, and the remaining 60% is obtained from the local information, thereby adaptively controlling the flow of local and global context features, so that the processing network can balance the importance of local information and global context features according to the degree of association between each position and other positions, effectively realizing feature fusion and improving the accuracy of model processing images.

[0097] Figure 5 A schematic diagram of a fusion network according to some embodiments of the present disclosure is shown.

[0098] like Figure 5 As shown, the fusion network introduces a gating mechanism and convolution operation, and the calculation formula is as follows.

[0099]

[0100]

[0101] Z aug =Z g *σ(Z l )+Z l *(1-σ(Z l )).

[0102] Among them, l represents local, g represents global, m represents the current stage, and W l and W g represents the learnable convolution kernel parameter matrix, σ represents the Sigmoid function, Represent local and global information respectively, * represents convolution, and feature set Z aug It contains global information aggregated by self-attention and local information extracted by convolution.

[0103] Figure 6 A schematic diagram of an image processing model according to some embodiments of the present disclosure is shown.

[0104] The following will be combined Figure 6 , which introduces in detail the image processing model according to some embodiments of the present disclosure.

[0105] like Figure 6 As shown, the image processing model includes, for example, a plurality of stages, where N1, N2, N3, and N4 represent the number of times the corresponding stage is repeated.

[0106] The local information extraction network and the global information extraction network are, for example, multi-stage stacked structures, that is, all or part of N3 and N4 are greater than 1, thereby achieving efficient hierarchical representation learning of local-global context features.

[0107] The “block embedding” in the figure is, for example, a dynamic convolutional network for downsampling. Considering that the dimension of the feature map in the first two stages is too large, if attention is calculated, it will have a significant impact on the computational burden, so a local information extraction network is used in the first stage. Subsequently, a parallel local-global context extraction structure is used to extract local information and global information independently. Finally, the local and global information are fused for image classification. The feature set Z is obtained by globally averaging the spatial features output by the fusion network and projecting them using a fully connected layer. aug It is obtained as a binary classification tensor to obtain the image classification result.

[0108] Global information and local information are not only integrated at the end, for example, Figure 6 In the process of repeating stage 3 and stage 4, at any stage, the global information and local information of the two branches can be fused, and then the global information and local information can be extracted from the fusion result, so as to promote better fusion and alignment of the global information and the local information.

[0109] In the global information extraction network, in order to process the spatial dimensions of the image (such as depth D, height H, and width W), the image is converted into a one-dimensional tensor and input into the fully connected layer. According to some embodiments of the present disclosure, since the multi-head attention network is located at the later stage of the entire network and processes feature maps that have been downsampled in multiple stages and have a smaller size, the previous operation of converting the image into a one-dimensional tensor and inputting it into the fully connected layer has a smaller impact on the computational burden of the multi-head attention network, thereby achieving a balance between computational efficiency and accuracy.

[0110] According to some embodiments of the present disclosure, the image processing method includes using a local information extraction network to determine the local information of the image, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information extraction network includes a multi-head convolutional network; using a global information extraction network to determine the global information of the image, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency relationship of the image; and using a processing network to generate an image processing result based on the local information of the image and the global information of the image.

[0111] The above scheme of the present disclosure proposes a dual-branch structure to extract local and global context features at the same time, in which one branch uses convolution operation and the other branch calculates attention distribution. Among them, by designing the multi-head convolutional network as a multi-head structure unified with the attention mechanism, before entering the processing network, the features extracted by the two branches (i.e., the local information extracted by the multi-head convolutional network and the global information extracted by the attention mechanism) can be aligned, so that the model can better integrate the local and global dimensions to understand the image and improve the accuracy of image processing.

[0112] The image processing method according to some embodiments of the present disclosure can be applied to the classification of structural magnetic resonance imaging (sMRI), for example, to classify structural magnetic resonance imaging and classify the image as belonging to or not belonging to Alzheimer's disease (AD).

[0113] For AD classification tasks, changes in brain structure sometimes involve multiple local areas and subtle feature changes. In addition, sMRI data usually have high-dimensional features. According to the image classification method of some embodiments of the present disclosure, it is possible to better understand sMRI images from local and global dimensions, and effectively integrate local and global information to classify sMRI. The multi-head convolutional network can extract local discriminant features in sMRI within a limited receptive field while retaining the spatial structure of the feature map, and focus on local areas and subtle feature changes of the brain structure through local information. At the same time, through global information, overfitting caused by the high-dimensional features of sMRI can be avoided, and the generalization ability of the model can be improved.

[0114] In the AD classification task, the input of the model is sMRI image data. The standardized template MNI152 T1 has a standard spatial coordinate system and brain structure information. Considering that spatial resolution is an important parameter affecting image quality and resolution, in the preprocessing operation of the original data, the T1-weighted sMRI is paired with the MNI152 T1 1mm template to obtain a more accurate description of the brain structure. At the same time, in order to avoid the computational burden caused by too many sMRI data dimensions, the image is cropped to 148×192×156 (D×H×W). Among them, D, H, and W represent the number of channels, height, and width, respectively. These preprocessing methods can improve the quality and effectiveness of sMRI data and lay the foundation for subsequent analysis.

[0115] Figure 7 A schematic diagram showing sMRI preprocessing according to some embodiments of the present disclosure.

[0116] Figure 7In the figure, (a), (b) and (c) are the original sMRI images acquired from the axial view, coronal view and sagittal view, respectively. (d), (e) and (f) are the preprocessed brain images, cropped from the corresponding views and matched with the MNI152 T1 1mm template. Figure 7 It can be seen that the preprocessed sMRI images have better image quality.

[0117] Experiments have shown that for sMRI-based AD classification tasks, the accuracy of the image processing method according to some embodiments of the present disclosure exceeds 90%.

[0118] In addition to image classification tasks, the local-global information extraction results according to some embodiments of the present disclosure are also expected to improve the performance of other tasks such as segmentation or recognition tasks for sMRI images.

[0119] Figure 8 A flowchart of a method for training an image processing model according to some embodiments of the present disclosure is shown.

[0120] like Figure 8 As shown, the training method of the image processing model includes steps S21 to S25. The image processing model includes a local information extraction network, a global information extraction network and a processing network. In some embodiments, the training method of the image processing model is performed by a training device of the image processing model.

[0121] In step S21, an image training sample is obtained, wherein the image training sample includes an image and a label.

[0122] In step S22, the local information of the image is determined using a local information extraction network, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information represents the dependency relationship between local neighborhoods in the image.

[0123] In some embodiments, the multi-head convolutional network includes multiple convolution kernels, and uses a local information extraction network to determine the local information of the image, including: using multiple convolution kernels of the multi-head convolutional network to extract local information of the image from the image in parallel.

[0124] In some embodiments, multiple convolution kernels of a multi-head convolutional network are used to extract local information of an image from an image in parallel, including: obtaining a first feature map having multiple channels; using each of the multiple convolution kernels to convolve some of the multiple channels of the first feature map to obtain a convolution result, wherein some of the channels corresponding to different convolution kernels are not repeated; and determining the local information of the image based on the convolution results of the multiple convolution kernels.

[0125] In some embodiments, local information of an image is determined based on convolution results of multiple convolution kernels, including: concatenating the convolution results of multiple convolution kernels along a channel dimension to obtain local information of the image.

[0126] In some embodiments, obtaining a first feature map having multiple channels includes: generating a convolutional position code of an image; and generating the first feature map according to the image and the convolutional position code.

[0127] In some embodiments, the local information extraction network also includes a dynamic convolutional network, and the local information extraction network is used to determine the local information of the image, including: using the dynamic convolutional network to downsample the image to obtain second input information; based on the second input information, using a multi-head convolutional network to determine the local information of the image.

[0128] In some embodiments, the local information extraction network also includes a first feedforward neural network.

[0129] In step S23, the global information of the image is determined using a global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency of the image.

[0130] In some embodiments, the global information extraction network also includes a dynamic convolutional network, and the global information extraction network is used to determine the global information of the image, including: using the dynamic convolutional network to downsample the image to obtain first input information; based on the first input information, using a multi-head attention network to determine the global information of the image.

[0131] In some embodiments, the global information extraction network also includes a second feedforward neural network and a residual attention network.

[0132] In some embodiments, at least one of the first feedforward neural network and the second feedforward neural network includes, in sequence, a first linear layer, an activation layer, and a second linear layer, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer.

[0133] In step S24, the processing network is used to generate an image processing result based on the local information of the image and the global information of the image.

[0134] In some embodiments, an image processing result is generated based on local information of an image and global information of the image using a processing network, including: using a convolution kernel of a first size to convolve the local information of the image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, the first size exceeds the size of multiple convolution kernels in a multi-head convolutional network, and the first size does not exceed the second size; an image processing result is generated based on the local convolution result and the global information of the image.

[0135] In some embodiments, an image processing result is generated based on local information of an image and global information of the image using a processing network, including: fusing local information of an image and global information of an image to obtain fused information; during the fusion process, using a gating unit to adaptively amplify at least one of the local information and the global information of the image, or adaptively suppress at least one of the local information and the global information of the image; and generating an image processing result based on the fused information.

[0136] In some embodiments, an image processing result is generated based on local information of the image and global information of the image, including: performing point-by-point convolution on the global information of the image to obtain a global convolution result; and generating an image processing result based on the local information of the image and the global convolution result.

[0137] In step S25, the image processing model is trained based on the image processing results and labels.

[0138] Fig. 9 A block diagram of an image processing apparatus according to some embodiments of the present disclosure is shown.

[0139] like Fig. 9 As shown, the image processing device 9 includes a local determination module 91 , a global determination module 92 , and a generation module 93 .

[0140] The local determination module 91 is configured to determine the local information of the image using a local information extraction network, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information extraction network includes a multi-head convolutional network, for example, performing Figure 1 Step S11 is shown.

[0141] The global determination module 92 is configured to determine the global information of the image using a global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency of the image, for example, performing Figure 1 Step S12 is shown.

[0142] The generation module 93 is configured to generate an image processing result by using a processing network according to the local information of the image and the global information of the image, for example, performing the following steps: Figure 1 Step S13 is shown.

[0143] In some embodiments, the multi-head convolutional network includes multiple convolution kernels, and the local determination module 91 is further configured to: utilize the multiple convolution kernels of the multi-head convolutional network to extract local information of the image from the image in parallel.

[0144] In some embodiments, the local determination module 91 is further configured to: obtain a first feature map having multiple channels; use each of the multiple convolution kernels to convolve some of the multiple channels of the first feature map to obtain a convolution result, wherein some of the channels corresponding to different convolution kernels are not repeated; determine the local information of the image based on the convolution results of the multiple convolution kernels.

[0145] In some embodiments, the local determination module 91 is further configured to: concatenate the convolution results of multiple convolution kernels along the channel dimension to obtain local information of the image.

[0146] In some embodiments, the local determination module 91 is further configured to: generate a convolutional position code of the image; and generate a first feature map based on the image and the convolutional position code.

[0147] In some embodiments, the generation module 93 is further configured to: use a convolution kernel of a first size to convolve local information of the image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, and the first size exceeds the size of multiple convolution kernels in the multi-head convolutional network, and the first size does not exceed the second size; generate an image processing result based on the local convolution result and the global information of the image.

[0148] In some embodiments, the processing network includes a gating unit, and the generation module 93 is further configured to: use the processing network to fuse the local information of the image and the global information of the image to obtain fused information; during the fusion process, use the gating unit to adaptively amplify at least one of the local information and the global information of the image, or adaptively suppress at least one of the local information and the global information of the image; generate an image processing result based on the fused information.

[0149] In some embodiments, the generation module 93 is further configured to: perform point-by-point convolution on the global information of the image to obtain a global convolution result; and generate an image processing result based on the local information of the image and the global convolution result.

[0150] In some embodiments, the global information extraction network also includes a dynamic convolutional network, and the global determination module 92 is further configured to: use the dynamic convolutional network to downsample the image to obtain first input information; and determine the global information of the image based on the first input information using a multi-head attention network.

[0151] In some embodiments, the local information extraction network also includes a dynamic convolutional network, and the local determination module 91 is further configured to: use the dynamic convolutional network to downsample the image to obtain second input information; and determine the local information of the image based on the second input information using a multi-head convolutional network.

[0152] In some embodiments, the local information extraction network also includes a first feedforward neural network; the global information extraction network also includes a second feedforward neural network and a residual attention network.

[0153] In some embodiments, at least one of the first feedforward neural network and the second feedforward neural network includes, in sequence, a first linear layer, an activation layer, and a second linear layer, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer.

[0154] Fig.10 A block diagram of a training apparatus for an image processing model according to some embodiments of the present disclosure is shown.

[0155] like Fig.10 As shown, the training device 10 of the image processing model includes an acquisition module 101, a local determination module 102, a global determination module 103, a generation module 104 and a training module 105. The image processing model includes a local information extraction network, a global information extraction network and a processing network.

[0156] The acquisition module 101 is configured to acquire image training samples, wherein the image training samples include images and labels, for example, by performing the following steps: Figure 8 Step S21 is shown.

[0157] The local determination module 102 is configured to determine the local information of the image using a local information extraction network, wherein the local information represents the dependency relationship between local neighborhoods in the image, and the local information represents the dependency relationship between local neighborhoods in the image, for example, by performing Figure 8 Step S22 is shown.

[0158] The global determination module 103 is configured to determine the global information of the image using a global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency of the image, for example, performing Figure 8 Step S23 is shown.

[0159] The generation module 104 is configured to generate an image processing result using a processing network according to local information of the image and global information of the image, for example, by performing the following steps: Figure 8 Step S24 is shown.

[0160] The training module 105 is configured to train the image processing model according to the image processing results and labels, for example, Figure 8 Step S25 is shown.

[0161] In some embodiments, the multi-head convolutional network includes multiple convolution kernels, and the local determination module 101 is further configured to: utilize the multiple convolution kernels of the multi-head convolutional network to extract local information of the image from the image in parallel.

[0162] In some embodiments, the local determination module 101 is further configured to: obtain a first feature map having multiple channels; use each of the multiple convolution kernels to convolve some of the multiple channels of the first feature map to obtain a convolution result, wherein some of the channels corresponding to different convolution kernels are not repeated; determine the local information of the image based on the convolution results of the multiple convolution kernels.

[0163] In some embodiments, the local determination module 101 is further configured to: concatenate the convolution results of multiple convolution kernels along the channel dimension to obtain local information of the image.

[0164] In some embodiments, the local determination module 101 is further configured to: generate a convolutional position code of the image; and generate a first feature map according to the image and the convolutional position code.

[0165] In some embodiments, the generation module 103 is further configured to: use a convolution kernel of a first size to convolve local information of the image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, and the first size exceeds the size of multiple convolution kernels in the multi-head convolutional network, and the first size does not exceed the second size; generate an image processing result based on the local convolution result and the global information of the image.

[0166] In some embodiments, the processing network includes a gating unit, and the generation module 103 is further configured to: use the processing network to fuse the local information of the image and the global information of the image to obtain fused information; during the fusion process, use the gating unit to adaptively amplify at least one of the local information and the global information of the image, or adaptively suppress at least one of the local information and the global information of the image; generate an image processing result based on the fused information.

[0167] In some embodiments, the generation module 103 is further configured to: perform point-by-point convolution on the global information of the image to obtain a global convolution result; and generate an image processing result based on the local information of the image and the global convolution result.

[0168] In some embodiments, the global information extraction network also includes a dynamic convolutional network, and the global determination module 102 is further configured to: use the dynamic convolutional network to downsample the image to obtain first input information; and determine the global information of the image based on the first input information using a multi-head attention network.

[0169] In some embodiments, the local information extraction network also includes a dynamic convolutional network, and the local determination module 101 is further configured to: use the dynamic convolutional network to downsample the image to obtain second input information; and determine the local information of the image based on the second input information using a multi-head convolutional network.

[0170] In some embodiments, the local information extraction network also includes a first feedforward neural network; the global information extraction network also includes a second feedforward neural network and a residual attention network.

[0171] In some embodiments, at least one of the first feedforward neural network and the second feedforward neural network includes, in sequence, a first linear layer, an activation layer, and a second linear layer, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer.

[0172] Fig.11 A block diagram of an image processing apparatus according to some other embodiments of the present disclosure is shown.

[0173] like Fig.11 As shown, the image processing device 11 includes a memory 1101; and a processor 1102 coupled to the memory 1101, and the memory 1101 is used to store and execute the image processing method. The processor 1102 is configured to execute the image processing method described in any embodiment of the present disclosure or the training method of the image processing model described in any embodiment of the present disclosure based on the instructions stored in the memory 1101. The present disclosure provides a computer program product, including computer program instructions, which, when executed by the processor, implements the image processing method described in any embodiment of the present disclosure or the training method of the image processing model described in any embodiment of the present disclosure.

[0174] Fig.12 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0175] like Fig.12 As shown, the computer system 120 may be embodied in the form of a general-purpose computing device. The computer system 120 includes a memory 1210, a processor 1220, and a bus 1200 that connects the various system components.

[0176] The memory 1210 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing the image processing method or the training method of the image processing model in any of the embodiments of the present disclosure. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, and the like.

[0177] The processor 1220 can be implemented by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0178] The bus 1200 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0179] The computer system 120 may also include an input / output interface 1230, a network interface 1240, a storage interface 1250, etc. These interfaces 1230, 1240, 1250, the memory 1210, and the processor 1220 may be connected via a bus 1200. The input / output interface 1230 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 1240 may provide a connection interface for various networked devices. The storage interface 1250 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0180] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of image processing methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame and combination of frames in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0182] These computer-readable program instructions can also be readable and stored in a computer-readable memory, which cause the computer to work in a specific manner to produce a product, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0183] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0184] The image processing method and device, and the computer-readable storage medium in the above-mentioned embodiments improve the accuracy of image processing.

[0185] So far, the image processing method and device, and computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

Claims

1. An image processing method, comprising: Determining local information of the image using a local information extraction network, wherein the local information represents a dependency relationship between local neighborhoods in the image, and the local information extraction network includes a multi-head convolutional network; Using a global information extraction network to determine the global information of the image, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency of the image; According to the local information of the image and the global information of the image, the image processing result is generated by using the processing network.

2. The image processing method according to claim 1, wherein: The multi-head convolution network includes a plurality of convolution kernels, and the local information extraction network is used to determine the local information of the image, including: The multiple convolution kernels of the multi-head convolutional network are used to extract local information from the image in parallel.

3. The image processing method according to claim 2, wherein: The method uses multiple convolution kernels of the multi-head convolution network to extract local information of the image from the image in parallel, including: Obtaining a first feature map having a plurality of channels; Using each of the multiple convolution kernels, convolve some of the multiple channels of the first feature map to obtain a convolution result, wherein the some channels corresponding to different convolution kernels are not repeated; According to the convolution results of multiple convolution kernels, the local information of the image is determined.

4. The image processing method according to claim 3, wherein: Determining local information of the image according to the convolution results of the multiple convolution kernels includes: The convolution results of multiple convolution kernels are concatenated along the channel dimension to obtain the local information of the image.

5. The image processing method according to claim 3, wherein: The obtaining of a first feature map having multiple channels includes: Generate convolutional positional encodings of images; The first feature map is generated according to the image and the convolution position encoding.

6. The image processing method according to claim 1, wherein: The method of generating an image processing result by using a processing network according to local information of the image and global information of the image includes: Using a convolution kernel of a first size, convolving local information of the image to obtain a local convolution result, wherein the local information of the image is a second feature map of a second size, the first size exceeds the size of multiple convolution kernels in the multi-head convolution network, and the first size does not exceed the second size; The image processing result is generated based on the local convolution result and the global information of the image.

7. The image processing method according to claim 1, wherein: The processing network includes a gating unit, and the image processing result is generated by using the processing network according to the local information of the image and the global information of the image, including: Using the processing network, the local information of the image and the global information of the image are fused to obtain fused information; In the fusion process, using the gating unit, adaptively amplifies at least one of the local information and the global information of the image, or adaptively suppresses at least one of the local information and the global information of the image; Generate image processing results based on the fusion information.

8. The image processing method according to claim 1, wherein: The generating of the image processing result according to the local information of the image and the global information of the image includes: Perform point-by-point convolution on the global information of the image to obtain the global convolution result; The image processing result is generated based on the local information of the image and the global convolution result.

9. The image processing method according to claim 1, wherein: The global information extraction network also includes a dynamic convolutional network. The global information extraction network is used to determine the global information of the image, including: Using a dynamic convolutional network, down-sample the image to obtain first input information; Based on the first input information, a multi-head attention network is used to determine the global information of the image.

10. The image processing method according to claim 1, wherein: The local information extraction network also includes a dynamic convolutional network. The local information extraction network is used to determine the local information of the image, including: Using a dynamic convolutional network, downsample the image to obtain the second input information; According to the second input information, a multi-head convolutional network is used to determine the local information of the image.

11. The image processing method according to claim 1, wherein: The local information extraction network also includes a first feedforward neural network; The global information extraction network also includes a second feedforward neural network and a residual attention network.

12. The image processing method according to claim 1, wherein: At least one of the first feedforward neural network and the second feedforward neural network includes a first linear layer, an activation layer, and a second linear layer in sequence, and the number of input channels of the first linear layer is the same as the number of output channels of the second linear layer.

13. A method for training an image processing model, wherein: The image processing model includes a local information extraction network, a global information extraction network and a processing network, and the training method of the image processing model includes: Obtaining an image training sample, wherein the image training sample includes an image and a label; Determining local information of the image using a local information extraction network, wherein the local information represents a dependency relationship between local neighborhoods in the image, and the local information represents a dependency relationship between local neighborhoods in the image; Using a global information extraction network to determine the global information of the image, wherein the global information extraction network includes a multi-head attention network, and the global information represents the global dependency of the image; Using the processing network, the image processing result is generated according to the local information of the image and the global information of the image; Train the image processing model based on the image processing results and labels.

14. An image processing device, comprising: A local determination module is configured to determine local information of an image using a local information extraction network, wherein the local information represents a dependency relationship between local neighborhoods in the image, and the local information extraction network includes a multi-head convolutional network; A global determination module is configured to determine global information of an image using a global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents a global dependency relationship of the image; The generation module is configured to generate an image processing result by using a processing network according to local information of the image and global information of the image.

15. A training device for an image processing model, wherein: The image processing model includes a local information extraction network, a global information extraction network and a processing network, and the training method of the image processing model includes: An acquisition module is configured to acquire an image training sample, wherein the image training sample includes an image and a label; A local determination module is configured to determine local information of an image using a local information extraction network, wherein the local information represents a dependency relationship between local neighborhoods in the image, and the local information represents a dependency relationship between local neighborhoods in the image; A global determination module is configured to determine global information of an image using a global information extraction network, wherein the global information extraction network includes a multi-head attention network, and the global information represents a global dependency relationship of the image; A generation module is configured to generate an image processing result according to local information of the image and global information of the image by using a processing network; The training module is configured to train the image processing model according to the image processing results and labels.

16. An image processing device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the image processing method according to any one of claims 1 to 12 or the training method of the image processing model according to claim 13 based on instructions stored in the memory.

17. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the image processing method according to any one of claims 1 to 12 or the training method of the image processing model according to claim 13.

18. A computer program product, comprising computer program instructions, which, when executed by a processor, implement the image processing method according to any one of claims 1 to 12 or the training method of the image processing model according to claim 13.