Image processing method and device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device and a storage medium, the method comprising: determining a first image block feature corresponding to a target image, and dividing the first image block feature into a second image block feature and a third image block feature; performing feature enhancement on the second image block feature based on a global attention mechanism to obtain a fourth image block feature, and performing feature enhancement on the third image block feature based on a local attention mechanism to obtain a fifth image block feature; determining a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature; performing a target image processing operation on the target image according to the target image block feature to obtain an image processing result. The embodiments of the present disclosure can improve the accuracy of the image processing result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] Recently, self-attention networks have been widely used in natural language processing. Self-attention networks strengthen features by establishing connections between features, thereby improving the final performance of the network. With the introduction of visual self-attention networks, self-attention networks have also been widely used in the field of computer vision, showing great potential. However, the design of existing visual self-attention networks simply copies the design in natural language processing, and does not improve the characteristics of computer vision, resulting in poor performance of visual self-attention networks. Summary of the invention

[0003] The present disclosure proposes a technical solution of an image processing method and device, an electronic device, and a storage medium.

[0004] According to one aspect of the present disclosure, an image processing method is provided, including: determining a first image block feature corresponding to a target image, and dividing the first image block feature into a second image block feature and a third image block feature; performing feature enhancement on the second image block feature based on a global attention mechanism to obtain a fourth image block feature, and performing feature enhancement on the third image block feature based on a local attention mechanism to obtain a fifth image block feature; determining a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature; and performing a target image processing operation on the target image according to the target image block feature to obtain an image processing result.

[0005] In a possible implementation, the number of channels corresponding to the first image block feature is a target channel number; dividing the first image block feature into a second image block feature and a third image block feature includes: dividing the first image block feature on a channel dimension according to the target channel number and a target ratio to obtain the second image block feature and the third image block feature.

[0006] In a possible implementation, the feature enhancement of the second image block feature based on the global attention mechanism to obtain the fourth image block feature includes: determining a first feature vector, a second feature vector and a third feature vector based on the second image block feature; determining an attention feature map based on the first feature vector and the second feature vector; and determining the fourth image block feature based on the attention feature map and the third feature vector.

[0007] In a possible implementation, the feature enhancement of the third image block feature based on the local attention mechanism to obtain the fifth image block feature includes: performing convolution processing on the third image block feature according to a target convolution kernel and a first channel expansion rate to obtain the fifth image block feature.

[0008] In a possible implementation, determining a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature includes: performing feature conversion on the fourth image block feature and the fifth image block feature according to a second channel expansion rate to obtain the target image block feature.

[0009] In a possible implementation, the image processing method is implemented by a self-attention neural network, wherein the self-attention neural network includes at least one attention module, and at least one of the attention modules includes a global attention submodule, a local attention submodule and a feedforward submodule; the feature enhancement of the second image block feature based on the global attention mechanism to obtain the fourth image block feature includes: using the global attention submodule to enhance the second image block feature based on the global attention mechanism to obtain the fourth image block feature; the feature enhancement of the third image block feature based on the local attention mechanism to obtain the fifth image block feature includes: using the local attention submodule to enhance the third image block feature based on the local attention mechanism to obtain the fifth image block feature; the determining the target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature includes: using the feedforward submodule to perform feature conversion on the fourth image block feature and the fifth image block feature to obtain the target image block feature.

[0010] In a possible implementation, the method also includes: constructing a first network structure search space and a second network structure search space, wherein the first network structure search space includes multiple module distribution hyperparameters, and the second network structure search space includes multiple module structure hyperparameters; according to the first network structure search space, determining a target module distribution hyperparameter from the multiple module distribution hyperparameters, wherein the target module hyperparameter is used to indicate the number of the global attention sub-modules and the number of the local attention sub-modules included in each attention module; according to the target module distribution hyperparameter and the second network structure search space, determining a target module structure hyperparameter from the multiple module structure hyperparameters, wherein the target module structure hyperparameter is used to indicate the target number of channels corresponding to each attention module, the target convolution kernel and the first channel expansion rate corresponding to each local attention sub-module, and the second channel expansion rate corresponding to each feedforward module; constructing the self-attention neural network according to the target module distribution hyperparameter and the target module structure hyperparameter.

[0011] In a possible implementation, determining the target module distribution hyperparameters from the multiple module distribution hyperparameters according to the first network structure search space includes: constructing a first super network according to the first network structure search space, wherein the first super network includes multiple first optional network structures constructed according to the multiple module distribution hyperparameters; determining a first target network structure from the multiple first optional network structures by performing network training on the first super network; and determining the module distribution hyperparameters corresponding to the first target network structure as the target module distribution hyperparameters.

[0012] In a possible implementation, the method of determining a target module structure hyperparameter from the multiple module structure hyperparameters according to the target module distribution hyperparameter and the second network structure search space includes: constructing a second super network according to the target module distribution hyperparameter and the second network structure search space, wherein the second super network includes multiple second optional network structures constructed according to the multiple module structure hyperparameters; determining a second target network structure from the multiple second optional network structures by performing network training on the second super network; and determining the module structure hyperparameter corresponding to the second target network structure as the target module structure hyperparameter.

[0013] In a possible implementation, in the same attention module, each local attention sub-module corresponds to the same target convolution kernel and the first channel expansion rate.

[0014] According to one aspect of the present disclosure, an image processing device is provided, comprising: a feature determination module, configured to determine a first image block feature corresponding to a target image, and to divide the first image block feature into a second image block feature and a third image block feature; an attention module, configured to perform feature enhancement on the second image block feature based on a global attention mechanism to obtain a fourth image block feature, and to perform feature enhancement on the third image block feature based on a local attention mechanism to obtain a fifth image block feature; a determination module, configured to determine a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature; and a target image processing module, configured to perform a target image processing operation on the target image according to the target image block feature to obtain an image processing result.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0016] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0017] In the disclosed embodiment, the first image block feature corresponding to the target image is determined, and the first image block feature is divided into a second image block feature and a third image block feature; the second image block feature is feature-enhanced based on the global attention mechanism to obtain a fourth image block feature, and the third image block feature is feature-enhanced based on the local attention mechanism to obtain a fifth image block feature; the target image block feature corresponding to the target image is determined based on the fourth image block feature and the fifth image block feature; the target image block feature is subjected to a target image processing operation based on the target image block feature to obtain an image processing result. The image block features are feature-enhanced based on the global attention mechanism and the local attention mechanism respectively, so that the target image block features after feature enhancement include both global information and local information, which effectively improves the semantic expression ability of the target image block features, and then after the target image processing operation is performed using the target image block features with higher semantic expression ability, the accuracy of the image processing result can be improved.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0020] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;

[0021] Figure 2 A schematic diagram showing a method of determining features of a plurality of first image blocks corresponding to a target image according to an embodiment of the present disclosure;

[0022] Figure 3 A network structure diagram of a self-attention neural network in the related art is shown;

[0023] Figure 4 A network structure diagram of a self-attention neural network according to an embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic diagram showing a hierarchical network structure search according to an embodiment of the present disclosure;

[0025] Figure 6 A block diagram of an image processing device according to an embodiment of the present disclosure is shown;

[0026] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0027] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0029] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0031] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0032] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown. The image processing method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server. Figure 1 As shown, the image processing method may include:

[0033] In step S11, a first image block feature corresponding to a target image is determined, and the first image block feature is divided into a second image block feature and a third image block feature.

[0034] The target image here is an image to be processed that needs to be processed. In order to better obtain the internal correlation of the target image, the target image can be segmented and divided into multiple image blocks. By extracting features from each image block, multiple first image block features corresponding to the target image can be obtained. The number of multiple image blocks can be determined according to actual conditions, and this disclosure does not make specific restrictions on this.

[0035] Figure 2 FIG. 2 is a schematic diagram showing a method for determining the first image block feature corresponding to a target image according to an embodiment of the present disclosure. Figure 2 As shown, the target image is segmented and divided into L image blocks, and features are extracted from the L image blocks respectively to obtain L first image block features.

[0036] In order to facilitate subsequent processing of the multiple first image block features, the multiple first image block features can be converted into a first image block feature sequence. The specific method of converting the multiple first image block features into the first image block feature sequence can be determined according to actual conditions, and this disclosure does not specifically limit this.

[0037] The first image block feature is divided into two parts, the second image block feature and the third image block feature, so that the second image block feature and the third image block feature can be enhanced based on the global attention mechanism and the local attention mechanism respectively. The process of dividing the first image block feature into the second image block feature and the third image block feature will be described in detail later in conjunction with possible implementation methods of the present disclosure, and will not be repeated here.

[0038] In step S12, the second image block feature is enhanced based on the global attention mechanism to obtain the fourth image block feature, and the third image block feature is enhanced based on the local attention mechanism to obtain the fifth image block feature.

[0039] The global attention mechanism may refer to a process of determining the global attention relationship between all features, and then determining a new feature based on the global attention relationship. Therefore, based on the global attention mechanism, the global attention relationship between the features of each second image block is determined, and then the fourth image block feature is determined based on the global attention relationship to achieve feature enhancement. Among them, since the fourth image block feature is determined based on the global attention relationship, the fourth image block feature includes global information of the target image. The process of feature enhancement based on global attention will be described in detail later based on possible implementation methods of the present disclosure, which will not be repeated here.

[0040] The local attention mechanism may refer to the process of determining the local attention relationship between locally adjacent features, and then determining a new feature based on the local attention relationship. Therefore, based on the local attention mechanism, the local attention relationship between locally adjacent third image block features is determined, and then the fifth image block feature is determined based on the local attention relationship to achieve feature enhancement. Among them, since the fifth image block feature is determined based on the local attention relationship, the fifth image block feature includes local information of the target image. The process of feature enhancement based on local attention will be described in detail later based on possible implementation methods of the present disclosure, which will not be repeated here.

[0041] In step S13, a target image block feature corresponding to the target image is determined according to the fourth image block feature and the fifth image block feature.

[0042] Since the fourth image block feature includes the global information of the target image and the fifth image block feature includes the local information of the target image, the target image block feature determined based on the fourth image block feature and the fifth image block feature includes both the global information and the local information of the target image and has a higher semantic expression capability.

[0043] In step S14, a target image processing operation is performed on the target image according to the target image block feature to obtain an image processing result.

[0044] Since the target image block features include both global information and local information and have high semantic expression capabilities, according to image processing requirements, target image processing operations can be performed using the target image block features to obtain image processing results with higher accuracy.

[0045] In the disclosed embodiment, multiple first image block features corresponding to the target image are determined, and the first image block features are divided into second image block features and third image block features; the second image block features are feature enhanced based on the global attention mechanism to obtain a fourth image block feature, and the third image block features are feature enhanced based on the local attention mechanism to obtain a fifth image block feature; the target image block features corresponding to the target image are determined based on the fourth image block features and the fifth image block features; the target image block features are subjected to target image processing operations based on the target image block features to obtain image processing results. Feature enhancement is performed on the image block features based on the global attention mechanism and the local attention mechanism respectively, so that the target image block features after feature enhancement include both global information and local information, which effectively improves the semantic expression ability of the target image block features, and then after the target image processing operation is performed using the target image block features with higher semantic expression ability, the accuracy of the image processing results can be improved.

[0046] In a possible implementation, the number of channels corresponding to the first image block feature is a target number of channels; dividing the first image block feature into a second image block feature and a third image block feature includes: dividing the first image block feature on a channel dimension according to the target number of channels and a target ratio to obtain the second image block feature and the third image block feature.

[0047] By dividing the first image block feature in the channel dimension, a second image block feature for global attention processing and a third image block feature for local attention processing can be obtained. The number of the second image block feature and the third image block feature meets the target ratio, which prepares for the subsequent feature enhancement process. The specific values ​​of the target number of channels and the target ratio can be determined according to actual conditions, and this disclosure does not make specific limitations on this.

[0048] Still taking the above L first image block features as an example, the target channel number is d k , the target ratio is a:b. Therefore, the L first image block features can be evenly divided into N parts along the channel dimension, and the N parts The part is determined as A second image block feature group, and the N parts The part is determined as Each second image block feature group includes L second image block features, and the number of channels corresponding to each second image block feature is d k / N; each third image block feature group includes L third image block features, and the number of channels corresponding to each third image block feature is d k / N. The specific value of N can be determined according to actual conditions, and the present disclosure does not make any specific limitation on this.

[0049] In a possible implementation, the second image block feature is enhanced based on the global attention mechanism to obtain the fourth image block feature, including: determining the first feature vector, the second feature vector and the third feature vector according to the second image block feature; determining the attention feature map according to the first feature vector and the second feature vector; determining multiple second image block features according to the attention feature map and the third feature vector.

[0050] By determining the attention feature map, the global attention relationship between the features of each second image block can be obtained, and then according to the attention feature map, feature enhancement based on the global attention mechanism can be achieved to effectively obtain the fourth image block features including global information.

[0051] Still with the above Taking the second image block feature group as an example, for any second image block feature group, the L second image block features included in the second image block feature group are converted into three different feature vectors: a first feature vector Q, a second feature vector K and a third feature vector V, wherein the number of feature channels corresponding to the first feature vector Q, the second feature vector K and the third feature vector V is d k / N, and then use the following formula (1) to enhance the features based on the global attention mechanism:

[0052]

[0053] Among them, Softmax(·) represents the normalization function, and · represents the vector dot product. As shown in the above formula (1), by performing a dot product operation on the feature vector Q and the feature vector K, the dot product result Q·(K T ), using the number of feature channels d k / N and the normalization function Softmax(·), the dot product result Q·(K T ) is normalized to obtain an attention feature map indicating the global attention relationship Using Attention Feature Map A dot product operation is performed with the feature vector V to obtain Att(Q, K, V), which is the output result corresponding to the feature group of the second image block obtained after feature enhancement based on the global attention mechanism.

[0054] Based on the above method, we can obtain The output result corresponding to the second image block feature group, and then The output results corresponding to the second image block feature groups are fused in the channel dimension to obtain L fourth image block features after feature enhancement based on the global attention mechanism.

[0055] In a possible implementation, the feature of the third image block is enhanced based on a local attention mechanism to obtain multiple fifth image block features, including: performing convolution processing on the third image block feature to obtain the fifth image block feature.

[0056] By performing convolution processing on the third image block feature to achieve feature enhancement based on the local attention mechanism, multiple fifth image block features including local information can be effectively obtained according to the convolution result. Since the third image block feature is one-dimensional, the third image block feature can be subjected to one-dimensional convolution processing to obtain the fifth image block feature.

[0057] Still with the above Taking a third image block feature group as an example, for any third image block feature group, one-dimensional convolution processing is performed on the L third image block features included in the third image block feature group to obtain a convolution result corresponding to the third image block feature group.

[0058] In a possible implementation, the third image block feature is enhanced based on a local attention mechanism to obtain a fifth image block feature, including: performing convolution processing on the third image block feature according to a target convolution kernel and a first channel expansion rate to obtain a fifth image block feature.

[0059] Since the third image block feature is one-dimensional, the third image block feature can be subjected to one-dimensional convolution processing according to the target convolution kernel and the first channel expansion rate to obtain the fifth image block feature.

[0060] Still with the above For example, for any third image block feature group, according to the target convolution kernel and the first channel expansion rate, the L third image block features included in the third image block feature group are convolved, specifically including: according to the first channel expansion rate E, the L third image block features are subjected to one-dimensional pointwise convolution processing to obtain a first convolution sub-result, wherein the number of channels corresponding to the first convolution result is d k / N×E; According to the target convolution kernel K, the first convolution result is processed by one-dimensional layer-by-layer convolution (depthwise convolution) to obtain the second convolution sub-result, where the number of channels corresponding to the second convolution sub-result is d k / N×E; according to the first channel expansion rate E, the second convolution sub-result is subjected to one-dimensional pointwise convolution processing to obtain the convolution result corresponding to the third image block feature group, wherein the number of channels of the convolution result corresponding to the third image block feature group is d k / N. The specific values ​​of the first channel expansion rate E and the target convolution kernel K can be determined according to actual conditions, and the present disclosure does not make specific limitations on this.

[0061] Based on the above method, we can obtain The convolution result corresponding to the third image block feature group is then The convolution results corresponding to the third image block feature groups are fused in the channel dimension to obtain L fifth image block features after feature enhancement based on the local attention mechanism.

[0062] In a possible implementation, the feature enhancement of the third image block based on the local attention mechanism can be achieved by reducing the size of the processed data in addition to the one-dimensional convolution processing method mentioned above. Taking a third image block feature group as an example, for any third image block feature group, only some of the third image block features (for example, L / 2 third image block features) among the L third image block features included in the third image block feature group are subjected to feature enhancement as shown in the above formula (1), thereby achieving feature enhancement based on the local attention mechanism by reducing the size of the processed data. Other processing forms can also be used for feature enhancement of the third image block based on the local attention mechanism according to actual needs, and the present disclosure does not specifically limit this.

[0063] In a possible implementation, determining a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature includes: performing feature conversion on the fourth image block feature and the fifth image block feature according to the second channel expansion rate to obtain the target image block feature.

[0064] Through the second channel expansion rate, the fourth image block feature and the fifth image block feature are further transformed to obtain the target image block feature including both global information and local information, which effectively improves the semantic expression ability of the target image block feature.

[0065] Still using the L fourth image block features and the L fifth image block features, the L fourth image block features and the L fifth image block features are fused in the channel dimension to obtain a fusion result, and then the second channel expansion rate d z , perform feature conversion on the fusion result so that the number of channels of the fusion result is converted to the same as the target number of channels corresponding to the first image block feature before feature enhancement, so that the number of channels is d kThe second channel expansion rate d z The specific value of can be determined according to actual conditions, and this disclosure does not make any specific limitation on this.

[0066] In a possible implementation, the target image processing operation includes one of the following: target detection, target tracking, image recognition, and image classification.

[0067] The target image processing operation is performed using the target image block features with higher semantic expression ability, so that a higher precision image processing result can be obtained.

[0068] For example, according to the target image block features, the target image is classified to obtain an image classification result corresponding to the target image with high classification accuracy. In addition to the above-mentioned target detection, target tracking, image recognition, and image classification, the target image processing operation may also include other image processing operations according to actual image processing requirements, which is not specifically limited in the present disclosure.

[0069] In a possible implementation, the image processing method is implemented by a self-attention neural network, wherein the self-attention neural network includes at least one attention module, and at least one attention module includes a global attention submodule, a local attention submodule and a feedforward submodule; based on the global attention mechanism, the second image block feature is feature-enhanced to obtain a fourth image block feature, including: using the global attention submodule, based on the global attention mechanism, the second image block feature is feature-enhanced to obtain a fourth image block feature; based on the local attention mechanism, the third image block feature is feature-enhanced to obtain a fifth image block feature, including: using the local attention submodule, based on the local attention mechanism, the third image block feature is feature-enhanced to obtain a fifth image block feature; according to the fourth image block feature and the fifth image block feature, the target image block feature corresponding to the target image is determined, including: using the feedforward submodule, the fourth image block feature and the fifth image block feature are feature-converted to obtain the target image block feature.

[0070] By setting an attention module in the self-attention neural network, the attention module includes a global attention submodule, a local attention submodule and a feedforward submodule, so that the global attention submodule can be used to perform feature enhancement based on the global attention mechanism, the local attention submodule can be used to perform feature enhancement based on the local attention mechanism, and the feedforward module can be used to further perform feature conversion on the image block features after feature enhancement based on the global attention mechanism and the local attention mechanism, so as to obtain the target image block features that include both global information and local information, thereby effectively improving the semantic expression ability of the target image block features, and thereby effectively improving the network performance of the self-attention neural network.

[0071] Figure 3 FIG. 1 shows a network structure diagram of a self-attention neural network in the related art. Figure 3 As shown in , the self-attention neural network includes M attention modules (G-Block_1 to G-Block_M), and each attention module adopts the multi-head attention (MHA) mechanism. Figure 3 As shown, each attention module includes G=3 global attention sub-modules (ie, each global attention module includes 3 attention heads) and a feedforward sub-module (FeedForward Neural Network, FFN).

[0072] Still taking the above L first image block features as an example, for G-Block_1, the number of target channels corresponding to G-Block_1 is d k In the case of G-Block_1, since G-Block_1 includes G = 3 global attention submodules, the L first image block features are evenly divided into N = 3 parts along the channel dimension, and each part includes d channels. k / 3 L image block features, and then the number of channels in the three parts is d k The L image block features of / 3 are sent to each global attention submodule respectively, so as to perform the feature enhancement based on the global attention mechanism shown in the above formula (1) in each global attention submodule respectively, and obtain the output result of each global attention submodule. Then, the output result of each global attention submodule is fused in the channel dimension and input into the FFN submodule for further feature conversion to obtain the output result of G-Block_1, which is the channel number d. k The L image block features; then the number of channels output by G-Block_1 is d k The L image block features are input into G-Block_2, and the above feature enhancement process is repeated until the output result of G-Block_M is obtained.

[0073] like Figure 3 The self-attention neural network in the related art shown only includes a global attention sub-module and an FFN sub-module in each attention module. Therefore, the self-attention neural network in the related art can only realize feature enhancement based on the global attention mechanism, resulting in ignoring local information in the target image, making the network performance of the self-attention neural network poor.

[0074] Figure 4 FIG. 4 shows a network structure diagram of a self-attention neural network according to an embodiment of the present disclosure. Figure 4As shown, the self-attention neural network proposed in the present disclosure includes M attention modules (GL-Block_1 to GL-Block_M). At least one attention module includes G global attention sub-modules, L local attention sub-modules and one FFN sub-module. Figure 3 and Figure 4 , G-Block_2 includes G=3 global attention sub-modules. After replacing two of the global attention sub-modules with two local attention sub-modules, GL-Block_2 can be obtained. That is, GL-Block_2 includes G=1 global attention sub-module, L=2 local attention sub-modules and one FFN sub-module.

[0075] Still taking the above L first image block features as an example, for GL-Block_1, the number of target channels corresponding to G-Block_1 is d k , and GL-Block_1 includes G = 1 global attention submodule and L = 2 local attention submodules, the L first image block features are evenly divided into N = 3 parts along the channel dimension, and each part includes d channels. k / 3 of the L image block features, and then a part of the channels is d k / 3 of the L image block features are sent to the global attention submodule, and the remaining two parts with d channels are k The L image block features of / 3 are sent to two local attention sub-modules respectively.

[0076] For GL-Block_1, in G = 1 global attention submodule, the feature enhancement based on the global attention mechanism shown in the above formula (1) is performed to obtain the output result of the global attention submodule; in L = 2 local attention submodules, one-dimensional convolution processing is performed respectively to realize the feature enhancement based on the local attention mechanism, and the output results of L = 2 local attention submodules are obtained; then, the output results of 1 global attention submodule and 2 local attention submodules are fused in the channel dimension, and input into the FFN submodule for further feature conversion to obtain the output result of GL-Block_1, which is the channel number d. k The L image block features; then the number of channels output by GL-Block_1 is d k The L image block features are input into G-Block_2, and according to the distribution of the global attention submodule and the local attention submodule in G-Block_2, the above feature enhancement process is repeated until the output result of GL-Block_M is obtained.

[0077] The output result of GL-Block_M is the image block feature after M times of global-local feature enhancement, that is, the target image block feature corresponding to the above target image, such as Figure 4 As shown, the self-attention neural network also includes an image processing module (Head module shown in the figure), and the target image block features output by GL-Block_M are input into the Head module for corresponding target image processing operations to obtain image processing results. The Head module can be set according to actual image processing requirements, for example, it can be an image classification module, a target detection module, an image recognition module, an image tracking module, etc., and the present disclosure does not make specific limitations on this.

[0078] By introducing a local attention sub-module into the self-attention neural network, both global information and local information can be utilized in the feature enhancement process, so that the final target image block features have higher semantic expression ability. After performing target image processing operations on the target image using the target image features, a higher-precision image processing result can be obtained, thereby effectively improving the network performance of the self-attention neural network.

[0079] In one possible implementation, the local attention submodule may include two pointwise convolutional layers and a depthwise convolutional layer between the two pointwise convolutional layers. The first pointwise convolutional layer corresponds to the first channel expansion rate E to achieve an E-fold expansion of the number of channels. After that, the depthwise convolutional layer corresponds to the target convolution kernel K and does not change the number of channels. The last pointwise convolutional layer restores the number of channels to the number of input channels. The feature enhancement process of the local attention submodule based on the local attention mechanism is similar to the above description and will not be repeated here.

[0080] In a possible implementation, the FFN submodule corresponds to the second channel expansion rate d z , which is used to perform a one-step feature conversion on the output results of each global attention submodule and each local attention submodule. The feature conversion process of the FFN submodule is similar to the above description and will not be repeated here.

[0081] like Figure 4 The self-attention neural network shown in FIG. 1 includes the following components: when constructing the self-attention neural network, the module distribution parameters corresponding to each attention module (the number of global attention submodules G and the number of local attention submodules L included in each attention module), the module structure hyperparameters corresponding to each attention module (the number of target channels d corresponding to each attention module), and the number of local attention submodules L. k , the target convolution kernel K and the first channel expansion rate E corresponding to each local attention submodule, and the second channel expansion rate d corresponding to each FFN submodulez ), are all network hyperparameters that need to be considered.

[0082] In a possible implementation, the image processing method also includes: constructing a first network structure search space and a second network structure search space, wherein the first network structure search space includes multiple module distribution hyperparameters, and the second network structure search space includes multiple module structure hyperparameters; in the first network structure search space, determining a target module distribution hyperparameter from multiple module distribution hyperparameters, wherein the target module hyperparameter is used to indicate the number of global attention sub-modules and the number of local attention sub-modules included in each attention module; determining a target module structure hyperparameter from multiple module structure hyperparameters according to the target module distribution hyperparameter and the second network structure search space, wherein the target module structure hyperparameter is used to indicate the number of target channels corresponding to each attention module, the target convolution kernel and the first channel expansion rate corresponding to each local attention sub-module, and the second channel expansion rate corresponding to each feedforward module; constructing a self-attention neural network according to the target module distribution hyperparameter and the target module structure hyperparameter.

[0083] By constructing the first network structure search space and the second network structure search space, it is possible to perform a hierarchical network structure search on the target module distribution hyperparameters and the target structure hyperparameters based on the first network structure search space and the second network structure search space, so as to reduce the size of the search space corresponding to the network structure search at each level, effectively improve the search efficiency, and thus quickly construct a self-attention neural network.

[0084] Table 1 shows an example of the first network structure search space and the second network structure search space.

[0085] Table 1

[0086]

[0087] As shown in Table 1, the first network structure search space includes the possible distribution of global attention submodules and local attention submodules in each attention module. For example, (G, L) = (1, 2) is used to indicate that the attention module includes G = 1 global attention submodule and L = 2 local attention submodules.

[0088] The second network structure search space includes the module structure parameters corresponding to each attention module. For example, the number of target channels d corresponding to each attention module k (options include 96, 192, 384), the target convolution kernel K corresponding to each local attention submodule (options include 17, 31, 45) and the first channel expansion rate E (options include 1, 2, 4), and the second channel expansion rate d corresponding to each FFN submodule z(options include 2, 4, 6).

[0089] Module distribution hyperparameters (G, L) and module structure hyperparameters (K, E, d k d z ) may include, in addition to the options shown in Table 1 above, other options may be determined based on actual conditions. The number of options corresponding to each hyperparameter may also be determined based on actual conditions, and the present disclosure does not make any specific limitations on this.

[0090] Figure 5 A schematic diagram of a hierarchical network structure search according to an embodiment of the present disclosure is shown. Figure 5 As shown, first, a high-level network structure search is performed in the first network structure search space, and the target module distribution hyperparameters (G, L) are determined for each self-attention module from multiple module distribution hyperparameters. Then, the target module distribution hyperparameters (G, L) in each self-attention module are fixed, and a low-level network structure search is performed in the second network structure search space, and the target module structure hyperparameters (K, E, d k , d z ).

[0091] In a possible implementation, a target module distribution hyperparameter is determined from a plurality of module distribution hyperparameters according to a first network structure search space, including: constructing a first super network according to the first network structure search space, wherein the first super network includes a plurality of first optional network structures constructed according to the plurality of module distribution hyperparameters; determining a first target network structure from the plurality of first optional network structures by performing network training on the first super network; and determining the module distribution hyperparameter corresponding to the first target network structure as the target module distribution hyperparameter.

[0092] According to the multiple module distribution hyperparameters included in the first network structure search space, multiple first optional network structures can be constructed, and then a first super network including multiple first optional network structures can be constructed. By training the first super network, a high-level network structure search can be implemented in the first network structure search space, and the target first network structure with the best performance is determined from the multiple first optional network structures, and the module distribution hyperparameters corresponding to the target first network structure are determined as the target module distribution hyperparameters.

[0093] For example, the first super network is trained to obtain multiple first optional network architectures after training; the accuracy of the multiple first optional network architectures after training is verified based on an evolutionary algorithm, and a first preset number of first optional network architectures with the best accuracy ranking are selected from them; the selected preset number of first optional network architectures are trained again to determine a target first optional network architecture whose network performance meets the requirements, and then the module distribution hyperparameter corresponding to the target first optional network architecture is determined as the target module distribution hyperparameter. The size of the first preset number can be determined according to actual conditions, and the present disclosure does not make specific limitations on this.

[0094] like Figure 5 As shown, after performing the above-mentioned high-level network structure search in the first network structure search space shown in Table 1, the target module distribution hyperparameters corresponding to GL-Block_1 (G=1, L=2); the target module distribution hyperparameters corresponding to GL-Block_2 (G=3, L=0); ...; the target module distribution hyperparameters corresponding to GL-Block_M (G=0, L=3) are obtained.

[0095] In a possible implementation, target module structure hyperparameters are determined from a plurality of module structure hyperparameters according to the target module distribution hyperparameters and the second network structure search space, including: constructing a second super network according to the target module distribution hyperparameters and the second network structure search space, wherein the second super network includes a plurality of second optional network structures constructed according to the plurality of module structure hyperparameters; determining a second target network structure from a plurality of second optional network structures by performing network training on the second super network; and determining the module structure hyperparameters corresponding to the second target network structure as the target module structure hyperparameters.

[0096] The target module distribution hyperparameters corresponding to each attention module are fixed unchanged, and then according to the multiple module structure hyperparameters included in the second network structure search space, multiple second optional network structures can be constructed, and then a second super network including multiple second optional network structures can be constructed. By training the second super network, a low-level network structure search can be implemented in the second network structure search space, and the target second network structure with the best performance is determined from the multiple second optional network structures, and the module structure hyperparameters corresponding to the target second network structure are determined as the target module structure hyperparameters.

[0097] In one possible implementation, in the same attention module, each local attention sub-module corresponds to the same target convolution kernel and first channel expansion rate.

[0098] The construction principle of the self-attention neural network may include: for the same attention module, each local attention submodule in the attention module corresponds to the same target convolution kernel K and the first channel expansion rate E. For example, for the above GL-Block_1, GL-Block_1 includes L=2 local attention submodules, and the L=2 local attention submodules correspond to the same target convolution kernel K and the first channel expansion rate E. Based on the construction principle of the self-attention neural network, the second optional network structure included in the second super network that does not conform to the principle can be deleted to reduce the size of the second network search space and improve the subsequent search efficiency for the target second network structure.

[0099] After the second network structure search space is narrowed down based on the construction principle of the self-attention neural network, the second super network includes all the second optional network structures that conform to the construction principle of the self-attention neural network. The second super network is trained to perform low-level network structure search in the second network structure search space and determine the target module structure hyperparameters.

[0100] For example, the second super network is trained to obtain multiple second optional network architectures after training; the accuracy of the multiple second optional network architectures after training is verified based on the evolutionary algorithm, and the second optional network architectures with the best accuracy ranking of the second preset number are selected from them; the selected preset number of second optional network architectures are trained again to determine the target second optional network architecture whose network performance meets the requirements, and then the module structure hyperparameters corresponding to the target second optional network architecture are determined as the target structure hyperparameters. The size of the first preset number can be determined according to actual conditions, and the present disclosure does not make specific limitations on this.

[0101] like Figure 5 As shown in Table 1, after performing the above low-level network structure search in the second network structure search space, the target module structure hyperparameters (K = 17, E = 4, d k =92,d z =2); GL-Block_2 corresponding to the target module structure hyperparameters (K = 45, E = 1); ...; GL-Block_M corresponding to the target module structure hyperparameters (d k =192,d z =3).

[0102] A self-attention neural network can be constructed based on the target module distribution hyperparameters obtained by searching the high-level network structure and the target module structure hyperparameters obtained by searching the low-level network structure. The self-attention neural network includes at least one self-attention module, and at least one self-attention module includes a global attention submodule and a local attention submodule, so that the self-attention neural network can use both global information and local information during feature enhancement, thereby effectively improving the network performance of the self-attention neural network.

[0103] According to the needs of image processing, the self-attention neural network disclosed in the present invention can be applied to image processing tasks such as target detection, target tracking, image recognition, image classification, etc., and the present disclosure does not make specific limitations on this.

[0104] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0105] In addition, the present disclosure also provides an image processing device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0106] Figure 6 FIG. 1 is a block diagram of an image processing device according to an embodiment of the present disclosure. Figure 6 As shown, the device 60 includes:

[0107] A feature determination module 61, configured to determine a first image block feature corresponding to a target image, and to divide the first image block feature into a second image block feature and a third image block feature;

[0108] an attention module 62, configured to enhance the features of the second image block based on a global attention mechanism to obtain a fourth image block feature, and enhance the features of the third image block based on a local attention mechanism to obtain a fifth image block feature;

[0109] A first determining module 63, configured to determine a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature;

[0110] The target image processing module 64 is used to perform a target image processing operation on the target image according to the target image block characteristics to obtain an image processing result.

[0111] In a possible implementation, the number of channels corresponding to the first image block feature is the target number of channels;

[0112] The feature determination module 61 is specifically used for:

[0113] According to the target number of channels and the target ratio, the first image block feature is divided in the channel dimension to obtain the second image block feature and the third image block feature.

[0114] In a possible implementation, the attention module 62 includes: a global attention submodule, specifically configured to:

[0115] Determine a first feature vector, a second feature vector, and a third feature vector according to the second image block feature;

[0116] Determine an attention feature map according to the first feature vector and the second feature vector;

[0117] A fourth image block feature is determined based on the attention feature map and the third feature vector.

[0118] In a possible implementation, the attention module 62 further includes: a local attention submodule, specifically configured to:

[0119] The third image block feature is convolved according to the target convolution kernel and the first channel expansion rate to obtain a fifth image block feature.

[0120] In a possible implementation, the first determining module 63 is specifically configured to:

[0121] According to the second channel expansion rate, feature conversion is performed on the fourth image block feature and the fifth image block feature to obtain a target image block feature.

[0122] In a possible implementation, the device 60 implements the image processing method through a self-attention neural network, wherein the self-attention neural network includes at least one attention module 62, and the at least one attention module 62 includes a global attention submodule, a local attention submodule, and a feedforward submodule;

[0123] A global attention submodule, used for enhancing the features of the second image block based on a global attention mechanism to obtain features of the fourth image block;

[0124] A local attention submodule, used for enhancing the features of the third image block based on the local attention mechanism to obtain the features of the fifth image block;

[0125] The feedforward submodule is used to perform feature conversion on the fourth image block feature and the fifth image block feature to obtain a target image block feature.

[0126] In a possible implementation, the device 60 further includes:

[0127] A search space construction module, used to construct a first network structure search space and a second network structure search space, wherein the first network structure search space includes a plurality of module distribution hyperparameters, and the second network structure search space includes a plurality of module structure hyperparameters;

[0128] A second determination module is used to determine a target module distribution hyperparameter from a plurality of module distribution hyperparameters according to the first network structure search space, wherein the target module hyperparameter is used to indicate the number of global attention submodules and the number of local attention submodules included in each attention module;

[0129] A third determination module is used to determine a target module structure hyperparameter from a plurality of module structure hyperparameters according to the target module distribution hyperparameter and the second network structure search space, wherein the target module structure hyperparameter is used to indicate the number of target channels corresponding to each attention module, the target convolution kernel and the first channel expansion rate corresponding to each local attention submodule, and the second channel expansion rate corresponding to each feedforward module;

[0130] A network building module is used to build a self-attention neural network according to the target module distribution hyperparameters and the target module structure hyperparameters.

[0131] In a possible implementation manner, the second determining module is specifically configured to:

[0132] Constructing a first super network according to the first network structure search space, wherein the first super network includes a plurality of first optional network structures constructed according to a plurality of module distribution hyperparameters;

[0133] Determine a first target network structure from a plurality of first optional network structures by performing network training on the first super network;

[0134] The module distribution hyperparameter corresponding to the first target network structure is determined as the target module distribution hyperparameter.

[0135] In a possible implementation, the third determining module is specifically configured to:

[0136] Constructing a second super network according to the target module distribution hyperparameters and the second network structure search space, wherein the second super network includes a plurality of second optional network structures constructed according to a plurality of module structure hyperparameters;

[0137] By performing network training on the second super network, a second target network structure is determined from a plurality of second optional network structures;

[0138] The module structure hyperparameters corresponding to the second target network structure are determined as the target module structure hyperparameters.

[0139] In one possible implementation, in the same attention module, each local attention sub-module corresponds to the same target convolution kernel and first channel expansion rate.

[0140] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0141] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0142] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0143] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0144] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0145] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0146] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0147] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0148] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0149] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0150] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0151] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0152] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0153] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0154] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0155] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0156] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0157] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 8 As shown, the electronic device 1900 may be provided as a server. Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0158] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0159] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0160] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0161] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0162] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0163] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0164] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0165] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0166] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0167] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0168] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0169] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: include: Determining a first image block feature corresponding to a target image, and dividing the first image block feature into a second image block feature and a third image block feature; Performing feature enhancement on the second image block feature based on a global attention mechanism to obtain a fourth image block feature, and performing feature enhancement on the third image block feature based on a local attention mechanism to obtain a fifth image block feature; Determining a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature; According to the target image block features, a target image processing operation is performed on the target image to obtain an image processing result.

2. The method according to claim 1, characterized in that The number of channels corresponding to the first image block feature is the target number of channels; The dividing the first image block feature into a second image block feature and a third image block feature comprises: According to the target number of channels and the target ratio, the first image block feature is divided in the channel dimension to obtain the second image block feature and the third image block feature.

3. The method according to claim 1 or 2, characterized in that: The step of enhancing the feature of the second image block based on the global attention mechanism to obtain the feature of the fourth image block includes: Determine a first feature vector, a second feature vector and a third feature vector according to the second image block feature; Determine an attention feature map according to the first feature vector and the second feature vector; Determine the fourth image block feature based on the attention feature map and the third feature vector.

4. The method according to claim 1 or 2, characterized in that: The step of enhancing the third image block feature based on the local attention mechanism to obtain the fifth image block feature includes: The third image block feature is convolved according to the target convolution kernel and the first channel expansion rate to obtain the fifth image block feature.

5. The method according to claim 1 or 2, characterized in that: The determining, according to the fourth image block feature and the fifth image block feature, a target image block feature corresponding to the target image comprises: According to the second channel expansion rate, feature conversion is performed on the fourth image block feature and the fifth image block feature to obtain the target image block feature.

6. The method according to claim 1 or 2, characterized in that: The image processing method is implemented by a self-attention neural network, wherein the self-attention neural network includes at least one attention module, and at least one of the attention modules includes a global attention submodule, a local attention submodule and a feedforward submodule; The step of enhancing the feature of the second image block based on the global attention mechanism to obtain the feature of the fourth image block includes: Using the global attention submodule, based on the global attention mechanism, feature enhancement is performed on the second image block feature to obtain the fourth image block feature; The step of enhancing the third image block feature based on the local attention mechanism to obtain the fifth image block feature includes: Using the local attention submodule, based on the local attention mechanism, feature enhancement is performed on the third image block feature to obtain the fifth image block feature; The determining, according to the fourth image block feature and the fifth image block feature, a target image block feature corresponding to the target image comprises: The feedforward submodule is used to perform feature conversion on the fourth image block feature and the fifth image block feature to obtain the target image block feature.

7. The method according to claim 6, characterized in that The method further comprises: Constructing a first network structure search space and a second network structure search space, wherein the first network structure search space includes a plurality of module distribution hyperparameters, and the second network structure search space includes a plurality of module structure hyperparameters; According to the first network structure search space, determining a target module distribution hyperparameter from the multiple module distribution hyperparameters, wherein the target module hyperparameter is used to indicate the number of the global attention submodules and the number of the local attention submodules included in each attention module; Determine a target module structure hyperparameter from the multiple module structure hyperparameters according to the target module distribution hyperparameter and the second network structure search space, wherein the target module structure hyperparameter is used to indicate the target number of channels corresponding to each attention module, the target convolution kernel and the first channel expansion rate corresponding to each local attention submodule, and the second channel expansion rate corresponding to each feedforward module; The self-attention neural network is constructed according to the target module distribution hyperparameters and the target module structure hyperparameters.

8. The method according to claim 7, characterized in that The step of determining a target module distribution hyperparameter from the plurality of module distribution hyperparameters according to the first network structure search space includes: Constructing a first super network according to the first network structure search space, wherein the first super network includes a plurality of first optional network structures constructed according to the plurality of module distribution hyperparameters; Determine a first target network structure from the plurality of first optional network structures by performing network training on the first super network; The module distribution hyperparameter corresponding to the first target network structure is determined as the target module distribution hyperparameter.

9. The method according to claim 7, characterized in that: The step of determining a target module structure hyperparameter from the plurality of module structure hyperparameters according to the target module distribution hyperparameter and the second network structure search space comprises: Constructing a second super network according to the target module distribution hyperparameters and the second network structure search space, wherein the second super network includes a plurality of second optional network structures constructed according to the plurality of module structure hyperparameters; Determine a second target network structure from the plurality of second optional network structures by performing network training on the second super network; The module structure hyperparameters corresponding to the second target network structure are determined as the target module structure hyperparameters.

10. The method according to claim 7, characterized in that In the same attention module, each local attention sub-module corresponds to the same target convolution kernel and the first channel expansion rate.

11. An image processing device, characterized in that: include: A feature determination module, used to determine a first image block feature corresponding to a target image, and divide the first image block feature into a second image block feature and a third image block feature; an attention module, configured to perform feature enhancement on the second image block feature based on a global attention mechanism to obtain a fourth image block feature, and perform feature enhancement on the third image block feature based on a local attention mechanism to obtain a fifth image block feature; A first determining module, configured to determine a target image block feature corresponding to the target image according to the fourth image block feature and the fifth image block feature; The target image processing module is used to perform a target image processing operation on the target image according to the target image block characteristics to obtain an image processing result.

12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 10.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images