Image processing method and system based on neural network, storage medium and electronic equipment

Abstract

The invention provides an image processing method and system based on a neural network, a storage medium and electronic equipment, and relates to the technical field of neural networks, and the method comprises the steps: obtaining an initial tensor of a to-be-optimized feature image; in the first path, based on the initial tensor, performing convolution extraction to obtain a first tensor containing a channel dimension attention weight factor; in the second path, after the space dimension is converted into the channel dimension based on the initial tensor, a second tensor containing a space attention weight factor is obtained through convolution extraction; obtaining an attention weight matrix based on the first tensor and the second tensor; and based on the initial tensor and the attention weight matrix, obtaining the tensor of the feature image after attention enhancement. According to the method, the problem of insufficient global information extraction capability of the convolutional neural network is solved, pixel-level channel and spatial dimension attention modeling can be performed on the tensor of the feature image, and the correlation extraction and modeling capability of the convolutional neural network on the image features under a large scale is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to an image processing method, system, computer-readable storage medium and electronic device based on a neural network. Background Art

[0002] In recent years, deep learning image recognition technology based on deep convolutional neural networks (CNN) has been widely used in image classification, semantic segmentation and object detection tasks. Convolutional neural networks extract and condense image features through a series of stacked convolutional layers and pooling layers, thereby realizing a variety of recognition functions.

[0003] In the prior art, the calculation of spatial attention is to model channel attention and spatial attention respectively using a fully connected layer and a convolutional layer in series, which is mainly to strengthen the modeling of spatial attention. Since the features in each channel are treated equally, the information interaction between channels is ignored; while channel attention directly processes the information in a channel globally, which easily ignores the information interaction in space. In addition, the series connection method easily causes attention features of different dimensions to interfere with each other, affecting the final modeling effect. Summary of the invention

[0004] In response to the above problems, the present invention proposes an image processing method, system, computer-readable storage medium and electronic device based on a neural network, which can perform pixel-level channel and spatial attention modeling on feature images to improve image recognition performance.

[0005] In a first aspect, a neural network-based image processing method is provided, comprising:

[0006] Obtain the initial tensor of the feature image to be optimized after compression in the spatial dimension;

[0007] In the first path, based on the initial tensor, convolution extraction obtains a first tensor including a channel dimension attention weight factor;

[0008] In the second path, after transforming the spatial dimension to the channel dimension based on the initial tensor, convolution extraction obtains a second tensor containing the spatial attention weight factor;

[0009] Obtaining an attention weight matrix based on the first tensor and the second tensor;

[0010] Based on the initial tensor and the attention weight matrix, a tensor of the feature image after attention enhancement is obtained.

[0011] Furthermore, the step of obtaining an initial tensor of the feature image to be optimized after being compressed in the spatial dimension includes:

[0012] Based on the feature image to be optimized, multi-scale group convolution extraction is performed to obtain a feature enhanced processing tensor;

[0013] Based on the processed tensor, down-sampling is performed along the width direction and the height direction of the feature image to be optimized to obtain the initial tensor.

[0014] Furthermore, the multi-scale grouped convolution extraction based on the feature image to be optimized to obtain a feature enhanced processing tensor includes:

[0015] Based on the original tensor of the feature image to be optimized, splitting is performed along the channel direction to obtain multiple groups of split tensors;

[0016] Based on the multiple groups of split tensors, convolution kernels of different sizes are used to perform convolution respectively to obtain multiple first convolution tensors;

[0017] The processing tensor is obtained by performing a connection operation based on multiple first convolution tensors.

[0018] Furthermore, in the first path, based on the initial tensor, convolution extraction obtains a first tensor containing a channel dimension attention weight factor, including:

[0019] Perform multiple convolutions based on the initial tensor to obtain a first convolution tensor;

[0020] Performing a convolution on the first convolution tensor to obtain a second convolution tensor, where the number of channels of the second convolution tensor is the same as the number of channels of the initial tensor;

[0021] The first tensor is obtained by upsampling the second convolution tensor.

[0022] Furthermore, in the second path, after transforming the spatial dimension to the channel dimension based on the initial tensor, convolution extraction obtains a second tensor containing a spatial attention weight factor, including:

[0023] Performing a first reshaping operation based on the initial tensor to obtain a first reshaping tensor;

[0024] Perform multiple convolutions based on the first reshaped tensor to obtain a third convolution tensor,

[0025] Performing a second reshaping operation based on the third convolution tensor to obtain a second reshaped tensor, wherein the length, width, and number of channels of the second reshaped tensor are respectively the same as the length, width, and number of channels of the initial tensor;

[0026] The second tensor is obtained by upsampling based on the second reshaped tensor.

[0027] Furthermore, obtaining an attention weight matrix based on the first tensor and the second tensor includes:

[0028] Perform a connection operation in a channel direction based on the first tensor and the second tensor to obtain a connection tensor;

[0029] Performing convolution based on the connection tensor to obtain a fourth convolution tensor, wherein the number of channels of the fourth convolution tensor is the same as the number of channels of the initial tensor;

[0030] Substitute the fourth convolution tensor into the sigmoid activation function to obtain the attention weight matrix.

[0031] Further, based on the initial tensor and the attention weight matrix, a tensor after feature image attention enhancement is obtained, including:

[0032] A Hadamard product operation is performed based on the initial tensor and the attention weight matrix to obtain a tensor of the feature image after attention enhancement.

[0033] In a second aspect, a neural network-based image processing system is provided, comprising:

[0034] An acquisition module is used to obtain the initial tensor of the feature image to be optimized after being compressed in the spatial dimension;

[0035] A processing module, configured to, in a first path, convolute and extract, based on the initial tensor, a first tensor including a channel dimension attention weight factor;

[0036] A processing module is also used in the second path to convolve and extract a second tensor containing a spatial attention weight factor after transforming the spatial dimension to the channel dimension based on the initial tensor;

[0037] The processing module is further used to obtain an attention weight matrix based on the first tensor and the second tensor;

[0038] The processing module is also used to obtain a tensor after feature image attention enhancement based on the initial tensor and the attention weight matrix.

[0039] In a third aspect, a computer-readable storage medium is provided, storing a program or instruction, which, when executed on a computer, enables the computer to execute the neural network-based image processing method as described in any of the above schemes.

[0040] In a fourth aspect, an electronic device is provided, comprising: a processor, the processor being coupled to a memory,

[0041] The processor is used to read and execute the computer program stored in the memory to implement the neural network-based image processing method described in any of the above solutions.

[0042] One or more technical solutions provided by this application have at least the following technical effects or advantages:

[0043] First, the input feature image to be processed is subjected to multi-scale group convolution to extract features at different scales, providing more information for establishing the attention relationship model between features; then the extracted information is input into the dual-path attention module for attention modeling between features. This module includes two parallel network structures, which respectively perform attention calculations on the two dimensions of the channel relationship of the feature map and the image spatial relationship of each channel. The design of downsampling-attention relationship calculation-upsampling is used to achieve attention modeling for each pixel of the feature map while reducing the amount of calculation; finally, the attention information of the channel and spatial directions is fused through the feature fusion module to obtain the attention weight matrix. Through the processing of the above three modules, this algorithm can perform pixel-level channel and spatial attention modeling on the feature map tensor, effectively improving the ability of the convolutional neural network to extract and model the correlation of image features at a large scale.

[0044] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a flow chart of an image processing method based on a neural network provided in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the data geometry structure of a tensor provided in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a multi-scale convolution algorithm flow chart provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of a tensor dimension transformation process provided by an embodiment of the present invention;

[0050] Figure 5 Attention map (a) and overlay map (b) of the test sample with F+P microstructure;

[0051] Figure 6Attention map (a) and superposition map (b) of the S-back test sample with microstructure;

[0052] Figure 7 A schematic diagram of the structure of an image processing system based on a neural network provided by an embodiment of the present invention;

[0053] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The algorithm of the present invention needs to be integrated into the CNN backbone network to work, so a CNN image classification model with a 14-layer structure is first built. The network consists of 4 convolutional feature extractors and 1 fully connected classifier. Each feature extractor consists of 2 3×3 convolutional layers and 1 downsampling maximum pooling layer. The first extractor uses 4 times downsampling and the rest are 2 times. The fully connected classifier consists of 1 fully connected layer and 1 softmax output layer, which are used to output image classification results. The network input is a 224×224×3 color image, and the shape changes after 4 feature extraction layers to 224→56→28→14→7, and the number of channels changes to 3→64→144→256→576. The image processing method or image processing system of the present invention is added to the 3rd to 4th feature extractors. Taking the 3rd feature extractor as an example, the input tensor is set to X(256,28,28), where the height×width of X is 28×28 and the number of channels is 256.

[0056] Figure 1 The following is a flowchart of a neural network-based image processing method provided by an embodiment of the present invention. Figure 1 As shown, an image processing method based on a neural network in an embodiment of the present invention specifically includes the following steps:

[0057] S101: Obtain the initial tensor of the feature image to be optimized after compression in the spatial dimension.

[0058] The specific steps include:

[0059] S1011: Based on the feature image to be optimized, multi-scale group convolution extraction is performed to obtain a feature enhanced processing tensor.

[0060] S10111: Based on the original tensor of the feature image to be optimized, split it along the channel direction to obtain multiple groups of split tensors.

[0061] The feature image to be optimized l Z(C,H,W) is input into a feature extractor with an m-layer convolutional layer structure, and the original tensor of the feature image to be optimized is X(C,H,W). The process can be expressed by formula (1).

[0062] X(C,H,W)=f(Conv( 1 Z,k,g,s=1) m ) (1)

[0063] Among them, f() is the activation function, the convolution kernel scale, number of groups and other parameters in the Conv operator can be freely selected according to actual needs, C is the number of channels of X (Channel) and should be a square number, H and W are the height (Height) and width (width) of X respectively, and its data geometry structure is as follows Figure 2 The process of multi-scale group convolution is as follows Figure 3 As shown, it is mainly divided into the following steps:

[0064] The input tensor X is split along the C axis. The recommended number of splits is n. After splitting, a set of split tensors Xi with a width of W, a height of H, and a number of channels ci are obtained. The process can be expressed by formula (2).

[0065] X i (c i ,H,W) i＝1～n = split[X(c i ,H,W),axis=0] i＝1～n (2)

[0066] Among them, split() represents the tensor splitting operator; axis is the axis (dimension) direction along which the splitting is performed, axis=0 represents splitting along the channel direction, axis=1 represents splitting along the height direction, and axis=2 represents splitting along the width direction; c=[ci]i=1~n represents the splitting channel, satisfying

[0067] For example, the input tensor X(C,H,W) is split along the C axis, the number of splits is 4, and after splitting, a set of split tensors X1~X4 are obtained. This process can be expressed by formula (2-1).

[0068]

[0069] S10112: Based on the multiple groups of split tensors, convolution kernels of different sizes are used to perform convolution to obtain multiple first convolution tensors.

[0070] Split the tensor X into each group i Use convolution kernels of different sizes to perform convolution and obtain the first convolution tensor Y i : The process can be expressed by formula (3).

[0071] Y i (c i ,H,W) i＝1～n =Conv(X i ,k i ,g i ,d i ,s=1) i＝1～n (3)

[0072] Among them, Conv() is the convolution operator, s=1 represents the convolution step size is 1; k=[k i ] i＝1～n is the kernel size, i.e. the width and height of the convolution kernel; g = [g i ] i＝1～n is the number of convolution groups, gi and ci satisfy the integer divisibility relationship, d = [d i ] i＝1～n is the dilation rate of the convolution kernel. When d=1, conventional convolution is performed. When d≠1, dilated convolution is performed. The shape k' of the dilated convolution kernel and the dilation rate d satisfy the relationship d=(k'-1) / 2.

[0073] For example, each group of split tensors X1~X4 is convolved with convolution kernels of different sizes to obtain outputs Y1~Y4 respectively. This process can be expressed by formula (3-1).

[0074]

[0075] Among them, Conv() is the convolution operator, k = [5, 3, 3, 3] is the convolution kernel size; g = [126, 83, 10, 37] is the number of convolution groups, and d = [2, 3, 3, 1] is the expansion rate of the convolution kernel.

[0076] S10113: Perform a connection operation based on multiple first convolution tensors to obtain a processing tensor.

[0077] The first convolution tensor Y i The concatenation operation is performed along axis = 0 (channel direction) to obtain the processed tensor Y: This process can be expressed by formula (4).

[0078] Y(C,H,W)=concat(Y i ,axis=0) (4)

[0079] Among them, concat() represents the tensor connection operator, which means concatenating tensors along a certain axis.

[0080] For example, the convolution outputs Y1~Y4 are connected along axis=0 (channel direction) to obtain the processing tensor Y. This process can be expressed by formula (4-1).

[0081] Y(256,28,28)=concat(Y1,Y2,Y3,Y4,axis=0) (4-1)

[0082] S1012: Based on the processed tensor, downsample along the width direction and the height direction of the feature image to be optimized to obtain an initial tensor.

[0083] Down sampling is performed on the processing tensor Y along the width and height directions, that is, sampling is performed with a window of a certain size and step size, and the maximum value of the pixels in each window or the average value of the pixels in the window is taken as the output to obtain the initial tensor Y1 with a reduced shape. This process is expressed by formula (5).

[0084]

[0085] Where r1 is the spatial compression ratio and satisfies r1>1, downsample() is the downsampling operator, which can be average pooling, maximum pooling, convolution, etc.

[0086] For example, the maximum pooling method is used here for downsampling, and the process is given by formula (5-1).

[0087] Y1(256,5,5)=MaxPool[Y(256,28,28)] (5-1)

[0088] Where r1=5 is the spatial compression ratio, and MaxPool() is the maximum pooling operator.

[0089] S102: In the first path, based on the initial tensor, convolution extraction obtains a first tensor containing a channel dimension attention weight factor.

[0090] Specifically, S1021: Perform multiple convolutions based on the initial tensor to obtain a first convolution tensor.

[0091] Perform multiple 1×1 convolutions (the width and height of the convolution kernel are both 1) on the downsampled initial tensor Y1 to obtain the first convolution tensor Y 11 , the process is expressed by formula (6).

[0092]

[0093] Wherein, r2 is the channel compression ratio and satisfies r2>1.

[0094] For example, perform two 1×1 convolutions on the initial tensor Y1 to obtain the first convolution tensor Y 11 , the process is given by formula (6-1).

[0095] Y 11 (16,5,5)=Conv(Conv(Y1(256,5,5),k=1,s=1),k=1,s=1) (6-1)

[0096] Among them, r2=16 is the channel compression ratio.

[0097] S1022: Perform a convolution based on the first convolution tensor to obtain a second convolution tensor, where the number of channels of the second convolution tensor is the same as the number of channels of the initial tensor.

[0098] For the first convolution tensor Y 11 Perform a 1×1 convolution so that the second convolution tensor Y 12 The number of channels of is the same as that of Y1, and the process is expressed by equation (7).

[0099]

[0100] For example, for the first convolution tensor Y 11 Perform 1×1 convolution to obtain the second convolution tensor Y 12 , Y 12 The number of channels is the same as Y1, and the process is expressed by formula (7-1).

[0101] Y 12 (256, 5, 5) = Conv(Y 11 (16,5,5),k=1,s=1) (7-1)

[0102] S1023: Obtain a first tensor based on upsampling the second convolution tensor.

[0103] For the second convolution tensor Y 12 Upsampling is performed to increase the spatial resolution (increase the number of pixels in the width and height directions), and the first tensor Y 13 The width and height of should be the same as the processing tensor Y. The process is expressed by formula (8).

[0104]

[0105] Among them, upsample() is an upsampling operator, which can be operations such as unpooling, linear interpolation, and transpose convolution.

[0106] For example, for the second convolution tensor Y 12 Upsampling is performed, and bilinear interpolation upsampling is used here. The width and height of the output result should be the same as the processed tensor Y. The process is expressed by formula (8-1).

[0107] Y 13 (256,28,28)=interpolate[Y 12 (256,5,5), mode=bilinear] (8-1)

[0108] In formula (25), interpolate() is a linear interpolation operator, and “bilinear” means that a bilinear interpolation method is used.

[0109] S103: In the second path, after transforming the spatial dimension to the channel dimension based on the initial tensor, convolution extraction is performed to obtain a second tensor containing a spatial attention weight factor.

[0110] Specifically include:

[0111] S1031: Perform a first reshaping operation based on the initial tensor to obtain a first reshaping tensor.

[0112] Reshape the initial tensor Y1 to obtain the first reshaped tensor Y 21 This operation can be divided into three steps. First, all pixels of the initial tensor Y1 are expanded in the width and height directions to obtain a transition form with a shape of (C, HW); second, all channels are arranged in a square manner in the channel direction to obtain a shape of The transition form of ; finally, the tensor direction axis is rearranged to obtain the final shape The output result of this process is to transform the spatial dimensions (width and height) of the feature map into the channel direction, such as Figure 4 As shown, the process is generally represented by equation (9).

[0113]

[0114] Among them, reshape() represents a tensor deformation operator, which can transform the shape of the input tensor along different axis directions.

[0115] For example, reshape the initial tensor Y1 to obtain the first reshaped tensor Y 21This operation can be divided into three steps. First, expand all pixels of Y1 in the width and height directions to obtain a transition form with a shape of (256,25); second, arrange all channels in a square manner in the channel direction to obtain a shape of transition form; finally, the tensor direction axes are rearranged to obtain the output result with a final shape of (25, 16, 16). The process is generally expressed by formula (9-1).

[0116] Y 21 (25,16,16)=reshape[Y1(256,5,5)] (9-1)

[0117] Among them, reshape() represents a tensor deformation operator, which can transform the shape of the input tensor along different axis directions.

[0118] S1032: Perform multiple convolutions based on the first reshaped tensor to obtain a third convolution tensor.

[0119] First reshape the tensor Y 21 After multiple 1×1 convolutions, the third convolution tensor Y is obtained. 22 , the process is expressed by formula (10).

[0120]

[0121] For example, Y 22 (25,16,16)=Conv(Conv(Y 21 (25,16,16),k=1,s=1),k=1,s=1)(10-1)

[0122] S1033: Perform a second reshaping operation based on the third convolution tensor to obtain a second reshaped tensor, wherein the length, width, and number of channels of the second reshaped tensor are respectively the same as the length, width, and number of channels of the initial tensor.

[0123] For the third convolution tensor Y 22 Perform a reshape operation to obtain the second reshape tensor Y 23 , so that the second reshape tensor Y 23 The shape of is restored to the same as the initial tensor Y1, and this process is expressed by equation (11).

[0124]

[0125] For example, for the third convolution tensor Y 22 Perform a reshape operation to obtain the second reshape tensor Y 23 , its shape is restored to the same as the initial tensor Y1, and this process is expressed by formula (11-1).

[0126] Y 23(256,5,5)=reshape[Y 22 (25,16,16)] (11-1)

[0127] S1034: Perform upsampling based on the second reshaped tensor to obtain a second tensor.

[0128] Reshape the second tensor Y 23 Perform upsampling operation to obtain the second tensor Y 24 , so that Y 24 The shape of is restored to the original tensor X(C,H,W), and the process is expressed by formula (12).

[0129]

[0130] For example, for Y 23 Perform bilinear interpolation upsampling to restore its shape to (256, 28, 28). The process is expressed by formula (12-1).

[0131] Y 24 (256,28,28)=interpolate[Y 23 (256,5,5), mode=bilinear] (12-1)

[0132] S104: Obtain an attention weight matrix based on the first tensor and the second tensor.

[0133] Specifically include:

[0134] S1041: Perform a connection operation in the channel direction based on the first tensor and the second tensor to obtain a connection tensor.

[0135] The first tensor Y 13 and the second tensor Y 24 Perform a connection operation along the channel direction to obtain the connection tensor Y 41 , the process is expressed by formula (13).

[0136] Y 41 (2C,H,W)=concat(Y 13 ,Y 24 ,axis=0) (13)

[0137] For example, the first tensor Y 13 and the second tensor Y 24 Perform the connection operation along the channel direction to obtain the connection tensor Y 41 , the process is expressed by formula (13-1).

[0138] Y 41 (512,28,28)=concat(Y 13 ,Y24 ,axis=0) (13-1)

[0139] S1042: Perform convolution based on the connection tensor to obtain a fourth convolution tensor, where the number of channels of the fourth convolution tensor is the same as the number of channels of the initial tensor.

[0140] Connect the tensor Y 41 Perform 1×1 convolution to obtain the fourth convolution tensor Y 42 , so that the fourth convolution tensor Y 42 The number of channels is restored to the initial value C.

[0141] For example, for Y 41 Perform a 1×1 convolution to obtain the fourth convolution tensor Y 42 The number of channels is restored to the initial value of 256.

[0142] S1043: The fourth convolution tensor Y 42 Substitute the sigmoid activation function to get the attention weight matrix Y re , the process is expressed by formula (14).

[0143] Y re (C,H,W)=sigmoid(Conv(Y 42 ,k=1,k=1)) (14)

[0144] The expression of the sigmoid function is:

[0145] y = sigmoid(x) = (1 + exp(-x)) -1 (15)

[0146] For example, Y 42 Then bring in the sigmoid activation function to get the attention weight matrix Y re , the process is expressed by formula (14-1).

[0147] Y re (256,28,28)=sigmoid(Conv(Y 42 (512,28,28),k=1,k=1)) (14-1)

[0148] S105: Based on the initial tensor and the attention weight matrix, a tensor of the feature image after attention enhancement is obtained.

[0149] Specifically, we can use the initial tensor X(C,H,W) and the attention weight matrix Y re Perform Hadamard product operation, that is, multiply the corresponding elements to obtain the tensor after feature image attention enhancement l+1Z, the process can be expressed by formula (15).

[0150]

[0151] If residual connection is considered, equation (16) can be further written as:

[0152]

[0153] In formula (15), when the number of channels of the output feature map is different from that of the input, it is necessary to transform the number of channels of the input X to the same as that of the output through a 1×1 convolution. l+1 Z is the same, and then the addition operation is performed.

[0154] The CNN backbone network equipped with the neural network-based image processing method described in this embodiment is trained. The data set used is a steel microstructure classification data set, which includes 8 types of typical microstructures, a total of 4000 training samples, 400 verification samples and 400 test samples. The model is trained with an adam optimizer at a learning rate of 0.001 for 50 cycles and then stopped.

[0155] After the training is completed, an image with a microstructure of F+P is selected as a test sample and input into the model. Check a channel of the attention map output by the attention algorithm of the present invention in the third feature extractor. For the convenience of observation, it is interpolated to 224×224, such as Figure 5 shown.

[0156] Select the test image input model with S microstructures again, check a channel of the attention map output by the attention algorithm of the present invention in the third feature extractor, and interpolate it to 224×224, such as Figure 6 shown.

[0157] Figure 7 The present invention also provides an image processing system based on a dual-channel convolutional neural network, such as Figure 7 As shown, the system comprises:

[0158] An acquisition module 201 is used to acquire an initial tensor of the feature image to be optimized after being compressed in the spatial dimension;

[0159] A processing module 202 is used for, in a first path, convolutionally extracting, based on the initial tensor, obtaining a first tensor with an increased spatial dimension attention weight;

[0160] The processing module 202 is further configured to, in a second path, transform the spatial dimension into the channel dimension based on the initial tensor, and then perform convolution extraction to obtain a second tensor;

[0161] The processing module 202 is further configured to obtain an attention weight matrix based on the first tensor and the second tensor;

[0162] The processing module 202 is also used to obtain a tensor after feature image attention enhancement based on the initial tensor and the attention weight matrix.

[0163] It should be noted here that the above-mentioned acquisition module 201 and processing module 202 correspond to steps S101 to S105 in the embodiment of the image processing method based on a neural network. The examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiments.

[0164] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0165] First, for the input feature image to be processed, features at different scales are extracted through multi-scale grouped convolution to provide more information for establishing the attention relationship model between features; then the extracted information is input into the dual-path attention module for attention modeling between features. This module includes two parallel network structures, which perform attention calculations on the two dimensions of the channel relationship of the feature map and the image spatial relationship of each channel. The design of downsampling-attention relationship calculation-upsampling is used to achieve attention modeling for each pixel in the feature map while reducing the amount of calculation; finally, the feature fusion module is used to fuse the attention information in the channel and spatial directions to obtain the attention weight matrix. Through the processing of the above three modules, this algorithm can perform pixel-level channel and spatial attention modeling on the feature map tensor, effectively improving the convolutional neural network's ability to extract and model the correlation of image features at large scales.

[0166] An embodiment of the present invention also provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed on a computer, the computer executes the image processing method based on the dual-channel convolutional neural network as described in Embodiment 1.

[0167] like Figure 8 As shown, an embodiment of the present invention further provides an electronic device 300, including: a processor 301, the processor 301 is coupled to a memory 302,

[0168] The processor 301 is used to read and execute the computer program stored in the memory to implement the image processing method based on the dual-channel convolutional neural network as described in the above method embodiment.

[0169] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network-based image processing method, characterized in that: include: Obtain the initial tensor of the feature image to be optimized after compression in the spatial dimension; In the first path, based on the initial tensor, convolution extraction obtains a first tensor including a channel dimension attention weight factor; In the second path, after transforming the spatial dimension to the channel dimension based on the initial tensor, convolution extraction obtains a second tensor containing the spatial attention weight factor; Obtaining an attention weight matrix based on the first tensor and the second tensor; Based on the initial tensor and the attention weight matrix, a tensor of the feature image after attention enhancement is obtained.

2. The method according to claim 1, characterized in that The step of obtaining an initial tensor of the feature image to be optimized after being compressed in the spatial dimension includes: Based on the feature image to be optimized, multi-scale group convolution extraction is performed to obtain a feature enhanced processing tensor; Based on the processed tensor, down-sampling is performed along the width direction and the height direction of the feature image to be optimized to obtain the initial tensor.

3. The method according to claim 2, characterized in that The method of extracting a feature-enhanced processing tensor based on the feature image to be optimized by multi-scale grouping convolution includes: Based on the original tensor of the feature image to be optimized, splitting is performed along the channel direction to obtain multiple groups of split tensors; Based on the multiple groups of split tensors, convolution kernels of different sizes are used to perform convolution respectively to obtain multiple first convolution tensors; The processing tensor is obtained by performing a connection operation based on multiple first convolution tensors.

4. The method according to claim 1, characterized in that: In the first path, based on the initial tensor, convolution extraction obtains a first tensor containing a channel dimension attention weight factor, including: Perform multiple convolutions based on the initial tensor to obtain a first convolution tensor; Performing a convolution on the first convolution tensor to obtain a second convolution tensor, where the number of channels of the second convolution tensor is the same as the number of channels of the initial tensor; The first tensor is obtained by upsampling the second convolution tensor.

5. The method according to claim 1, characterized in that In the second path, after transforming the spatial dimension to the channel dimension based on the initial tensor, convolution extraction obtains a second tensor containing the spatial attention weight factor, including: Performing a first reshaping operation based on the initial tensor to obtain a first reshaping tensor; Perform multiple convolutions based on the first reshaped tensor to obtain a third convolution tensor, Performing a second reshaping operation based on the third convolution tensor to obtain a second reshaped tensor, wherein the length, width, and number of channels of the second reshaped tensor are respectively the same as the length, width, and number of channels of the initial tensor; The second tensor is obtained by upsampling based on the second reshaped tensor.

6. The method according to claim 1, characterized in that The obtaining an attention weight matrix based on the first tensor and the second tensor includes: Perform a connection operation in a channel direction based on the first tensor and the second tensor to obtain a connection tensor; Performing convolution based on the connection tensor to obtain a fourth convolution tensor, wherein the number of channels of the fourth convolution tensor is the same as the number of channels of the initial tensor; Substitute the fourth convolution tensor into the sigmoid activation function to obtain the attention weight matrix.

7. The method according to claim 1, characterized in that Based on the initial tensor and the attention weight matrix, a tensor after feature image attention enhancement is obtained, including: A Hadamard product operation is performed based on the initial tensor and the attention weight matrix to obtain a tensor of the feature image after attention enhancement.

8. An image processing system based on a neural network, characterized in that: include: An acquisition module is used to obtain the initial tensor of the feature image to be optimized after being compressed in the spatial dimension; A processing module, configured to, in a first path, convolute and extract, based on the initial tensor, a first tensor including a channel dimension attention weight factor; A processing module is also used in the second path to convolve and extract a second tensor containing a spatial attention weight factor after transforming the spatial dimension to the channel dimension based on the initial tensor; The processing module is further used to obtain an attention weight matrix based on the first tensor and the second tensor; The processing module is also used to obtain a tensor after feature image attention enhancement based on the initial tensor and the attention weight matrix.

9. A computer-readable storage medium, characterized in that: A program or instruction is stored, and when the program or instruction is run on a computer, the computer is enabled to execute the neural network-based image processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: a processor coupled to the memory, The processor is used to read and execute the computer program stored in the memory to implement the neural network-based image processing method as described in any one of claims 1 to 7.

Images