Image processing method, apparatus, device, and storage medium
By combining global average pooling and global maximum pooling with one-dimensional channel convolution kernels, the problem of feature loss and increased computation caused by existing attention mechanisms is solved, thereby improving the performance of computer vision models and the accuracy of object detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUOQI INTELLIGENT CONTROL (CHONGQING) TECH CO LTD
- Filing Date
- 2023-04-23
- Publication Date
- 2026-06-09
Smart Images
Figure CN116468905B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology
[0002] Attention mechanisms are widely used in computer vision models such as object detection and image classification. They can extract more relevant information about the target from image features and eliminate interfering information, thereby improving the performance of the model.
[0003] The mainstream attention mechanism in existing technologies mainly involves passing the input features through two fully connected layers, then obtaining the weights for different channels through activation functions, and finally multiplying the input features by the weights of each channel to obtain the output result.
[0004] However, this approach is prone to feature loss when input features pass through fully connected layers, leading to a decrease in the performance of the computer vision model and resulting in poor computer vision results. Summary of the Invention
[0005] This application provides an image processing method, apparatus, device, and storage medium to address the problem that existing attention mechanism methods have defects that lead to poor computer vision results.
[0006] In a first aspect, embodiments of this application provide an image processing method, including:
[0007] Obtain the input features and number of channels of the image, and calculate the global average pooling value and global maximum pooling value of the input features;
[0008] Based on the number of channels, a one-dimensional channel convolution kernel is calculated;
[0009] Based on the one-dimensional channel convolution kernel, calculate the convolution result of the global average pooling value and the convolution result of the global maximum pooling value;
[0010] The weight of each channel in the input feature is determined based on the convolution result of the global average pooling value and the convolution result of the global maximum pooling value.
[0011] Based on the input features and the weight of each channel, a target feature map is calculated, which is used for target detection.
[0012] In one possible design of the first aspect, calculating the global average pooling value and the global maximum pooling value of the input feature includes:
[0013] Based on the input features, obtain the width, height, and pixel value of the i-th row and j-th column of the image, where i and j are both positive integers;
[0014] The global average pooling value is calculated based on the width, height, and pixel value of the i-th row and j-th column of the image.
[0015] Obtain the maximum value among the pixel values in the i-th row and j-th column, and use it as the global maximum pooling value.
[0016] In another possible design of the first aspect, calculating the global average pooling value based on the width, height, and pixel value in the i-th row and j-th column of the image includes:
[0017]
[0018] In the above formula, g(x) represents the global average pooling value, W represents the width of the image, H represents the height of the image, and x i，j This represents the pixel value in the i-th row and j-th column;
[0019] Correspondingly, obtaining the maximum value among the elements in the i-th row and j-th column, as the global maximum pooling value, includes:
[0020]
[0021] In the above formula, h(x) represents the global maximum pooling value, W represents the width of the image, H represents the height of the image, and x i，j This represents the pixel value in the i-th row and j-th column.
[0022] In another possible design of the first aspect, the calculation of the one-dimensional channel convolution kernel based on the number of channels includes:
[0023]
[0024] In the above formula, k represents the size of the one-dimensional channel convolution kernel, b and γ are constants, C represents the number of channels, and odd represents taking the odd number closest to the calculated value.
[0025] In another possible design of the first aspect, the step of calculating the convolution result of the global average pooling value and the convolution result of the global maximum pooling value based on the one-dimensional channel convolution kernel includes:
[0026]
[0027]
[0028] In the above formula, w 1i w represents the convolution result of the global average pooling value. 2i The convolution result representing the global maximum pooling value, α j Ω is the j-th element in the one-dimensional channel convolution kernel α. 1iΩ represents the set of elements in every k adjacent channels of g(x). 2i Let h(x) represent the set of elements in every k adjacent channels, and C represent the number of element sets. Denotes the i-th set Ω 1i The element of the j-th channel, Denotes the i-th set Ω 2i The element of the j-th channel, where i and j are both positive integers.
[0029] In another possible design of the first aspect, determining the weight of each channel in the input feature based on the convolution result of the global average pooling value and the convolution result of the global maximum pooling value includes:
[0030] ω i =σ(w1i+w 2i ), i∈C
[0031] In the above formula, ω represents the weight of the i-th channel, σ represents the activation function, and w 1i w represents the convolution result of the global average pooling value. 2i This represents the convolution result of the global maximum pooling value, where C represents the number of elements in the set.
[0032] In another possible design of the first aspect, the step of calculating the target feature map based on the input features and the weights of each channel includes:
[0033] The input features are multiplied by the weights to obtain the multiplication result;
[0034] The input features are input into a convolutional block, and the result of the convolutional block is calculated. The convolutional block includes at least a convolutional layer, a normalization layer, and an activation function.
[0035] The target feature map is obtained by summing the multiplication result and the convolution block result.
[0036] Secondly, embodiments of this application provide an image processing apparatus, comprising:
[0037] The pooling value calculation module is used to obtain the input features and number of channels of the image, and calculate the global average pooling value and the global maximum pooling value of the input features;
[0038] The convolution kernel calculation module is used to calculate a one-dimensional channel convolution kernel based on the number of channels;
[0039] The convolution result calculation module is used to calculate the convolution result of the global average pooling value and the convolution result of the global maximum pooling value based on the one-dimensional channel convolution kernel.
[0040] The weight determination module is used to determine the weight of each channel in the input feature based on the convolution result of the global average pooling value and the convolution result of the global maximum pooling value.
[0041] The feature map calculation module is used to calculate the target feature map based on the input features and the weight of each channel, and the target feature map is used for target detection.
[0042] Thirdly, embodiments of this application provide a computer device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method described above.
[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement the above-described method.
[0044] The image processing method, apparatus, device, and storage medium provided in this application obtain global average pooling value and global maximum pooling value by using global average pooling and global maximum pooling, and calculate the convolution result of global average pooling value and global maximum pooling value by sharing a one-dimensional channel convolution kernel for the two pooling values. This can reduce the number of parameters and avoid changes in the number of ascending and descending channels, reduce the occupation of computing resources and feature loss, and improve computer vision effects. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application;
[0046] Figure 1 A schematic diagram of the road environment provided for an embodiment of this application;
[0047] Figure 2 A schematic flowchart of the image processing method provided in the embodiments of this application;
[0048] Figure 3 This is a schematic diagram of the module structure provided in an embodiment of this application;
[0049] Figure 4 A schematic diagram of a one-dimensional channel convolution operation provided in an embodiment of this application;
[0050] Figure 5 A schematic diagram of the activation function provided in the embodiments of this application;
[0051] Figure 6The detection results provided in this application are based on YOLOv3 with the new module SECA-s applied.
[0052] Figure 7 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;
[0053] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this application.
[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0056] Computer vision is the science of enabling machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. In the field of computer vision, attention mechanisms are widely used for target detection and image classification. These mechanisms extract features from images to obtain information more relevant to the target, eliminating interfering information and thus improving model performance. For example... Figure 1 A schematic diagram of the road environment provided for an embodiment of this application, such as... Figure 1As shown, autonomous vehicles can acquire images 11 through sensors such as cameras, including vehicles 110, 111, 112, and several pedestrians. These are relatively important pieces of information that can provide reference for autonomous driving, while environmental background and other factors are relatively less important and need to be excluded from the images. However, in practical applications, since the image 11 captured contains relatively little feature information about vehicle 112, if the attention mechanism method is not effective, it may be impossible to obtain information about the key vehicle 112, which may lead to safety hazards for autonomous driving. In related technologies, attention mechanism methods mainly include the following three types: (I) Squeeze-and-Excitation Networks (SE-Net). In SE-Net, the input features are first processed through two fully connected layers. Let the first fully connected layer be fc1, and the input features are adjusted to a 1x1xC1 vector, i.e., x1 = fc1(x), x1 ∈ 1×1×C1. Let the second fully connected layer be fc2, i.e., x2 = fc2(x1), x2 ∈ 1×1×C. Then, x2 is passed through the sigmoid activation function to obtain the weights w = sigmoid(x2) for each channel. Finally, multiplying the input feature x by the corresponding channel w yields the result after SE-Net. Since this method goes through two fully connected layers, the input features are first reduced in dimensionality and then increased in dimensionality, which can cause feature loss and greatly increases the computational cost, making it unsuitable for practical deployment. (II) Convolutional Block Attention Module (CBAM) is a mechanism that includes a channel attention mechanism and a spatial attention mechanism. It has a higher computational cost and consumes more computational resources. It also suffers from feature loss due to changes in dimensionality. (III) Efficient Channel Attention for Deep Convolutional Neural Networks (ECA-Net) employs a non-dimensionality reduction approach, using one-dimensional channel convolutions instead of fully connected layers to achieve interaction between adjacent channels, significantly reducing the number of parameters. It can be embedded into other convolutional networks, achieving a significant improvement in accuracy without sacrificing detection speed, while also enhancing model interpretability. However, ECA-Net suffers from significant mean shift in estimation and cannot extract texture information of targets in images.
[0057] To address the shortcomings of the aforementioned attention mechanisms, this application provides an image processing method, apparatus, device, and storage medium. Based on the ECA channel attention mechanism, a new module SECA-s is obtained and applied to a computer vision model. This module can acquire more useful information for inference, thereby improving the overall performance of the model.
[0058] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0059] Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. This method can be applied to a computer device. Taking a computer device as the executing entity as an example, as... Figure 2 As shown, the method may specifically include the following steps:
[0060] Step S201: Obtain the input features and number of channels of the image, and calculate the global average pooling value and global maximum pooling value of the input features.
[0061] For example, the image can be an image collected by sensors mounted on an autonomous vehicle in the field of intelligent driving. It is usually a color image, which includes three color channels: R, G, and B, i.e., the number of channels is 3. In addition, the color image usually includes information such as road signs, pedestrians, other vehicles, and background environment. By applying attention mechanism methods to the color image, useful information (such as pedestrians and other vehicles) can be extracted, while the attention to other less useful information (such as background environment) can be reduced.
[0062] In this embodiment, the image can be represented by a tensor (H, W, N), where H represents the height of the image, W represents the width of the image, and N represents the number of channels of the image. For example, the image can be input into a convolutional kernel to extract this tensor.
[0063] For example, Figure 3 This is a schematic diagram of the module structure provided in an embodiment of this application. This module is used to implement the image processing method provided in this application. Figure 3 As shown, the region enclosed by the first dashed box 300 can be considered a new module SECA-s, which is used to implement the attention mechanism method provided in this application. The first dashed box 300 includes a second dashed box 301, which is used to subsequently perform channel convolution on the global average pooling value and the global maximum pooling value to obtain the convolution result of the global average pooling value and the convolution result of the global maximum pooling value. Furthermore, Figure 3 middle This indicates that corresponding elements are added together. The expression represents element-wise multiplication, σ represents the sigmoid activation function, and x represents the input feature. Additionally, the first dashed box 301 includes two pooling layers (i.e., a first pooling layer and a second pooling layer). The first pooling layer performs max pooling on the input features to obtain the global maximum pooling value, while the second pooling layer performs mean pooling on the input features to obtain the global average pooling value. It should be noted that the first and second pooling layers are parallel and can be executed simultaneously, outputting different results (i.e., the global average pooling value and the global maximum pooling value).
[0064] Step S202: Calculate the one-dimensional channel convolution kernel based on the number of channels.
[0065] In this embodiment, the value k represents the size of the one-dimensional channel convolution kernel. The size k of the one-dimensional channel convolution kernel is calculated based on the number of channels of the input feature. One-dimensional channel convolution enables information interaction among the channels of the input feature within the range of the kernel size. This interaction range varies depending on the number of input feature channels and the type of convolution kernel. Therefore, there is a correlation between the value k and the number of input feature channels: the more channels, the larger the interaction range; conversely, the fewer channels, the smaller the interaction range. In other words, there is a mapping relationship between the number of channels and the value k, and the size of the one-dimensional channel convolution kernel can be determined based on this mapping relationship.
[0066] For example, the size of the one-dimensional channel convolution kernel can also be set manually, such as by manually adjusting the size of the one-dimensional channel convolution kernel based on the number of channels of the input features.
[0067] Step S203: Calculate the convolution result of the global average pooling value and the convolution result of the global maximum pooling value based on the one-dimensional channel convolution kernel.
[0068] In this embodiment, the global average pooling value and the global maximum pooling value share a single channel convolution kernel (i.e., a one-dimensional channel convolution kernel). For example, continue to refer to... Figure 3 After performing channel convolution using a one-dimensional channel convolution kernel, the convolution results with global average pooling value and global maximum pooling value are output respectively.
[0069] Step S204: Determine the weight of each channel in the input feature based on the convolution results of the global average pooling value and the global maximum pooling value.
[0070] In this embodiment, the sum of the two convolution results can be normalized to a value between 0 and 1 using an activation function, and this value can be used as the weight for each channel. Different channels have corresponding convolution results; for example, the convolution results using the global average pooling value include the global average pooling values for a total of C channels.
[0071] Step S205: Calculate the target feature map based on the input features and the weight of each channel. The target feature map is used for target detection.
[0072] In this embodiment, the input features and the weights of each channel are multiplied to calculate the target feature map. This target feature map can be used for subsequent target detection, such as detecting useful feature information, which may include pedestrians, vehicles, road signs, etc.
[0073] The embodiments of this application obtain global average pooling value and global maximum pooling value by using global average pooling and global maximum pooling, and share a one-dimensional channel convolution kernel for the two pooling values to calculate the convolution result of the global average pooling value and the convolution result of the global maximum pooling value. This can reduce the number of parameters and avoid changes in the number of ascending and descending channels, reduce the occupation of computing resources and feature loss, and improve the computer vision effect.
[0074] In other embodiments, the step S201 above, "calculating the global average pooling value and the global maximum pooling value of the input features," can be implemented through the following steps: Based on the input features, obtain the width, height, and pixel value of the i-th row and j-th column of the image; calculate the global average pooling value based on the width, height, and pixel value of the i-th row and j-th column of the image; obtain the maximum value among the pixel values of the i-th row and j-th column as the global maximum pooling value. Here, i and j are both positive integers.
[0075] In this embodiment, it can be assumed that the input feature is x∈R W×H×C Here, W represents the width of the image, H represents the height of the image, and C represents the number of channels. The global average pooling value g(x) and the global maximum pooling value h(x) can be calculated from these values, both with dimensions 1x1xC.
[0076]
[0077] In the above formula, g(x) represents the global average pooling value, W represents the width of the image, H represents the height of the image, and x i，j x i，j This represents the pixel value in the i-th row and j-th column;
[0078]
[0079] In the above formula, h(x) represents the global maximum pooling value, W represents the width of the image, H represents the height of the image, and x i，j This represents the pixel value in the i-th row and j-th column.
[0080] In this embodiment, the simultaneous use of max pooling and mean pooling enables the extraction of more useful information for subsequent object detection, which is a significant improvement over current Efficient Channel Attention (ECA) modules. (Continue to refer to...) Figure 3 Mean pooling calculates the average of all elements in a region during forward propagation. During weight updates, it distributes the gradient of a subsequent element, passed down as n equal parts, to the previous layer, ensuring consistent gradients before and after pooling. This reduces the variance of the estimate caused by limited neighborhood size and preserves more background information. Max pooling calculates the maximum value in a region and records its position, discarding other elements. During backpropagation, it also adheres to the gradient sum invariance principle, directly passing the gradient of subsequent elements to the maximum value. This reduces interference from useless information, improves computation speed, reduces mean shift caused by convolutional layer parameter errors, preserves more texture information, increases nonlinearity and feature invariance, reduces overfitting risk, and improves model generalization ability.
[0081] This application embodiment uses global average pooling and global maximum pooling simultaneously to obtain global average pooling value and global maximum pooling value respectively. This can expand the receptive field while reducing the number of parameters, improve the performance of the computer vision model, and obtain more comprehensive target and background information, thereby improving the computer vision effect.
[0082] In other embodiments, the calculation of the one-dimensional channel convolution kernel size can be achieved through the following steps:
[0083]
[0084] In the above formula, k represents the size of the one-dimensional channel convolution kernel, b and γ are constants, C represents the number of channels, and odd represents taking the odd number closest to the calculated value.
[0085] In this embodiment, there may be some kind of mapping between the number of channels C and the size k of the one-dimensional channel convolution kernel:
[0086] C = ψ(k)
[0087] In the above formula, the mapping Ψ is unknown. It can be analyzed that k has a non-linear relationship with the number of channels C. Therefore, a parameterized exponential function can be chosen to approximate this non-linear mapping relationship Ψ:
[0088] C=ψ(k)≈exp(γ*kb)
[0089] In convolutional neural networks, since the number of channels C is usually an integer power of 2, the number of channels C can also be expressed as:
[0090] C=ψ(k)≈2 (γ*k-b)
[0091] Here, k can be determined based on the number of channels:
[0092]
[0093] In the above formula, k represents the size of the one-dimensional channel convolution kernel, b and γ are constants, C represents the number of channels, and odd represents taking the odd number closest to the calculated value.
[0094] This application embodiment calculates a one-dimensional channel convolution kernel using the number of channels, which enables the global average pooling value and the global maximum pooling value to share the same one-dimensional channel convolution kernel, avoiding changes in the number of ascending and descending channels, thereby reducing feature loss in the image and further improving the computer vision effect.
[0095] Based on the above embodiments, in some other embodiments, the calculation of the convolution result of the global average pooling value and the convolution result of the global maximum pooling value can be achieved through the following steps:
[0096]
[0097]
[0098] In the above formula, w 1i w represents the convolution result of the global average pooling value. 2i The convolution result representing the global maximum pooling value, α j Ω is the j-th element in the one-dimensional channel convolution kernel α. 1i Ω represents the set of elements in every k adjacent channels of g(x). 2i Let h(x) represent the set of elements in every k adjacent channels, and C represent the number of element sets. Denotes the i-th set Ω 1i The element of the j-th channel, Denotes the i-th set Ω 2i The element of the j-th channel, where i and j are both positive integers.
[0099] For example, Figure 4 This is a schematic diagram of a one-dimensional channel convolution operation provided in an embodiment of this application, as shown below. Figure 4 As shown, the feature dimension after pooling is 1x1xC, where, This indicates the addition of corresponding elements. W is obtained by calculating using the above formula. 11 W 1i , ..., W 1C .
[0100] This embodiment of the application performs channel convolution on the global average pooling value and the global maximum pooling value, sharing a single channel convolution kernel. This avoids changes in the number of ascending and descending channels, reducing feature loss. Simultaneously, it enables interaction between adjacent feature channels and fusion of target information from different channels, which is beneficial for subsequent feature inference.
[0101] Based on the above embodiments, in some other embodiments, determining the weight of each channel can be achieved through the following steps:
[0102] ω i =σ(w 1i +w 2i ), i∈C
[0103] In the above formula, ω i Let w represent the weight of the i-th channel, σ represent the activation function, and w represent the weight of the i-th channel. 1i w represents the convolution result of the global average pooling value. 2i This represents the convolution result of the global maximum pooling value, where C represents the number of elements in the set.
[0104] In this embodiment, the activation function is the Sigmoid function, for example, Figure 5 A schematic diagram of the activation function provided in the embodiments of this application, such as... Figure 5 As shown, after passing through the Sigmoid function, the two convolution results (i.e., the convolution result of the global average pooling value and the convolution result of the global maximum pooling value) are summed and normalized to a value between 0 and 1, which is used as the weight of each channel.
[0105] The embodiments of this application calculate the weight of each channel. Based on the weight, the importance of information in different channels can be distinguished, which is more advantageous for extracting target information from complex interference background information, thereby further improving the computer vision effect.
[0106] In some embodiments, step S205 can be implemented by the following steps: multiplying the input features with weights to obtain the multiplication result; inputting the input features into a convolutional block to obtain the convolutional block result, wherein the convolutional block includes at least a convolutional layer, a normalization layer, and an activation function; and summing the multiplication result and the convolutional block result to obtain the target feature map.
[0107] In this embodiment, the formula for calculating the multiplication result is as follows:
[0108] U = w * x
[0109] In the above formula, U is the result of multiplication, U∈R W×H×C w represents the weights, and x represents the input features.
[0110] In this embodiment, the convolutional block can be preset, which can be a CBL module (composed of three network layers: Conv, BatchNormalization, and Leaky ReLU). The Conv layer is a convolutional layer that processes the input image using multiple different convolutional kernels to obtain different response feature maps.
[0111] Batch Normalization (BN) layers are normalization layers. As a layer in the neural network, BN is placed before the activation function and after the convolutional layers. When the number of feature maps is m and the feature map size is w*h (i.e., the number of image pixels), the amount of data for BN is m*w*h. The main operation steps of the BN layer are: calculating the mean and variance of all batch data; then, the difference between the pixel value and the mean, divided by the variance, is used for normalization; simultaneously, offset and scale factors are added to control the normalized values. The values of these factors are learned by the neural network during training.
[0112] The Leaky ReLU function is an activation function, a variant of the ReLU function. In the ReLU function, when the input is negative, the learning speed of ReLU can become very slow, or even render the neuron ineffective. This is because the input is less than zero and the gradient is zero, so its weights cannot be updated, and it remains silent for the rest of the training process. When the input value of ReLU is negative, the output is always 0, and its first derivative is also always 0, which prevents the neuron from updating its parameters, meaning the neuron stops learning. To address this shortcoming of the ReLU function, a leaky value is introduced in the negative half-region, hence the name Leaky ReLU. This function's output has a very small slope for negative inputs. Since the derivative is always non-zero, this reduces the occurrence of silent neurons, allowing gradient-based learning (although it will be very slow), and solves the problem of neurons not learning when the ReLU function enters the negative region.
[0113] The CBL module outputs CBL(X) after the input features pass through it. The formula for calculating the target feature map V is as follows:
[0114] V=CBL(X)+U
[0115] This embodiment of the application obtains a convolution block result by passing the input features through a 3×3 CBL module, and then superimposes the result with the convolution block result as the final result of the input features. This realizes the superposition of the input features after the attention mechanism with the original input features, increases the amount of available information, and can further improve the computer vision effect.
[0116] For example, in some embodiments, the above can be... Figure 3The new SECA-s module, defined by the first dashed bounding box 300, is applied to YOLOv3. YOLO (You Only Look Once) is a fast and accurate real-time object detection algorithm. For example, as described above... Figure 1 Taking image 11 as an example, which is used as the input image for YOLOv3, Figure 6 The detection results provided in this application embodiment are based on YOLOv3 obtained by applying the new module SECA-s, such as Figure 6 As shown, it extracts more useful information (selected by dashed boxes) and has better computer vision results.
[0117] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0118] Figure 7 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application, such as... Figure 7 As shown, the image processing apparatus 700 includes a pooling value calculation module 710, a convolution kernel calculation module 720, a convolution result calculation module 730, a weight determination module 740, and a feature map calculation module 750. The pooling value calculation module 710 is used to acquire the input features and number of channels of the image, and calculate the global average pooling value and the global maximum pooling value of the input features. The convolution kernel calculation module 720 is used to calculate a one-dimensional channel convolution kernel based on the number of channels. The convolution result calculation module 730 is used to calculate the convolution result of the global average pooling value and the convolution result of the global maximum pooling value based on the one-dimensional channel convolution kernel. The weight determination module 740 is used to determine the weight of each channel in the input features based on the convolution result of the global average pooling value and the convolution result of the global maximum pooling value. The feature map calculation module 750 is used to calculate a target feature map based on the input features and the weight of each channel; the target feature map is used for target detection.
[0119] Optionally, the pooling value calculation module 710 can be specifically used to: obtain the width, height, and pixel value of the i-th row and j-th column of the image based on the input features, where i and j are both positive integers; calculate the global average pooling value based on the width, height, and pixel value of the i-th row and j-th column of the image; and obtain the pixel value of the i-th row and j-th column...
[0120]
[0121] The maximum value among the pixel values is used as the global maximum pooling value.
[0122] Optionally, the pooling value calculation module 710 can be specifically used for:
[0123] In the above formula, g(x) represents the global average pooling value, W represents the width of the image, H represents the height of the image, and x i，j This represents the pixel value in the i-th row and j-th column;
[0124] Correspondingly, the maximum value among the elements in the i-th row and j-th column is obtained as the global maximum pooling value, including:
[0125]
[0126] In the above formula, h(x) represents the global maximum pooling value, W represents the width of the image, H represents the height of the image, and x i，j This represents the pixel value in the i-th row and j-th column.
[0127] Optionally, the convolution kernel calculation module can be used specifically for:
[0128]
[0129] In the above formula, k represents the size of the one-dimensional channel convolution kernel, b and γ are constants, C represents the number of channels, and odd represents taking the odd number closest to the calculated value.
[0130] Optionally, the convolution result calculation module can be used specifically for:
[0131]
[0132]
[0133] In the above formula, w 1i w represents the convolution result of the global average pooling value. 2i The convolution result representing the global maximum pooling value, α j Ω is the j-th element in the one-dimensional channel convolution kernel α. 1i Ω represents the set of elements in every k adjacent channels of g(x). 2i Let h(x) represent the set of elements in every k adjacent channels, and C represent the number of element sets. Denotes the i-th set Ω 1i The element of the j-th channel, Denotes the i-th set Ω 2i The element of the j-th channel, where i and j are both positive integers.
[0134] Optionally, the weight determination module can be used specifically for:
[0135] ω i =σ(w 1i +w 2i ), i∈C
[0136] In the above formula, ω iLet w represent the weight of the i-th channel, σ represent the activation function, and w represent the weight of the i-th channel. 1i w represents the convolution result of the global average pooling value. 2i This represents the convolution result of the global maximum pooling value, where C represents the number of elements in the set.
[0137] Optionally, the feature map calculation module can be used to: multiply the input features by the weights to obtain the multiplication result; input the input features into a convolutional block to obtain the convolutional block result, wherein the convolutional block includes at least a convolutional layer, a normalization layer, and an activation function; and sum the multiplication result and the convolutional block result to obtain the target feature map.
[0138] The apparatus provided in this application embodiment can be used to execute the methods in the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.
[0139] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the pooling value calculation module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.
[0140] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this application. For example... Figure 8 As shown, the computer device 800 includes at least one processor 801, a memory 802, a bus 803, and a communication interface 804. The processor 801, communication interface 804, and memory 802 communicate with each other via the bus 803. The communication interface 804 is used to communicate with other devices. This communication interface includes a communication interface for data transmission and a display interface or operation interface for human-computer interaction. The processor executes computer execution instructions stored in the memory, specifically performing the relevant steps in the methods described in the above embodiments.
[0141] The processor may be a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs. Memory is used to store programs. Memory may include high-speed RAM and may also include non-volatile memory, such as at least one disk storage device.
[0142] This embodiment also provides a computer-readable storage medium storing computer instructions, which, when executed by at least one processor of a computer device, enable the computer device to perform the methods provided in the various embodiments described above.
[0143] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0144] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. An image processing method, characterized in that, include: The input features of the image and the corresponding number of channels are obtained. Global average pooling and global max pooling are then performed on the input features to obtain the global average pooling value and the global max pooling value. Based on the number of channels, determine the one-dimensional channel convolution kernel for channel-dimensional convolution; Using the same one-dimensional channel convolution kernel, one-dimensional channel convolution is performed on the global average pooling value and the global maximum pooling value respectively to obtain the corresponding convolution results; Based on the convolution results of the global average pooling value and the global maximum pooling value, the attention weights of each channel in the input features are fused to generate the attention weights. The input features are multiplied by the attention weights of each channel to obtain the target feature map, which is used for target detection.
2. The method according to claim 1, characterized in that, The step of performing global average pooling and global max pooling on the input features to obtain global average pooling values and global max pooling values includes: Based on the input features, obtain the width, height, and pixel value of the i-th row and j-th column of the image, where i and j are both positive integers; The global average pooling value is calculated based on the width, height, and pixel value of the i-th row and j-th column of the image. Obtain the maximum value among the pixel values in the i-th row and j-th column, and use it as the global maximum pooling value.
3. The method according to claim 2, characterized in that, The step of calculating the global average pooling value based on the width, height, and pixel value in the i-th row and j-th column of the image includes: In the above formula, This represents the global average pooling value, where W represents the image width and H represents the image height. This represents the pixel value in the i-th row and j-th column; Correspondingly, obtaining the maximum value among the elements in the i-th row and j-th column, as the global maximum pooling value, includes: In the above formula, This represents the global maximum pooling value, where W represents the width of the image and H represents the height of the image. This represents the pixel value in the i-th row and j-th column.
4. The method according to claim 1, characterized in that, The step of determining the one-dimensional channel convolution kernel for channel-dimensional convolution based on the number of channels includes: In the above formula, k represents the size of the one-dimensional channel convolution kernel, b and γ are constants, C represents the number of channels, and odd represents taking the odd number closest to the calculated value.
5. The method according to claim 1, characterized in that, The step of using the same one-dimensional channel convolution kernel to perform one-dimensional channel convolution on the global average pooling value and the global maximum pooling value respectively to obtain the corresponding convolution results includes: In the above formula, The convolution result representing the global average pooling value. This represents the convolution result of the pooling value with the global maximum value. Let j be the j-th element in the one-dimensional channel convolution kernel α. Let g(x) represent the set of elements in every k adjacent channels. Let h(x) represent the set of elements in every k adjacent channels, and C represent the number of element sets. Represents the i-th set The element of the j-th channel, Represents the i-th set The element of the j-th channel, where i and j are both positive integers.
6. The method according to claim 5, characterized in that, The step of fusing the convolution results of the global average pooling value and the global maximum pooling value to generate attention weights for each channel in the input features includes: In the above formula, This represents the attention weight of the i-th channel. This represents the activation function. The convolution result representing the global average pooling value. This represents the convolution result of the global maximum pooling value, where C represents the number of elements in the set.
7. The method according to any one of claims 1-6, characterized in that, The step of multiplying the input features and the attention weights of each channel to obtain the target feature map includes: The input features are multiplied by the attention weights of each channel to obtain the multiplication result; The input features are input into a convolutional block, and the result of the convolutional block is calculated. The convolutional block includes at least a convolutional layer, a normalization layer, and an activation function. The target feature map is obtained by summing the multiplication result and the convolution block result.
8. An image processing apparatus, characterized in that, include: The pooling value calculation module is used to obtain the input features of the image and the corresponding number of channels, and to perform global average pooling and global max pooling on the input features to obtain the global average pooling value and the global max pooling value. The kernel determination module is used to determine a one-dimensional channel convolution kernel for channel-dimensional convolution based on the number of channels; The convolution result calculation module is used to perform one-dimensional channel convolution on the global average pooling value and the global maximum pooling value respectively using the same one-dimensional channel convolution kernel to obtain the corresponding convolution result; The weight generation module is used to generate attention weights for each channel in the input features by fusing the convolution results of the global average pooling value and the convolution results of the global maximum pooling value. The feature map generation module is used to multiply the input features and the attention weights of each channel to obtain the target feature map, which is used for target detection.
9. A computer device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-7.