A method and apparatus for convolution of feature images

Abstract

This application discloses a method and apparatus for convolution of feature images to improve the training effect of convolutional neural network models. The convolution method includes: acquiring a convolutional unit, the convolutional unit including an attention generation module and a feature convolution module; acquiring an input feature image, the input feature image being an image used in training the convolutional neural network model; inputting the input feature image into the attention generation module to generate an attention feature image, the attention feature image containing attention values ​​of pixels in the input feature image; inputting the input feature image and the attention feature image into the feature convolution module; and using the feature convolution module and the attention values ​​in the attention feature image to perform convolution processing on the pixels in the input feature image to generate convolutional data.

Description

Technical Field

[0001] This application relates to the field of convolutional neural network models, and more particularly to a method and apparatus for convolution of feature images. Background Technology

[0002] In recent years, deep learning has flourished in the field of image processing as an emerging technology. Its ability to autonomously learn image data features greatly avoids the tediousness of manually designing algorithms. Furthermore, it possesses accurate detection performance, high detection efficiency, and good generalization performance across various image tasks, leading to its widespread application in image processing, including image detection, image classification, and image reconstruction. Convolutional operations, as the core operator of deep learning in image processing, possess three major characteristics: local perception, weight sharing, and downsampling. Due to its excellent image feature extraction performance, it has become the cornerstone of deep learning's success in the image processing field.

[0003] In the entire input feature image, the pixels containing the target object are important and should be given more attention during convolution, while background pixels should receive less attention to avoid interfering with the feature extraction of the target object. Traditional convolution treats all pixels in the input feature image equally during the sliding operation, without distinguishing between pixels and giving equal attention to target object pixels and background pixels. Traditional convolution operations are based on receptive fields of convolution kernels with fixed shapes, such as 3x3 and 5x5 kernels. Currently, there are two main types of convolution kernels: dilated convolution and deformable convolution.

[0004] Deformable convolution introduces an offset into the receptive field, and this offset is learnable. This allows the receptive field to no longer be a rigid square, but rather closely resemble the actual shape of the object. Subsequent convolutional regions always cover the area around the object's shape, regardless of its deformation, which can be handled by the learned offset. Deformable convolution allows for free control of the size and shape of the receptive field by increasing the offset at the convolution position. However, these offsets are fixed in each convolution process. Since objects in an image may have different sizes and shapes, using a receptive field of the same size and shape cannot adapt to different objects. Therefore, its effectiveness varies significantly depending on the input feature images.

[0005] Dilated convolution, also known as expanding convolution or dilated convolution, is simply the process of expanding the convolution kernel by adding spaces (zeros) between its elements. The basic principle of dilated convolution differs from traditional convolution. Traditional convolution first defines the pixel region of the input feature image based on the kernel's effective range, and then performs convolution operations on these regions. Dilated convolution defines a pixel region with a larger effective range than the kernel. Within this region, pixels are selected according to predetermined rules, and these selected pixels are combined to form a new pixel region with the same range as the kernel. For example, a 5x5 receptive field paired with a 3x3 kernel for dilated convolution: First, the 5x5 receptive field defines a 5x5 pixel region. Within this region, nine pixels are selected, primarily based on pixel value or pixel position. This can be done by selecting the nine largest pixels or nine evenly spaced pixels. Finally, a 3x3 kernel is used for the final convolution operation. Dilated convolution expands the receptive field to some extent, but the shape of the receptive field is still a regular rectangle, and the size of the receptive field is also fixed.

[0006] However, the actual target objects in the input feature images vary in size and shape, and the receptive field of the fixed-size convolution kernel limits the convolution effect on the actual objects in the image. It cannot match the target objects of different sizes and shapes well, which affects the feature extraction effect and thus reduces the training effect of the convolutional neural network model. Summary of the Invention

[0007] This application discloses a method and apparatus for convolution of feature images, which can be used to improve the training effect of convolutional neural network models.

[0008] Specifically, this application discloses a novel variable receptive field convolution method based on pixel attention. On the one hand, the convolution kernel utilizes the attention of each pixel in the input feature image, selecting pixels with higher attention for convolution operations. This allows the convolution to better target the more prominent features in the image, improving upon the traditional approach of treating all pixels equally. This enables more pixels related to the object to be processed to enter the convolution operation, while fewer irrelevant pixels are captured, improving the efficiency of convolution in extracting object features and reducing interference from irrelevant pixels. On the other hand, it expands the limitation of the fixed receptive field of the traditional convolution kernel, making the receptive field dynamically adjustable within a certain range for each convolution operation. Furthermore, the shape of the receptive field is variable, breaking the limitation of the fixed rectangular receptive field of the traditional convolution kernel. This allows for matching different shapes and sizes of receptive fields to different objects, better preserving object features.

[0009] The first aspect of this application provides a convolution method for feature images, comprising:

[0010] Obtain a convolutional unit, wherein the convolutional unit includes an attention generation module and a feature convolution module;

[0011] Obtain the input feature image, which is an image used in the training of the convolutional neural network model;

[0012] The input feature image is input into the attention generation module to generate an attention feature image, which contains the attention values ​​of the pixels in the input feature image.

[0013] The input feature image and the attention feature image are input into the feature convolution module;

[0014] The feature convolution module is used to perform convolution processing on the pixels in the input feature image based on the attention values ​​in the attention feature image to generate convolution data.

[0015] Optionally, the step of inputting the input feature image into the pixel attention generation module to generate an attention feature image includes:

[0016] The pixel attention generation module compresses the feature channels of the input feature image to generate compressed features.

[0017] The pixel attention generation module restores the compressed features into a feature matrix of the same size as the input feature image.

[0018] The pixel attention generation module calculates the attention value for each pixel in the feature matrix to generate an attention feature image.

[0019] Optionally, the pixel attention generation module includes a BatchNorm-Conv-ReLU layer, a BatchNorm-Conv layer, and a SigMoid function layer;

[0020] The step of compressing the input feature image through the pixel attention generation module to generate compressed features includes:

[0021] The input feature image is compressed by the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features.

[0022] Optionally, the step of restoring the compressed features into a feature matrix of the same size as the input feature image through the pixel attention generation module includes:

[0023] The compressed features are restored to a feature matrix of the same size as the input feature image by the BatchNorm-Conv layer in the pixel attention generation module.

[0024] Optionally, the step of calculating an attention value for each pixel in the feature matrix and generating an attention feature image using the pixel attention generation module includes:

[0025] The attention feature image is generated by calculating the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module.

[0026] Optionally, the feature extraction module includes at least one attention convolution kernel, which includes an attention receptive field and a preset convolution kernel, wherein the effective range of the attention receptive field is larger than that of the preset convolution kernel;

[0027] The step of using the feature extraction module and performing convolution processing on the pixels in the input feature image based on the attention values ​​in the attention feature image to generate convolution data includes:

[0028] Based on the attention receptive field and the attention feature image, pixel filtering and stitching processing is performed on the input feature image to generate deformable features with the same size and the effective range of the preset convolution kernel;

[0029] The deformed features are convolved using the preset convolution kernel to generate convolutional data.

[0030] Optionally, the step of performing pixel filtering and stitching processing on the input feature image based on the attention receptive field and the attention feature image to generate deformable features with the same size and effective range as the preset convolutional kernel includes:

[0031] The region to be filtered is determined in the input feature image based on the attention receptive field.

[0032] Based on the effective range of the preset convolutional kernel and the attention feature image, a set of pixels whose attention values ​​meet the preset threshold are selected in the region to be filtered.

[0033] The set of pixels whose attention values ​​meet the preset threshold is subjected to pixel filtering and splicing to generate a deformable feature with the same size and effective range as the preset convolution kernel.

[0034] Optionally, after selecting a set of pixels whose attention values ​​meet a preset threshold in the region to be filtered based on the effective range of the preset convolution kernel and the attention feature image, and before performing pixel filtering and stitching processing on the set of pixels whose attention values ​​meet the preset threshold to generate a deformable feature with the same size as the effective range of the preset convolution kernel, the convolution method further includes:

[0035] An attention difference is generated by calculating the first attention value set of the pixel set and the second attention value set of other pixels in the region to be filtered.

[0036] When the attention difference is less than a preset threshold, the effective range of the attention receptive field and the preset convolution kernel is reset.

[0037] The first aspect of this application provides a convolution apparatus for a feature image, comprising:

[0038] The first acquisition unit is used to acquire convolutional units, wherein the convolutional unit includes an attention generation module and a feature convolution module;

[0039] The second acquisition unit is used to acquire the input feature image, which is an image used in the training of the input convolutional neural network model.

[0040] The first input unit is used to input the input feature image into the attention generation module to generate an attention feature image, wherein the attention feature image contains the attention values ​​of the pixels in the input feature image;

[0041] The second input unit is used to input the input feature image and the attention feature image into the feature convolution module;

[0042] A convolutional unit is used to perform convolution processing on pixels in the input feature image using the feature convolution module and based on the attention values ​​in the attention feature image to generate convolutional data.

[0043] Optionally, the first input unit includes:

[0044] The first generation module is used to compress the feature channels of the input feature image through the pixel attention generation module to generate compressed features;

[0045] The compression and restoration module is used to restore the compressed features into a feature matrix of the same size as the input feature image through the pixel attention generation module;

[0046] The second generation module is used to calculate the attention value for each pixel in the feature matrix through the pixel attention generation module, and generate an attention feature image.

[0047] Optionally, the pixel attention generation module includes a BatchNorm-Conv-ReLU layer, a BatchNorm-Conv layer, and a SigMoid function layer;

[0048] The first generation module includes:

[0049] The input feature image is compressed by the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features.

[0050] Optionally, the compression and restoration module includes:

[0051] The compressed features are restored to a feature matrix of the same size as the input feature image by the BatchNorm-Conv layer in the pixel attention generation module.

[0052] Optionally, the second generation module includes:

[0053] The attention feature image is generated by calculating the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module.

[0054] Optionally, the feature extraction module includes at least one attention convolution kernel, which includes an attention receptive field and a preset convolution kernel, wherein the effective range of the attention receptive field is larger than that of the preset convolution kernel;

[0055] The convolutional unit includes:

[0056] The first generation module is used to perform pixel filtering and stitching processing on the input feature image based on the attention receptive field and the attention feature image to generate deformable features with the same size and the effective range of the preset convolutional kernel.

[0057] The second generation module is used to perform convolution processing on the deformed features using the preset convolution kernel to generate convolution data.

[0058] Optionally, the first generation module includes:

[0059] A determination submodule is used to determine the region to be filtered in the input feature image based on the attention receptive field.

[0060] The selection submodule is used to select a set of pixels whose attention values ​​meet a preset threshold in the region to be filtered, based on the effective range of the preset convolutional kernel and the attention feature image.

[0061] The first generation submodule is used to perform pixel filtering and splicing processing on the set of pixels whose attention values ​​meet the preset threshold, and generate deformable features with the same size and range of effect as the preset convolution kernel.

[0062] Optionally, the first generation module further includes:

[0063] The second generation submodule is used to calculate and generate an attention difference based on the first attention value set of the pixel set and the second attention value set of other pixels in the region to be filtered.

[0064] The setting submodule is used to reset the effective range of the attention receptive field and the preset convolution kernel when the attention difference is less than a preset threshold.

[0065] A third aspect of this application provides an electronic device, comprising:

[0066] Processor, memory, input / output units, and bus;

[0067] The processor is connected to memory, input / output units, and a bus;

[0068] The memory holds a program, which the processor calls to execute, such as the first aspect and any optional convolution method of the first aspect.

[0069] The fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs convolution methods as described in the first aspect and any optional convolution method of the first aspect.

[0070] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0071] In this invention, a convolutional unit is first obtained, comprising an attention generation module and a feature convolution module. The attention generation module can be located inside the convolutional neural network model and connected to other working layers, or it can be located outside the convolutional neural network model and used independently. Next, an input feature image is obtained. This input feature image is an image input into the convolutional neural network model during training. This input feature image can be a feature output from a certain stage in the convolutional neural network model, which requires convolution. In this case, the input feature image can be first input into the attention generation module, and then the output data can be fed into subsequent convolution stages. The input feature image is input into the attention generation module to generate an attention feature image, which contains the attention values ​​of the pixels in the input feature image. The input feature image and the attention feature image are then input into the feature convolution module. The feature convolution module uses the attention values ​​in the attention feature image to perform convolution processing on the pixels in the input feature image, generating convolutional data. In this invention, an attention feature image is generated using an attention generation module. Then, pixels in the input feature image are filtered based on the attention values ​​in the attention feature image. The filtered pixels are then convolved. This allows the generated data to utilize the attention of each pixel in the input feature image, selecting pixels with higher attention for convolution. This enables the convolution to better target areas with more prominent features in the image, improving upon the traditional approach of treating all pixels equally. More pixels relevant to the object being processed are included in the convolution operation, while fewer irrelevant pixels are captured, increasing the efficiency of feature extraction and reducing interference from irrelevant pixels. This, in turn, improves the training performance of the convolutional neural network model. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of an embodiment of the convolution method for feature images in this application;

[0074] Figure 2-1 This is a schematic diagram of an embodiment of the first stage of the convolution method for feature images in this application;

[0075] Figure 2-2 This is a schematic diagram of an embodiment of the second stage of the convolution method for feature images in this application;

[0076] Figure 3This is a schematic diagram of an embodiment of the convolution apparatus for the feature image of this application;

[0077] Figure 4 This is a schematic diagram of another embodiment of the convolution apparatus for the feature image of this application;

[0078] Figure 5 This is a schematic diagram of one embodiment of the electronic device of this application. Detailed Implementation

[0079] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0080] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0081] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0082] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0083] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0084] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0085] In existing technologies, pixels containing the target object are considered important in the entire input feature image and should receive more attention during convolution, while background pixels should receive less attention to avoid interfering with feature extraction. Traditional convolution treats all pixels in the input feature image equally during the sliding operation, without distinguishing between them and giving equal attention to target and background pixels. Traditional convolution operations are based on receptive fields of kernels with fixed shapes, such as 3x3 and 5x5 kernels. Currently, there are two main types of convolution kernels: dilated convolution and deformable convolution. Deformable convolution introduces an offset into the receptive field, and this offset is learnable. This allows the receptive field to no longer be a rigid square but rather closely resemble the actual shape of the object. This ensures that subsequent convolutional regions always cover the area around the object's shape, regardless of object deformation, which can be handled through the learned offset. Deformable convolution allows for free control of the receptive field size and shape by increasing the offset of the convolution position. However, these offsets are fixed in each convolution process. Since objects in an image may vary in size and shape, using a receptive field of the same size and shape cannot adapt to different objects, resulting in significant differences in its effectiveness for different input feature images. Dilated convolution, also called dilated convolution or diaphoretic convolution, simply involves adding spaces (zeros) between the elements of the convolution kernel to expand the kernel. The basic principle of dilated convolution is that traditional convolution first determines the pixel region of the input feature image according to the kernel's effective range, and then performs the convolution operation on these pixel regions. Dilated convolution, on the other hand, determines the pixel region with a range larger than the kernel's effective range, selects pixels within this region according to predetermined rules, and combines these selected pixels into a new pixel region with the same range as the kernel. For example, dilated convolution uses a 5x5 receptive field paired with a 3x3 convolution kernel. First, the 5x5 receptive field defines a 5x5 pixel region. Within this region, nine pixels are selected, primarily based on pixel value or position. This can be done by selecting the nine largest pixels or nine evenly spaced pixels. Finally, a 3x3 convolution kernel is used for further processing. Dilated convolution expands the receptive field to some extent, but its shape remains a regular rectangle, and its size is fixed. However, the actual size and shape of the target objects in the input feature image vary greatly. The fixed-size receptive field limits the convolution effect on objects of different sizes and shapes, hindering feature extraction and ultimately reducing the training performance of the convolutional neural network model.

[0086] Based on this, this application discloses a method and apparatus for convolution of feature images, which can be used to improve the training effect of convolutional neural network models.

[0087] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0088] The method described in this application can be applied to servers, devices, terminals, or other devices with logical processing capabilities; therefore, this application does not limit its application. For ease of description, the following description uses a terminal as the executing entity.

[0089] Please see Figure 1 This application provides an embodiment of a convolution method for feature images, comprising:

[0090] 101. Obtain a convolutional unit, wherein the convolutional unit includes an attention generation module and a feature convolution module;

[0091] The terminal acquires a convolutional unit, which includes an attention generation module and a feature convolution module. The attention generation module is used to perform attention judgment on the pixels on the input feature image, determine which pixels are more important, and generate a unique attention value for each pixel.

[0092] The feature convolution module is used in conjunction with the attention value output by the attention generation module to perform convolution operations on the pixels in the input feature image.

[0093] 102. Obtain the input feature image, which is an image used in the training of the convolutional neural network model;

[0094] The terminal acquires an input feature image, which is an image used in the training of the convolutional neural network model. It should be noted that the input feature image can be a complete image or simply feature data generated by processing an image in a specific layer of the convolutional neural network model; in this embodiment, it is uniformly referred to as input feature data.

[0095] 103. Input the input feature image into the attention generation module to generate an attention feature image, wherein the attention feature image contains the attention values ​​of the pixels in the input feature image;

[0096] The terminal inputs the input feature image into the attention generation module to generate an attention feature image, which contains the attention values ​​of the pixels in the input feature image.

[0097] Deep learning's attention mechanism is a biomimetic of human visual attention, essentially a resource allocation mechanism. The physiological principle is that human visual attention can receive high-resolution signals from a specific area of ​​an image while perceiving its surrounding areas at lower resolution, and the viewpoint can change over time. In other words, the human eye quickly scans the entire image, finds the target area requiring attention, and then allocates more attention to that area to acquire more detailed information and suppress other useless information. This improves the efficiency of convolutional unit representation.

[0098] In convolutional neural network models, the attention mechanism can be considered a resource allocation mechanism. It can be understood as redistributing resources that were originally evenly distributed according to the importance of the attention objects. Important units receive more resources, while unimportant or less important units receive less. In the structural design of deep neural networks, the resources that attention needs to allocate are basically weights.

[0099] 104. Input the input feature image and the attention feature image into the feature convolution module;

[0100] The terminal inputs the input feature image and the attention feature image into the feature convolution module.

[0101] 105. Using the feature convolution module and based on the attention values ​​in the attention feature image, perform convolution processing on the pixels in the input feature image to generate convolution data.

[0102] The convolution process involves traversing the input feature image from top to bottom and from left to right using a sliding window. The result of each traversal is a weighted sum of the elements at the corresponding positions.

[0103] This can be viewed as a weighted summation of a local area; it corresponds to local perception. Its principle is that when observing an object, we cannot observe every pixel or the whole at once, but rather start by recognizing the local area, which corresponds to convolution. Convolutional kernels are typically 1x1, 3x3, or 5x5 in size (usually odd x odd). For example, with an input of 224x224x3 (three RGB channels) and an output of 32-bit depth, the convolutional kernel size is 5x5. We would need 32 convolutional kernels, each 5x5x3 (the last 3 being the original image's RGB depth of 3). Each layer of each kernel is 5x5 (3 layers in total), convolved with each layer of the original image (224x224). The resulting three images are then superimposed (arithmetic summation) to form a new feature map. By performing this operation on each kernel, we can obtain 32 new feature maps (feature convolutional images).

[0104]

[0105] In the above formula, D is the depth, F is the size of the convolution kernel, and w d,m,n Let x(d, i+m, j+n) represent the weight of the convolution kernel in the m-th row and n-th column of the d-th layer, and x(d, i+m, j+n) be the offset. The convolution kernel moves across the image to perform convolution. The result of this formula is the output of one layer of a convolution kernel. i,j w represents the pixel in the i-th row and j-th column of the d-th layer of the image. b For bias.

[0106] The terminal uses the feature convolution module and performs convolution processing on the pixels in the input feature image based on the attention values ​​in the attention feature image to generate convolution data.

[0107] In this embodiment, firstly, a convolutional unit is obtained, which includes an attention generation module and a feature convolution module. The attention generation module can be located inside the convolutional neural network model and used in conjunction with other working layers, or it can be located outside the convolutional neural network model and used independently. Next, an input feature image is obtained. The input feature image is the image input into the convolutional neural network model during training. This input feature image can be a feature output from a certain stage in the convolutional neural network model, which requires convolution. In this case, the input feature image can be first input into the attention generation module, and then the output data can be fed into subsequent convolution stages. The input feature image is input into the attention generation module to generate an attention feature image, which contains the attention values ​​of the pixels in the input feature image. The input feature image and the attention feature image are then input into the feature convolution module. The feature convolution module uses the attention values ​​in the attention feature image to perform convolution processing on the pixels in the input feature image, generating convolutional data. In this invention, an attention feature image is generated using an attention generation module. Then, pixels in the input feature image are filtered based on the attention values ​​in the attention feature image. The filtered pixels are then convolved. This allows the generated data to utilize the attention of each pixel in the input feature image, selecting pixels with higher attention for convolution. This enables the convolution to better target areas with more prominent features in the image, improving upon the traditional approach of treating all pixels equally. More pixels relevant to the object being processed are included in the convolution operation, while fewer irrelevant pixels are captured, increasing the efficiency of feature extraction and reducing interference from irrelevant pixels. This, in turn, improves the training performance of the convolutional neural network model.

[0108] Please see Figure 2-1 and Figure 2-2 This application provides an embodiment of a convolution method for feature images, comprising:

[0109] 201. Obtain a convolutional unit, wherein the convolutional unit includes an attention generation module and a feature convolution module;

[0110] 202. Obtain the input feature image, which is an image used in the training of the convolutional neural network model;

[0111] Steps 201 to 202 in this embodiment are similar to steps 101 to 102 in the previous embodiment, and will not be repeated here.

[0112] 203. The input feature image is compressed by the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features;

[0113] 204. The compressed features are restored to a feature matrix of the same size as the input feature image through the BatchNorm-Conv layer in the pixel attention generation module;

[0114] 205. Calculate the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module to generate an attention feature image;

[0115] The terminal compresses the feature channels of the input feature image through the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features, and then restores the compressed features into a feature matrix of the same size as the input feature image through the BatchNorm-Conv layer in the pixel attention generation module. Finally, the terminal calculates the attention value for each pixel in the feature matrix through the SigMoid function in the pixel attention generation module to generate an attention feature image.

[0116] Specifically, in this embodiment, the terminal first needs to compress the feature channels of the input feature image using the BatchNorm-3*3 Conv-ReLU layer in the pixel attention generation module to generate compressed features. Then, the compressed features are restored to a feature matrix of the same size as the input feature image using the BatchNorm-3*3 Conv layer in the pixel attention generation module. Finally, the terminal calculates the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module to generate an attention feature image. The BatchNorm-3*3 Conv-ReLU layer is an optimal size; due to the differences among pixels in the input feature image, a BatchNorm-1*1 Conv-ReLU layer cannot be used, as this would increase the convolution workload.

[0117] 206. Input the input feature image and the attention feature image into the feature convolution module;

[0118] Step 206 in this embodiment is similar to step 104 in the previous embodiment, and will not be repeated here.

[0119] 207. Determine the region to be filtered in the input feature image based on the attention receptive field;

[0120] 208. Select a set of pixels in the region to be filtered whose attention values ​​meet the preset threshold based on the effective range of the preset convolution kernel and the attention feature image;

[0121] 209. Calculate and generate an attention difference based on the first attention value set of the pixel set and the second attention value set of other pixels in the region to be filtered;

[0122] 210. When the attention difference is less than a preset threshold, reset the attention receptive field and the effective range of the preset convolution kernel;

[0123] In this embodiment, the terminal determines the region to be filtered in the input feature image based on the attention receptive field, that is, the receptive field determines the region that needs to be convolved. Then, the terminal selects a set of pixels in the region to be filtered whose attention values ​​meet a preset threshold based on the effective range of the preset convolution kernel and the attention feature image. The terminal calculates an attention difference based on a first set of attention values ​​for the pixel set and a second set of attention values ​​for other pixels in the region to be filtered. This generates an attention difference, which determines whether the difference between pixels in the currently selected region is too small. If it is too small, the terminal needs to reset the attention receptive field and the effective range of the preset convolution kernel to increase the region to be filtered and incorporate more meaningful pixels.

[0124] In addition to the methods described above, the general approach used in this embodiment is to obtain the region to be filtered through the receptive field, determine the number of pixels needed based on the effective range of the convolution kernel, select the pixel with the largest attention value from the region to be filtered, and splice them into a new convolution region.

[0125] For example, in the receptive fields of a 3x3 convolution kernel and a 5x5 convolution kernel, there are 25 pixels and 25 attention values ​​in the region to be selected. From these 25 pixels, the 9 pixels with the largest attention values ​​are selected and spliced ​​together to form a new convolution region.

[0126] Furthermore, in this embodiment, when the difference between the 9 pixels and the other 16 pixels is not significant, it is necessary to increase the size of the receptive field and re-select the pixels.

[0127] 211. Perform pixel filtering and splicing on the set of pixels whose attention values ​​meet the preset threshold to generate a deformable feature with the same size and effective range as the preset convolution kernel;

[0128] 212. Perform convolution processing on the deformed features using the preset convolution kernel to generate convolution data.

[0129] The terminal performs pixel filtering and splicing on the set of pixels whose attention values ​​meet the preset threshold to generate a deformable feature with the same size as the effective range of the preset convolution kernel, and then performs convolution processing on the deformable feature through the preset convolution kernel to generate convolution data.

[0130] In this embodiment, firstly, a convolutional unit is obtained, which includes an attention generation module and a feature convolution module. The attention generation module can be located inside the convolutional neural network model and used in conjunction with other working layers, or it can be located outside the convolutional neural network model and used independently. Next, an input feature image is obtained. The input feature image is an image input into the convolutional neural network model during training. This input feature image can be a feature output from a certain stage in the convolutional neural network model, which requires convolution. In this case, the input feature image can be first input into the attention generation module, and then the output data can be fed into subsequent convolution stages. The terminal compresses the feature channels of the input feature image through the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features. The terminal restores the compressed features to a feature matrix of the same size as the input feature image through the BatchNorm-Conv layer in the pixel attention generation module. The terminal calculates the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module to generate an attention feature image. The input feature image and the attention feature image are then input into the feature convolution module. The terminal determines a region to be filtered based on the attention receptive field in the input feature image. The terminal selects a set of pixels in the region to be filtered whose attention values ​​meet a preset threshold, based on the effective range of the preset convolutional kernel and the attention feature image. An attention difference is generated by calculating the first set of attention values ​​for the pixel set and the second set of attention values ​​for other pixels in the region to be filtered. When the attention difference is less than the preset threshold, the attention receptive field and the effective range of the preset convolutional kernel are reset. The terminal performs pixel filtering and stitching on the set of pixels whose attention values ​​meet the preset threshold to generate a deformed feature with the same size as the effective range of the preset convolutional kernel. The terminal performs convolution processing on the deformed feature using the preset convolutional kernel to generate convolutional data. In this invention, an attention feature image is generated using an attention generation module. Then, pixels in the input feature image are filtered based on the attention values ​​in the attention feature image. The filtered pixels are then convolved. This allows the generated data to utilize the attention of each pixel in the input feature image, selecting pixels with higher attention for convolution. This enables the convolution to better target areas with more prominent features in the image, improving upon the traditional approach of treating all pixels equally. More pixels relevant to the object being processed are included in the convolution operation, while fewer irrelevant pixels are captured, increasing the efficiency of feature extraction and reducing interference from irrelevant pixels. This, in turn, improves the training performance of the convolutional neural network model.

[0131] Furthermore, this embodiment improves upon the shortcomings of traditional convolutional kernels that treat all pixels equally and have fixed receptive field sizes and shapes. Before convolution, attention is calculated on the input feature map to obtain attention feature maps for each pixel in the input feature map. The attention feature maps are the same size as the original input feature map, and each element represents the attention value of the pixel at the corresponding position in the input feature map, i.e., the importance of that pixel in the entire feature map. The attention value is a number between 0 and 1, with higher values ​​indicating higher importance. The receptive field of the attention convolution kernel is expanded from the traditional fixed size and shape (e.g., a 3×3 pixel square) to a variable size and shape receptive field within a certain range (e.g., 5×5 pixels). From the original input feature map block within this expanded range (5×5 pixels), 3×3 positions with high attention are selected in the corresponding attention feature block, and then 3×3 pixels are obtained from the corresponding positions in the original input feature map block before convolution. This allows the receptive field of the attention convolution kernel to be variable in size and shape within a certain range, which can better match the appearance features of objects and better focus on more important pixel values ​​based on attention, reducing interference from irrelevant pixels.

[0132] Since the attention of each block of the input feature is different, when using an attention convolution kernel to perform a convolution operation on an input feature map, the receptive field of the convolution kernel is different for each operation. This improves upon the traditional convolution kernel, which uses the same size receptive field on the input feature map each time. This allows the attention convolution kernel to correspond to the appropriate receptive field for target objects of different sizes on the same input feature map, and can extract features better for objects of different sizes and shapes on the same input feature map.

[0133] In this embodiment, the attention convolution kernel avoids modification of traditional convolution kernels by processing the input feature map. It can directly replace the original convolution kernel in the neural network without modifying the network structure. The attention feature map is generated autonomously through deep learning and can be performed synchronously during the training of the neural network without additional computation.

[0134] Please see Figure 3 This application provides an embodiment of a convolution apparatus for a feature image, comprising:

[0135] The first acquisition unit 301 is used to acquire a convolutional unit, wherein the convolutional unit includes an attention generation module and a feature convolutional module;

[0136] The second acquisition unit 302 is used to acquire an input feature image, wherein the input feature image is an image used in training the convolutional neural network model.

[0137] The first input unit 303 is used to input the input feature image into the attention generation module to generate an attention feature image, wherein the attention feature image has the attention values ​​of the pixels in the input feature image;

[0138] The second input unit 304 is used to input the input feature image and the attention feature image into the feature convolution module;

[0139] Convolution unit 305 is used to perform convolution processing on the pixels in the input feature image using the feature convolution module and according to the attention value in the attention feature image to generate convolution data.

[0140] Please see Figure 4 This application provides an embodiment of a convolution apparatus for a feature image, comprising:

[0141] The first acquisition unit 401 is used to acquire a convolutional unit, wherein the convolutional unit includes an attention generation module and a feature convolutional module;

[0142] The second acquisition unit 402 is used to acquire an input feature image, wherein the input feature image is an image used in training the convolutional neural network model.

[0143] The first input unit 403 is used to input the input feature image into the attention generation module to generate an attention feature image, wherein the attention feature image has the attention values ​​of the pixels in the input feature image;

[0144] Optionally, the first input unit 403 includes:

[0145] The first generation module 4031 is used to compress the feature channels of the input feature image through the pixel attention generation module to generate compressed features.

[0146] The compression and restoration module 4032 is used to restore the compressed features into a feature matrix of the same size as the input feature image through the pixel attention generation module;

[0147] The second generation module 4033 is used to calculate the attention value for each pixel in the feature matrix through the pixel attention generation module, and generate an attention feature image.

[0148] Optionally, the pixel attention generation module includes a BatchNorm-Conv-ReLU layer, a BatchNorm-Conv layer, and a SigMoid function layer;

[0149] The first generation module 4031 includes:

[0150] The input feature image is compressed by the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features.

[0151] Optionally, the compression and restoration module 4032 includes:

[0152] The compressed features are restored to a feature matrix of the same size as the input feature image by the BatchNorm-Conv layer in the pixel attention generation module.

[0153] Optionally, the second generation module 4033 includes:

[0154] The attention feature image is generated by calculating the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module.

[0155] The second input unit 404 is used to input the input feature image and the attention feature image into the feature convolution module;

[0156] Convolution unit 405 is used to perform convolution processing on the pixels in the input feature image using the feature convolution module and according to the attention value in the attention feature image to generate convolution data.

[0157] Optionally, the feature extraction module includes at least one attention convolution kernel, which includes an attention receptive field and a preset convolution kernel, wherein the effective range of the attention receptive field is larger than that of the preset convolution kernel;

[0158] The convolutional unit 405 includes:

[0159] The first generation module 4051 is used to perform pixel filtering and splicing processing on the input feature image according to the attention receptive field and the attention feature image to generate deformable features with the same size and the effective range of the preset convolution kernel.

[0160] The second generation module 4052 is used to perform convolution processing on the deformed features through the preset convolution kernel to generate convolution data.

[0161] Optionally, the first generation module 4051 includes:

[0162] The determination submodule 40511 is used to determine the region to be filtered in the input feature image based on the attention receptive field.

[0163] The selection submodule 40512 is used to select a set of pixels whose attention values ​​meet a preset threshold in the region to be filtered, based on the effective range of the preset convolutional kernel and the attention feature image.

[0164] The first generation submodule 40513 is used to perform pixel screening and splicing processing on the set of pixels whose attention values ​​meet the preset threshold, and generate deformable features with the same size and range of effect as the preset convolution kernel.

[0165] Optionally, the first generation module 4051 further includes:

[0166] The second generation submodule 40514 is used to calculate and generate an attention difference based on the first attention value set of the pixel set and the second attention value set of other pixels in the region to be filtered.

[0167] The submodule 40515 is configured to reset the effective range of the attention receptive field and the preset convolutional kernel when the attention difference is less than a preset threshold.

[0168] Please see Figure 5 This application provides an electronic device, including:

[0169] Processor 501, memory 503, input / output unit 502 and bus 504.

[0170] The processor 501 is connected to the memory 503, the input / output unit 502, and the bus 504.

[0171] The memory 503 stores a program, and the processor 501 calls the program to execute it, such as... Figure 1 , Figure 2-1 , Figure 2-2 The convolution method in [the context of the text].

[0172] This application provides a computer-readable storage medium on which a program is stored, and when the program is executed on a computer, it performs the following... Figure 1 , Figure 2-1 , Figure 2-2 The convolution method in [the context of the text].

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A convolution method for feature images, characterized in that, include: Obtain a convolutional unit, wherein the convolutional unit includes a pixel attention generation module and a feature convolution module; Obtain the input feature image, which is an image used in the training of the convolutional neural network model; The input feature image is input into the pixel attention generation module to generate an attention feature image, which contains the attention values ​​of the pixels in the input feature image. The step of inputting the input feature image into the pixel attention generation module to generate an attention feature image includes: compressing the feature channels of the input feature image through the pixel attention generation module to generate compressed features; restoring the compressed features into a feature matrix of the same size as the input feature image through the pixel attention generation module; and calculating an attention value for each pixel in the feature matrix through the pixel attention generation module to generate an attention feature image. The input feature image and the attention feature image are input into the feature convolution module; The feature convolution module is used to perform convolution processing on the pixels in the input feature image based on the attention values ​​in the attention feature image to generate convolution data.

2. The convolution method according to claim 1, characterized in that, The pixel attention generation module includes a BatchNorm-Conv-ReLU layer, a BatchNorm-Conv layer, and a SigMoid function layer; The step of compressing the input feature image through the pixel attention generation module to generate compressed features includes: The input feature image is compressed by the BatchNorm-Conv-ReLU layer in the pixel attention generation module to generate compressed features.

3. The convolution method according to claim 2, characterized in that, The step of restoring the compressed features into a feature matrix of the same size as the input feature image through the pixel attention generation module includes: The compressed features are restored to a feature matrix of the same size as the input feature image by the BatchNorm-Conv layer in the pixel attention generation module.

4. The convolution method according to claim 2, characterized in that, The step of calculating an attention value for each pixel in the feature matrix and generating an attention feature image by means of the pixel attention generation module includes: The attention feature image is generated by calculating the attention value for each pixel in the feature matrix using the SigMoid function in the pixel attention generation module.

5. The convolution method according to any one of claims 1 to 4, characterized in that, The feature extraction module includes at least one attention convolution kernel, which includes an attention receptive field and a preset convolution kernel. The effective range of the attention receptive field is larger than that of the preset convolution kernel. The step of using the feature extraction module and performing convolution processing on the pixels in the input feature image based on the attention values ​​in the attention feature image to generate convolution data includes: Based on the attention receptive field and the attention feature image, pixel filtering and stitching processing is performed on the input feature image to generate deformable features with the same size and the effective range of the preset convolution kernel; The deformed features are convolved using the preset convolution kernel to generate convolutional data.

6. The convolution method according to claim 5, characterized in that, The step of performing pixel filtering and stitching processing on the input feature image based on the attention receptive field and the attention feature image to generate deformable features with the same size and effective range as the preset convolutional kernel includes: The region to be filtered is determined in the input feature image based on the attention receptive field. Based on the effective range of the preset convolutional kernel and the attention feature image, a set of pixels whose attention values ​​meet the preset threshold are selected in the region to be filtered. The set of pixels whose attention values ​​meet the preset threshold is subjected to pixel filtering and splicing to generate a deformable feature with the same size and effective range as the preset convolution kernel.

7. The convolution method according to claim 5, characterized in that, After selecting a set of pixels whose attention values ​​meet a preset threshold in the region to be filtered based on the effective range of the preset convolution kernel and the attention feature image, and before performing pixel filtering and splicing processing on the set of pixels whose attention values ​​meet the preset threshold to generate a deformable feature with the same size as the effective range of the preset convolution kernel, the convolution method further includes: An attention difference is generated by calculating the first attention value set of the pixel set and the second attention value set of other pixels in the region to be filtered. When the attention difference is less than a preset threshold, the effective range of the attention receptive field and the preset convolution kernel is reset.

8. A convolution device for a feature image, characterized in that, include: The first acquisition unit is used to acquire a convolutional unit, wherein the convolutional unit includes a pixel attention generation module and a feature convolutional module; The second acquisition unit is used to acquire the input feature image, which is an image used in the training of the input convolutional neural network model. The first input unit is used to input the input feature image into the pixel attention generation module to generate an attention feature image, wherein the attention feature image contains the attention values ​​of the pixels in the input feature image; The first input unit includes: a first generation module, configured to compress the feature channels of the input feature image through the pixel attention generation module to generate compressed features; a compression and decompression module, configured to restore the compressed features to a feature matrix of the same size as the input feature image through the pixel attention generation module; and a second generation module, configured to calculate an attention value for each pixel in the feature matrix through the pixel attention generation module to generate an attention feature image. The second input unit is used to input the input feature image and the attention feature image into the feature convolution module; A convolutional unit is used to perform convolution processing on pixels in the input feature image using the feature convolution module and based on the attention values ​​in the attention feature image to generate convolutional data.

Images