Adaptive multi-scale visual feature expression method and system based on deep attention

Abstract

The present invention discloses an adaptive multi-scale visual feature expression method and system based on deep attention, comprising: obtaining an original neural network model as a trunk branch of an SDA network module for extracting hierarchical features; designing a lightweight deep attention branch of the SDA network module for dynamically weighting hierarchical features; stacking multiple SDA network modules to form a neural network, which is a neural network method for adaptive multi-scale visual feature expression based on deep attention; and optimizing parameters of an adaptive multi-scale feature expression neural network SDA‑Net, wherein the obtained neural network model can be used as the trunk of visual tasks such as image classification, target detection and semantic segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of multi-scale visual feature expression, and in particular to an adaptive multi-scale visual feature expression method and system. Background Art

[0002] Since the target objects are of different sizes, multi-scale feature expression has important practical significance for various computer vision tasks, including image classification, object detection, semantic segmentation, and instance segmentation. Therefore, multi-scale feature expression is widely used in the design of convolutional neural networks (CNNs), such as Res2Net, PyConv, and EPSANet. Among them, multi-kernel convolution is one of the most commonly used methods, which uses different kernel sizes or multi-kernel cascades to extract features of different scales and fuse rich multi-scale information.

[0003] While such multi-core methods can perceive different scales, they tend to quickly increase the size of the receptive field, which is a disadvantage when recognizing small target objects. For example, the recently proposed EPSANet, a variant neural network of ResNet, replaces the 3×3 convolution with a pyramid segmentation attention module to improve the multi-scale feature expression capability. The module consists of several groups of convolutions with different kernel sizes, such as 3, 5, 7, and 9. Therefore, the output of this network consists of multiple components with different sizes of receptive fields. The 3×3 convolution contributes to the receptive field of the smallest part, which is comparable to ResNet in terms of receptive field setting. Other convolutions with larger kernels increase the size of the receptive field. In other words, EPSANet achieves a wide range of effective receptive fields, ranging from the smallest part equivalent to ResNet.

[0004] However, this large receptive field tends to recognize large target objects and ignore small target objects. Therefore, how to design an adaptive multi-scale neural network is the key to extracting receptive fields from small to large ranges to adapt to targets of different scales. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present application provides an adaptive multi-scale visual feature expression method and system based on deep attention.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect is the adaptive multi-scale visual feature expression method based on deep attention, including:

[0008] The original neural network model is obtained as the main branch of the SDA network module for extracting hierarchical features;

[0009] Design a lightweight deep attention branch of the SDA network module for dynamically weighted hierarchical features;

[0010] Stacking multiple SDA network modules to form a neural network is a neural network method for adaptive multi-scale visual feature expression based on deep attention;

[0011] The parameters of the adaptive multi-scale feature expression neural network SDA-Net are optimized, and the obtained neural network model can be used as the backbone of visual tasks such as image classification, object detection and semantic segmentation.

[0012] In a second aspect, one or more embodiments provide an adaptive multi-scale visual feature expression system based on deep attention, including:

[0013] The trunk branch module obtains the original neural network model as the trunk branch of the SDA network module for extracting hierarchical features;

[0014] Deep attention branch module, designing a lightweight deep attention branch of the SDA network module for dynamically weighted hierarchical features;

[0015] Stacking module, stacking multiple SDA network modules to form a neural network, which is a neural network method for adaptive multi-scale visual feature expression based on deep attention;

[0016] The optimization module optimizes the parameters of the adaptive multi-scale feature expression neural network SDA-Net. The obtained neural network model can be used as the backbone of visual tasks such as image classification, target detection and semantic segmentation.

[0017] In a third aspect, the present application also provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the first aspect above.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, completes the method described in the first aspect.

[0019] In a fifth aspect, the present application also provides a computer program (product), including a computer program, which is used to implement any method of the aforementioned first aspect when running on one or more processors.

[0020] Compared with the prior art, the beneficial effects of this application are:

[0021] 1. Solve the problem of target scale changes in computer vision tasks, so that the constructed neural network can adaptively process targets of different scales, thereby achieving better performance. To this end, this application proposes an adaptive multi-scale visual feature expression method based on deep attention. This method dynamically selects hierarchical features of different receptive fields based on the deep attention mechanism, and obtains an adaptive multi-scale feature expression after fusion;

[0022] 2. This application constructs a new independent module based on the attention mechanism of the depth dimension, enriching the module family of the attention neural network;

[0023] 3. The deep attention mechanism of this application can not only work independently, but also be combined with other attention mechanisms, including channel attention, spatial attention and branch attention, to achieve further performance improvement;

[0024] 4. The method of this application can be used in various traditional CNN frameworks, including ResNet, PyConv, Res2Net, SENet, CBAM and EPSANet;

[0025] 5. Comprehensive experiments confirm that the SDA method of this application achieves advanced classification performance on the ImageNet dataset, as well as advanced object detection and instance segmentation performance on the COCO dataset in various CNN frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural diagram of the SDA method of the first embodiment.

[0027] Figure 2 This is a neural network framework diagram of the method of the first embodiment.

[0028] Figure 3 This is a graph comparing the classification performance of the method of the first embodiment and several SOTA multi-scale methods on the ImageNet-1k dataset.

[0029] Figure 4 This is a graph comparing the classification performance of the method of the first embodiment and several SOTA attention methods on the ImageNet-1k dataset.

[0030] Figure 5 This is a graph comparing the target detection results of the method of the first embodiment and several different methods on the COCO dataset.

[0031] Figure 6 This is a comparison chart of instance segmentation results on the COCO dataset between the method of the first embodiment and several different methods. DETAILED DESCRIPTION

[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0035] Existing multi-scale solutions often focus on increasing the size of the receptive field and risk ignoring small receptive fields. Therefore, it is a challenging problem to effectively build adaptive neural networks to recognize objects of various spatial scales. To address this problem, in addition to existing attention dimensions (such as channels, spaces, and branches), this application introduces a new attention dimension, namely depth, and proposes a novel selective deep attention network to comprehensively handle multi-scale objects in various visual tasks. Specifically, given a neural network, such as ResNet, its blocks in each stage output hierarchical feature maps of the same resolution but with different receptive field sizes. Based on this structural feature, this application designs a staged building block, called SDA, which includes a trunk branch and an SE (Squeeze and Selection)-like attention branch. The block outputs of the trunk branch are fused together, and their deep attention allocation is globally guided by the attention branch. According to the proposed attention mechanism, different deep features can be dynamically selected, which helps to adaptively adjust the receptive field size of input objects of variable sizes. In this way, the information interaction across blocks leads to long-range dependencies along the depth direction. Compared with other multi-scale methods, the SDA method of this application combines multiple receptive fields of different depth blocks in each stage into the stage output, thereby providing a wider and richer range of effective receptive fields. In addition, this method can be used as a pluggable module for other multi-scale networks and attention networks, and the combined network is called SDA-Net. Such a combination further expands the range of effective receptive fields, making them develop towards small receptive fields, thereby realizing interpretable neural networks. A large number of experiments have shown that the SDA method of this application achieves state-of-the-art performance and outperforms other multi-scale and attention methods in many computer vision tasks such as image classification, object detection, and instance segmentation.

[0036] Embodiment 1

[0037] This embodiment discloses an adaptive multi-scale visual feature expression method based on deep attention, comprising the following steps:

[0038] S1: Get the original neural network model as the main branch of the SDA network module for extracting hierarchical features:

[0039] Independent of the specific CNN backbone structure, here we take ResNet, SENet and Res2Net as the backbone branches of SDA-Net. Extract a feature map sequence Z = [Z1, Z2, ..., Z m ]. Here, m is the number of blocks, Z i ∈R h×w×crepresents the output of the ith block, where h, w, and c are the height, width, and number of channels, respectively. Since the number of channels is the same, the resolution is the same, but the receptive field size is different, Z is essentially a hierarchical feature structure with gradient semantics from low to high.

[0040] S2: Design a lightweight deep attention branch of the SDA network module for dynamically weighted hierarchical features:

[0041] like Figure 1 As shown, we first use the element summation method to merge the hierarchical features of the trunk branches to obtain the fusion feature F, which is expressed as

[0042]

[0043] where F∈R h×w×c Then, GAP is used to aggregate the spatial information of the F feature map to generate a global spatial context descriptor u∈R 1×1×c In other words, u is generated by reducing the spatial dimension h×w of F;

[0044] Specifically, the kth element of the descriptor u is represented as

[0045]

[0046] Subsequently, u is input into two 1×1 convolutions, expressed as

[0047]

[0048] Where β and γ represent BN and ReLU functions respectively;

[0049] W1∈R d×1×1×c is a learnable linear transformation that maps u to a lower-dimensional space for improved efficiency, W2∈R mc×1×d Used to restore the channel dimension. Here, d = max(c / r, L), where r is the reduction ratio and L is the threshold;

[0050] Furthermore, v is aligned along the depth dimension and reshaped to obtain v T , and then use the softmax activation function to implement the soft weighting mechanism, expressed as

[0051]

[0052] Where δ represents the softmax function, s T is the set of attention vectors;

[0053] Finally, at the end of each SDA module, the semantic information of different scales is adaptively fused by soft weighting across block features according to the scale of the input object. Therefore, the output of the SDA module can be expressed as

[0054]

[0055] where s T is reshaped back to align along the channel dimension s, and O represents the final output of the SDA module.

[0056] S3: Multiple SDA modules are stacked to form a neural network, which is a neural network method for adaptive multi-scale visual feature expression based on deep attention:

[0057] SDA-Net is constructed by stacking multiple SDA modules. Based on ResNet-50, the number of blocks in each stage is adjusted from (3, 4, 6, 3) to (5, 6, 12, 5), and the second convolution in each block is set to 8 groups to align the FLOPs with ResNet-50. Due to the proposed deep attention mechanism, SDA-Net provides a wide effective receptive field for flexible multi-scale expression capabilities. Specifically, if SDA-Net pays more attention to the deep blocks in each stage, it outputs higher-level semantic features with a large receptive field, which makes it more favorable for large-scale objects. On the contrary, if more attention is paid to the shallow blocks in each stage, lower-level features are obtained with a small receptive field, which is suitable for learning small-scale objects. The original ResNet is regarded as a special case of SDA-Net, which generates fixed-scale features for single-scale objects through the last block. In contrast, SDA-Net adaptively learns dynamic multi-scale contexts, achieving better performance for objects of different scales by leveraging intermediate features rather than just the features of the last block;

[0058] More importantly, due to the newly proposed deep attention dimension, the SDA method is orthogonal to other multi-scale attention methods such as EPSANet, SENet, CBAM, PyConv, and Res2Net. Therefore, the SDA method further expands the range of effective receptive fields to small receptive fields to enhance the model expression ability, and its complexity is similar to EPSANet-101, SENet-101, CBAM-101, PyConv-101, and Res2Net-101, respectively. The comparison of ResNet-50, SDA-ResNet-86, SDA-SENet-86 and SDA-Res2Net-86 neural architectures can be found in Figure 2 Found in.

[0059] S4: Parameter optimization of the adaptive multi-scale feature expression neural network SDA-Net is performed. The resulting neural network model can be used as the backbone of visual tasks such as image classification, object detection, and semantic segmentation:

[0060] The stochastic gradient descent algorithm SGD is used to optimize the SDA-Net weights. Specifically, the stochastic gradient descent algorithm SGD with weight decay is used to optimize the weights. The corresponding loss function is expressed as

[0061]

[0062] Among them, the first term is the cross entropy loss function, and the second term is the weight decay term. Adjust the first and second terms for the weighting coefficients, W is the neural network weight parameter.

[0063] Embodiment 2

[0064] This embodiment provides an adaptive multi-scale visual feature expression system based on deep attention, including:

[0065] The trunk branch module obtains the original neural network model as the trunk branch of the SDA network module for extracting hierarchical features;

[0066] Deep attention branch module, designing a lightweight deep attention branch of the SDA network module for dynamically weighted hierarchical features;

[0067] Stacking module, multiple SDA modules are stacked to form a neural network, which is a neural network method for adaptive multi-scale visual feature expression based on deep attention;

[0068] The optimization module optimizes the parameters of the adaptive multi-scale feature expression neural network SDA-Net. The obtained neural network model can be used as the backbone of visual tasks such as image classification, object detection and semantic segmentation;

[0069] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions;

[0070] The description of each embodiment in the above embodiments has different focuses. For parts not described in detail in one embodiment, reference can be made to the relevant descriptions of other embodiments.

[0071] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, and the division of the modules described above is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0072] Embodiment 3

[0073] This embodiment further provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in the first embodiment;

[0074] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.;

[0075] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type;

[0076] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0077] Embodiment 4

[0078] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0079] Simulation experiment

[0080] This application conducts a large number of experiments to verify the effectiveness of SDA-Net on many CV task benchmarks, namely image classification, object detection and instance segmentation;

[0081] In order to evaluate the effectiveness of the SDA-Net of this application in image classification tasks, we conducted experiments on the large-scale ImageNet-1k dataset and deployed the typical ResNet-50 as the backbone branch. Considering the combination with other multi-scale and attention methods, we also deployed several other backbone branches, including EPSANet, SENet, CBAM, PyConv, and Res2Net. The ImageNet-1K dataset contains 1000 object classes, of which 1.28M training images are used for training samples and 50K validation images are used for test samples;

[0082] This application uses the same data augmentation and hyperparameter settings as ResNet-50. Specifically, the training images are resized to 256×256 and then randomly cropped to 224×224, followed by random horizontal flipping. All models are trained from scratch by stochastic gradient descent (SGD) with weight decay of 1e-4, momentum of 0.9, and batch size of 256. The initial learning rate is set to 0.1 and reduced by 10 times every 30 learning epochs until 120 learning epochs. During testing, we resize the validation images to 256×256 and then use center crop 224×224 for evaluation. Label smoothing is used to regularize the network;

[0083] The SDA method is evaluated on the MS COCO dataset for other downstream CV tasks. Multiple detectors such as Faster R-CNN and Mask R-CNN are used in this application. The COCO-2017 dataset consists of approximately 118K training images and 5K validation images for testing over 80 foreground object classes and 1 background class. These detectors are implemented with the MMDetection toolbox and configured using its default settings. Specifically, during training, the input images are resized to 800 pixels on the shorter side. We train all models using SGD for 12 epochs with an initial learning rate of 0.02, which is reduced by a factor of 10 at the 8th and 11th epochs. Other hyperparameters are set as follows: weight decay is 1e-4, momentum is 0.9, batch size is 16 (4 images per GPU);

[0084] Figure 3 The classification performance comparison results of the SDA method of this application and several SOTA multi-scale methods on the ImageNet-1k dataset are shown. Under similar FLOPs, the SDA method achieves higher classification results;

[0085] Figure 4The classification performance comparison results of this application and several SOTA attention methods on the ImageNet-1k dataset are shown. Under similar FLOPs, the SDA method outperforms the current most advanced attention model;

[0086] Figure 5 The SDA method of this application is compared with several different methods in the COCO dataset. At a slightly lower FLOPs, the SDA method achieves higher object detection accuracy;

[0087] Figure 6 The example segmentation comparison results of the SDA method of this application on the COCO dataset are shown. At a slightly lower FLOPs, the SDA method achieves higher instance segmentation accuracy.

[0088] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0089] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An adaptive multi-scale visual feature expression method based on deep attention, characterized by: include: The original neural network model is obtained as the main branch of the SDA network module for extracting hierarchical features; Design a lightweight deep attention branch of the SDA network module for dynamically weighted hierarchical features, specifically including: obtaining the fused feature F by merging the hierarchical features of the trunk branch using the element summation method, aggregating the spatial information of the F feature map using GAP, and generating a global spatial context descriptor u, which is input into two 1×1 convolutions, v=W2(γ(β(W1u))), where W2 is a learnable linear transformation for restoring the channel dimension, γ represents the ReLU function, β represents the BN function, W1 is a learnable linear transformation, W1u represents mapping u to a low-dimensional space, aligning v along the depth dimension, and obtaining v through the reshaping operation. T , and then use the softmax activation function to implement the soft weighting mechanism, s t =δ(v), where δ represents the softmax function, s T is a set of attention vectors. At the end of each SDA module, soft weighting is performed through cross-block features according to the scale of the input object to adaptively fuse semantic information of different scales. Stacking multiple SDA network modules to form a neural network is a neural network method for adaptive multi-scale visual feature expression based on deep attention. The parameters of the adaptive multi-scale feature expression neural network SDA-Net are optimized, and the obtained neural network model is used as the network backbone to perform image classification, object detection and instance segmentation on the input image.

2. The method according to claim 1, characterized in that: The original neural network model is obtained as the main branch of the SDA network module for extracting hierarchical features; The invention also includes: extracting a feature map sequence Z = [Z1, Z2, ..., Z m ], m is the number of blocks.

3. An adaptive multi-scale visual feature expression system based on deep attention, characterized by: include: The trunk branch module obtains the original neural network model as the trunk branch of the SDA network module for extracting hierarchical features; Deep attention branch module, design a lightweight deep attention branch of the SDA network module for dynamically weighted hierarchical features, specifically including: obtaining the fused feature F by merging the hierarchical features of the trunk branch using the element summation method, aggregating the spatial information of the F feature map using GAP, and generating a global spatial context descriptor u, which is input into two 1×1 convolutions, v=W2(γ(β(W1u))), where W2 is a learnable linear transformation for restoring the channel dimension, γ represents the ReLU function, β represents the BN function, W1 is a learnable linear transformation, W1u represents mapping u to a low-dimensional space, aligning v along the depth dimension, and obtaining v through the reshaping operation. T , and then use the softmax activation function to implement the soft weighting mechanism, s T =δ(v), where δ represents the softmax function, s T It is a set of attention vectors. At the end of each SDA module, soft weighting is performed through cross-block features according to the scale of the input object to adaptively fuse semantic information of different scales. In the stacking module, multiple SDA modules are stacked to form a neural network, which is a neural network method for adaptive multi-scale visual feature expression based on deep attention. In the optimization module, parameters of the adaptive multi-scale feature expression neural network SDA-Net are optimized, and the obtained neural network model is used as the network backbone to perform image classification, target detection and instance segmentation on the input image.

4. An electronic device, characterized in that: include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory to enable the electronic device to execute the method described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1-2.

Images