Method and device for constructing visual backbone network based on interlayer mixed attention

Abstract

The invention relates to the technical field of large model architecture, and provides a visual backbone network construction method based on interlayer mixed attention. The method comprises the following steps: constructing a feature input module; constructing at least one linear processing stage for extracting detail features from the sequence features; at least one hybrid processing stage is constructed, an interlayer hybrid mode is adopted for construction, and the interlayer hybrid mode refers to that a global self-attention layer is inserted in a plurality of continuously deployed linear attention layers based on a preset interval; and constructing a hidden state bridging module which is used for carrying out sparse sampling and linear projection on hidden state features output by the linear attention layer in the linear processing stage and then injecting the hidden state features into input features of the global self-attention layer in the hybrid processing stage. According to the construction method and device of the visual backbone network based on the interlayer mixed attention provided by the invention, the efficient visual backbone network which has linear calculation efficiency and global modeling capability and can effectively maintain long-distance detail features can be constructed.

Description

Method and apparatus for constructing a visual backbone network based on interlayer hybrid attention Technical Field

[0001] This invention relates to the field of large model architecture technology, and in particular to a method and apparatus for constructing a visual backbone network based on interlayer hybrid attention. Background Technology

[0002] With the successful application of the Transformer architecture in computer vision, the Vision Transformer (ViT) has become one of the mainstream backbone networks. However, the standard Softmax self-attention mechanism in ViT has a computational complexity that is quadratic with the length of the input sequence (O(N)). 2 The inherent limitations of this technology lead to significant computational and memory pressure when processing high-resolution images.

[0003] To overcome the aforementioned efficiency bottlenecks, existing technologies have proposed several improvement schemes. The first category is window-based attention mechanisms, such as the Swin Transformer, which reduce computation by limiting global attention to a local window. However, this approach sacrifices the ability to model long-range dependencies. The second category is linear attention models, such as Mamba and RWKV, which reduce computational complexity to linear (O(N)) through mechanisms like state-space models. However, research shows that pure linear attention suffers from information decay in long sequence processing; that is, as the sequence length increases, the model's ability to capture distant features decreases exponentially, leading to the loss of detailed information and insufficient performance in dense prediction tasks.

[0004] Furthermore, existing technologies attempt to combine linear attention with standard self-attention to balance efficiency and performance. Existing hybrid strategies mainly fall into two categories: one is to compute both types of attention in parallel within the same module (intra-block hybridization), but this still requires computationally expensive Softmax; the other is to use different attention at different stages of the network (stage-level hybridization), for example, using convolutional or linear attention in shallow layers and self-attention in deeper layers. However, these strategies are often rigid, either failing to fundamentally eliminate the quadratic computational burden or being forced to degenerate into window attention in deeper layers for efficiency, further weakening global modeling capabilities, and typically relying on convolutional neural networks as the underlying structure for downsampling or feature extraction.

[0005] Therefore, existing visual backbone networks struggle to achieve efficient, explicit global feature interaction while maintaining linear computational complexity and preserving necessary fine-grained spatial information. Summary of the Invention

[0006] This application provides a method and apparatus for constructing a visual backbone network based on interlayer hybrid attention, which can solve the technical problem of not being able to balance computational efficiency, global modeling ability and detail preservation ability in high-resolution image processing, and achieve efficient and accurate visual feature extraction with near-linear computational complexity.

[0007] In a first aspect, embodiments of this application provide a method for constructing a visual backbone network based on interlayer hybrid attention, comprising: constructing a feature input module for converting an input image into sequential features; constructing at least one linear processing stage, each linear processing stage consisting of multiple sequentially connected linear attention layers for extracting detail features from the sequential features; constructing at least one hybrid processing stage, wherein the hybrid processing stage is constructed using an interlayer hybrid mode, the interlayer hybrid mode referring to inserting a global self-attention layer at a preset interval among multiple consecutively deployed linear attention layers; and constructing a hidden state bridging module for sparsely sampling and linearly projecting the hidden state features output by the linear attention layers in the linear processing stage, and then injecting them into the input features of the global self-attention layer in the hybrid processing stage.

[0008] In one embodiment, the hidden state features output by the linear attention layer in the linear processing stage are sparsely sampled and linearly projected, and then injected into the input features of the global self-attention layer in the hybrid processing stage. Specifically, this can be achieved by: sparsely sampling the hidden state features to reduce the spatial dimension; linearly projecting the sampled features; and fusing the projected features with the features to be input to the global self-attention layer in the hybrid processing stage through a gated weighting method.

[0009] In another embodiment, a global self-attention layer is inserted based on a preset interval. Specifically, this can be implemented as follows: after every k consecutive linear attention layers are arranged, a global self-attention layer is inserted, where k is an integer greater than 1.

[0010] In yet another embodiment, the constructed visual backbone network includes four sequentially connected processing stages, wherein the first two stages are linear processing stages and the last two stages are hybrid processing stages.

[0011] Secondly, embodiments of this application provide an application of a visual backbone network in image processing, including: acquiring an image to be processed; inputting the image to be processed into a constructed visual backbone network to extract image features; and performing object detection or semantic segmentation tasks based on the extracted image features.

[0012] In one embodiment, the aforementioned visual backbone network is a pre-trained network; when performing object detection or semantic segmentation tasks, only some parameters of the visual backbone network or task-specific network head parameters are fine-tuned.

[0013] Thirdly, embodiments of this application provide a construction apparatus for a visual backbone network based on interlayer hybrid attention, comprising: a construction module.

[0014] The aforementioned construction module is used to construct a feature input module to convert the input image into sequential features; the aforementioned construction module is also used to construct at least one linear processing stage, each of which consists of multiple sequentially connected linear attention layers, used to extract detailed features from the sequential features; the aforementioned construction module is also used to construct at least one hybrid processing stage, which is constructed using an inter-layer hybrid mode, whereby a global self-attention layer is inserted into multiple consecutively deployed linear attention layers at a preset interval; the aforementioned construction module is also used to construct a hidden state bridging module, used to sparsely sample and linearly project the hidden state features output by the linear attention layers in the linear processing stage, and then inject them into the input features of the global self-attention layer in the hybrid processing stage.

[0015] Thirdly, embodiments of this application provide an application apparatus for a visual backbone network in image processing, comprising: an acquisition module, an extraction module, and a processing module.

[0016] The acquisition module is used to acquire the image to be processed; the extraction module is used to input the image to be processed into the constructed visual backbone network to extract image features; the processing module is used to perform object detection or semantic segmentation tasks based on the extracted image features.

[0017] Fifthly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program. When the processor executes the computer program, it implements the steps of the method for constructing a visual backbone network based on interlayer hybrid attention in the first aspect, or the steps of applying the visual backbone network in image processing in the second aspect.

[0018] The method and apparatus for constructing a visual backbone network based on interlayer hybrid attention provided in this application reduce the overall computational complexity of the network from quadratic to approximately linear when processing high-resolution images by constructing a hybrid processing stage with linear attention layers as the main body and sparsely inserting global self-attention layers at key locations. This significantly reduces the computational burden of high-resolution image processing. Furthermore, by constructing a hidden state bridging module, the detailed hidden state features from the linear processing stage are directly injected into the deep global attention layer after optimization, providing high-resolution detail supplementation from the shallow layer for deep global information interaction. This mechanism, working in conjunction with the sparse global attention layer, overcomes the defect of pure linear models where detailed information is easily forgotten in long sequences, improving the network's ability to model subtle targets and edge features in images. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a flowchart illustrating the method for constructing a visual backbone network based on interlayer hybrid attention according to an embodiment of this application; Figure 2 is a structural diagram of the visual backbone network according to an embodiment of this application; Figure 3 is a flowchart illustrating the application of the visual backbone network according to an embodiment of this application in image processing; Figure 4 is a structural diagram of the device for constructing a visual backbone network based on interlayer hybrid attention according to an embodiment of this application; Figure 5 is a structural diagram of the device for applying the visual backbone network in image processing according to an embodiment of this application; Figure 6 is a structural diagram of the electronic device according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] This invention aims to solve the following problems existing in the visual backbone network: 1. The standard self-attention has high computational complexity and cannot efficiently process high-resolution images.

[0023] 2. Pure linear attention models suffer from information decay, resulting in weak long-distance feature interaction capabilities and insufficient performance in high-resolution dense prediction tasks.

[0024] 3. Existing linear attention hybrid architecture strategies are rigid and cannot achieve efficient global feature correction with a very low proportion of self-attention layers.

[0025] 4. Existing hybrid attention architectures typically rely on convolutional neural network downsampling, which fails to demonstrate the advantages of a fully attention-based architecture.

[0026] The following section, with reference to Figures 1 to 6, details the method and apparatus for constructing a visual backbone network based on interlayer hybrid attention provided in the embodiments of this application.

[0027] Figure 1 is a flowchart illustrating the method for constructing a visual backbone network based on interlayer hybrid attention according to an embodiment of this application. Referring to Figure 1, an embodiment of this application provides a method for constructing a visual backbone network based on interlayer hybrid attention, which may include: Step S101: Constructing a feature input module.

[0028] The feature input module receives the input image and converts it into sequence features suitable for processing by the Transformer encoder.

[0029] In one embodiment, the feature input module extracts and downsamples the input image through a convolutional layer. Specifically, a convolutional layer with a stride of 4 is used to downsample the input image by a factor of 4, embedding image patches into a feature sequence, so that the spatial size of the output feature map becomes H / 4×W / 4.

[0030] This convolution operation maps each local region of the input image into a high-dimensional feature vector, achieving a transformation from pixel space to feature space. The feature map is then unfolded into a feature sequence according to its spatial location, and positional encoding and classification labels are added to form sequence features. This design reduces subsequent computational complexity while preserving the local structural information of the image.

[0031] Step S102: Construct at least one linear processing stage.

[0032] The linear processing stage consists of multiple sequentially connected linear attention layers, which are used to extract detailed features from the sequence features.

[0033] Specifically, linear attention layers can be implemented using attention mechanisms based on recursive updates of hidden states, such as weighted key-value (WKV) or receptance weighted key-value (RWKV) operators. These operators replace traditional attention matrix calculations with recursive accumulation of hidden states, thereby reducing computational complexity to linear and supporting efficient autoregressive inference.

[0034] WKV achieves linear complexity by assigning weights that decay over time (or position) to historical information (past keys k and values ​​v) and accumulating them into a hidden state, thus avoiding the need to compute all pairwise similarities as in traditional attention.

[0035] The RWKV operator can be understood as an attention mechanism that controls the inflow of information through a learnable receiving gate (R) and performs weighted aggregation of historical key values ​​(kv).

[0036] The core of hidden state recursive update lies in compressing historical information through recursively updated hidden state H_t, thereby achieving attention computation with linear complexity.

[0037] In one embodiment, the output vector of the linear attention layer can be represented as: in, This represents the output vector of the linear attention layer; Represents a linear transformation function (usually ReLU, SiLU, or exponential function, etc.); A query vector representing the current time step (or spatial location); This represents the hidden state matrix that is recursively updated with time step (spatial location), accumulating and compressing all historical information from the starting position to the current time step; This represents the dot product of a vector and a matrix.

[0038] As shown in Figure 2, the first two processing stages (Stage 1, Stage 2) can be constructed as linear processing stages. Each linear processing stage consists of two linear processing layers connected sequentially.

[0039] It should be noted that the feature resolution in the linear processing stage is relatively high. Using stacked pure linear attention layers can minimize computational complexity and memory usage, achieving low latency in the early stages of feature extraction.

[0040] Step S103: Construct at least one hybrid processing stage.

[0041] The hybrid processing stage is constructed using an inter-layer hybrid mode, which refers to inserting a global self-attention layer at a preset interval among multiple consecutively deployed linear attention layers.

[0042] In one embodiment, as shown in Figure 2, the third and fourth processing stages (Stage 3, Stage 4) can be constructed as a hybrid processing stage.

[0043] In the hybrid processing stage, the feature map resolution has been significantly reduced by the previous downsampling (e.g., reduced to 1 / 16 and 1 / 32 of the input resolution, respectively). At this time, the introduction of a self-attention layer with global interaction capability reduces the computational and memory overhead required by the originally computationally complex global attention layer to an acceptable range.

[0044] In the embodiments of this application, the preset interval for inserting global self-attention layers can be an alternating stacking of linear attention layers-linear attention layers-global self-attention layers-linear attention layers-linear attention layers-global self-attention layers, or an alternating stacking of linear attention layers-global self-attention layers-linear attention layers-global self-attention layers.

[0045] That is, after establishing local features through k consecutive linear attention layers, a global self-attention layer is inserted to perform information interaction across the entire graph, where k is an integer greater than 1.

[0046] For example, as shown in Figure 2, after establishing local features through two consecutive linear attention layers in stage 3, a global self-attention layer is inserted; that is, in stage 3, the layers are stacked alternately in the pattern of linear attention layer-linear attention layer-global self-attention layer-linear attention layer-linear attention layer-global self-attention layer.

[0047] In stage 4, after each consecutive linear attention layer has been used to build local features, a global self-attention layer is inserted. That is, in stage 4, linear attention layers and global self-attention layers are stacked alternately.

[0048] This invention employs an interlayer hybrid approach, sparsely inserting a very small number of global self-attention layers between consecutively stacked linear attention layers. The effective receptive field of stacked linear layers grows only sublinearly with depth, making it difficult to establish complete global interactions. Therefore, the inserted sparse self-attention layers act as a "shortcut," efficiently reconstructing explicit global pairwise dependencies after the sequence information is progressively compressed. This approach uses an alternating hybrid mode in the mid-to-late stages of the model, significantly reducing computational overhead while achieving performance superior to densely used self-attention layers.

[0049] Step S104: Construct the hidden state bridging module.

[0050] The hidden state bridging module is used to connect the linear processing stage and the hybrid processing stage.

[0051] The hidden state bridging module processes the hidden state features output by the linear attention layer in the linear processing stage and injects them into the input of the global self-attention layer in the hybrid processing stage to achieve cross-layer and cross-resolution feature enhancement.

[0052] Specifically, constructing a hidden state bridging module may include the following sub-steps: Obtaining source features: Extracting the hidden state features H_shallow generated during the computation process from the output of a linear attention layer in a specified linear processing stage (e.g., Stage 1 or Stage 2).

[0053] Sparse sampling: Since H_shallow preserves high-resolution details, in order to reduce the amount of data and focus on key information, a sparse sampling function (such as the equidistant sampling function Samp) is used to select some feature tokens, and their spatial size is aligned to the size of the target deep feature through downsampling.

[0054] Feature injection: The sampled features are transformed using a learnable linear projection matrix W_HSB. Simultaneously, the transformed features are modulated using a gating weight (obtained from the previous features of the target layer through linear transformation and activation function σ).

[0055] Finally, the modulated features are added to the features X_out(deep-1) to be input into the deep global self-attention layer to complete the injection.

[0056] Specifically, the input features of the global self-attention layer satisfy the following formula: in, This represents the input features of the current deepest (the deepest) global self-attention layer; This represents the output features of the layer preceding the current deepest layer (the -1st layer); This represents the activation function, usually the Sigmoid function; it is used to generate a gated weight vector with values ​​between (0,1) to adaptively control the strength of the injected features. Represents the gate weight matrix, a learnable linear projection parameter that maps X (usually X_out(deep-1)) to a space that matches the gate dimension; This represents the input characteristics of gating computation; This represents element-wise multiplication; The projection matrix representing the hidden state bridging; Represents a sparse sampling function; This represents the hidden state features extracted from the shallow linear attention layer.

[0057] The above formula shows that the input to a deep global self-attention layer consists of the output of the previous global self-attention layer plus a gated modulated detail feature from a shallow layer. In this way, the deep global self-attention layer can directly "see" and utilize the high-resolution detail information from the shallow layers of the network during computation, thereby effectively mitigating the problem of long-distance detail information attenuation that may be caused by multiple linear attention stackings.

[0058] For example, as shown in Figure 2, a visual backbone network may include a feature input module and four sequentially connected processing stages (Linear Processing Stage 1, Linear Processing Stage 2, Hybrid Processing Stage 3, Hybrid Processing Stage 4) to achieve feature extraction from fine-grained to abstract semantics.

[0059] Feature input module: The input image (size is H×W, channel is C1, usually C1=3) first goes through input slicing and tile embedding operations.

[0060] For example, an image can be divided into non-overlapping patches by a convolutional layer with a stride of 4 and each patch can be projected as an embedding vector, thereby converting a two-dimensional image into a one-dimensional sequence feature of length (H / 4×W / 4).

[0061] Linear processing stages (Stage 1 and Stage 2): As shown in Figure 2, the first stage (Stage 1) and the second stage (Stage 2) are composed of multiple tile fusion modules stacked sequentially.

[0062] The tile fusion module is the basic unit for feature interaction in the processing stage: the tile fusion module in the linear processing stage consists only of linear attention layers. Its purpose is to efficiently extract local detail features with linear computational complexity at high resolutions (H / 4×W / 4 to H / 16×W / 16 as shown in Figure 2), ensuring the computational efficiency of the network in the early stages.

[0063] Hybrid processing stages (Stage 3 and Stage 4): As shown in Figure 2, the third stage (Stage 3) and the fourth stage (Stage 4) are also composed of stacked tile fusion modules, but they adopt an inter-layer blending mode internally.

[0064] Among them, the tile fusion module is the basic unit for feature interaction in the processing stage: the tile fusion module in the hybrid processing stage includes a linear attention layer and a global self-attention layer, which are deployed alternately at preset intervals.

[0065] For example, Stage 3 can be constructed using an alternating pattern of linear attention layer - linear attention layer - global attention layer, while Stage 4 can be constructed using an alternating pattern of linear attention layer - global attention layer. At this stage, global interactions are introduced at the semantic level with extremely low overhead at a further reduced resolution (e.g., H / 32×W / 32), achieving efficient context modeling.

[0066] Hidden State Bridging Module: Figure 2 clearly shows three hidden state bridging paths. These paths directly inject the high-resolution, detailed hidden state features generated by the linear attention layer into the global self-attention layer after sparse sampling and transformation. This connection ensures that deep networks can directly access and utilize the fine-grained information from shallow layers during global inference, thereby effectively mitigating the information decay problem.

[0067] Feature Flow and Resolution Variation: Figure 2 clearly illustrates the changes in data flow and spatial resolution. After embedding, the sequence spatial resolution starts at H / 4×W / 4. With each processing stage, which typically involves downsampling operations (such as convolution), the feature map spatial size decreases step by step (e.g., to H / 16×W / 16, H / 32×W / 32).

[0068] The method for constructing a visual backbone network based on interlayer hybrid attention provided in this application constructs a hidden state bridging module to efficiently transfer high-resolution details from shallow layers to deep layers, thereby achieving both powerful global context modeling and fine local detail preservation in a network dominated by linear computational complexity.

[0069] The visual backbone network constructed based on the aforementioned method, due to its combination of linear computational efficiency and powerful multi-level feature representation capabilities, can be widely applied to various high-resolution image understanding tasks. The following section provides a detailed explanation using typical application scenarios.

[0070] Figure 3 is a flowchart illustrating the application of a visual backbone network in image processing according to an embodiment of this application. Referring to Figure 3, an embodiment of this application provides an application of a visual backbone network in image processing, which may include: Step S301: Acquiring the image to be processed.

[0071] Step S302: Input the image to be processed into the visual backbone network and extract image features.

[0072] Step S303: Based on the extracted image features, perform object detection or semantic segmentation tasks.

[0073] In some embodiments, the aforementioned visual backbone network is a pre-trained network, and when performing object detection or semantic segmentation tasks, only some parameters of the visual backbone network or task-specific network head parameters are fine-tuned.

[0074] It should be noted that this invention can be widely applied to fields requiring the processing of high-resolution input and real-time decision-making. For example, in autonomous driving systems, vehicle-mounted cameras need to continuously acquire high-definition street view images. Using the network constructed in this invention as the backbone network of its perception module enables real-time analysis of high-resolution images while maintaining linear computational complexity. Its hidden-state bridging mechanism effectively transmits and retains key details such as distant traffic signs, pedestrians, and small obstacles, overcoming the problem of feature forgetting caused by long distances and small pixel counts. This significantly improves the reliability of detecting distant and small objects, providing more accurate and timely perception results for autonomous driving path planning and decision-making.

[0075] The visual backbone network of this invention is also well-suited for deployment on edge devices with limited computing power, memory, and power consumption, such as smartphones, tablets, and embedded IoT devices. Compared to traditional Visual Transformers (ViTs), this invention significantly reduces memory usage and computational latency during model inference by using linear attention as the primary method and sparse global attention. This enables real-time execution of tasks such as high-precision image classification, background blurring, and instant semantic segmentation (e.g., subject segmentation in photo editing) on ​​mobile devices, improving user experience while meeting stringent requirements for device battery life and responsiveness.

[0076] The method for constructing a visual backbone network based on interlayer hybrid attention provided in this application has the following significant advantages: (1) By adopting the "interlayer hybrid" strategy, a small number of global self-attention layers are sparsely inserted only at key locations, while linear attention is used in the vast majority of layers. This makes the overall computational complexity of the network increase approximately linearly with the input resolution, significantly reducing the computational burden of high-resolution image processing. At the same time, these sparse global attention layers effectively reconstruct the pairwise feature interactions across the entire image, ensuring the model's representational ability.

[0077] (2) The hidden state features, rich in detail, output from the shallow high-resolution linear attention layer are injected across scales into the input of the deep global self-attention layer through the hidden state bridging module, so that the deep network can directly "access" the fine-grained information of the shallow layer when performing global computation. This design fundamentally makes up for the defect of pure linear models in which detailed features are easily forgotten in long sequences, and significantly improves the model's ability to perceive details such as edges and small objects in images.

[0078] The foregoing mainly describes the solution provided in this application. Accordingly, this application also provides a device for constructing a visual backbone network based on inter-layer hybrid attention, which is used to implement the above-described method embodiments.

[0079] The following describes the apparatus for constructing a visual backbone network based on interlayer hybrid attention provided in the embodiments of this application. The apparatus for constructing a visual backbone network based on interlayer hybrid attention described below can be referred to in conjunction with the method for constructing a visual backbone network based on interlayer hybrid attention described above.

[0080] Figure 4 is a schematic diagram of the structure of the construction device for a visual backbone network based on interlayer hybrid attention provided in an embodiment of this application. As shown in Figure 4, this embodiment of the application provides a construction device for a visual backbone network based on interlayer hybrid attention, which may include: a construction module 401, used to construct a feature input module for converting an input image into sequential features; the construction module 401 is also used to construct at least one linear processing stage, each of which consists of multiple sequentially connected linear attention layers, for extracting detail features from the sequential features; the construction module 401 is also used to construct at least one hybrid processing stage, which is constructed using an interlayer hybrid mode, whereby a global self-attention layer is inserted into multiple consecutively deployed linear attention layers at a preset interval; the construction module 401 is also used to construct a hidden state bridging module, used to sparsely sample and linearly project the hidden state features output by the linear attention layers in the linear processing stage, and then inject them into the input features of the global self-attention layer in the hybrid processing stage.

[0081] In one embodiment, the construction module 401 is specifically used to: perform sparse isometric sampling on the latent state features to reduce the spatial dimension; perform linear projection on the sampled features; and fuse the projected features with the features to be input to the global self-attention layer in the hybrid processing stage through a gated weighting method.

[0082] In another embodiment, the construction module 401 is specifically used to: insert a global self-attention layer after every k consecutive linear attention layers are arranged, where k is an integer greater than 1.

[0083] In yet another embodiment, the construction module 401 is specifically used for: the linear attention layer being an attention unit based on recursive updates of hidden states.

[0084] In another embodiment, the construction module 401 is specifically used for: the constructed visual backbone network includes four sequentially connected processing stages, wherein the first two stages are linear processing stages and the last two stages are hybrid processing stages.

[0085] The following describes the application apparatus of the visual backbone network in image processing provided in the embodiments of this application. The application apparatus of the visual backbone network in image processing described below can be referred to in conjunction with the application of the visual backbone network in image processing described above.

[0086] Figure 5 is a schematic diagram of the structure of the visual backbone network application device in image processing provided in the embodiment of this application. As shown in Figure 5, the embodiment of this application provides an application device for a visual backbone network in image processing, which may include: an acquisition module 501 for acquiring an image to be processed; an extraction module 502 for inputting the image to be processed into the constructed visual backbone network to extract image features; and a processing module 503 for performing object detection or semantic segmentation tasks based on the extracted image features.

[0087] In one implementation, the aforementioned visual backbone network is a pre-trained network; when performing object detection or semantic segmentation tasks, only some parameters of the visual backbone network or task-specific network head parameters are fine-tuned.

[0088] In some embodiments, the apparatus for constructing a visual backbone network based on interlayer hybrid attention includes hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-described functions. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0089] This application embodiment can divide the construction device of the visual backbone network based on interlayer hybrid attention into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into a feature extraction module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0090] Figure 6 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. As shown in Figure 6, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute the steps of the method for constructing a visual backbone network based on interlayer hybrid attention. The method includes: constructing a feature input module for converting an input image into sequential features; constructing at least one linear processing stage, each of which consists of multiple sequentially connected linear attention layers for extracting detail features from the sequential features; constructing at least one hybrid processing stage, wherein the hybrid processing stage is constructed using an interlayer hybrid mode, which refers to inserting a global self-attention layer at a preset interval among multiple consecutively deployed linear attention layers; and constructing a hidden state bridging module for sparsely sampling and linearly projecting the hidden state features output by the linear attention layers in the linear processing stage and then injecting them into the input features of the global self-attention layer in the hybrid processing stage.

[0091] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the construction method of the visual backbone network based on interlayer hybrid attention provided in the above embodiments, such as: constructing a feature input module for converting an input image into sequential features; constructing at least one linear processing stage, each of which consists of multiple sequentially connected linear attention layers for extracting detail features from the sequential features; constructing at least one hybrid processing stage, wherein the hybrid processing stage is constructed using an interlayer hybrid mode, which refers to inserting a global self-attention layer at a preset interval among multiple consecutively deployed linear attention layers; and constructing a hidden state bridging module for sparsely sampling and linearly projecting the hidden state features output by the linear attention layer in the linear processing stage and then injecting them into the input features of the global self-attention layer in the hybrid processing stage.

[0093] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the methods provided in the above embodiments, such as: constructing a feature input module for converting an input image into sequential features; constructing at least one linear processing stage, each linear processing stage consisting of multiple sequentially connected linear attention layers for extracting detail features from the sequential features; constructing at least one hybrid processing stage, the hybrid processing stage being constructed using an inter-layer hybrid mode, the inter-layer hybrid mode referring to inserting a global self-attention layer based on a preset interval among multiple consecutively deployed linear attention layers; and constructing a hidden state bridging module for sparsely sampling and linearly projecting the hidden state features output by the linear attention layers in the linear processing stage, and then injecting them into the input features of the global self-attention layer in the hybrid processing stage.

[0094] Processor-readable storage media can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing a visual backbone network based on inter-layer hybrid attention, characterized in that, The method includes: constructing a feature input module for converting an input image into sequential features; constructing at least one linear processing stage, each of which consists of multiple sequentially connected linear attention layers for extracting detail features from the sequential features; constructing at least one hybrid processing stage, wherein the hybrid processing stage is constructed using an inter-layer hybrid mode, the inter-layer hybrid mode referring to inserting a global self-attention layer at a preset interval among multiple consecutively deployed linear attention layers; and constructing a hidden state bridging module for sparsely sampling and linearly projecting the hidden state features output by the linear attention layers in the linear processing stage, and then injecting them into the input features of the global self-attention layer in the hybrid processing stage.

2. The method for constructing a visual backbone network based on inter-layer hybrid attention according to claim 1, characterized in that, The step of sparsely sampling and linearly projecting the hidden state features output by the linear attention layer in the linear processing stage and then injecting them into the input features of the global self-attention layer in the hybrid processing stage includes: sparsely sampling the hidden state features to reduce the spatial dimension; linearly projecting the sampled features; and fusing the projected features with the features to be input to the global self-attention layer in the hybrid processing stage using a gated weighting method.

3. The method for constructing a visual backbone network based on interlayer hybrid attention according to claim 1 or 2, characterized in that, The insertion of the global self-attention layer based on a preset interval includes: inserting one of the global self-attention layers after every k consecutive linear attention layers are arranged, where k is an integer greater than 1.

4. The method for constructing a visual backbone network based on inter-layer hybrid attention according to claim 1, characterized in that, The linear attention layer is an attention unit based on recursive updates of hidden states.

5. The method according to claim 1, characterized in that, The constructed visual backbone network comprises four sequentially connected processing stages, wherein the first two stages are the linear processing stages and the last two stages are the hybrid processing stages.

6. An application of a visual backbone network in image processing, characterized in that, include: Obtain the image to be processed; input the image to be processed into the visual backbone network constructed by the method of any one of claims 1 to 5, and extract image features; Based on the extracted image features, object detection or semantic segmentation tasks are performed.

7. The application of the visual backbone network according to claim 6 in image processing, characterized in that, The visual backbone network is a pre-trained network; when performing the object detection or semantic segmentation task, only some parameters of the visual backbone network or task-specific network head parameters are fine-tuned.

8. A device for constructing a visual backbone network based on interlayer hybrid attention, characterized in that, include: The building module is used to construct the feature input module, which is used to convert the input image into sequence features; The building module is also used to build at least one linear processing stage, each of which consists of multiple sequentially connected linear attention layers, for extracting detailed features from the sequence features; The construction module is further configured to construct at least one hybrid processing stage, wherein the hybrid processing stage is constructed using an inter-layer hybrid mode, wherein the inter-layer hybrid mode refers to inserting a global self-attention layer at a preset interval among multiple consecutively deployed linear attention layers; the construction module is further configured to construct a hidden state bridging module, which is used to sparsely sample and linearly project the hidden state features output by the linear attention layer in the linear processing stage, and then inject them into the input features of the global self-attention layer in the hybrid processing stage.

9. An application device for a visual backbone network in image processing, characterized in that, include: The acquisition module is used to acquire the image to be processed; The extraction module is used to input the image to be processed into the visual backbone network constructed by the method of any one of claims 1 to 5, and extract image features; The processing module is used to perform object detection or semantic segmentation tasks based on the extracted image features.

10. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for constructing a visual backbone network based on interlayer hybrid attention as described in any one of claims 1 to 5, or the steps of applying the visual backbone network in image processing as described in any one of claims 6 to 7.

Images