Computer-implemented method and network for dense prediction

Abstract

The present disclosure relates to a computer-implemented network for dense prediction, the network comprising: a plurality of sub-networks, where the plurality of sub-networks are cascaded together, each sub-network in the plurality of sub-networks comprises a sub-trunk from bottom to top for multi-scale feature extraction and a feedback path from top to bottom for multi-scale feature fusion; a lateral jump connection between feature maps at corresponding feature levels in a bottom-up sub-trunk and a top-down feedback path in the sub-network; and a lateral hopping connection across adjacent sub-networks.

Description

Technical Field

[0001] Generally, aspects of the present disclosure relate to artificial intelligence, and more particularly, to methods and networks for dense prediction. Background Art

[0002] Multi-scale feature maps are essential for dense prediction tasks on images, including object detection, instance segmentation, and semantic segmentation. For example, instance segmentation aims to classify and localize individual objects using pixel-wise instance masks. Instance segmentation has a wide range of applications, including autonomous driving, robotics, video surveillance, etc. Learning multi-scale feature representations is of great significance because high-performance instance segmentation requires recognizing a varying number of instances over a wide range of scales and locations.

[0003] Feature Pyramid Network (FPN) is widely used in existing architectures for multi-scale processing. FPN usually exploits the inherent feature hierarchy generated by the classification network to build a feature pyramid with strong semantics at all scales by fusing adjacent features from the inherent feature hierarchy through lateral connections and top-down pathways.

[0004] However, FPN and many related methods decouple the feature learning process into two sequential parts, i.e., first extracting multi-scale feature maps through the classification network, and then fusing these feature maps through a lightweight module consisting of one or more paths. Such a design may be suboptimal for dense prediction tasks. In addition, since most of the parameters and computational budget have been allocated to the classification network as the backbone, the feature fusion process may be insufficient under a limited computational budget.

[0005] Therefore, it may be desirable to provide an improved architecture or technique to facilitate the performance of multi-scale processing for dense prediction. Summary of the invention

[0006] A simplified overview of one or more aspects of the present disclosure is presented below to provide a basic understanding of these aspects. This overview is not an extensive overview of all contemplated aspects, and is neither intended to identify key or essential elements of all aspects, nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0007] In one aspect of the present disclosure, a computer-implemented network for dense prediction is provided, the network comprising: a plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion; lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the sub-networks and in the top-down feedback paths; and lateral skip connections across adjacent sub-networks.

[0008] In another aspect of the present disclosure, a computer-implemented method for dense prediction using a network is provided, the method comprising: providing an input to the network consisting of a plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion, and wherein the network further comprises lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the sub-networks and in the top-down feedback paths, and lateral skip connections across adjacent sub-networks; and providing an output feature map of the network to a head for a task; and wherein the input comprises a digital image having a scale, and the output feature map is of different scales, and the task comprises image classification, instance segmentation, object detection, or semantic segmentation.

[0009] In another aspect of the present disclosure, there is provided an apparatus for dense prediction, the apparatus comprising a memory and at least one processor, the at least one processor being coupled to the memory and configured to perform a computer-implemented method of dense prediction using a network, the method comprising: providing an input to the network consisting of a plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion, and wherein the network further comprises lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the sub-networks and in the top-down feedback paths, and lateral skip connections across adjacent sub-networks; and providing an output feature map of the network to a head for a task; and wherein the input comprises a digital image having a scale, and the output feature map is of different scales, and the task comprises image classification, instance segmentation, object detection, or semantic segmentation.

[0010] In another aspect of the present disclosure, a computer program product for dense prediction is provided, the computer program product comprising processor executable computer code for a computer-implemented method for performing dense prediction using a network, the method comprising: providing input to the network consisting of a plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion, and wherein the network further comprises lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the sub-networks and in the top-down feedback paths, and lateral skip connections across adjacent sub-networks; and providing an output feature map of the network to a head for a task; and wherein the input comprises a digital image having a scale, and the output feature map is of different scales, and the task comprises image classification, instance segmentation, object detection, or semantic segmentation.

[0011] In another aspect of the present disclosure, a computer-readable medium storing computer code for dense prediction is provided, which, when executed by a processor, causes the processor to perform a computer-implemented method of dense prediction using a network, the method comprising: providing an input to the network consisting of a plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion, and wherein the network further comprises lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the sub-networks and in the top-down feedback paths, and lateral skip connections across adjacent sub-networks; and providing an output feature map of the network to a head for a task; and wherein the input comprises a digital image having a scale, and the output feature map is of different scales, and the task comprises image classification, instance segmentation, object detection, or semantic segmentation.

[0012] By splitting a single backbone into multiple sub-backbones and cascading them together with a top-down feedback path following each sub-backbone, extracting multi-scale feature maps and fusing multi-scale feature maps can be performed in an iterative manner, which can improve the performance of the multi-scale process without introducing additional computational cost.

[0013] Other aspects or variations of the present disclosure, as well as other advantages thereof, will become apparent by consideration of the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Aspects of the disclosure will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects.

[0015] Figure 1A , Figure 1B , Figure 1C and Figure 1D Several current architectures in the prior art for dense prediction tasks are shown.

[0016] Figure 2 A schematic diagram of a feature pyramid network (FPN) for obtaining multi-scale feature maps is shown.

[0017] Figure 3 A schematic diagram of an iterative architecture for obtaining a multi-scale feature map according to one or more aspects of the present disclosure is shown.

[0018] Figure 4A , Figure 4B and Figure 4C Schematic diagrams showing three alternatives of the architecture of sub-networks that may be implemented in the proposed iterative architecture according to one or more aspects of the present disclosure.

[0019] Figure 5 An exemplary schematic diagram illustrating common structures in a bottom-up path according to one or more aspects of the present disclosure.

[0020] Figure 6 A schematic diagram showing a design of a network block that may be employed in a sub-backbone according to one or more aspects of the present disclosure.

[0021] Figure 7 A schematic diagram illustrating another design for a focused network block that may be employed in a sub-backbone according to one or more aspects of the present disclosure.

[0022] Figure 8 An exemplary workflow of a method for dense prediction using a network of the proposed iterative architecture according to one or more aspects of the present disclosure is shown.

[0023] Fig. 9 An example of a hardware implementation of an apparatus according to one or more aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The present disclosure will now be discussed with reference to several exemplary implementations. It should be understood that these implementations are only discussed to enable those skilled in the art to better understand and thus implement embodiments of the present disclosure, rather than implying any limitation on the scope of the present disclosure.

[0025] Convolutional neural networks (CNNs) and transformer-based networks have achieved promising results in many computer vision tasks, including image classification, object detection, semantic segmentation, etc. For these tasks, most popular CNNs (e.g., AlexNet, VGGNet, ResNet, ConvNeXt) and recently developed transformer-based networks (e.g., SwinTransformer, Focal Transformer, PoolFormer) follow a sequential approach in the design of the architecture, i.e., gradually reducing the spatial size of feature maps and performing predictions based on the coarsest feature scale (with the lowest resolution). However, for many dense prediction tasks, such as object detection and instance segmentation, multi-scale features are required to handle objects of different scales.

[0026] Figure 1A , Figure 1B , Figure 1C and Figure 1D Several current architectures for dense prediction tasks such as object detection are shown, where images are represented by shading, feature maps are represented by outlines, and thicker outlines represent semantically stronger feature maps. Figure 1A The architecture 110 shows a feature pyramid built on an image pyramid, where feature maps are computed independently at each image scale, which is slow. Figure 1B Architecture 120 shows the above architecture, which uses only feature maps of a single scale for faster detection (e.g., via prediction 105). Figure 1C The architecture 130 shows an alternative approach to reusing the pyramid feature hierarchy computed by the ConvNet as if it were a feature pyramid built on an image pyramid. (e.g., in Figure 1B and Figure 1C The feature layers in this network produce feature maps with different spatial resolutions, but introduce large semantic gaps between them caused by different depths in the network (e.g., Figure 1B and Figure 1C ). Typically, high-resolution feature maps have low-level features that may compromise their representational capabilities for object recognition. However, in order to avoid using low-level features, some methods abandon high-resolution feature maps, which will miss the opportunity to utilize high-resolution feature maps to facilitate the detection of small objects.

[0027] Figure 1DThe architecture 140 shows an FPN that combines low-resolution, semantically strong features from a ConvNet with a pyramid-shaped feature hierarchy with high-resolution, semantically weak features via a top-down path and lateral connections. Based on FPN, many methods have been proposed to extract multi-scale feature maps and fuse multi-scale feature maps, for example, by adding more bottom-up and / or top-down paths.

[0028] However, these methods decouple the feature learning process into two sequential parts, i.e., first extracting multi-scale feature maps through a classification network (e.g., ConvNet), and then fusing these feature maps through a lightweight module consisting of one or more paths. Such a design may be suboptimal for dense prediction tasks. In addition, since most of the parameters and computational budget have been allocated to the classification network as the backbone, the feature fusion process may be insufficient under a limited computational budget.

[0029] The present disclosure proposes an iterative architecture for obtaining multi-scale feature maps to solve these problems. The main idea of ​​the iterative architecture may be to divide a single trunk into multiple sub-trunks and cascade them together with a top-down feedback path following each sub-trunk, except for the last sub-trunk. In this way, extracting multi-scale feature maps and fusing multi-scale feature maps may be performed in an iterative manner, which can improve the performance of the multi-scale process without introducing additional computational costs. For example, the proposed iterative architecture may maintain a model size and comparable computational complexity as FPN, while improving the performance of dense prediction.

[0030] Figure 2 FIG. 2 is a schematic diagram of an FPN 200 for obtaining a multi-scale feature map. The FPN 200 may have an architecture (eg, Figure 1DThe architecture 140 includes a backbone 210 and a relatively lightweight feature fusion network 220, wherein nodes C2, C3, C4, or C5 represent feature maps with different strides (e.g., C2, C3, C4, or C5 represent feature maps with strides of 4, 8, 16, or 32, respectively), and prefixes C and P are used to distinguish the bottom-up path of the backbone 210 from the top-down path of the relatively lightweight feature fusion network 220, respectively, and edges represent information flow between feature maps. An RGB input image 230 may first be fed into a lightweight stem block 240 to generate an initial feature map with a spatial resolution of H / 4×W / 4, where H and W represent the height and width of the input image 230. The backbone 210 is then used to extract multi-scale features, namely C2, C3, C4, and C5. The backbone 210 may be a heavy classification network composed of network blocks, e.g., more than one hundred layers in total. The feature maps with specific scales in the backbone 210 are transformed from the previous low-level features through several stacked network blocks. Afterwards, a relatively lightweight feature fusion network 220 is used to fuse the generated multi-scale feature maps. Finally, the output feature maps are fed to the specific task head.

[0031] Figure 3 A schematic diagram of an iterative architecture 300 for obtaining a multi-scale feature map according to one or more aspects of the present disclosure is shown, wherein nodes T2, T3, T4, or T5 represent one or more feature maps having the same scale (i.e., spatial size) as the scale of nodes C2, C3, C4, or C5, respectively. The iterative architecture 300 may include a plurality of subnetworks (e.g., subnetwork 310-a and subnetwork 310-b) cascaded together. Each of the plurality of subnetworks (e.g., subnetwork 310-a or subnetwork 310-b) may include a bottom-up subtrunk (e.g., subtrunk 311-a or subtrunk 311-b) for feature extraction, and a top-down feedback path (e.g., feedback path 312-a or feedback path 312-b) for feature fusion. In addition, the iterative architecture 300 may include an additional subtrunk 311-c in the final stage. The included sub-trunks (e.g., sub-trunks 311-a, 311-b, and 311-c) can be split from a single trunk (e.g., trunk 210), and each sub-trunk can have a reduced number of network blocks, for example, a total of tens of layers. The total size of the included sub-trunks (e.g., sub-trunks 311-a, 311-b, and 311-c) can be equivalent to the size of a single trunk (e.g., trunk 210).

[0032] In one aspect of the present disclosure, the iterative architecture 300 may use a stem block 330 to initially process an input image 320. For example, the input image 320 may be a digital image of a certain scale captured by a sensor or camera or generated by a computer (e.g., video, radar, LiDAR, ultrasound, motion or thermal image, etc.). The output feature map of the previous sub-network (e.g., sub-network 310-a) may be used as the input of the next sub-network (e.g., sub-network 310-b). In addition, skip connections may be added between adjacent feature maps in each row, which may involve a copy operation.

[0033] In another aspect of the present disclosure, the iterative architecture 300 can be configured to perform feature extraction and feature fusion in an iterative manner, which may be optimal for dense prediction. In addition, since the total size of the sub-trunks (e.g., sub-trunks 311-a, 311-b, and 311-c) included in the iterative architecture 300 can be equivalent to the size of a single trunk (e.g., trunk 210), there may be no additional computational cost.

[0034] As an example, the single backbone can be a general-purpose CNN (e.g., AlexNet, VGGN et, ResNet, ConvNeXt), a Transformer-based network (e.g., Swin Transformer, Focal Transformer, PoolFormer), or a variant thereof.

[0035] Those skilled in the art will appreciate that other backbones than the above-described networks may be suitable for splitting into sub-backbones for use in the proposed iterative architecture without departing from the present disclosure. Figure 3 The number of subnetworks (i.e., 2) and the number of feature levels (i.e., 4) in are merely illustrative, and the available implementations of the iterative architecture 300 may not be limited to this particular example, but rather other numbers of subnetworks (e.g., 3, 4, 5, 6, 7, 8, 9, etc.) and / or other numbers of feature levels (e.g., 2, 3, 5, 6, etc.) may be possible without departing from the present disclosure.

[0036] Figure 4A , 4B 4C are schematic diagrams showing three alternatives for the architecture of sub-networks that may be implemented in iterative architecture 300 according to one or more aspects of the present disclosure. Figure 4A The architecture 410 may have a symmetric structure in the bottom-up and top-down paths, i.e., its bottom-up and top-down paths share similar structures and have similar computational complexity. For example, the architecture 410 may be a UNet-style subnetwork.

[0037] Figure 4BThe architecture 420 can have an asymmetric structure in the bottom-up and top-down paths. The top-down path can be relatively lightweight than the bottom-up path. For example, between adjacent nodes in the top-down path (e.g., P2, P3, P4, and P5), a 1×1 convolution unit can be first used to reduce the channel dimension of the high-level features, followed by a bilinear interpolation layer to match the spatial size of the feature map. In addition, a 7×7 depthwise separable convolution unit can be used to fuse the element-wise summed features in each level before feeding them to the next level. For example, the architecture 420 can have an FPN-style structure (i.e., with Figure 1D and Figure 2 A similar structure to the top-down feedback path of the FPN shown in FIG. 1 ).

[0038] Figure 4C The architecture 430 may also have an asymmetric structure in the bottom-up and top-down paths. The top-down path may be extremely lightweight to include only 1×1 convolutional units between adjacent nodes (e.g., Q2, Q3, Q4, and Q5) that need to be learned. Such a design may allow information to be transmitted more directly and efficiently (in Figure 4C in the vertical direction) between different feature levels and (in Figure 4C In the horizontal direction of the top-down path, the top-down path flows between different sub-networks, which can promote multi-scale feature fusion. At the same time, due to the extremely lightweight structure of the top-down path, Figure 4C The architecture 430 may consume less computational resources. In addition, the architecture 430 may have an interpolation unit or an upsampling unit between adjacent nodes (e.g., Q2, Q3, Q4, and Q5) in the top-down path to match the spatial size of the feature map.

[0039] In one or more aspects of the present disclosure, architectures 410, 420, and 430 may all have similar structures in a bottom-up path, such as Figure 5 Given a feature from a stem block (e.g., stem block 330), a number (N 1 ) A first feature level 510 consisting of stacked network blocks (e.g., network block 501) can be used to transform these features. Within each subsequent feature level, a 2×2 convolution unit 502 with a stride of 2 can be used to reduce the spatial size of the feature map. In other words, the feature map within each feature level (e.g., Figure 2 , Figure 3 , FIG. 4A to FIG. 4C The numerals after each letter in the letters in ) can have the same spatial size (i.e., the same scale), while the feature maps can have different spatial sizes (i.e., multi-scale) across different feature levels. For example, Figure 5As shown, the spatial size of the feature map gradually decreases across different feature levels (e.g., in this example, H / 4×W / 4 in the first feature level 510 (e.g., corresponding to node C2, T2, P2, or Q2), H / 8×W / 8 in the second feature level 520 (e.g., corresponding to node C3, T3, P3, or Q3), H / 16×W / 16 in the third feature level 530 (e.g., corresponding to node C4, T4, P4, or Q4), and H / 32×W / 32 in the fourth feature level 540 (e.g., corresponding to node C5, T5, P5, or Q5)). For simplicity and generality, the feature map output by the last layer of a node can be passed to the next node as input through an edge by default, and the feature map can be a multi-channel feature map.

[0040] In one or more aspects of the present disclosure, if a feature level in a bottom-up path has more than one input, e.g. Figure 3 At the feature level of the sub-trunk 311-b or 311-c (represented by nodes C2, C3, C4 or C5), a 1×1 convolution unit can be introduced to fuse the input.

[0041] Those skilled in the art will appreciate that other core sizes and stride values ​​than those exemplified above may be suitable for use with sub-trunks of architectures 410 , 420 , and 430 and 300 without departing from the present disclosure.

[0042] Figure 6 FIG. 6 is a schematic diagram showing a design of a network block 600 that can be used in a sub-backbone according to one or more aspects of the present disclosure. Figure 5The network block 501 of FIG. 500 . The design of the network block 600 may include three convolution units 610 , 620 , and 630 , and a jump connection 601 between input features and adders for outputting features of the network block. The design of the network block 600 may also include some other layers or units between adjacent convolution units, such as normalization layers or activation units (e.g., ReLU or GELU), which are omitted for brevity. As an example, the convolution unit 610 may include a 1×1 convolution unit with n channels, the convolution unit 620 may include a 3×3 convolution unit with n channels, and the convolution unit 630 may include a 1×1 convolution unit with 4n channels, and the input features of the network block have a 4n channel dimension, which may be named a residual neural network (ResNet) block. As another example, the convolution unit 610 may include a 7×7 depthwise convolution unit (or a 3×3 convolution unit) with n channels, the convolution unit 620 may include a 1×1 convolution unit with 4n channels, and the convolution unit 630 may include a 1×1 convolution unit with n channels, and the input features of the network block have n channel dimensions, which may be named an iterative network block.

[0043] Figure 7 FIG. 1 is a schematic diagram showing another design of a focused network block 700 that can be employed in a sub-backbone according to one or more aspects of the present disclosure. The focused network block 700 can be employed to implement Figure 5 The network block 501 of FIG. 700 includes four convolution units 710, 720, 730, and 715 and three skip connections 701, 702, and 703. The focused network block 700 may also include some other layers or units between adjacent convolution units, such as normalization layers or activation units (e.g., ReLU or GELU), which are omitted for brevity. The convolution units 710, 720, and 730 may be similar to the convolution units 610, 620, and 630, for example, including a 7×7 depth convolution unit (or a 3×3 convolution unit) with n channels, a 1×1 convolution unit with 4n channels, and a 1×1 convolution unit with n channels, respectively. The additional convolution unit 715 may include a 7×7 depth convolution unit with a dilation rate of 3 and n channels. Two consecutive convolution units 710 and 715 may be used to capture local dense features and global sparse features, respectively. The architecture of the focusing network block 700 can combine both fine-grained local and coarse-grained global interactions to achieve a more efficient convolution mechanism.

[0044] In one aspect of the present disclosure, the design of the focused network block 700 can be applied only to the top feature level of each sub-network (e.g., feature level 540 corresponding to C5) to better balance performance and efficiency. By applying the focused network block to the network block 501 in the feature level 540 corresponding to C5 to expand the receptive field of high-level neurons, the learning of subsequent low-level features can be beneficial, for example, to enhance the semantics of the low-level features.

[0045] Typically, in order to capture a wide range of information, ConvNet introduces large convolution kernels to simulate the self-attention function. However, as the kernel size is larger than 7×7, the performance tends to saturate. This may be because as the network depth increases, the effective receptive field of the neuron saturates. In one or more aspects of the present disclosure, when a sub-trunk (e.g., sub-trunk 311-a, 311-b, or 311-c) is much shallower than a single trunk (e.g., trunk 210), the performance may not saturate as the kernel size becomes larger (i.e., larger than 7×7). For example, when a sub-trunk has a small number of network blocks (e.g., N 3 <4 and / or N 4 <2), the kernel size of the convolution unit 710 and / or the convolution unit 715 may be greater than 7×7, or the dilation rate may be greater than 3.

[0046] Figure 8 An exemplary workflow of a method 800 for dense prediction using a network of an iterative architecture 300 having architecture 410, architecture 420, or architecture 430 according to one or more aspects of the present disclosure is shown. At block 810, an input may be provided to a network consisting of a plurality of subnetworks, wherein the plurality of subnetworks are cascaded together, and each of the plurality of subnetworks may include a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion, and wherein the network may also include lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the subnetworks and in the top-down feedback path, and lateral skip connections across adjacent subnetworks. For example, the bottom-up sub-trunk and the top-down feedback path may have the architecture of architecture 410, architecture 420, or architecture 430 combined with one or more network blocks 600 and / or focused network blocks 700. At block 820, the output feature map of the network may be provided to a head for a dense prediction task. For example, the input may include a digital image of one scale, and the output feature map may be of a different scale, and the tasks may include image classification, instance segmentation, object detection, or semantic segmentation, etc.

[0047] Fig. 91 shows an example of a hardware implementation of an apparatus 900 according to one or more aspects of the present disclosure. The apparatus 900 for dense prediction may include a memory 910 and at least one processor 920. The processor 920 may be coupled to the memory 910 and configured to perform the method 800 using the architecture 300, wherein the architecture 300 is similar to the above referenced Figure 8 , Figure 3 , FIG. 4A to FIG. 4C and Figure 6 and / or Figure 7 The described architecture 410, 420 or 430 and the network block 600 and / or the focused network block 700 are combined. The processor 920 can be a general-purpose processor, or can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. The memory 910 can store input data, output data, data generated by the processor 920, and / or instructions executed by the processor 920.

[0048] The various operations, models, and networks described in conjunction with the disclosure herein may be implemented in hardware, software executed by a processor, firmware, a computer, or any combination thereof. According to one or more aspects of the present disclosure, a computer program product for dense prediction may include processor-executable computer code for performing method 800 using architecture 300, wherein architecture 300 is similar to the above-referenced Figure 8 , Figure 3 , FIG. 4A to FIG. 4C and Figure 6 and / or Figure 7 The architecture 410, 420, or 430 described above, and the network block 600 and / or the focused network block 700 are combined. According to another embodiment of the present disclosure, a computer readable medium may store a computer code for dense prediction, which, when executed by a processor, may cause the processor to perform the method 800 using the architecture 300, wherein the architecture 300 is similar to the above referenced Figure 8 , Figure 3 , FIG. 4A to FIG. 4C and Figure 6 and / or Figure 7 The described architecture 410, 420 or 430 and the network block 600 and / or the focus network block 700 combination. Computer-readable media include both non-transitory computer storage media and communication media including any medium that facilitates the transfer of a computer program from one place to another. Any connection can be properly referred to as a computer-readable medium. Other embodiments and implementations are within the scope of the present disclosure.

[0049] In an embodiment of the present disclosure, the iterative architecture may also include lateral jump connections across two adjacent subnetworks (e.g., subnetwork 310-a and subnetwork 310-b). For example, there may be lateral jump connections between feature maps at corresponding feature levels of a subnetwork (e.g., in both the bottom-up subtrunk 311-a and the top-down feedback path 312-a of subnetwork 310-a) and feature maps at corresponding feature levels in the bottom-up subtrunk (e.g., subtrunk 311-b) of an adjacent subnetwork (e.g., subnetwork 310-b). In other words, a node at a certain feature level of a subtrunk may use as input an output feature map from its previous subnetwork at the same feature level, and the output feature map may come from both the subtrunk and the top-down feedback path of the previous subnetwork, i.e., more than one input feature map. In this case, a 1×1 convolution unit may be introduced to fuse the input feature maps.

[0050] In an embodiment of the present disclosure, the cascaded sub-networks may have a size equivalent to that of a conventional classification model or network, and the cascaded architecture may enable the network to be configured to perform multi-scale feature extraction and multi-scale feature fusion in an iterative manner, thereby coupling feature extraction and feature fusion in multi-scale feature learning to achieve optimal performance for dense prediction.

[0051] In an embodiment of the present disclosure, each of the multiple subnetworks may have a symmetric structure, a feature pyramid network (FPN) structure, or an extremely lightweight feedback path structure. The extremely lightweight feedback path structure may include a 1 by 1 convolution unit, which is the only learnable unit in the extremely lightweight feedback path structure.

[0052] In an embodiment of the present disclosure, the bottom-up sub-backbone may include a residual neural network (ResNet) block or an iterative network block and / or a focused network block. For example, the bottom-up sub-backbone may include several feature levels, and each feature level may be composed of one or more network blocks. For example, a network block may be a ResNet block, an iterative network block, or a focused network block. For another example, only the top feature level (e.g., Figure 5 The fourth feature level 540) uses a focused network block.

[0053] In an embodiment of the present disclosure, the input of a network composed of multiple sub-networks may include a digital image with a scale captured by a sensor or camera or generated by a computer, and the output feature map of the network may be of different scales and may have strong semantics at all scales.

[0054] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to make or use the various embodiments. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the various embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the appended claims and the principles and novel features disclosed herein.

Claims

1. A computer-implemented network for dense prediction, comprising: A plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-backbone for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion; lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-backbones and the top-down feedback paths in the sub-networks; and Lateral skip connections across adjacent sub-networks.

2. The computer-implemented network of claim 1 , wherein: The lateral hop connection across the adjacent sub-networks includes: Lateral skip connections between feature maps at corresponding feature levels in a sub-network and in the bottom-up sub-backbones in adjacent sub-networks.

3. The computer-implemented network of claim 2, wherein: The network is configured to perform the multi-scale feature extraction and the multi-scale feature fusion in an iterative manner.

4. The computer-implemented network of claim 1 , wherein: Each of the multiple sub-networks has a symmetrical structure, a feature pyramid network (FPN) structure, or an extremely lightweight feedback path structure.

5. The computer-implemented network of claim 4, wherein: The extremely lightweight feedback path structure includes a 1x1 convolution unit, which is the only learnable unit in the extremely lightweight feedback path structure.

6. The computer-implemented network of claim 1 , wherein: The bottom-up sub-backbone includes a residual neural network (ResNet) block or an iterative network block.

7. The computer-implemented network of claim 6, wherein: The bottom-up sub-backbone also includes one or more focused blocks in the top feature level.

8. A computer-implemented method for performing dense prediction using a network, comprising: Providing input to the network consisting of a plurality of sub-networks, wherein the plurality of sub-networks are cascaded together, and each of the plurality of sub-networks comprises a bottom-up sub-trunk for multi-scale feature extraction and a top-down feedback path for multi-scale feature fusion, and wherein the network further comprises lateral skip connections between feature maps at corresponding feature levels in the bottom-up sub-trunks in the sub-networks and in the top-down feedback path, and lateral skip connections across adjacent sub-networks; and providing the output feature map of the network to a head for the task; and The input includes a digital image with a scale, the output feature map is of a different scale, and the task includes image classification, instance segmentation, object detection or semantic segmentation.

9. An apparatus for dense prediction, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the computer-implemented method of claim 8.

10. A computer program product for dense prediction, comprising: Processor executable computer code for performing the computer implemented method of claim 8.

11. A computer readable medium storing computer code for dense prediction, which, when executed by a processor, causes the processor to perform the computer-implemented method of claim 8.

Images