Method and apparatus for performing dense prediction using converter block

Abstract

The methods, apparatus, systems, and articles of manufacture disclosed herein perform dense prediction of input images using transducers at a heavy component level and at an encoder level of an image processing system. The disclosed apparatus includes an encoder having an embedder that converts an input image into a plurality of indicia representing features extracted from the input image. The indicia are embedded with a learnable location embedding. The encoder also includes one or more transducers configured in the sequence of stages to correlate the indicia to each other. The apparatus further includes a decoder including one or more of recombiners that assemble the markers into feature representations, one or more of fusion blocks that combine the feature representations to generate a final feature representation, and an output header that generates dense predictions based on the final feature representation and based on output tasks.

Description

Technical Field

[0001] The present disclosure relates generally to dense prediction in computer vision, and more particularly to methods and apparatus for performing dense prediction using transformer blocks for improving dense prediction in computer vision. Background Art

[0002] Computer vision refers to computerized techniques that allow a computer system to gain a high-level understanding of a digital image or video. The high-level understanding can then be used to inform whether an action will be taken / recommended. One aspect of computer vision called dense prediction is often performed using fully convolutional deep networks.

[0003] Dense prediction refers to the task of predicting a label (or multiple labels) that will be associated with each pixel in an image. For example, a semantic segmentation label identifies the class corresponding to the content represented in the image (e.g., buildings, sky, cars, people, etc.). In addition, a depth estimation label identifies the distance between the processed image and the observer of the image. Further, a keypoint detection label indicates the likelihood that a pixel is a keypoint. Keypoint pixels occupy points in the image that are associated with content of interest in the image (e.g., salient content). An optical flow label identifies the apparent motion of the labeled pixel in the current frame relative to another frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a block diagram of an image processing system that implements a dense predictor according to the teachings disclosed herein.

[0005] Figure 2A yes Figure 1 Block diagram of the transformer encoder / embedder of a dense predictor.

[0006] Figure 2B yes Figure 1 A block diagram of one transformer stage among a set of transformer stages of a dense predictor.

[0007] Figure 3 yes Figure 1 Block diagram of the reorganizer for the dense predictor.

[0008] Figure 4A yes Figure 1 Block diagram of the fusion module of the dense predictor.

[0009] Figure 4B It is a block diagram of the residual convolutional network.

[0010] Figure 5A is the input image and shows how it compares to the MiDaS fully convolutional architecture. Figure 1A collection of dense predictions of an image corresponding to monocular depth estimation.

[0011] Figure 5B It is shown that the use Figure 1 Table showing numerical comparisons of monocular depth estimates generated by a dense predictor of and using prior art in monocular depth estimation.

[0012] Fig. 6A Is the input image and display use Figure 1 A collection of dense predictors of and images of corresponding semantic segmentation estimates generated using other prediction techniques.

[0013] Figure 6B It is shown that the use Figure 1 Table showing numerical comparisons of semantic segmentation generated by the dense predictor and using other prediction techniques.

[0014] Figure 7 is a flow chart representing example machine readable instructions executable by an example processor circuit module to implement Figure 1 Dense predictor of .

[0015] Figure 8 is a flow chart representing example machine readable instructions executable by an example processor circuit module to implement Figure 1 Embedder for dense predictors of .

[0016] Fig. 9 is a flow chart representing example machine readable instructions executable by an example processor circuit module to implement Figure 1 A collection of transformer stages of a dense predictor.

[0017] Fig.10 is a flow chart representing example machine readable instructions executable by an example processor circuit module to implement Figure 1 A reorganizer of dense predictors.

[0018] Fig.11 is a flow chart representing example machine readable instructions executable by an example processor circuit module to implement Figure 1 The fusion block of the dense predictor.

[0019] Fig.12 is a flow chart representing example machine readable instructions executable by an example processor circuit module to implement Figure 1 The residual convolutional unit of the fusion block of the dense predictor.

[0020] Fig.13is a block diagram of an example processing platform including a processor circuit module configured to perform Figure 7-12 Example machine readable instructions to implement Figure 1 A dense predictor for and its individual components.

[0021] Fig.14 yes Fig.13 A block diagram of an example implementation of a processor circuit module.

[0022] Fig.15 yes Fig.13 A block diagram of another example implementation of a processor circuit module.

[0023] The drawings are not drawn to scale. In general, the same reference numerals will be used throughout the text to represent the same or similar parts in (one or more) drawings and the accompanying written description. Unless otherwise explicitly stated, descriptors such as "first", "second", "third", etc. are used herein, rather than being attributed to or otherwise indicating any meaning of priority, physical order, arrangement in a list, and / or sorting in any manner, but are only used as labels and / or arbitrary names to distinguish elements, so as to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" can be used to represent an element in the detailed description, and different descriptors such as "second" or "third" can be used in the claims to represent the same element. In such examples, it should be understood that such descriptors are only used to clearly identify those elements that may originally share the same name, for example. As used herein, the phrase "communicating with..." (including its variations) includes direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but also includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-time events. As used herein, a "processor circuit module" is defined to include: (i) one or more dedicated circuits configured to perform (one or more) specific operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors); and / or (ii) one or more general-purpose semiconductor-based circuits programmed with instructions to perform specific operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuit modules include programmed microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processor units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs or microcontrollers, and integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU may be implemented by a heterogeneous computing system that includes multiple types of processor circuit modules (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc. and / or combinations thereof) and (one or more) application programming interfaces ((one or more) APIs), which may assign (one or more) computing tasks to whichever (of the multiple types of processing circuit modules is most suitable for performing the (one or more) computing tasks. DETAILED DESCRIPTION

[0024] Dense prediction (the task of predicting / assigning a label to every pixel in an image) is an aspect of computer vision techniques that is typically performed using some combination of convolution and subsampling. These basic elements are used to learn multi-scale representations of an image that can exploit the spatial context of the image. Example classes of such dense prediction techniques include techniques that 1) progressively upsample representations that have been pooled at different levels, 2) use dilated convolutions to increase the receptive field, and 3) perform parallel multi-scale feature aggregation at multiple resolutions to increase the receptive field.

[0025] Unfortunately, the encoder of the fully convolutional technique progressively downsamples the input image in order to extract features at multiple scales. This act of downsampling enables an increase in the receptive field, the grouping of low-level features into abstract high-level features, and at the same time ensures that the memory and computational requirements of the network remain tractable. However, downsampling has significant disadvantages that are particularly prominent in dense prediction tasks. In particular, downsampling in dense prediction causes a loss of feature resolution and granularity in the deeper levels of the dense prediction model, and these aspects are difficult to recover in the downstream decoder. Previous solutions to mitigate such losses in granularity focus on the decoder.

[0026] In contrast, the methods, devices, systems, and articles of manufacture disclosed herein are based on an encoder-decoder design / architecture that utilizes a visual transformer (ViT) as the basic computational building block of the encoder. The bag-of-words representations provided by the visual transformer are reorganized into image-like feature representations at various resolutions, and the feature representations are progressively combined into a final dense prediction using a convolutional decoder.

[0027] Further, the methods, devices, systems and articles disclosed herein include an architecture that abandons explicit downsampling operations after the initial image embedding has been calculated and also maintains a representation with constant dimensionality at all image processing stages. Further, the architecture has a global receptive field (context) at each image processing stage.

[0028] These properties are especially advantageous for dense prediction tasks, since such properties naturally give rise to fine-grained and globally coherent predictions. The increased accuracy of dense predictions allows building higher quality products in various technological fields (depth sensing, camera ISP, autonomous driving, robotics).

[0029] Figure 1 1 is a block diagram of an image processing system 100 according to the teachings disclosed herein. In some examples, the example image processing system 100 includes an example image embedder 102, an example set of transformer stages 104, an example set of recombiners 106, an example set of fusion modules 108, and a headend 110 at the output of the system 100.

[0030] In some examples, the input image 112 is transformed into tokens 114 by the image embedder 102. The input image 112 is transformed into tokens 114 by (1) extracting non-overlapping tiles and then performing a linear projection of the flattened representation of the tokens, or (2) applying a feature extractor implemented using a residual network (e.g., ResNet-50). In some examples, position embedding and special tokens 114ST (also known as special tile independent readout tokens) are used to enhance the embedding performed by the image embedder 102 (as further described below).

[0031] In some examples, the labels 114 then pass through the example plurality of transformer stages 104. Subsequently, the example reassembler 106 reassembles the labels 114 provided by different transformer stages of the transformer stages into image-like representations at multiple resolutions. In some examples, the reassembler 106 reassembles the labels into feature maps having a spatial resolution of 1 / s, and progressively fuses and upsamples the image-like representations (e.g., feature maps) to generate fine-grained predictions (e.g., dense predictions). In some examples, the example set of fusion modules 108 fuses the feature maps using residual convolution units, and then upsamples the feature maps (as further described below).

[0032] Figure 1 The example image processing system 100 of the embodiment of the present invention operates as an encoder / decoder performing dense prediction. In some such examples, the example embedder 102 and the example set 104 of transformer stages operate together as an encoder. In addition, the example set 106 of transformers, the example set 108 of fusion blocks, and the head end operate as decoders.

[0033] Embedder Figure 2A yes Figure 1 200A is a block diagram of an example embedder 102. Figure 1 As described, an example set of transformer encoders (also called visual transformers ("ViT")) operate on a bag-of-words representation of an image, much like a natural language processor operates. Image patches, or alternatively deep features extracted from an image, that are individually embedded in a feature space act as "words". The term "feature" as used herein can also refer to deep features. In machine learning, a deep feature is a consistent response of a node or layer within a hierarchical model to an input that gives a response that is related to the final output of the model. A feature is considered "deeper" than another feature depending on how early the response is activated in a decision tree or other framework. The embedded words as used herein are called tokens.

[0034] In some examples, the example embedder 102 processes the size p of the image by 2All non-overlapping square patches of pixels are embedded in the image, and a patch embedding process is performed on the image. In some examples, the embedder 102 includes an example image segmenter 202, an example pixel flattener 204, an example linear projector 206, an example first cascader 208, and an example readout tag generator 210. The image segmenter 202 divides the image into non-overlapping tiles, and the pixel flattener 204 flattens the tiles into vectors. Further, the linear projector 206 embeds the patches individually. Alternatively, in some examples, a more sample-efficient variant of ViT extracts an embedding by applying a residual network (e.g., ResNet50) to the image to obtain a feature map, and then using the pixel features of the resulting feature map as tags.

[0035] Since the transformer is a set-to-set function, the embedder (which operates as a transformer) does not inherently preserve the spatial position information of the individual tags. Therefore, the example first cascader 208 cascades the embedding with the learned position embedding to add the spatial position to the image representation. The example readout tag generator 210 generates special tags 114ST (also called readout tags). The readout tags are not based on the input image (e.g., are tile independent) and serve as the final global image representation to be used for classification. Thus, for example, applying the embedder to an image of size H×W pixels results in the generation of t 0 = {t0 0, ..., t0 Np}, t0 n∈RD marking, where Np = HW p 2 , t 0 is the readout tag 114ST, the variable D is the feature dimension of each tag, and the variable R represents a set of real numbers.

[0036] In some examples, for illustrative purposes, the patch size "p" is specified as a value of 16. The embedding process of ViT-Base and ViT-Large projects the flattened patches to dimensions D=768 and D=1024, respectively (using pixel flattener 204) (these values ​​of D are just examples). In general, the network will learn better with larger values ​​of D, but will also require more computational resources. Since both feature dimensions are larger than the number of pixels in the input patch, the embedding process is able to learn to retain such information when it is beneficial to the task. Using the ViT-Base and ViT-Large projects, features from the input patch "p" can be parsed with pixel-level accuracy. Similarly, the ViT-Hybrid architecture extracts features at 1 / 16 input resolution, which is up to twice the lowest resolution features typically used with convolutional network backbones. Therefore, the recovery of features at the decoder is improved (where an example set of recombiners and fusion blocks are used to implement the decoder).

[0037] Applying Tags to Transformers In some examples, the tag is provided to a set of transformer stages 106. In some examples, the set of transformer stages includes L stages (e.g., L transformers). Using the set of L transformer stages 104 (see Figure 1 ) transforms the markup into a new representation t l , where l represents the output of the lth converter stage. In some examples, three different variants of converter stages can be used.

[0038] Figure 2B 1 is a block diagram of one of the example transformers 200B included in the example set of transformer stages. All transformers 200B have the same components that perform the same operations. In some examples, transformer 200B includes an example first normalizer 212, an example multi-head attention block 214, an example first adder 216, an example second normalizer 218, an example multi-layer perceptron 220, and an example second adder 222. The set 104 of transformer stages (see Figure 1 ) is normalized by the first normalizer 212 of the first transformer 200B of the example embedder 102 (see Figure 1 ) is a set of tags generated by the multi-headed attention block 214. The multi-headed attention block 214 then performs multi-headed attention on the normalized tags. A first adder 216 adds the output of the multi-headed attention block 214 to the embedded patch / tag 114 generated by the embedder 102. The added information is then fed to a second normalizer 218 and also to a second adder 222. The input data to the second normalizer is normalized and then provided to a multi-layer perceptron 220. The output of the multi-layer perceptron 220 is provided to a second adder 222, where the output is added to the output of the first adder 216. The output of the transformer 220B is provided to the next transformer 220B of the set 104 of transformer stages and is also provided to an example reorganizer 106 (see Figure 1 )'s corresponding recombiner.

[0039] The example first variant (referred to as the ViT base variant) uses a patch-based embedding process performed by the embedder 102 and features 12 transformer levels (e.g., L=12). The example second variant (referred to as the ViT-Large variant) also uses a patch-based embedding process performed by the embedder 102, with 24 transformer levels (e.g., L-24) and a wider feature size D. The variable "D" as used herein represents the dimension of the flattened label after being projected by the linear projector 206. The example third variant (referred to as the ViT-Hybrid variant) employs a residual network (e.g., ResNet50) to perform the image embedding process, followed by 12 transformer levels.

[0040] Thus, the operation of each transformer of the example transformer 200B relates each label to all other labels. In addition, each transformer stage maintains the same number of labels across all computations. Since the labels have a one-to-one correspondence with the image patch, the ViT encoder maintains the labels generated by the embedder 102 (see ) across all transformer stages (e.g., across all stages of the set of transformers 104). Figure 1 ) performs. Additionally, the multi-head self-attention operation is inherently global, as every token pays attention to and influences every other token. The global receptive field exists at every processing stage after the initial embedding.

[0041] Figure 3 yes Figure 1 A block diagram 300 of one of the example collections 106 of assemblers of Figure 1 As described, the set of assemblers, the example fusion block 108 and the example headend 110 operate as a decoder. In operation, the assembler 106 assembles the set of markers 114 into image-like feature representations at various resolutions, and the set of fusion blocks progressively fuses the feature representations into a final dense prediction.

[0042] In some examples, each of the reorganizers included in the set of reorganizers 106 includes an example tag reader 302, an example second cascade 304, an example storage 306, an example projector 308, and an example resampler 310. As an overview, the components of the reorganizer operate to perform a three-stage operation to recover an image-like representation from the output tags of any layer of the set of transformer stages 104.

[0043] In some examples, the three-stage operation performed by each of the reassemblers 106 can be represented as an evaluation of a variable "t" over an interval spanning from "D^" to "s", where the evaluation is calculated by multiplying the results of the "resampling operation", the "cascade operation", and the "read operation" for each value of "(t)". ​​In some examples, the variable "s" represents the output size ratio of the restored representation relative to the input image, and the variable "D^" represents the number of channels. For example, a high-dimensional image can have spatial dimensions represented by M×N and D channels (unlike a normal image having three channels (e.g., red, green, and blue)).

[0044] In some examples, the decoder (e.g., the example set 106 of assemblers, the example set 108 of fusion blocks, and the example headend 110) converts “N p +1" markers are mapped to N p is a set of tags that are amenable to spatial concatenation into an image-like representation. The mapping is denoted as “Read:RN p+1×D →R Np×D ”.

[0045] The mapping operation takes one of three actions (also called variants) depending on the goal of the image processing system. For example, the readout marker 114ST does not serve an explicit purpose for the task of dense prediction, but can be useful in some examples for capturing and distributing global information. Three different variants of the mapping are evaluated, including: (1) Readignore(t) = {t1, ..., tNp}, which ignores the readout marker 114ST; (2) Readadd(t) = {t1+t0, ..., tNp+t0}, which transfers information from the readout marker 114ST to all other markers by adding the representations; and (3) Readproj(t) = {mlp(cat(t1, t0)), ..., mlp(cat(t Np ,t0))}, which passes information to other tokens by concatenating the readout to all other tokens before projecting the representation to the original feature dimension D using a linear layer followed by a Gaussian error linear unit non-linearity.

[0046] In some examples, during the operation of the example tag reader 302 (eg, "Read: p+1×D →R Np×D ”), by placing each marker according to the position of the initial patch in the image, the generated N p The tokens are reshaped into an image-like representation. In some examples, the tokens are reshaped into an image-like representation. In some such examples, the second cascader 304 applies a spatial cascade operation that produces a feature map of size H / p×W / p with D channels. The operation performed by the example cascader 304 can be represented as “R Np×D →R H / p×W / p×D ” and generates an image-like representation that can be temporarily stored in the storage device 306. The generated image-like representation (also called feature representation) is passed to the example resampler 310, which scales the representation to a size H / s×W / s with D^ features per pixel. The feature representation is a more abstract representation of the input image. The resampling operation performed by the resampler 310 can be represented as “R H / p×W / p×D →R H / s×W / s×D ^".

[0047] Figure 4A yes Figure 1Block diagram of one of the example fusion blocks 108 of the set 108 of fusion blocks. In some examples, the fusion block 108 includes an example first residual convolutional unit 402, an example adder 404, an example second residual convolutional unit 406, an example resampler 408, and an example projector 410. In some examples, the output of the assembler 106 is provided to the input fusion block 108, where it is operated on by the first residual convolutional unit 402. The adder 404 adds the output of the first convolutional unit 420 to the output of an earlier fusion 108 (not shown). The second residual convolutional unit 406 operates on the output of the adder 404. The output of the second residual convolutional unit 406 provides the result to the resampler 0.5, and the resampled data is then projected by the projector 410.

[0048] Figure 4B Block diagram of example first and second residual convolutional units 402, 406. In some, each of the first and second residual convolutional units 402, 406 includes an example first rectified linear unit, an example 3×3 convolution 414, an example first batch normalizer 416, an example second rectified linear unit 418, an example strided 3×3 transposed convolution 420, and an example second batch normalizer 422 and an example adder 424. In some examples, the input provided to the residual convolutional units 402, 406 is provided to both the example adder 424 and the first rectified linear unit 412. The output of the first rectified linear unit 412 is provided to the example first convolutional unit (3×3) 414, and then normalized by the first batch normalizer 416. Thereafter, the operations of blocks 412 - 416 are repeated by blocks 418, 420, and 422 respectively. The output of the second normalizer 422 is provided to the adder 424 for addition to the input data set.

[0049] The resampler network operates as follows: First, it uses an example projector 308 to project the input representation to D^ using a 1×1 convolution, followed by a (strided) 3×3 convolution when s ≥ p or a strided 3×3 transposed convolution when s < p, thereby implementing both spatial downsampling and upsampling operations respectively.

[0050] Irrespective of the transformer backbone used in the image processing system, features are recombined in a set of recombiners 106 at a set of resolutions equal to the number of recombiners contained in the set. In some examples, the set includes four recombiners to produce four different resolutions. In some examples, features are assembled from deeper layers of the transformer at a lower resolution, while features from earlier layers are assembled at a higher resolution.

[0051] Thus, in some examples, when ViT-Large is used, layers l = {6, 12, 18, 24} are used to reorganize the labels, and when ViT-Base is used, layers l = {3, 6, 9, 12} are used to reorganize the labels. When ViT-Hybrid is employed, features from the first and second residual network blocks (e.g., from the embedding network) and levels l = {9, 12} are used. Some example architectures use projection as a readout operation, and thereby produce feature maps with D^ = 256 dimensions. These architectures are referred to as DPT-Base, DPT-Large, and DPT-Hybrid, respectively.

[0052] In some examples, a RefineNet-based feature fusion block is used to combine the extracted feature maps from consecutive stages, and the feature representations (maps) are progressively upsampled by a factor of two in each fusion stage. The final representation size has half the resolution of the input image. Task-specific output headers (e.g. Figure 1 The head end 110 of the feature image is attached to generate the final prediction. In some examples, the head end 110 generates the final result (e.g., semantics, depth, etc.) from the abstract representation encoded in the feature image.

[0053] Manipulates varying image sizes. Similar to fully convolutional networks, DPT can manipulate varying image sizes. As long as the image size is divisible by p, the embedding process can be applied and will produce a varying number of image tokens N p . As a set-to-set architecture, the transformer encoder can generally handle a varying number of tokens. However, the position embedding has a dependency on image size because it encodes the position of the tile in the input image. The position embedding is linearly interpolated to the appropriate size, which can be performed on the fly. After the embedding process and the transformer stage, both the reorganization and fusion modules can generally handle a varying number of tokens, given that the input image is aligned with the span of the convolutional decoder (32 pixels). That is, the side / side length in pixels of the given input must be divisible by 32.

[0054] Figure 5A is shown by using Figure 1 The encoder-decoder image processing system 100 outperforms other techniques for performing monocular depth estimation obtained with an improved set of images. Figure 5B is a table summarizing the numerical comparison with the state-of-the-art in monocular depth estimation. Significant increase in accuracy of the proposed architectures (DPT-Large and DPT-Hybrid) over the previous state-of-the-art (MiDaS) fully convolutional architecture is observed.

[0055] Fig. 6A is shown by using Figure 1The encoder-decoder image processing system 100 outperforms other techniques for performing semantic segmentation on a collection of improved images. Figure 6B is a table showing comparison with the prior art in semantic segmentation. Figure 1 When the proposed architecture (eg, system 100) is used, accuracy improvement can again be seen.

[0056] In some examples, the image processing system includes a component for converting an input image into a plurality of tags representing features extracted from the input image. For example, the component for converting may be Figure 1 2. In some examples, the embedder 102 may be implemented by machine executable instructions, such as by executing a processor circuit module. Figure 8 The processor circuit module may be implemented by at least blocks 802-808 of Fig.13 An example processor circuit module 1300, Fig.14 The example processor circuit module 1400 and / or Fig.15 The embedder 102 is implemented by an example field programmable gate array (FPGA) circuit module 1500. In other examples, the embedder 102 is implemented by other hardware logic circuit modules, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the embedder 102 may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, application specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware, but other structures are equally appropriate.

[0057] In some examples, the means for converting includes means for dividing the input image. For example, the means for dividing may be implemented by the image segmenter 202 of FIG. 2. In some examples, the means for dividing 202 may be implemented by machine executable instructions, such as by executing a processor circuit module. Figure 8 The instructions implemented by at least block 802 of the present invention may be implemented by the processor circuit module Fig.13 An example processor circuit module 13, Fig.14 The example processor circuit module 1400 and / or Fig.15The example field programmable gate array (FPGA) circuit module 1500 is implemented. In other examples, the component 202 for partitioning is implemented by other hardware logic circuit modules, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the component 202 for partitioning can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuits, FPGAs, application specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware, but other structures are equally appropriate.

[0058] In some examples, the means for converting includes means for cascading means for learning positions. For example, the means for cascading means for learning positions may be implemented by the example first cascader 208 of FIG. 2. In some examples, the means for cascading may be implemented by machine executable instructions, such as by executing a processor circuit module. Figure 8 The instructions implemented by at least block 802 of the present invention may be implemented by the processor circuit module Fig.13 An example processor circuit module 13, Fig.14 The example processor circuit module 1400 and / or Fig.15 The example field programmable gate array (FPGA) circuit module 1500 is implemented. In other examples, the components for cascading are implemented by other hardware logic circuit modules, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the component for cascading 208 can be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, application specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware, but other structures are equally appropriate.

[0059] In some examples, the image processing system includes a component for reassembling, which can include a component for reading a tag, a component for spatially concatenating outputs of a tag reader to generate a feature representation (also referred to as a feature map), and a component for scaling the feature representation. For example, the component for reassembling, which can include a component for reading a tag, a component for spatially concatenating outputs of a tag reader to generate a feature representation (also referred to as a feature map), and a component for scaling the feature representation, can be respectively composed of Figure 1The recombiner 106, the tag reader 302, the second cascade 304, and the resampler 310 are implemented. In some examples, the means for recombining, which can include means for reading tags, means for spatially cascading the outputs of the tag readers to generate a feature representation (also called a feature map), and means for scaling the feature representation, can be implemented by machine executable instructions, such as by being executed by a processor circuit module. Fig.10 The processor circuit module may be implemented by at least blocks 1002-1006 of Fig.13 An example processor circuit module 1300, Fig.14 The example processor circuit module 1400 and / or Fig.15 The example field programmable gate array (FPGA) circuit module 1500 is implemented. In other examples, the components for reading the tag, the components for spatially cascading the outputs of the tag readers to generate the feature representation (also known as the feature map), and the components for reorganizing the components for scaling the feature representation are implemented by other hardware logic circuit modules, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the components for reading the tag, the components for spatially cascading the outputs of the tag readers to generate the feature representation (also known as the feature map), and the components for reorganizing the components for scaling the feature representation may be implemented by at least one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, application specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware, but other structures are equally appropriate.

[0060] Although Figure 2A , Figure 2B , Figure 3 4 shows an example way to implement an image processing system 100 that performs dense prediction, but Figure 2A , Figure 2B , Figure 3One or more of the elements, processes and / or devices shown in and to FIG. 4 may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Further, the example embedder 102, the example image segmenter 202, the example pixel flattener 204, the example linear projector 206, the example first cascade 208, the example readout tag generator 210, the example transformer 200B, the example first normalizer 212, the example multi-head attention block 214, the example first adder 216, the example second normalizer 218, the example multi-layer perceptron 220, the example second adder 222, the example reorganizer 106, the example tag reader 302, the example second cascade 304, the example storage device 306, the example projector 308 and the example resampler The example fusion block 108, the example first residual convolution unit 402, the example adder 404, the example second residual convolution unit 406, the example resampler 408, the example projector 410, the example first and second residual convolution units 402 and 406, the example first rectified linear unit 412, the example 3×3 convolution 414, the example first batch normalizer 416, the example second rectified linear unit 418, the example stride 3×3 transposed convolution 420, the example second batch normalizer 422 and the example adder 424 can be implemented solely by hardware or by hardware in combination with software and / or firmware. Thus, for example, the example embedder 102, the example image segmenter 202, the example pixel flattener 204, the example linear projector 206, the example first cascade 208, the example readout tag generator 210, the example transformer 200B, the example first normalizer 212, the example multi-head attention block 214, the example first adder 216, the example second normalizer 218, the example multi-layer perceptron 220, the example second adder 222, the example reorganizer 106, the example tag reader 302, the example second cascade 304, the example storage 306, the example projector 308, and the example readout tag generator 210. 08 and example resampler 310, example fusion block 108, example first residual convolution unit 402, example adder 404, example second residual convolution unit 406, example resampler 408, example projector 410, example first and second residual convolution units 402 and 406, example first rectified linear unit 412, example 3×3 convolution 414, example first batch normalizer 416, example second rectified linear unit 418, example strided 3×3 transposed convolution 420, example second batch normalizer 422 and example adder 424 and / or more generally Figure 1Any of the example image processing systems of may be implemented by processor circuit modules, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units ((one or more) GPUs), (one or more) digital signal processors ((one or more) DSPs), (one or more) application specific integrated circuits ((one or more) ASICs), (one or more) programmable logic devices ((one or more) PLDs) and / or (one or more) field programmable logic devices ((one or more) FPLDs) such as field programmable gate arrays. Further, Figure 1 The example image processing system 100 may include one or more elements, processes and / or devices as Figure 2A , Figure 2B , Figure 3 4 and / or may include more than one or all of the elements, processes and / or devices shown.

[0061] Figure 7-12 The figure shows the method used to implement Figure 1 The image processing system 100 and / or Figure 2A , Figure 2B , Figure 3 , Figure 4A and Figure 4B Example hardware logic circuit modules, machine readable instructions, hardware implemented state machines and / or flowcharts of any combination thereof for components of the image processing system. The machine readable instructions may be instructions for a processor circuit module (such as the following in combination with Fig.13 The processor circuit module 1312 shown in the example processor platform 1300 and / or the following combination Fig.14 and / or Fig.15The example processor circuit module 1312) described herein may be one or more executable programs or (one or more) parts of executable programs executed by an executable program. The program may be embodied in software, and the software is stored on one or more non-transitory computer-readable storage media such as a CD, a floppy disk, a hard disk drive (HDD), a DVD, a Blu-ray disk, a volatile memory (e.g., any type of random access memory (RAM), etc.) or a non-volatile memory (e.g., FLASH memory, HDD, etc.) associated with a processor circuit module located in one or more hardware devices, but the entire program and / or part thereof may alternatively be executed by one or more hardware devices other than the processor circuit module and / or embodied in firmware or dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices, and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, a client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that may facilitate communication between a server and an endpoint client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. Further, although reference is made to Figure 7-12 , but many other methods of implementing the example image processing system 100 may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any one or all of the blocks may be implemented by one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware. The processor circuit modules may be distributed in different network locations and / or local to one or more hardware devices, such as single-core processors (e.g., single-core central processing units (CPUs)), multi-core processors in a single machine (e.g., multi-core CPUs), etc.), multiple processors distributed across multiple servers of a server rack, multiple processors distributed across one or more server racks, CPUs and / or FPGAs in the same package (e.g., the same integrated circuit (IC) package or in two or more independent housings, etc.).

[0062] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, an encapsulated format, etc. The machine-readable instructions as described herein may be stored as data or a data structure (e.g., as a portion of an instruction, a code, a representation of a code, etc.) that may be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations of a network or a collection of networks (e.g., in the cloud, in an edge device, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., so that they are directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, machine-readable instructions may be stored in multiple portions that are individually compressed, encrypted, and / or stored on separate computing devices, wherein the portions, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that may collectively form a program such as the program described herein.

[0063] In another example, the machine-readable instructions may be stored in a state in which they are readable by a processor circuit module, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., settings stored, data entered, network addresses recorded, etc.) before the machine-readable instructions and / or (one or more) corresponding programs are fully or partially executed. Therefore, a machine-readable medium as used herein may include machine-readable instructions and / or (one or more) programs, regardless of the specific format or state of the machine-readable instructions and / or (one or more) programs when stored or otherwise at rest or in transition.

[0064] The machine-readable instructions described herein can be expressed in any past, present or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions may be expressed using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0065] As described above, executable instructions (e.g., computer and / or machine readable instructions) may be used to implement Figure 7-12The example operations of the present invention are stored on one or more non-transitory computer and / or machine readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks, wherein information is stored for any length of time (e.g., for an extended period of time, permanently, for a short period of time, temporarily buffered, and / or cached information). The terms "non-transitory computer readable medium" and "non-transitory computer readable storage medium" as used herein are expressly defined to include any type of computer readable storage device and / or storage disk, and do not include propagating signals and do not include transmission media.

[0066] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim adopts any form of "include" or "comprising" (e.g., comprises, comprising, including, having, etc.) as an introduction or within any kind of claim recitation, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term, such as in the introduction of a claim, it is open-ended in the same manner as the terms "include" and "comprising" are open-ended. The term "and / or" when used, for example, in a form such as A, B, and / or C, means any combination or subset of A, B, and C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A and B" is intended to mean implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A or B" is intended to mean implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or operation of processes, instructions, acts, activities, and / or steps, the phrase "at least one of A and B" is intended to mean implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or operation of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to mean an implementation that includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0067] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude the plural. As used herein, an "a" or "an" object means one or more of that object. The terms "a" or "an," "one or more," and "at least one" are used interchangeably herein. Furthermore, although listed separately, multiple components, elements, or method actions may be implemented by, for example, the same entity or object. Additionally, although individual features may be included in different examples or claims, these may perhaps be combined, and the inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0068] Figure 7 is a flow diagram representative of example machine readable instructions and / or example operations 700 that may be executed and / or instantiated by a processor circuit module to perform dense prediction. Figure 7 The machine readable instructions and / or operations 700 of the present invention begin at block 702, where the example embedder 102 (see Figure 1 ) and an example set 104 of converter stages (see Figure 1 ) operate together to encode the input image. At block 704, an example set 106 of recombiners (see Figure 1 ), an example set of fusion blocks 108 (see Figure 1 ) and head end 110 (see Figure 1 ) operates to decode the encoded image to form a final dense prediction. After block 704, the operations and / or instructions of the flowchart end.

[0069] Figure 8 is a flow diagram representative of example machine readable instructions and / or example operations 800 that may be executed and / or instantiated by a processor circuit module to create a tag / embedding based on an input image. Figure 8 The machine readable instructions and / or operations 800 of the example begin at block 802, where the example image segmenter 202 (see FIG. 2) divides the image into non-overlapping tiles. At block 804, the example pixel flattener 204 (see FIG. 2) flattens the tiles into vectors. At block 806, the linear projector 206 (see FIG. 2) embeds the tiles individually. At block 808, the example first concatenator 208 concatenates the embeddings with the learned position embeddings to add the spatial positions to the image representation. At block 810, the example readout tag generator 210 generates readout tags (also known as special tags and / or Figure 1 The patch independent mark 114ST). Thereafter, the procedure / operation of the flowchart ends. In some examples, the operation is performed by the components as described in conjunction with FIG. 2.

[0070] Fig. 9is a flow chart representing example machine readable instructions and / or example operations 900 that may be executed by a processor circuit module and / or illustrate the process of transforming a tag. At block 902, a set 104 of transformer stages (see Figure 1 ) is normalized by the first normalizer 212 of the first transformer 200B of the example embedder 102 (see Figure 1 ) is a set of tags generated by the embedder 102. In box 904, the multi-head attention block 214 performs multi-head attention on the normalized tags. In box 906, the first adder 216 adds the output of the multi-head attention block 214 to the embedded patch / tag 114 generated by the embedder 102. In box 908, the added data is normalized and then provided to the multi-layer perceptron 220 in box 910, which operates as a feed-forward artificial neural network (or a network composed of multiple layers of perceptrons (with threshold activation)). In box 912, the output of the multi-layer perceptron 220 is provided to the second adder 222, where it is added to the output of the first adder 216. Finally, in box 912, the output of the transformer 220B is provided to the next transformer of the transformer 220B of the set of transformer stages 104, and is also provided to the example reorganizer 106 (see Figure 1 ) corresponding recombiner. Thereafter, Fig. 9 The method / operation / program instruction of the flowchart ends.

[0071] Fig.10 is a flow chart representing example machine readable instructions and / or example operations 1000 that may be executed and / or instantiated by a processor circuit module to reconstruct a transformed tag. At block 1002, the example tag reader 302 (see Figure 3 ) performs a read operation on a tag (for example, "Read:RN p+1×D →R Np×D ”). At block 1004, the generated N can be transformed by placing each marker according to the position of the initial tile in the image. p The tokens are reshaped into an image-like representation. In some examples, the tokens are reshaped into an image-like representation. In some such examples, the second cascader 304 applies a spatial cascade operation that produces a feature map of size H / p×W / p with D channels. The operation performed by the example cascader 304 can be represented as “R Np×D →R H / p×W / p×D ” and generates an image-like representation that can be temporarily stored in the storage device 306. At box 1006, the generated image-like representation is passed to the example resampler 310, which scales the representation to a size H / s×W / s with D^ features per pixel. The features represent a more abstract representation of the input image. The resampling operation performed by the resampler 310 can be represented as “RH / p×W / p×D →R H / s×W / s×D ^".

[0072] Fig.11 1 is a flow diagram representative of example machine readable instructions and / or example operations 1100 that may be executed and / or instantiated by a processor circuit module to fuse and upsample tags to generate dense predictions. At block 1102, the output of one of the assemblers 106 is provided to the input of a corresponding one of the fusion blocks 108 and is operated on by the example first residual convolution unit 402 of FIG. 4. At block 1104, the example adder 404 adds the output of the first convolution unit 420 to the output of an earlier fusion block 108 (not shown). At block 1106, the second residual convolution unit 406 operates on the output of the adder 404. At block 1108, the data is processed by the resampler 0.5 The resampled data is resampled, and at block 1110, the resampled data is projected by projector 410. Thereafter, the method / operation / program instructions of flowchart 1100 end.

[0073] Fig.12 1200 is a flowchart representing example machine readable instructions and / or example operations 1200 that may be executed and / or instantiated by a processor circuit module to perform residual convolution within an example fusion block of the fusion block 108. At block 1202, the inputs provided to the residual convolution units 402, 406 are provided to both the example adder 424 and to the first rectified linear unit 412. At block 1204, the output of the first rectified linear unit 412 is provided to the example first convolution unit (3×3) 414, and then normalized by the first batch normalizer 416 (at block 1206). Thereafter, the operations of blocks 412-416 are repeated by blocks 418, 420, and 422, respectively. Finally, the output of the second normalizer 422 is provided to the adder 424 for addition to the input data set, and then the method / operation / program instructions represented by the flowchart 1200 end.

[0074] Fig.13 is a block diagram of an example processor platform 1300 that is configured to execute and / or instantiate Figure 7-12 Machine-readable instructions and / or operations to implement Figure 1 2-4. The processor platform 1300 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), an Internet appliance, a DVD player, or any other type of computing device.

[0075] The processor platform 1300 of the illustrated example includes a processor circuit module 1312. The processor circuit module 1312 of the illustrated example is hardware. For example, the processor circuit module 1312 can be implemented by one or more integrated circuits, logic circuits, FPGA microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired series or manufacturer. The processor circuit module 1312 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, the processor circuit module 1312 implements the example dynamic serializer 412, the dynamic initializer 414, and the example dynamic converter 418.

[0076] The processor circuit module 1312 of the illustrated example includes a local memory 1313 (e.g., cache, registers, etc.). The processor circuit module 1312 of the illustrated example communicates with a main memory (including a volatile memory 1314 and a non-volatile memory 1316) via a bus 1318. The volatile memory 1314 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1316 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1314, 1316 of the illustrated example is controlled by a memory controller 1317.

[0077] The processor platform 1300 of the illustrated example also includes an interface circuit module 1320. The interface circuit module 1320 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, near field communication (NFC) interface, PCI interface and / or PCIe interface.

[0078] In the example shown, one or more input devices 1322 are connected to the interface circuit module 1320. The input device(s) 1322 permit a user to enter data and / or commands into the processor circuit module 1312. The input device(s) 1322 can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a trackball, an isopoint device, and / or a voice recognition system.

[0079] One or more output devices 1324 are also connected to the interface circuit module 1320 of the illustrated example. The output device 1324 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit module 1320 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit module (such as a GPU).

[0080] The interface circuit module 1320 of the illustrated example also includes a communication device (such as a transmitter, receiver, transceiver, modem, residential gateway, wireless access point and / or network interface) to facilitate the exchange of data with an external machine (e.g., any kind of computing device) through a network 1326. Communication can be carried out through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a direct-line wireless system, a cellular telephone system, an optical connection, etc.

[0081] The processor platform 1300 of the illustrated example also includes one or more mass storage devices 1328 for storing software and / or data. Examples of such mass storage devices 1328 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, redundant array of independent disks (RAID) systems, solid-state storage devices (such as flash memory devices), and DVD drives.

[0082] can be Fig.13 The machine-executable instructions 1332 implemented by the machine-readable instructions may be stored in the mass storage device 1328, in the volatile memory 1314, in the non-volatile memory 1316, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0083] Fig.14 yes Fig.13 1312 of the processor circuit module 1312. In this example, Fig.13The processor circuit module 1312 is implemented by the microprocessor 1400. For example, the microprocessor 1400 can implement a multi-core hardware circuit module, such as a CPU, a DSP, a GPU, an XPU, etc. Although it may include any number of example cores 1402 (for example, 1 core), the microprocessor 1400 of this example is a multi-core semiconductor device including N cores. The cores 1402 of the microprocessor 1400 can operate independently, or can cooperate to execute machine-readable instructions. For example, the machine code corresponding to a firmware program, an embedded software program, or a software program may be executed by one of the cores 1402, or may be executed by multiple cores of the core 1402 at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is split into threads and executed in parallel by two or more cores of the core 1402. The software program may correspond to a program executed by Figure 7-12 The flowchart may represent part or all of the machine-readable instructions and / or operations.

[0084] The core 1402 can communicate via an example bus 1404. In some examples, the bus 1404 can implement a communication bus to implement communications associated with the core(s) of the core 1402. For example, the bus 1404 can implement at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the bus 1404 can implement any other type of computing or power bus. The core 1402 can obtain data, instructions, and / or signals from one or more external devices via an example interface circuit module 1406. The core 1402 can output data, instructions, and / or signals to one or more external devices via an example interface circuit module 1406. While the cores 1402 of this example include example local memory 1420 (e.g., a level 1 (L1) cache that may be split into an L1 data cache and an L1 instruction cache), the microprocessor 1400 also includes example shared memory 1410 that may be shared by the cores (e.g., a level 2 (L2 cache)) for high-speed access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 1410. The local memory 1420 of each core of the cores 1402 and the shared memory 1410 may be part of a hierarchy of storage devices, including cache memory and main memory (e.g., Fig.13 In some embodiments, the cache memory may be a cache memory that is stored in a plurality of levels (e.g., main memory 1314, 1316). Typically, higher levels of memory in the hierarchy exhibit lower access times and have smaller storage capacities than lower levels of memory. Changes to various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0085] Each core 1402 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit module. Each core 1402 includes a control unit circuit module 1404, an arithmetic and logic (AL) circuit module (sometimes referred to as an ALU) 1416, a plurality of registers 1418, an L1 cache 1320, and an example bus 1422. Other structures may exist. For example, each core 1402 may include a vector unit circuit module, a single instruction multiple data (SIMD) unit circuit module, a load / store unit (LSU) circuit module, a branch / jump unit circuit module, a floating point unit (FPU) circuit module, etc. The control unit circuit module 1414 includes a semiconductor-based circuit that is configured to control (e.g., coordinate) data movement within the corresponding core 1402. The AL circuit module 1416 includes a semiconductor-based circuit that is configured to perform one or more mathematical and / or logical operations on data within the corresponding core 1402. Some examples of the AL circuit module 1416 perform integer-based operations. In other examples, the AL circuit module 1416 also performs floating-point operations. In still other examples, AL circuit module 1416 may include a first AL circuit module that performs integer-based operations and a second AL circuit module that performs floating-point operations. In some examples, AL circuit module 1416 may be referred to as an arithmetic logic unit (ALU). Register 1418 is a semiconductor-based structure to store data and / or instructions, such as the results of one or more operations performed by AL circuit module 1416 of corresponding core 1402. For example, register 1418 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general registers, (one or more) flag registers, (one or more) segment registers, (one or more) machine-specific registers, (one or more) instruction pointer registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, etc. Register 1418 may be as follows Fig.14 1402. Alternatively, registers 1418 may be organized in any other arrangement, format, or structure, including being distributed throughout core 1402 to reduce access time. Bus 1420 may implement at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0086] Each core 1402 and / or more generally the microprocessor 1400 may include additional and / or alternative structures to those shown and described above. For example, one or more clock circuits, one or more power supply devices, one or more power gates, one or more cache home agents (CHA), one or more aggregation / common grid stops (CMS), one or more shifters (e.g., (one or more) barrel shifters) and / or other circuit modules may exist. The microprocessor 1400 is a semiconductor device that is made to include many transistors that are interconnected to implement the above-described structures in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit module may include one or more accelerators and / or cooperate with the accelerator. In some examples, the accelerator is implemented by a logic circuit module to perform certain tasks faster and / or more efficiently than can be done by a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those described herein. A GPU or other programmable device can also be an accelerator. The accelerator may be on a processor circuit module, in the same chip package as the processor circuit module, and / or in one or more packages separated from the processor circuit module.

[0087] Fig.15 yes Fig.13 13. In this example, the processor circuit module 1312 is implemented by an FPGA circuit module 1500. The FPGA circuit module 1500 can be used, for example, to perform operations that could otherwise be performed by a processor executing corresponding machine-readable instructions. Fig.13 However, once configured, the FPGA circuit module 1500 instantiates machine-readable instructions in hardware and is therefore generally capable of performing operations faster than if the operations could be performed by a general-purpose microprocessor executing corresponding software.

[0088] More specifically, Fig.13 The microprocessor 1310 (which can be programmed to execute Figure 7-12 In contrast, a general purpose device that contains some or all of the machine-readable instructions represented by a flowchart, but whose interconnections and logic circuit modules are fixed once they are made, Fig.15 The example FPGA circuit module 1500 includes interconnect and logic circuit modules, which can be configured and / or interconnected in different ways after fabrication, such as by Figure 7-12In particular, FPGA 1500 can be thought of as an array of logic gates, interconnects, and switches. The switches can be programmed to change how the logic gates are interconnected by the interconnects, effectively forming one or more dedicated logic circuits (unless and until FPGA circuit module 1500 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuit module. Those operations may correspond to the operations performed by Figure 7-12 Thus, FPGA circuit module 1500 can be configured to effectively instantiate as a dedicated logic circuit. Figure 7-12 1500 can execute operations corresponding to those software instructions faster than a general-purpose microprocessor can execute operations. Fig.15 Some or all of the machine-readable instructions correspond to operations.

[0089] exist Fig.15 In the example of FPGA circuit module 1500, FPGA circuit module 1500 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig.15 FPGA circuit module 1500 includes an example input / output (I / O) circuit module 1502 to obtain and / or output data from / to an example configuration circuit module 1504 and / or external hardware (e.g., an external hardware circuit module) 1506. For example, configuration circuit module 1504 may implement an interface circuit module that may obtain machine-readable instructions to configure FPGA circuit module 1500 or (one or more) portions thereof. In some such examples, configuration circuit module 1504 may obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit module (e.g., a programming or dedicated circuit module) that may implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, external hardware 1506 may implement Fig.14 FPGA circuit module 1500 also includes an array of example logic gate circuit modules 1508, a plurality of example configurable interconnects 1510, and example storage circuit modules 1512. Logic gate circuit modules 1508 and interconnects 1510 may be configured to exemplify one or more operations (the operations may correspond to Figure 7-12 at least some of the machine-readable instructions of the device) and / or other desired operations. Fig.15The logic gate circuit modules 1508 shown in are made in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured as a logic circuit. In some examples, the electrical structure includes a logic gate (e.g., an AND gate, an OR gate, a NOR gate, etc.) that provides a basic building block for the logic circuit. An electrically controllable switch (e.g., a transistor) is present in each of the logic gate circuit modules 1508 so that the configuration of the electrical structure and / or the logic gate can form a circuit that performs the desired operation. The logic gate circuit module 1508 may include other electrical structures, such as a lookup table (LUT), a register (e.g., a flip-flop or a latch), a multiplexer, etc.

[0090] The interconnects 1510 of the illustrated example are conductive pathways, traces, vias, or the like, which may include electrically controllable switches (e.g., transistors) whose states can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gate circuit modules 1508 to program the desired logic circuit.

[0091] The storage circuit module 1512 of the illustrated example is configured to store one or more (one or more) results of the operations performed by the corresponding logic gates. The storage circuit module 1512 may be implemented by a register or the like. In the illustrated example, the storage circuit module 1512 is distributed between the logic gate circuit modules 1508 to facilitate access and increase execution speed.

[0092] Fig.15 The example FPGA circuit module 1500 also includes an example dedicated operation circuit module 1514. In this example, the dedicated operation circuit module 1514 includes a dedicated circuit module 1516, which can be called to implement common functions to avoid the need to program those functions on site. Examples of such dedicated circuit modules 1516 include memory (e.g., DRAM) controller circuit modules, PCIe controller circuit modules, clock circuit modules, transceiver circuit modules, memory, and multiplier-accumulator circuit modules. Other types of dedicated circuit modules may exist. In some examples, the FPGA circuit module 1500 may also include an example general-purpose programmable circuit module 1518, such as an example CPU 1520 and / or an example DSP 1522. Additionally or alternatively, other general-purpose programmable circuit modules 1518 may exist, such as GPUs, XPUs, etc., which can be programmed to perform other operations.

[0093] Although Fig.14 and Fig.15 Show Fig.13 These are two example implementations of the processor circuit module 1312, but many other implementations are contemplated. For example, as described above, a modern FPGA circuit module may include an onboard CPU, such as Fig.15One or more of the example CPUs 1520. Thus, Fig.13 The processor circuit module 1312 can also be combined Fig.14 An example microprocessor 1400 and Fig.15 In some such hybrid examples, by Figure 7-12 The first portion of the machine-readable instructions represented by the flowchart of Fig.14 One or more of the cores 1402 are executed by Figure 7-12 The second portion of the machine-readable instructions represented by the flowchart of Fig.15 The FPGA circuit module 1500 is used to execute.

[0094] In some examples, Fig.13 The processor circuit module 1312 may be in one or more packages. For example, Fig.14 The processor circuit module 1400 and / or Fig.15 The FPGA circuit module 1500 may be in one or more packages. In some examples, the XPU may be comprised of a Fig.13 The XPU may be implemented by the processor circuit module 1312. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in still another package, and an FPGA in still another package.

[0095] From the above, it will be appreciated that example systems, methods, devices, and articles of manufacture that use transformers to perform dense prediction have been disclosed. The disclosed systems, methods, devices, and articles of manufacture utilize transformer blocks in place of convolutions in the early stages of the network. In addition, the present methods, devices, and articles of manufacture abandon explicit downsampling operations after the initial image embedding has been calculated, and maintain representations with constant dimensions throughout all processing stages. In addition, the dense predictors disclosed herein have a global receptive field (context) at each level. These such properties are particularly advantageous for dense prediction tasks because they result in fine-grained and globally coherent predictions. Further, the increased accuracy of dense predictions allows for the manufacture of higher quality products in a variety of spaces (depth sensing, camera ISP, autonomous driving, robotics).

[0096] Example methods, devices, systems, and articles of manufacture for performing dense prediction using transformer blocks are disclosed herein. Further examples and combinations thereof include the following: Example 1 includes a device, comprising: an encoder, comprising: an embedder, the embedder converting an input image into a plurality of tokens, the plurality of tokens representing features extracted from the input image, and the embedder embedding the plurality of tokens using learnable positions; and a plurality of transformers, the plurality of transformers configured in a sequence of stages to relate each of the plurality of tokens to other tokens; a decoder, comprising: a plurality of recombiners, the plurality of recombiners being associated with corresponding transformers of the plurality of transformers, each of the plurality of recombiners receiving outputs from the corresponding transformers of the plurality of transformers, and assembling the tokens into feature representations; a plurality of fusion blocks, the plurality of fusion blocks combining the feature representations to form a final feature representation; and an output head, the output head generating dense predictions based on the final feature representation and an output task.

[0097] Example 2 includes the apparatus of Example 1, wherein the embedder further generates a special tile independent tag and adds the special tile independent tag to the plurality of tags.

[0098] Example 3 includes the apparatus of Example 1, wherein a same number of tags is maintained at each stage of the set of transformer stages.

[0099] Example 4 includes the apparatus of Example 1, wherein the embedder: divides the input image into non-overlapping patches of equal pixel size; flattens the patches into vectors; and embeds the patches individually using linear projection, the labels corresponding to the embedded patches.

[0100] Example 5 includes the apparatus of Example 1, wherein the reassembler comprises: a tag reader that reads the plurality of tags; a concatenator that performs a spatial concatenation operation on an output of the tag reader to generate a feature representation; and a resampler that scales the feature representation to a scalar height of the input image divided by a scalar and to a width of the input image divided by the same scalar.

[0101] Example 6 includes the apparatus of Example 1, wherein the reassembler: reassembles the labels into feature representations from deeper levels of the transformer stages at a lower resolution; and assembles the labels into feature representations from earlier levels of the transformer stages at a higher resolution.

[0102] Example 7 includes the apparatus of Example 1, wherein the reorganizer places each marker into a position occupied by each corresponding tile extracted from the input image, the markers forming a feature representation when placed into the corresponding positions.

[0103] Example 8 includes a device comprising: a memory; instructions that, when executed, cause at least one processor to: convert an input image into a plurality of (N) tags, respective tags of the N tags being based on respective non-overlapping patches of the input image, the N tags comprising position information identifying respective positions in which the respective non-overlapping patches fit within the input image; reorganize the N tags into feature representations after the tags have passed through a transformer stage, the tags of the tags output by a deeper transformer stage of the transformer stage being assembled at a first resolution, the tags of the tags output by an earlier transformer stage of the transformer stage being assembled at a second resolution, the first resolution being lower than the second resolution; progressively fuse the feature representations using successive stages of a residual network, and upsampling the respective representations output by the respective stages of the residual network by a factor of two in each stage of the residual network; and generate dense predictions based on the fused feature maps.

[0104] Example 9 includes the apparatus of Example 8, wherein the processor further generates a special tile independent marker and concatenates the special marker to the N markers.

[0105] Example 10 includes the apparatus of Example 8, wherein the same number of tags is maintained at each stage of the converter stages.

[0106] Example 11 includes the device of Example 8, wherein the processor further: divides the input image into the non-overlapping patches, the non-overlapping patches having the same number of pixels; flattens the N markers into vectors; and applies a linear projection to the N markers to embed the markers.

[0107] Example 12 includes the apparatus of Example 8, wherein, to reorganize the N markers, the processor: reads the N markers; spatially concatenates the N markers to generate a feature map; and resamples the feature map to generate a scaled representation of the input image, the scaled representation having a dimension related to the input image by a scalar.

[0108] Example 13 includes the apparatus of Example 8, wherein the recombiner: assembles the N markers at a lower resolution into a feature representation generated at a deeper transformer stage of the transformer stages; and assembles the N markers at a higher resolution into a feature representation generated at an earlier transformer stage of the transformer stages.

[0109] Example 14 includes a non-transitory computer-readable medium comprising instructions that, when executed, cause a machine to at least: convert an input image into tags, the tags representing features extracted from the input image; and transform the tags using information relating each tag to all other tags; recombining the transformed tags into a feature representation; progressively fusing the feature representations to generate a final feature representation, progressively upsampling the final feature representation by a factor of two; and generating a dense prediction based on the final feature representation.

[0110] Example 15 includes the non-transitory computer readable medium of Example 14, wherein the instructions, when executed, cause the machine to generate a special tile independent flag and add the special tile independent flag to the flag.

[0111] Example 16 includes the non-transitory computer readable medium of Example 14, wherein a same number of tags is maintained at each stage of a set of transformer stages used to transform the tags.

[0112] Example 17 includes the non-transitory computer-readable medium of Example 14, wherein, to convert the input image into a mark, the instructions, when executed, further cause the at least one machine to: divide the input image into non-overlapping tiles of equal pixel size; flatten the non-overlapping tiles into vectors; and add spatial information to the non-overlapping tiles to form the mark.

[0113] Example 18 includes the non-transitory computer-readable medium of Example 14, wherein, to reassemble the transformed tokens, the instructions, when executed, further cause the at least one machine to: read the plurality of transformed tokens to generate read information; spatially concatenate the read information; and scale the final feature representation to a first height and a first width, the first height and the first width being related to a second height and a second width, respectively, by a scalar, and the second height and the second width corresponding to a size of the input image.

[0114] Example 19 includes the non-transitory computer-readable medium of Example 14, wherein, to reorganize the markings, the instructions, when executed, further cause the at least one machine to: reorganize the markings from deeper levels of the transformer stage at a lower resolution; and reorganize the markings from earlier levels of the transformer stage at a higher resolution.

[0115] Example 20 includes a method comprising: converting an input image into tags by employing at least one processor to execute instructions, the tags representing features extracted from the input image; and transforming the tags into information including relating each tag to all other tags by employing the at least one processor to execute instructions; recombining the transformed tags into feature representations by employing the at least one processor to execute instructions, the transformed tags being output from corresponding transformer stages of a plurality of transformer stages; progressively fusing the feature representations to generate a final feature representation by employing the at least one processor to execute instructions, progressively upsampling the feature representation by a factor of two by employing the at least one processor to execute instructions; and generating a dense prediction based on the final feature representation by employing the at least one processor to execute instructions.

[0116] Example 21 includes the method of Example 20, further comprising: generating a special tile independent flag; and adding the special tile independent flag to each of the flags.

[0117] Example 22 includes the method of Example 20, wherein a same number of tags is maintained at each stage of a set of transformer stages used to transform the tags.

[0118] Example 23 includes the method of Example 20, further comprising: dividing the input image into non-overlapping patches of equal pixel size; flattening the non-overlapping patches into vectors; and embedding the non-overlapping patches using linear projection.

[0119] Example 24 includes the method of Example 20, wherein recombining the transformed tags comprises: reading the plurality of transformed tags to generate read information; spatially concatenating the read information to form a final feature; and scaling the final feature representation to a first height and a first width, the first height and the first width being related to a second height and a second width, respectively, by scalars, and the second height and the second width corresponding to a size of the input image.

[0120] Example 25 includes the method of Example 20, wherein recombining the marks further comprises: recombining the marks from a deeper level of the converter stage at a lower resolution; and recombining the marks from an earlier level of the converter stage at a higher resolution. Although certain example systems, methods, apparatus, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, apparatus, and articles that fall fully within the scope of the claims of this patent.

[0121] The following claims are hereby incorporated into this Detailed Description by this reference, with each claim standing on its own as a separate embodiment of the disclosure.

[0122] This application also discloses a set of technical solutions, as follows: 1. A device comprising: Encoder, including: an embedder that converts an input image into a plurality of tokens representing features extracted from the input image and embeds the plurality of tokens using learnable positions; and a plurality of transducers configured in a stage sequence to relate each of the plurality of indicia to the other indicia; Decoder, including: a plurality of recombiners associated with corresponding transformers of the plurality of transformers, each of the plurality of recombiners receiving output from the corresponding transformer of the plurality of transformers and assembling the tokens into a feature representation; a plurality of fusion blocks that combine the feature representations to form a final feature representation; and An output head is provided, wherein the output head generates a dense prediction based on the final feature representation and an output task.

[0123] 2. The device as described in technical solution 1, wherein the embedder further generates a special tile independent tag and adds the special tile independent tag to the multiple tags.

[0124] 3. An apparatus as described in technical solution 1, wherein the same number of tags is maintained at each stage of the set of converter stages.

[0125] 4. The device according to technical solution 1, wherein the embedder: Dividing the input image into non-overlapping patches of equal pixel size; Flattening the patch into a vector; and The patches are individually embedded using linear projection and the labels correspond to the embedded patches.

[0126] 5. The device according to technical solution 1, wherein the recombiner comprises: a tag reader that reads the plurality of tags; a concatenator that performs a spatial concatenation operation on the output of the tag reader to generate a feature representation; and a resampler that scales the feature representation to a scalar height of the input image divided by a scalar and to a width of the input image divided by the same scalar.

[0127] 6. The device according to technical solution 1, wherein the recombinator: recombining the tokens into deeper level feature representations from the transformer level at a lower resolution; and The signatures are assembled into feature representations from earlier stages of the transformer stage at a higher resolution.

[0128] 7. An apparatus as described in technical solution 1, wherein the reorganizer places each marker into a position occupied by each corresponding patch extracted from the input image, and the marker forms a feature representation when placed into the corresponding position.

[0129] 8. A device comprising: Memory; Instructions which, when executed, cause at least one processor to: converting an input image into a plurality of (N) tags, respective tags of the N tags being based on respective non-overlapping tiles of the input image, the N tags comprising position information identifying respective locations where the respective non-overlapping tiles fit within the input image; recombining the N markers into a feature representation after the markers have passed through transformer stages, markers of the markers output by deeper ones of the transformer stages being assembled at a first resolution and markers of the markers output by earlier ones of the transformer stages being assembled at a second resolution, the first resolution being lower than the second resolution; progressively fusing the feature representations using successive stages of a residual network, and upsampling in each stage of the residual network a corresponding representation output by a corresponding stage of the residual network by a factor of two; and Generate dense predictions based on the fused feature maps.

[0130] 9. The device as described in technical solution 8, wherein the processor further generates a special patch independent mark and cascades the special mark to the N marks.

[0131] 10. An apparatus as described in technical solution 8, wherein the same number of marks is maintained at each stage of the converter stage.

[0132] 11. The device according to technical solution 8, wherein the processor further: Dividing the input image into the non-overlapping tiles, the non-overlapping patches having the same number of pixels; Flattening the N tokens into a vector; and A linear projection is applied to the N markers to embed the markers.

[0133] 12. The device of claim 8, wherein, in order to reorganize the N tags, the processor: Reading the N tags; spatially concatenating the N markers to generate a feature map; and The feature map is resampled to generate a scaled representation of the input image, the scaled representation having a dimension related to the input image by a scalar.

[0134] 13. The device according to technical solution 8, wherein the recombinator: assembling the N markers into a feature representation generated at a deeper transformer stage of the transformer stage at a lower resolution; and The N markers are assembled into a feature representation generated at an earlier transformer stage of the transformer stage at a higher resolution.

[0135] 14. A non-transitory computer readable medium comprising instructions that, when executed, cause a machine to at least: converting an input image into tokens representing features extracted from the input image; and transforming the tags using information relating each tag to all other tags; Recombining the transformed tokens into feature representations; progressively fusing the feature representations to generate a final feature representation; asymptotically upsampling the final feature representation by a factor of two; and A dense prediction is generated based on the final feature representation.

[0136] 15. A non-transitory computer-readable medium as described in technical solution 14, wherein the instructions, when executed, cause the machine to generate a special tile-independent mark and add the special tile-independent mark to the mark.

[0137] 16. A non-transitory computer-readable medium as described in technical solution 14, wherein the same number of tags is maintained at each stage of a set of transformer stages used to transform the tags.

[0138] 17. The non-transitory computer-readable medium of technical solution 14, wherein, in order to convert the input image into a mark, the instructions, when executed, further cause the at least one machine to: Dividing the input image into non-overlapping patches of equal pixel size; Flattening the non-overlapping tiles into vectors; and Spatial information is added to the non-overlapping tiles to form the marker.

[0139] 18. The non-transitory computer-readable medium of technical solution 14, wherein, in order to reorganize the transformed mark, the instruction, when executed, further causes the at least one machine to: reading the plurality of transformed marks to generate read information; Cascading read information in space; and The final feature representation is scaled to a first height and a first width, the first height and the first width are respectively related to a second height and a second width by a scalar, and the second height and the second width correspond to a size of the input image.

[0140] 19. The non-transitory computer-readable medium of technical solution 14, wherein, in order to reorganize the mark, the instructions, when executed, further cause the at least one machine to: recombining said markings from deeper levels of said transformer level at a lower resolution; and The signatures from an earlier stage of the converter stage are reconstructed at a higher resolution.

[0141] 20. A method comprising: converting an input image into a markup by executing instructions using at least one processor, the markup representing features extracted from the input image; and transforming the indicia into information including relating each indicia to all other indicia by executing instructions with the at least one processor; recombining the transformed indicia into a feature representation by executing instructions with the at least one processor, the transformed indicia being output from a corresponding transformer stage of the plurality of transformer stages; The method includes progressively fusing the feature representations by employing the at least one processor-executable instruction to generate a final feature representation, progressively upsampling the feature representations by a factor of two by employing the at least one processor-executable instruction; and generating a dense prediction based on the final feature representation by employing the at least one processor-executable instruction.

[0142] 21. The method according to technical solution 20 further comprises: Generate unique patch-independent markers; and The special patch independent tag is added to each tag of the tag.

[0143] 22. A method as described in technical solution 20, wherein the same number of tags is maintained at each stage of a set of transformer stages used to transform the tags.

[0144] 23. The method according to technical solution 20 further comprises: Dividing the input image into non-overlapping patches of equal pixel size; Flattening the non-overlapping tiles into vectors; and The non-overlapping tiles are embedded using linear projection.

[0145] 24. The method of technical solution 20, wherein the marker transformed by the recombination comprises: reading the plurality of transformed marks to generate read information; spatially concatenating the read information to form a final feature; and The final feature representation is scaled to a first height and a first width, the first height and the first width are respectively related to a second height and a second width by a scalar, and the second height and the second width correspond to a size of the input image.

[0146] 25. The method of technical solution 20, wherein recombining the marker further comprises: recombining said markings from deeper levels of said transformer level at a lower resolution; and The signatures from an earlier stage of the converter stage are reconstructed at a higher resolution.

Claims

1. A device for assigning labels to pixels of an image, the device comprising: an encoder for generating a first set of features and a second set of features based on the image; a transformer for processing the first set of features; and a decoder coupled to the transformer for generating a label based on an output of the transformer, The decoder comprises a first convolution unit and a second convolution unit, and wherein the second set of features is to be processed by the second convolution unit after being processed by the first convolution unit.

2. The device according to claim 1, wherein: The encoder comprises a convolutional neural network applied to the image.

3. The device according to claim 1, wherein: The second set of features is independent of patches of the image.

4. The device according to claim 1, wherein: The transformer includes a multi-head self-attention block.

5. The device according to claim 4, wherein: The transformer includes a plurality of normalizers and adders.

6. The device according to claim 1, wherein: The decoder comprises a recombiner, a fusion block, and an output head, and wherein the fusion block comprises the first convolution unit and the second convolution unit.

7. The device according to claim 1, wherein: The label identifies at least one class associated with the pixel in the image.

8. One or more machine-readable media having stored thereon instructions executable by a machine to: An encoder for generating a first set of features and a second set of features based on the image; a transformer for processing the first set of features; and a decoder coupled to the transformer for generating a label based on an output of the transformer, The decoder comprises a first convolution unit and a second convolution unit, and wherein the second set of features is to be processed by the second convolution unit after being processed by the first convolution unit.

9. One or more machine-readable media according to claim 8, wherein: The encoder comprises a convolutional neural network applied to the image.

10. The one or more machine-readable media of claim 8, wherein: The second set of features is independent of patches of the image.

11. The one or more machine-readable media of claim 8, wherein: The transformer includes a multi-head self-attention block.

12. One or more machine-readable media according to claim 11, wherein: The transformer includes a plurality of normalizers and adders.

13. The one or more machine-readable media of claim 8, wherein: The decoder comprises a recombiner, a fusion block, and an output head, and wherein the fusion block comprises the first convolution unit and the second convolution unit.

14. The one or more machine-readable media of claim 8, wherein: The label identifies at least one class associated with the pixel in the image.

15. A computing device comprising: A camera to capture the image; a memory for storing instructions; and A processor coupled to the memory, configured to execute the instructions to implement: an encoder for generating a first set of features and a second set of features based on the image; a transformer for processing the first set of features; and a decoder coupled to the transformer for generating a label based on an output of the transformer, The decoder comprises a first convolution unit and a second convolution unit, and wherein the second set of features is to be processed by the second convolution unit after being processed by the first convolution unit.

16. The computing device of claim 15, wherein: The encoder comprises a convolutional neural network applied to the image.

17. The computing device of claim 15, wherein: The second set of features is independent of patches of the image.

18. The computing device of claim 15, wherein: The transformer includes a multi-head self-attention block.

19. The computing device of claim 18, wherein: The transformer includes a plurality of normalizers and adders.

20. The computing device of claim 15, wherein: The decoder comprises a recombiner, a fusion block, and an output head, and wherein the fusion block comprises the first convolution unit and the second convolution unit.

21. The computing device of claim 15, wherein: The label identifies at least one class associated with the pixel in the image.

Images