Dynamic Header for Object Detection

Abstract

Systems and methods for object detection generate a feature pyramid corresponding to image data and rescale the feature pyramid to a scale corresponding to the median level of the feature pyramid, where the rescaled feature pyramid is a four-dimensional (4D) tensor. The 4D tensor is reshaped into a three-dimensional (3D) tensor, which has separate perspectives including scale features, spatial features, and task features corresponding to different dimensions of the 3D tensor. The 3D tensor is used together with multiple attention layers to update multiple feature maps associated with the image data. Object detection is performed on the image data using the updated feature maps.

Description

Background Technology

[0001] Many tasks in computer vision utilize a backbone, which is typically pre-trained. The backbone is then used as a feature extractor, its output representing the input feature map. Using this feature map, one or more tasks, such as detection and segmentation, can be performed. These tasks are usually performed by applying, for example, a detection head to the feature map, such that the head is connected to the backbone.

[0002] Developing object detection heads presents numerous challenges. First, it is desirable for the head to be scale-aware, as multiple objects with drastically different scales often coexist in an image. Second, it is desirable for the head to be spatially aware, as objects typically appear with very different shapes, rotations, and positions from different viewpoints. Third, it is desirable for the head to be task-aware, as objects can have various representations (e.g., bounding boxes, center points, and corner points) with entirely different objectives and constraints. Traditional systems are configured to address one of these challenges or to address them using separate systems. For example, designing a unified head can be viewed as an attention learning problem. However, because traditional methods use a fully self-attention mechanism on this tensor, the optimization problem is extremely difficult to solve, resulting in very high computational costs and often making implementation impractical. Summary of the Invention

[0003] This summary is provided to introduce a set of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0004] A computational method for object detection includes: generating a feature pyramid corresponding to image data; and rescaling the feature pyramid to a scale corresponding to the median level of the feature pyramid, wherein the rescaled feature pyramid is a four-dimensional (4D) tensor. The computational method also includes reshaping the 4D tensor into a three-dimensional (3D) tensor, the 3D tensor having separate perspectives including scale features, spatial features, and task features corresponding to different dimensions of the 3D tensor. The computational method further includes updating multiple feature maps associated with the image data using the 3D tensor and multiple attention layers, and performing object detection on the image data using the updated multiple feature maps.

[0005] Many of the accompanying features will become easier to understand by referring to the following detailed description taken in conjunction with the accompanying drawings. Attached Figure Description

[0006] This specification will be better understood from the following detailed description, which is read with reference to the accompanying drawings, in which:

[0007] Figure 1 This is a block diagram illustrating the system based on the example;

[0008] Figure 2 This is a block diagram illustrating an object detection system based on an example;

[0009] Figure 3 It is a block diagram based on the example dynamic header architecture;

[0010] Figure 4 It is based on the block diagram of the example header;

[0011] Figure 5 This is a block diagram illustrating a first-level detector based on an example;

[0012] Figure 6 This is a block diagram illustrating a two-stage detector based on an example;

[0013] Figure 7 This is a flowchart illustrating the operation of a computing device performing object detection according to an example; and

[0014] Figure 8 A computing device is shown as a functional block diagram based on an example.

[0015] Throughout the accompanying drawings, corresponding reference numerals denote the corresponding components. The system is illustrated in schematic form within the drawings. The drawings may not be drawn to scale. Detailed Implementation

[0016] The computing apparatus and methods described herein are configured to provide a unified head that is configured to be scale-aware, spatial-aware, and task-aware. In one or more examples, the unified head is a dynamic head configured to simultaneously unify scale-awareness, spatial-awareness, and task-awareness. This disclosure does not construct a fully self-attention mechanism, but rather deploys an attention mechanism on specific dimensions of features, such as hierarchy, space, and channels. In one example, a feature pyramid corresponding to image data is generated and then rescaled to a scale corresponding to the median level of that feature pyramid, making the rescaled feature pyramid a four-dimensional (4D) tensor. The 4D tensor is reshaped into a three-dimensional (3D) tensor, which corresponds to scale features, spatial features, and task features, which correspond to an overall view comprising multiple feature maps.

[0017] In various examples, object detection is performed using multiple feature maps (which are updated and / or reshaped as described in this paper) leveraging rescaled and reshaped 3D tensors. Specifically, scale-aware attention processors are deployed only in the level dimension and configured to learn the relative importance of each semantic level to enhance individual object features at the appropriate level based on scale. Spatial-aware attention processors are deployed in the spatial dimension, learning coherent discriminative representations of spatial locations. Task-aware attention processors are deployed intra-channel and guide different feature channels to favor different tasks (e.g., classification, box regression, and center / keypoint learning) based on different convolutional kernel responses from objects. Therefore, the resulting system with dynamic heads performs operations on rescaled and reshaped tensors with reduced computational cost compared to traditional methods. In this way, processors are used in unconventional ways to allow for more efficient and reliable object detection when programmed to perform the operations described in this paper, resulting in an improved user experience.

[0018] In various examples, the dynamic head framework is configured to unify the object detection head with attention. By coherently combining multiple self-attention mechanisms between feature levels (rescaling and reshaping tensors) for scale awareness, between spatial locations for spatial awareness, and within the output channel for task awareness, this disclosure improves the representational power of the object detection head without increasing computational overhead.

[0019] The process described in this article is not limited to object detection, but can be implemented using different types of computing tasks in different applications. Object detection processes, such as performing localization and classification in object detection, can be implemented in a data processing system 100 deployed as a cloud service (e.g., an object detection system). Figure 1 As shown. In this example, the data processing system 100 implements the object detection process described herein to allow for efficient object detection using different attention mechanisms. That is, the data processing system 100 operates using a unified header framework, which in some examples depends on different attention mechanisms for each of specific dimensions of features such as hierarchy, space, and channel.

[0020] The data processing system 100 includes one or more computers 102 for storing, for example, images, and a memory 104. It should be understood that other data may be stored in the memory 104 and processed by the one or more computers 102 using this disclosure.

[0021] In some examples, the data processing system 100 is connected to one or more end-user computing devices, such as a desktop computer 106, a smartphone 108, a laptop computer 110, and an augmented reality headset 112 (e.g., Microsoft). For example, data processing system 100 is shown connected to end-user computing devices via computer network 114, illustrated as the Internet.

[0022] Data processing system 100 receives input data, such as image data, from an end-user computing device or a server. The data is uploaded to data processing system 100 for processing, such as object detection processing to identify one or more objects within the image. It should be understood that some or all of the functions of data processing system 100 may be implemented within the end-user computing device.

[0023] The data processing system 100 in this example implements a dynamic head 116 that performs object detection using separate attention mechanisms on different layers of feature information in a feature pyramid (e.g., a feature map representation of the input image), allowing for efficient and effective object detection. Utilizing the dynamic head 116, in some examples, the feature map is improved after processing by each of a plurality of attention processors, as discussed in more detail herein. In some examples, the functionality of the data processing system 100 described herein is performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0024] Therefore, this disclosure allows for the performance of improved object detection using fewer computational resources. In this way, computational accuracy can be maintained while the "cost" (e.g., computational and / or storage requirements) of the operations performed on less complex optimization problems is reduced. For example, the dynamic header 116 of this disclosure enables the performance of improved object detection in situations that would otherwise be computationally infeasible.

[0025] Various examples include, for example Figure 2The object detection system 200 is shown. In one example, the object detection system 200 uses a unified head with different attention mechanisms to process a 3D tensor (which has been rescaled and reshaped as described herein) to generate an output 212, which in one example is an image with detected objects. More specifically, the object detection system 200 includes an object detection processor 202, which in some examples is configured as a processing engine to perform object detection on image data 204 comprising one or more images. It should be noted that the image data 204 may include different types of image data configured in different ways. It should also be noted that this disclosure can be applied to different types of data, including non-image data. In some examples, efficient processing of multiple feature maps is performed in various examples by providing a rescaled and reshaped tensor (3D tensor).

[0026] The object detection processor 202 can access input data 204, such as image data. For example, the object detection processor 202 accesses image data acquired by a camera or other image acquisition device as input data 204 for performing object detection. It should be understood that the object detection processor 202 is configured to perform object detection tasks in various application domains. For example, in addition to the image datasets described herein, this disclosure also provides object detection for different types of datasets.

[0027] In the example shown, input data 204 includes image data, where object detection processor 202 first processes input data 204 using backbone processor 206, which in some examples is configured to generate an improved 3D tensor 220. That is, the 3D tensor 220 in various examples is processed more efficiently with fewer computational resources. As described in more detail herein, a feature pyramid corresponding to input data 204 is generated and then rescaled to a scale corresponding to the median level of that feature pyramid, resulting in a 4D tensor. The 4D tensor is reshaped into a 3D tensor 220 corresponding to scale features, spatial features, and task features, which correspond to an overall view including multiple feature maps, and then output to a unified head processor 208, which generates incrementally improved feature maps 210 (e.g., feature maps that allow for more reliable or accurate object detection).

[0028] In one example, the unified head processor 208 first performs scale-aware attention processing to dynamically fuse scale features from different scales, and then performs spatial-aware attention processing (e.g., separating attention processing in each spatial dimension) across the spatial dimensions of the 3D tensor 220. In one example, the spatial-aware processing is configured to aggregate spatial features across levels at the same spatial location. Then, after performing scale-aware and spatial-aware processing, the unified head processor performs task-aware processing within the channels of the 3D tensor 220. In one example, the task-aware processing is configured to generalize different representations of objects, where multiple feature maps are reconstructed into different activations based on the different object detection tasks to be performed. It should be understood that the order of processing is merely an example of the processing steps performed. That is, processing can be performed in different orders, partial processing can be performed, processing steps can be repeated, etc.

[0029] In the case of generating feature map 210, object detection processor 202 performs object detection on image data 204 using multiple updated and / or modified feature maps to generate output 212. That is, object detection processor 202 reliably and accurately identifies one or more objects in image data 204.

[0030] Figure 3 An example of a dynamic head architecture 300 implemented by an object detection processor 200 is shown. The dynamic head architecture 300 is configured to perform one or more tasks described herein on multiple images. The dynamic head architecture 300 in this example implements a dynamic head framework to unify the object detection head with attention when processing rescaled and reshaped 3D tensors 220. By coherently combining multiple self-attention mechanisms between feature levels for scale awareness, between spatial locations for spatial awareness, and within the output channel for task awareness—which can be performed as a result of generating rescaled and reshaped 3D tensors 220—this disclosure improves the representational power of the object detection head without increasing computational overhead.

[0031] In various examples, the dynamic head architecture 300 is configured to perform object detection, for example, in computer vision applications. For instance, the dynamic head architecture 300 is configured as an object detector to perform deep learning using a backbone 302 for feature extraction and a unified head 304 for localization and classification tasks. As discussed herein, the unified head 304 is scale-aware, spatial-aware, and task-aware. That is, in some examples, the unified head 304 is a dynamic head configured to unify scale-awareness, spatial-awareness, and task-awareness. In the illustrated example, the output of the backbone 302 (e.g., the input to the unified head 304, which acts as the detection head) is a 3D tensor 220 with level × space × channel dimensions, and is processed by the unified head 304 using a feature-specific attention mechanism (e.g., level, space, and channel), rather than as a fully self-attention mechanism implemented by the 3D tensor 220.

[0032] In the example shown, processing is performed using scale-aware attention 310, spatial-aware attention 312, and task-aware attention 314, respectively. For example, scale-aware attention 310 is deployed only in the level dimension and learns the relative importance of various semantic levels to enhance features for a single object at the appropriate level based on its scale. Spatial-aware attention 312 is deployed in the spatial dimension (i.e., height × width) and learns coherent discriminative representations in spatial locations. Task-aware attention 314 is deployed intra-channel and guides different feature channels to favor different tasks (e.g., classification, bounding box regression, and center / keypoint learning) based on different convolutional kernel responses from the object.

[0033] Therefore, a unified attention mechanism for the detection head is implemented. It should be noted that although the attention mechanisms are applied to different dimensions of the feature tensor, i.e., a 3D tensor 220, the performance of each attention mechanism can complement each other to allow learning better representations that can be used to improve all types of object detection models, using less training time but with improved performance. For example, scale-aware attention 310 in the unified head 304 is configured to adapt the importance levels of various features to the input. Spatial-aware attention 312 in the unified head 304 is configured not only to apply attention to each spatial location but also to adaptively aggregate multiple feature levels together for more discriminative representation learning. Task-aware attention 314 in the unified head 304 is configured to allow attention to be deployed on channels, which can adaptively favor various tasks, such as for single-level or two-level detectors, or box / center / keypoint-based detectors.

[0034] More specifically, given L different levels of feature cascades in feature pyramid 306 The dynamic header architecture 300 uses upsampling or downsampling to scale continuous-level features to median-level features, denoted as: Therefore, the rescaled feature pyramid 306 is a 4D tensor. Where L represents the level in the pyramid, and H, W, and C represent the height, width, and number of channels of the median level feature, respectively. Furthermore, S = H × W is defined to reshape the tensor into a 3D tensor. Based on this representation, each tensor dimension has a different role, including the following:

[0035] Differences in target scale are related to features at different levels. By improving the representation of this disclosure, across different levels, based on various examples, Learning can be beneficial for scale awareness in object detection;

[0036] Various geometric transformations of different object shapes are related to the characteristics of different spatial locations. By improving the representation of this disclosure, across... Learning different spatial locations can improve spatial awareness for object detection; and

[0037] Different object representations and tasks can be associated with features from different channels. By improving the representation of this disclosure, across... Learning different channels can be beneficial for task awareness in object detection.

[0038] This disclosure unifies these roles within an effective attention learning problem, which in various examples involves combining multiple attentions across all three dimensions to form a unified head 304 for maximizing improvement.

[0039] In a specific example, given a feature tensor The general formula for applying self-attention is defined as: Where π(·) is the attention function. A naive solution to this attention function is to implement it through fully connected layers. However, due to the high dimensionality of tensors, directly learning the attention function across all dimensions is computationally expensive and practically unaffordable. Instead, in various examples, the attention function is transformed into three sequential attentions, each focusing on only one viewpoint, as follows: πL(·), πS(·), and πC(·) are three different attention functions, applicable to dimensions L, S, and C, respectively.

[0040] Three attention mechanisms will now be described: scale-aware attention 310, spatial awareness attention 312, and task-aware attention 314. In this example, scale-aware attention 310 is π. LAnd it is configured to dynamically fuse features from different scales based on semantic importance as follows: Where f(·) is a linear function approximated by a 1×1 convolutional layer, and σ(x)=max(0,min(1,x+1 / 2)) is a hard sigmoid function.

[0041] Continuing with this example, spatial awareness attention 312 is π S Furthermore, it focuses on distinguishable regions that coexist consistently between spatial location and feature level, based on fused features. Considering the high dimensionality in S, spatially aware attention 312 is decomposed into two steps: first, the attention learning problem is made sparse through deformable convolution, and then features are aggregated across levels at the same spatial location as follows: Where K is the number of sparse sampling locations, p K +Δp K It is achieved through self-learning spatial offset Δp k The location of the shift is focused on distinguishing regions, while Δm k It is position p k The self-learning importance scalar at the location. Both of these features originate from... Learned from median-level input features.

[0042] Continuing with this example, task-aware attention 314 is π. C It is configured to jointly learn and generalize different representations of an object. In this example, task-aware attention 314 is used as the final step or end of the processing pipeline. In some examples, task-aware attention 314 dynamically opens and closes feature channels to support different tasks, such as:

[0043]

[0044] in, It is the feature slice of the c-th channel, and [α] 1 ,α 2 ,β 1 ,β 2 ] T =θ(·) is a hyperfunction that learns to control the activation threshold. In one example, θ(·) is configured to first perform a global average merging of the L×S dimensions to reduce dimensionality, followed by a neural network with two fully connected layers and a normalization layer. The output is then normalized to [-1, 1] using a shifted sigmoid function.

[0045] It should be noted that, since the above three attention mechanisms (i.e., scale-aware attention 310, spatial-aware attention 312, and task-aware attention 314) are applied sequentially, The formula can be nested multiple times to efficiently combine multiple π.L π S and π C Blocks 402, 404, and 406 are stacked together, as follows: Figure 4 The head block 400 shown is illustrated. That is, the head block 400 shows a structure with π... L π S and π C The implementation of the unified head 304 representing scale-aware attention 310, spatial-aware attention 312, and task-aware attention 314 is defined by the above formula.

[0046] Therefore, object detection can be performed using the dynamic head architecture 400. It should be understood that any type of backbone network can be used to extract the feature pyramid 306, which is further scaled to the same size to form a 3D tensor 220 (illustrated as an overall view 308 with dimensions L×S×C), which is then used as input to the dynamic head.

[0047] It should be noted that variations and modifications can be considered, for example, by sequentially stacking multiple head blocks 400 that include scale-aware, spatial-aware, and task-aware attention. It should also be understood that the output of the dynamic head can be used for representations of different tasks and object detection, such as classification, center / box regression, etc.

[0048] Reference again Figure 2 and Figure 3 Feature maps 316, 318, 320, and 322 represent the outputs of the initial backbone processing, which involves scale-aware attention 310, spatial awareness attention 312, and task-aware attention 314. It should be understood that the initial feature map 316, the output from the backbone processing 302, is noisy because the domain differs from the pre-trained domain. However, after processing with scale-aware attention 310, feature map 318 becomes more sensitive to scale differences in the foreground object. Then, after processing with spatial awareness attention 312, feature map 320 becomes sparser and focuses on discriminating the spatial location of the foreground object. Finally, after processing with task-aware attention 314, feature map 322 is transformed into different activations based on the requirements of different downstream tasks. Therefore, different processing stages or steps lead to incremental improvements.

[0049] Furthermore, regarding the object detection processor 202, the operator can specify various parameters, etc. For example, the operator can use the graphical user interface 216 to specify the weighting values ​​of different layers of the neural network topology, the sensitivity of different attention, etc. For example, after the operator configures one or more parameters, the object detection processor 202 is configured to perform the object detection described herein. It should be noted that in some examples, once the training of one or more neural networks is completed (e.g., after the training data is exhausted), the trained object detector 218 is stored and loaded into one or more end-user devices, such as a smartphone 208, a wearable augmented reality computing device 212, a laptop computer 210, or other end-user computing devices. The end-user computing devices can use the trained object detector 218 to perform one or more tasks, such as object detection.

[0050] The dynamic header structure 300 can be integrated into different object detectors to improve detector performance. For example, Figure 5 The diagram illustrates a Level 1 detector 500 configured to predict object location by densely sampling locations from one or more feature maps, simplifying detector design. Instead of multiple branches to a backbone network for extracting dense features and multiple task-specific sub-network branches for handling different tasks, Level 1 detector 500 is configured to have a unified branch rather than multiple branches to a backbone. Therefore, Level 1 detector 500 is operable to perform multiple tasks simultaneously by advantageously using multiple attention mechanisms (i.e., scale-aware attention 310, spatial-aware attention 312, and task-aware attention 314) in each of a plurality of sequentially aligned heads 502. This further simplifies the architecture and improves efficiency. The architecture also eliminates the need to attach, for example, centrality prediction or keypoint prediction to classification or regression branches. Instead, by deploying a dynamic head architecture 300, in some examples, a more flexible detector is provided that only various types of predictions are attached to the head ends.

[0051] As another example, Figure 6 The diagram illustrates a two-stage detector 600 with multiple heads 602, wherein bounding box proposal and ROI-pooling layers 604 are derived from the feature pyramid of the backbone network (e.g., Figure 3 The feature pyramid 306 shown extracts intermediate representations. To provide this property, the two-stage detector 600 is configured to first apply scale-aware attention 310 and spatial-aware attention 312 to the feature pyramid 306 before passing through the ROI-pooling layer 604. Then, task-aware attention 314 is used to replace the fully connected layers.

[0052] One or more of the various examples can be modified or configured in different implementations. For example, in one implementation, sparse sampling is introduced, such as utilizing backbone 302 ( Figure 3 As shown in the diagram, deformable convolutions are used as transformation learning for convolutional layers to enhance feature representations. For example, deformable convolutions can be used in conjunction with this disclosure in a unified head 304 (object detection head), where, in the configuration shown, the deformable convolutions model only the S-dimensional sub-dimension. In this example, deformable processing is used in the backbone 302, and this deformable processing complements the dynamic head structure 300.

[0053] In some examples, nonlocal networks are implemented to enhance object detection performance by leveraging attention processing. In this example, the nonlocal network uses a simple dot product formula to enhance pixel features by fusing features from other pixels at different spatial locations. This behavior can be viewed as modeling only the L×S sub-dimension in the dynamic head architecture 300.

[0054] In some examples, the transformer is imported from natural language processing into computer vision tasks. This transformer provides a simple solution by applying multi-head fully connected layers to learn cross-attention correspondences and fuse features from different modes. This behavior can be viewed as modeling only the S×C sub-dimensions of the dynamic head architecture 300.

[0055] Therefore, object detection is performed in various examples using a rescaled and reshaped feature pyramid 306 with scale-aware attention 310, spatial awareness attention 312, and task-aware attention 314. Table 1 shows the results of ablation studies conducted using this disclosure.

[0056]

[0057] Table 1

[0058] Specifically, Table 1 shows the results of a controlled study of the effectiveness of different components in the dynamic head architecture 300 by gradually adding components to the baseline. As shown in Table 1, “L.”, “S.”, and “C.” represent scale-aware attention 310, spatial-aware attention 312, and task-aware attention 314, respectively. It can be seen that adding each component individually to the baseline improves its performance by 0.9 mAP, 2.4 mAP, and 1.3 mAP, respectively. Since spatial-aware attention 312 is the dominant dimension among the three modules, the largest gain is expected for spatial-aware attention 312. Then, both “L.” and “S.” are added to the baseline, resulting in a sustained performance improvement of 2.9 mAP. Finally, the fully dynamic head block with scale-aware attention 310, spatial-aware attention 312, and task-aware attention 314 significantly improves the baseline by 3.6 mAP. The results indicate that the different components function as a coordinated system or network.

[0059] Therefore, for various examples, a unified scale-aware, spatial-aware, and task-aware attention is provided for the object detection head within a single framework using rescaled and reshaped feature pyramids (or tensors). It should be understood that, as a plugin, the dynamic head is flexible and can be integrated into any type of object detector framework to improve performance. Furthermore, effective learning is achieved through the implementation of various examples.

[0060] As should be understood, various examples can be used for operating different types of neural networks and for object detection of different types. Furthermore, various examples can be used to perform object detection using different types of data. Figure 7 A flowchart of a method 700 for performing various examples of object detection is shown. The operations shown in the flowchart described herein may be performed in a different order than those shown, may include additional or fewer steps, and may be modified as needed or desired. Furthermore, one or more operations may be performed simultaneously, concurrently, or sequentially. In some examples, method 700 is performed on a computing device, such as a server or computer with sufficient processing power to perform the operations, such as a graphics processing unit (GPU).

[0061] Referring to method 700, which illustrates a method for object detection, at 702, the computing device generates a feature pyramid corresponding to the image data. That is, the feature pyramid is an image pyramid with features extracted from the image data (e.g., backbone features identified in the feature map). At 704, the computing device rescales the feature pyramid to a scale corresponding to an intermediate level of the feature pyramid. In one example, the rescaled feature pyramid is a 4D tensor as described herein.

[0062] At point 706, the computing device reshapes the 4D tensor into 3D tensors corresponding to scale features, spatial features, and task features, which correspond to an overall view comprising multiple feature maps. That is, in some examples, the feature pyramid is transformed into an overall view in 3D, where each feature level is three-dimensional. In one example, the height and width dimensions are combined as spatial features to form geometric features.

[0063] At point 708, the computing device uses a reshaped tensor and an attention layer to generate updated feature maps for multiple feature maps. That is, the reshaped 3D tensor is used to generate updated feature maps using the attention layer. In some examples described herein, the attention layer is configured to apply scale-aware attention, spatial-aware attention, and task-aware attention. In one example, scale-aware attention is applied along the horizontal dimension of the 3D tensor, where it is configured to dynamically fuse scale features from different scales. Spatial-aware attention is applied (e.g., deployed) along the spatial dimension of the 3D tensor, where it is configured to aggregate cross-level spatial features at the same spatial location. After applying scale-aware and spatial-aware attention, task-aware attention is applied within the channels of the 3D tensor, where it is configured to generalize different representations of objects. Multiple feature maps are thus transformed into different activations based on the different object detection tasks to be performed.

[0064] At 710, the computing device performs object detection on the image data using multiple updated feature maps. For example, as described herein, object detection is performed using recalibrated feature maps with improved detection characteristics or attributes.

[0065] Therefore, in some examples, method 700 can be used to perform object detection using rescaled and reshaped tensors. Using rescaled and reshaped tensors to generate output using attention layers are some examples.

[0066] Exemplary operating environment

[0067] According to Figure 8 The functional block diagram 800 in this disclosure is an example of a device that can operate in conjunction with a computing device 802. In one example, components of the computing device 802 may be implemented as part of an electronic device according to one or more embodiments described herein. The computing device 802 includes one or more processors 804, which may be a microprocessor, a controller, or any other suitable type of processor for processing computer-executable instructions to control the operation of the electronic device. Platform software, including an operating system 806 or any other suitable platform software, may be provided on the device 802 to enable application software 808 to execute on the device. According to one example, object detection 810 operating using a rescaled and reshaped tensor 812 may be implemented in software.

[0068] Computer-executable instructions can be provided using any computer-readable medium accessible to computing device 802. Computer-readable media can include, for example, computer storage media, such as memory 814, and communication media. Computer storage media, such as memory 814, include volatile and non-volatile, removable and non-removable media implemented with any method or technique for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. Conversely, communication media can embody computer-readable instructions, data structures, program modules, etc., in modulated data signals such as carrier waves or other transmission mechanisms. As defined herein, computer storage media does not include communication media. Therefore, computer storage media itself should not be construed as propagating signals. Propagated signals are not examples of computer storage media. Although the computer storage medium (memory 814) is shown within the computing device 802, those skilled in the art will understand that the storage may be distributed or located remotely and accessed via a network or other communication link (e.g., using communication interface 816).

[0069] The computing device 802 may include an input / output controller 818 configured to output information to one or more input devices 820 and output devices 822, such as a display or speaker, which may be separate from or integrated into the electronic device. The input / output controller 818 may also be configured to receive and process input from one or more input devices 820, such as a keyboard, microphone, or touchpad. In one embodiment, the output device 822 may also act as an input device 820. An example of such a device could be a touch-sensitive display. The input / output controller 818 may also output data to devices other than the output device 822, such as a locally connected printing device. In some embodiments, a user may provide input to (or receive output from) the input devices 820 and / or receive output from (or receive output from) the output devices 822.

[0070] In some examples, computing device 802 detects voice input, user gestures, or other user actions and provides a natural user interface (NUI). This user input can be used to create electronic ink, view content, select ink controls, play videos with electronic ink overlays, and for other purposes. In some examples, input / output controller 818 outputs data to a device other than a display device, such as a locally connected printing device.

[0071] The functions described herein can be performed, at least in part, by one or more hardware logic components. According to one embodiment, computing device 802 is configured by program code to perform examples and implementations of the described operations and functions when executed by (a plurality of) 804. Alternatively or additionally, the functions described herein can be performed, at least in part, by one or more hardware logic components. Examples, but not limited to, illustrative types of hardware logic components that can be used include FPGAs, ASICs, ASSPs, SOCs, CPLDs, and GPUs.

[0072] At least some of the functions of the various elements in the accompanying drawings can be performed by other elements in the drawings or entities not shown in the drawings (e.g., processors, network services, servers, applications, computing devices, etc.).

[0073] Although described in conjunction with exemplary computing system environments, the examples of this disclosure can be implemented using many other general-purpose or special-purpose computing system environments, configurations, or devices.

[0074] Examples of known computing systems, environments, and / or configurations suitable for use with any aspect of this disclosure include, but are not limited to, mobile or portable computing devices (e.g., smartphones), personal computers, server computers, handheld (e.g., tablets) or laptop devices, multiprocessor systems, game consoles or controllers, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or wearable or accessory-type communication devices (e.g., watches, glasses, headphones or earphones), network PCs, microcomputers, mainframe computers, distributed computing environments including any of the aforementioned systems or devices, etc. In general, this disclosure can operate with any device having processing capabilities, enabling it to execute instructions such as those described herein. Such systems or devices can accept input from users in any manner, including input devices such as keyboards or pointing devices, input via gestures, proximity input (such as by hover), and / or input via voice.

[0075] Examples of this disclosure can be described in the general context of computer-executable instructions, such as program modules executed by one or more computers or other devices, including software, firmware, hardware, or combinations thereof. Computer-executable instructions can be organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. Aspects of this disclosure can be implemented with any number and organization of such components or modules. For example, aspects of this disclosure are not limited to the specific computer-executable instructions or specific components or modules shown in the figures and described herein. Other examples of this disclosure may include different computer-executable instructions or components having more or fewer functions than those shown and described herein.

[0076] In examples involving general-purpose computers, when configured to execute the instructions described herein, aspects of this disclosure transform a general-purpose computer into a special-purpose computing device.

[0077] Other examples include:

[0078] A computerized method for object detection, the computerized method comprising:

[0079] - Generate a feature pyramid corresponding to the image data;

[0080] - Rescale the feature pyramid to a scale corresponding to the median level of the feature pyramid, wherein the rescaled feature pyramid is a four-dimensional (4D) tensor;

[0081] - Reshape the 4D tensor into a three-dimensional (3D) tensor, which corresponds to scale features, spatial features and task features, which correspond to an overall view of multiple feature maps;

[0082] - Use the 3D tensor and multiple attention layers to update the multiple feature maps; and

[0083] - Perform object detection on the image data using the updated feature maps. Other examples include:

[0084] A system for object detection, the system comprising:

[0085] -At least one processor; and

[0086] - At least one memory including computer program code, said at least one memory and said computer program code being configured to utilize said at least one processor to cause said at least one processor to:

[0087] - Generate a feature pyramid corresponding to the image data;

[0088] - Rescale the feature pyramid to a scale corresponding to the median level of the feature pyramid, wherein the rescaled feature pyramid is a four-dimensional (4D) tensor;

[0089] - Reshape the 4D tensor into a three-dimensional (3D) tensor, the 3D tensor having a separate perspective including scale features, spatial features and task features corresponding to different dimensions of the 3D tensor;

[0090] - Use the 3D tensor and multiple attention layers to update multiple feature maps associated with the image data; and

[0091] - Perform object detection on the image data using the updated feature maps. Other examples include:

[0092] One or more computer storage media having computer-executable instructions for object detection, which, when executed by a processor, cause the processor to at least:

[0093] - Receive a three-dimensional (3D) tensor from the backbone network, which is generated based on the feature pyramid;

[0094] - A dynamic head with multiple attention mechanisms is combined with the 3D tensor to simultaneously perform multiple object detection tasks on image data, the multiple attention mechanisms including multiple attention patterns from different viewpoints arranged sequentially; and

[0095] - Output image data of the detected objects obtained by performing the plurality of object detection tasks.

[0096] Alternatively, or in addition to the examples above, examples include any combination of the following: - wherein the rescaling includes using either upsampling or downsampling to resize continuous-level features according to a scale of the median level of the features to transform the feature pyramid into the overall view.

[0097] -The reshaping includes combining the height and width dimensions of the 4D tensor as spatial features in the overall view.

[0098] -The 4D tensor corresponding to the rescaled feature comprises multiple levels and is represented by dimensions corresponding to the height, width, and number of channels of the median-level feature.

[0099] - Where scale-aware attention, spatial-aware attention, and task-aware attention correspond to the height, width, and number of channels of the median-level feature, respectively.

[0100] -The use of the 3D tensor and multiple attention layers to update the multiple feature maps includes:

[0101] - Apply scale-aware attention processing to the level dimension of the 3D tensor, the scale-aware attention processing being configured to dynamically fuse scale features from different scales;

[0102] - Apply spatial awareness processing to the spatial dimensions of the 3D tensor, the spatial awareness processing being configured to aggregate cross-level spatial features at the same spatial location; and

[0103] - After deploying the scale-aware attention processing and the spatial-aware attention processing, task-aware processing is applied within the channels of the 3D tensor, wherein the task-aware processing is configured to generalize different representations of objects, and wherein the plurality of feature maps are remodeled into different activations based on different object detection tasks to be performed.

[0104] -The object detection performed on the image data using multiple updated feature maps includes at least one computer vision task.

[0105] Any range or device value given herein can be extended or changed without losing the desired effect, as will be apparent to those skilled in the art.

[0106] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

[0107] It should be understood that the benefits and advantages described above may apply to one instance or multiple instances. These examples are not limited to those that solve any or all of the stated problems, nor are they limited to those that have any or all of the stated benefits and advantages. It should also be understood that a reference to 'one' project refers to one or more of these projects.

[0108] The examples described herein, as well as those not specifically described herein but within the scope of the claims, constitute exemplary means for training neural networks. One or more processors 1004 shown, together with computer program code stored in memory 1014, constitute exemplary processing means for fusing multimodal data.

[0109] In this specification, the term "comprising" means including (multiple) features or (multiple) actions that follow, without excluding the presence of one or more additional features or actions.

[0110] In some examples, the operations shown in the figures can be implemented as software instructions encoded on a computer-readable medium, hardware programmed or designed to perform the operations, or both. For example, aspects of this disclosure can be implemented as a system-on-a-chip or other circuitry comprising multiple interconnected conductive elements.

[0111] Unless otherwise stated, the execution or order of the operations in the disclosed examples shown and described herein is not required. That is, operations may be performed in any order unless otherwise stated, and examples of this disclosure may include more or fewer operations than those disclosed herein. For example, specific operations are expected to be performed before, simultaneously with, or after another operation within the scope of various aspects of this disclosure.

[0112] When introducing elements or examples of aspects of this disclosure, the articles “a,” “an,” “the,” and “described” are intended to indicate the presence of one or more elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that other elements besides those listed may exist. The term “exemplary” is intended to mean “an example.” The phrase “one or more of the following: A, B, and C” means “at least one A and / or at least one B and / or at least one C.”

[0113] The phrase “one or more of the following: A, B, and C” means “at least one A and / or at least one B and / or at least one C”. The phrase “and / or” as used in the specification and claims should be understood to refer to “one or two” of the elements so connected, i.e., elements that exist in combination in some cases and separately in others. Multiple elements listed with “and / or” should be interpreted in the same way, i.e., “one or more” of the elements so connected. Other elements may optionally exist besides those specifically identified by the “and / or” clause, whether or not they are related to those specifically identified. Thus, as a non-limiting example, when used in conjunction with open-ended language such as “comprising,” a reference to “A and / or B” can mean, in one implementation, only A (optionally including elements other than B); in another implementation, only B (optionally including elements other than A); in yet another implementation, both A and B (optionally including other elements); and so on.

[0114] As used in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, meaning it includes multiple elements or at least one of the elements in the list, but also includes more than one, and optionally includes other items not listed. Only terms that explicitly indicate the opposite, such as "only one" or "exactly one of them," or when used in a claim as "consisting of...", will refer to multiple elements or exactly one of the elements in the list. Generally, when preceded by an exclusive term, such as "any one," "one of them," "only one," or "exactly one," the term "or" should only be interpreted as indicating an exclusive choice (i.e., "one or the other, but not both at the same time"). The phrase "mainly composed of..." as used in the claims should have the general meaning in the field of patent law.

[0115] As used in the specification and claims, when referring to a list of one or more elements, the phrase "at least one" should be understood to mean at least one element selected from any one or more elements in the list, but does not necessarily include at least one of each element specifically listed in the list, and does not exclude any combination of elements in the list. This definition also allows for the optional presence of elements other than those specifically identified in the list referred to by the phrase "at least one," whether related to or unrelated to those specifically identified elements. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B," or equivalently "at least one of A and / or B") could mean, in one implementation, at least one (optionally including more than one) A, without B (and optionally including elements other than B); in another implementation, at least one (optionally including more than one) B, without A (and optionally including elements other than A); in yet another implementation, at least one (optionally including more than one) A, and at least one (optionally including more than one) B (and optionally including other elements); and so on.

[0116] Having described in detail the various aspects of this disclosure, it will be apparent that modifications and variations are possible without departing from the scope of the various aspects of this disclosure as defined in the appended claims. Since various changes can be made to the above-described structures, products, and methods without departing from the scope of the various aspects of this disclosure, it is intended that all content contained in the foregoing description and shown in the accompanying drawings be interpreted as illustrative rather than restrictive.

Claims

1. A computerized method for object detection, the computerized method comprising: Generate a feature pyramid corresponding to the image data; The feature pyramid is rescaled to a scale corresponding to the median level of the feature pyramid, wherein the rescaled feature pyramid is a four-dimensional 4D tensor; The 4D tensor is reshaped into a three-dimensional 3D tensor, which has a separate perspective including scale features, spatial features and task features corresponding to different dimensions of the 3D tensor, wherein the dimensions of the 3D tensor include level dimension, spatial dimension and channel dimension. The 3D tensor and multiple attention layers are used to update multiple feature maps associated with the image data, wherein the multiple attention layers include scale-aware attention corresponding to the level dimension of the 3D tensor, spatial-aware attention corresponding to the spatial dimension of the 3D tensor, and task-aware attention corresponding to the channel dimension of the 3D tensor. The updated feature maps are used to perform object detection on the image data; as well as An output signal is sent based on the object detection performed on the image data.

2. The computerized method of claim 1, wherein the rescaling includes using either upsampling or downsampling to resize sequential level features according to a scale of the median level of the feature pyramid to convert the feature pyramid into an overall view.

3. The computerized method of claim 1, wherein the reshaping includes combining the height and width dimensions of the 4D tensor.

4. The computerized method of claim 1, wherein the 4D tensor corresponding to the median level of the rescaled feature pyramid comprises multiple levels and is represented by dimensions corresponding to the height, width, and number of channels of the median level feature.

5. The computerized method of claim 1, wherein updating the plurality of feature maps using the 3D tensor and the plurality of attention layers comprises: Scale-aware attention processing is applied to the level dimension of the 3D tensor, and the scale-aware attention processing is configured to dynamically fuse scale features from different scales. Spatial awareness processing is applied to the spatial dimension of the 3D tensor, and the spatial awareness processing is configured to aggregate cross-level spatial features at the same spatial location. as well as After applying the scale-aware attention processing and the spatial-aware processing, task-aware processing is applied within the channels of the 3D tensor, wherein the task-aware processing is configured to generalize different representations of objects, and wherein the plurality of feature maps are remodeled into different activations based on different object detection tasks to be performed.

6. The computerized method of claim 1, wherein performing object detection on the image data using the updated plurality of feature map pairs comprises at least one computer vision task.

7. The computerized method of claim 1, wherein the 4D tensor includes dimensions corresponding to the height, width, and number of channels of the median level of the feature pyramid.

8. A system for object detection, the system comprising: At least one processor; as well as At least one memory including computer program code, the at least one memory and the computer program code being configured to utilize the at least one processor to cause the at least one processor to: Generate a feature pyramid corresponding to the image data; The feature pyramid is rescaled to a scale corresponding to the median level of the feature pyramid, wherein the rescaled feature pyramid is a four-dimensional 4D tensor; The 4D tensor is reshaped into a three-dimensional 3D tensor, which has a separate perspective including scale features, spatial features and task features corresponding to different dimensions of the 3D tensor, wherein the dimensions of the 3D tensor include level dimension, spatial dimension and channel dimension. The 3D tensor and multiple attention layers are used to update multiple feature maps associated with the image data, wherein the multiple attention layers include scale-aware attention corresponding to the level dimension of the 3D tensor, spatial-aware attention corresponding to the spatial dimension of the 3D tensor, and task-aware attention corresponding to the channel dimension of the 3D tensor. The updated feature maps are used to perform object detection on the image data; as well as An output signal is sent based on the object detection performed on the image data.

9. The system of claim 8, wherein rescaling the feature pyramid comprises using either upsampling or downsampling to resize consecutive-level features according to a scale at the median level of the feature pyramid to convert the feature pyramid into a global view.

10. The system of claim 8, wherein reshaping the 4D tensor includes combining the height dimension and the width dimension of the 4D tensor.

11. The system of claim 8, wherein the 4D tensor corresponding to the rescaled feature pyramid comprises multiple levels and is represented by dimensions corresponding to the height, width, and number of channels of the median level of the feature pyramid.

12. The system of claim 8, wherein updating the plurality of feature maps using the 3D tensor and the plurality of attention layers comprises: Scale-aware attention processing is applied to the level dimension of the 3D tensor, and the scale-aware attention processing is configured to dynamically fuse scale features from different scales. Spatial awareness processing is applied to the spatial dimension of the 3D tensor, and the spatial awareness processing is configured to aggregate cross-level spatial features at the same spatial location. as well as After applying the scale-aware attention processing and the spatial-aware processing, task-aware processing is applied within the channels of the 3D tensor, wherein the task-aware processing is configured to generalize different representations of objects, and wherein the plurality of feature maps are remodeled into different activations based on different object detection tasks to be performed.

13. The system of claim 8, wherein performing object detection on the image data using the updated plurality of feature maps comprises performing at least one computer vision task.

14. The system of claim 8, wherein the 4D tensor includes dimensions corresponding to the height, width, and number of channels of the median level of the feature pyramid.

15. A computerized method for object detection, the computerized method comprising: Generate a feature pyramid corresponding to the image data; The feature pyramid is rescaled into a 4D tensor based on the median level of the feature pyramid. The 4D tensor includes a level dimension L representing the number of levels in the feature pyramid, a height dimension H representing the height of the median level of the feature pyramid, a width dimension W representing the width of the median level of the feature pyramid, and a channel dimension C representing the number of channels in the median level of the feature pyramid. The 4D tensor is reshaped into the 3D tensor by setting the spatial dimension S of the 3D tensor to be equal to the product of the W dimension and the H dimension of the 4D tensor, wherein the 3D tensor has the same L dimension and C dimension as the 4D tensor. The feature map associated with the image data is updated by applying scale-aware attention only to the L dimension of the 3D tensor, spatial-aware attention only to the S dimension of the 3D tensor, and task-aware attention only to the C dimension of the 3D tensor. Object detection is performed on the image data using the updated feature map; and The output signal is sent based on the object detection performed on the image data.

16. The computerized method of claim 15, wherein rescaling the feature pyramid to the 4D tensor comprises upsampling to resize the feature pyramid according to the median level of the feature pyramid to convert the feature pyramid into a global view.

17. The computerized method of claim 15, wherein rescaling the feature pyramid to the 4D tensor comprises downsampling to resize the feature pyramid according to the median level of the feature pyramid to convert the feature pyramid into a global view.

18. The computerized method of claim 15, wherein updating the feature map comprises: Scale-aware attention processing is applied to the level dimension of the 3D tensor, and the scale-aware attention processing is configured to dynamically fuse scale features from different scales. Spatial awareness processing is applied to the spatial dimension of the 3D tensor, and the spatial awareness processing is configured to aggregate cross-level spatial features at the same spatial location. as well as After applying the scale-aware attention processing and the spatial-aware processing, task-aware processing is applied within the channels of the 3D tensor, wherein the task-aware processing is configured to generalize different representations of objects, and wherein the feature maps are remodeled into different activations based on different object detection tasks to be performed.

19. The computerized method of claim 15, wherein performing object detection on the image data using the updated feature map comprises performing at least one computer vision task.

20. The computerized method of claim 15, wherein the output signal includes an image having detected an object.

Images