Object detection transformer for detecting objects in digital images with selective routing of object requests within the decoder

Abstract

A method for learning an object detection model in an image, said model being of the detection transformer type and comprising a first neural network (N1) adapted to determine features, an encoder (NE) to generate information from these features, and a decoder comprising a succession of layers including a self-attention layer (SL), a cross-attention layer (CL), and a detection layer (N3), the decoder being adapted to generate predictions (P1, P2…PN) from queries (Q1, Q2…QN) based on information (EO) provided by the encoder (NE), and wherein additional layers are trained so that a portion of the outputs of one layer of the decoder are passed as input to the subsequent layer of the succession, such that the complementary portion bypasses at least one layer of the succession. Figure for the abstract: Fig. 2

Description

Title of the invention: Detection transformer for object detection in digital images with selective routing of object requests within the decoder. FIELD OF THE INVENTION

[0001] The present invention relates to the field of computer vision, and more specifically to the field of object detection in a digital image.

[0002] One of the goals of object detection in a digital image is to predict a set of bounding boxes in the image, each containing a detected object, and to assign to each box a class corresponding to the type of object detected.

[0003] Artificial intelligence (AI) has revolutionized the field of computer vision, enabling machines to perceive and understand the visual world at an unprecedented level.

[0004] Various technologies have been implemented to solve this general problem.

[0005] In 2014, regional convolutional neural networks, or “RCNNs” (for “Region-based Convolutional Neural Network”), were proposed. They consist of two main stages and are therefore classified in a family of two-stage detectors. The approach first calculates a list of proposed regions of interest in the form of a bounding box in a stage called “region proposals.” Then, in a second stage, these regions are passed to a convolutional neural network (or “CNN”) to locate and classify the objects.

[0006] In parallel, another category of one-stage detector algorithms is being developed. Among the best known is the “YOLO” architecture, which was first presented in Redmon, J., Divvala, SK, Girshick, RB, & Farhadi, A. (2015). “You Only Look Once: Unified, Real-Time Object Detection”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779-788. YOLO achieved very fast speeds by eliminating the region proposal step and replacing it with a single network that handles end-to-end detection. Accuracy remained lower than that of two-stage detector methods, but with the evolution of YOLO to its fourth version in 2020, numerous improvements were made to increase accuracy.

[0007] More recently, it has been proposed to use the transformer mechanism, already used in the separate field of speech recognition and machine translation, for object detection.

[0008] One of the first successful attempts to integrate transformers into computer vision was the Vision Transformer (ViT) model, introduced in Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit and Neil Houlsby. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, ArXiv abs / 2010.11929 (2020).

[0009] The ViT model treats an image as a sequence of non-overlapping pixel blocks (or "patches") and applies a standard transformer architecture to process these patches. This approach has demonstrated competitive performance on image classification tasks, even surpassing leading CNNs in some cases.

[0010] Following the success of ViT, several other transformer-based models have been proposed for various computer vision tasks.

[0011] For example, the DETR model, introduced in Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko, “End-to-End Object Detection with Transformers”, ArXiv abs / 2005.12872 (2020) exploits transformers for object detection and obtains competitive results compared to traditional object detection methods.

[0012] This DETR model (for "DEtection Transformer" in English) is based on an architecture in 4 neural networks: a first network processes the digital image to provide a feature map, this map is provided to a second neural network, or encoder, whose output is provided to a third neural network, or decoder.

[0013] The decoder also takes object queries as input in order to predict, in collaboration with a neural decoding layer, a result (i.e. a bounding box associated with a class) for each query, using the information provided by the decoder.

[0014] Many developments of this DETR model have been proposed more recently, in particular to specify how to determine the queries and their influence on the predicted results.

[0015] In particular, it is observed that information redundancy between object requests can generate negative impacts on their evolution within the decoder and consequently on the performance of the DETR model. Thus, via the decoder's attention layers, irrelevant requests can negatively influence relevant requests.

[0016] The article by Tharsan Senthivel, Ngoc-Son Vu, Boris Borzic. “Detection Transformer with Diversified Object Queries”, IEEE ICIP 2023, Oct 2023, Kuala Lampur, Malaysia, ffhal-04304226, in particular, explains and proposes solutions to this problem of information redundancy in object queries.

[0017] Other research undertaken to study how to select object queries can also be cited, such as Chen, F., Zhang, H., Hu, K., Huang, YK, Zhu, C., & Savvides, M. (2023), “Enhanced training of query-based object detection via selective query recollection” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 23756-23765), or Zhengdong Hu, Yifan Sun, Jingdong Wang, Yi Yang, “DAC-DETR: Divide the Attention Layers and Conquer” in Advances in Neural Information Processing Systems 36 (NeurIPS 2023).

[0018] Manually selecting object queries based on their average performance (AP) in the detection mechanism improves the overall performance of the DETR model, but such manual selection is difficult to implement in practice in a context other than academic research.

[0019] There is therefore a need to improve current state-of-the-art proposals to make the best use of the information conveyed by the different object queries according to their relevance for prediction. Summary of the invention

[0020] To this end, according to a first aspect, the present invention can be implemented by a method for learning a model for detecting objects in a digital image, said model being of the detection transformer type and comprising a first neural network adapted to determine features within said digital image, an encoder for generating information from said features and a decoder comprising a succession of layers including a self-attention layer, a cross-attention layer and a detection layer, said decoder being adapted to generate predictions from object queries based on information provided by said encoder, and wherein additional layers are trained so that a portion of the outputs of at least one layer of said decoder are transmitted as input to the subsequent layer of said succession,so that the complementary part bypasses at least one layer of said succession,

[0021] According to preferred embodiments, the invention comprises one or more of the following features which can be used separately or in partial combination with each other or in total combination with each other: - said additional layers include a routing layer adapted to determine a score for each of said object requests, and a grouping layer adapted to transmit the corresponding output to the subsequent layer or bypass said at least one layer of said succession according to said score. - said additional layers include a reordering layer adapted to reorder the forecasts according to the ordering of object queries. - Routing and grouping layers are located at the beginning of said succession, and adapted to allow bypassing said self-attention layer. - Routing and grouping layers are located immediately after said self-attention layer, and adapted to allow bypassing said cross-attention and detection layers. - Routing and grouping layers are located at the beginning of said succession and adapted to allow bypassing said succession. - said digital image is derived from a stream of images generated by a video camera.

[0022] Another aspect relates to a method for detecting objects within a digital image, said method comprising a learning method as previously described.

[0023] According to one embodiment, said digital image is derived from a stream of images generated by a video camera.

[0024] Another aspect of the invention relates to a computer program suitable for implementation on an access point to a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, performs the steps as previously described.

[0025] Another aspect of the invention relates to a processing device comprising means for implementing the process as previously described.

[0026] Other features and advantages of the invention will become apparent from the following description of a preferred embodiment of the invention, given by way of example and with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE FIGURES

[0027] The attached drawings illustrate the invention: Figure [1] illustrates an example of the implementation of a process according to one embodiment of the invention;

[0028] [Fig.2] schematically illustrates an architecture of a possible implementation of a detection transformer according to embodiments of the invention;

[0029] Fig. 3 illustrates the evolution of the average accuracy as a function of the number of object requests for a DETR transformer decoder.

[0030] [Fig.4] illustrates a decoder associated with a detection layer, according to an embodiment of the invention, during the learning phase;

[0031] Figures 5a, 5b, 5c illustrate examples of embodiments of decoder models incorporating routing mechanisms as proposed

[0032] DETAILED DESCRIPTION OF EMBODIMENT MODES OF THE INVENTION

[0033] The [Fig. 1] illustrates an example of implementation of a process according to an embodiment of the invention.

[0034] This example illustrates a digital image IMG in which two objects are represented: a dog, 01 and a kennel, 02.

[0035] The method aims to detect these objects, that is to say on the one hand to determine a bounding box, respectively PI, P2, corresponding to each of these objects, and, for each object determine an associated class.

[0036] Bounding boxes are geometric shapes (generally rectangular) containing the detected object. The process also aims to optimize these boxes, that is, to find the box with the smallest surface area encompassing the object.

[0037] The determination of the classes depends on the learning that has been carried out on a labeled training set. It therefore consists of assigning to the object detected in a bounding box a class corresponding to one of the labels learned during the learning phase.

[0038] In the illustrated example, it can be assumed that the training set contained numerous images representing dogs of different breeds, in different positions and at different scales. The neural network is then able to generalize the different representations associated with the same label "dog" so as to be able to associate a new representation with the class "dog".

[0039] Such a mechanism can be used in various object detection tasks in digital images. In particular, it can be applied to detection in sequences of digital images, or video.

[0040] These digital images can be acquired by appropriate sensors (photographic or video camera, possibly infrared camera, etc.), and processed by a processing platform comprising at least one microprocessor, one or more memories and possible additional electronic circuits, intended to implement the proposed process in collaboration.

[0041] In particular, the platform can process images acquired continuously in order to enable real-time or near-real-time object detection.

[0042] For example, transformer-based models have been used for tasks such as video understanding, scene understanding, and 3D point cloud processing.

[0043] Many uses are possible for object detection mechanisms in images or image streams, for example in the sports field, but also in any other field (video surveillance, etc.)

[0044] The detection of an object in an image or in a sequence of images (or videos) can make it possible to position a marker on the detected object, or any other information associated with the object and, for example, retrievable via a database.

[0045] Depending on the use cases, and / or the choices of a user, different types of information can thus be displayed superimposed on the image (or images) and positioned so that the user can properly associate this information with the detected object.

[0046] Several objects can be processed in parallel: for example in a race, or in a match of a team sport, the names of the different participants can thus be displayed superimposed and alongside their position on the image.

[0047] Also, a human-machine interface can be provided so that the user can select a particular object (player, horse, bicycle, car...) and follow it throughout the event (race, match...) by means of a particular marker allowing it to be distinguished from other objects.

[0048] Another example of use is the extraction of detected objects to feed models of virtual events (races, matches, etc.), particularly for realistic video games, or for analysis or reporting videos based on the data thus collected.

[0049] Particularly interesting scenes can also be extracted for rebroadcast or modeled, or even marketed in other forms (NFT...).

[0050] In the field of sports racing, for example, one can detect runners, bicycles, cars, or horses crossing a finish line.

[0051] For example, in a horse race, such a mechanism can: - identify all the horses in a race in real time to specify their positions on a video to a user precisely, integrating some metadata such as speed, name, age, race history, etc., via cross-referencing with a database, - to identify the starters in a race in order to determine if all the horses are ready to start, - to detect anomalies during a race, for example, whether the correct number of horses are on the track, - to detect and count the people present in the racecourse, - to collect data on a horse's positioning on the racecourse throughout the race in order to increase knowledge about horse behavior, - to use this knowledge to display a "ghost" horse during another race in order to allow a visual comparison between the behaviors of the same horse during different races, - to build statistical databases from a set of races in order to determine indicators on horses, jockeys,

[0052] These examples can obviously be applied to other types of races: car races, bicycle races, foot races (100 meters, marathon...), etc.

[0053] In the context of team sports (football, rugby, etc.), it is also possible to track the different players as mentioned previously, for various applications. It is also possible to detect other types of objects (ball, etc.).

[0054] This allows for the automatic and immediate classification of a race, thus facilitating race management for an operator.

[0055] The proposed object detection method is based on the DETR model

[0056] It is proposed to improve the learning process of the DETR model, in order to constitute a more efficient model allowing, in the prediction phase, better detection of objects in the submitted digital images.

[0057] The proposed method is independent of the training set. The latter only has to associate a set of digital images with labels or "ground truth".

[0058] These labels provide a basis for comparison for the predictions established by the transformer for each image. The errors resulting from these comparisons allow the transformer's state (the various internal parameters, or "synaptic weights") to be iteratively modified until convergence.

[0059] The training set can be a publicly available database, such as the Coco database distributed by Microsoft (for "Common Object in Context"). This database was described in the article Lin, Tsung-Yi, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar and C. Lawrence Zitnick. “Microsoft COCO: Common Objects in Context.” European Conference on Computer Vision (2014).

[0060] As mentioned previously, the DETR model (for “DEtection Transformer” in English, i.e. Detection Transformer) was introduced by the article by Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. “End-to-End Object Detection with Transformers.” ArXiv abs / 2005.12872 (2020).

[0061] Fig. 2 illustrates an architecture of a possible implementation of a DETR detection transformer.

[0062] Such a DETR transformer can be broken down into three main components: - a first component Cl comprising a first neural network, NI, designed to determine a set of features for each digital image submitted as input; - a second component C2 comprising a second neural network of the type NE encoder - ND decoder; - a third component C3 comprising a set of neural networks (detection layer) N3 intended to determine a detection prediction from the output layer of the decoder of the second neural network N2.

[0063] According to one embodiment, such a DETR transformer can be implemented on an artificial intelligence platform having the functionality of proposing classical neural network models, such as convolutional neural networks and encoder-decoder neural networks.

[0064] The article by Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andréas Kôpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai and Soumith Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” Neural Information Processing Systems (2019), proposes a library providing components that allow for the easy implementation of a DETR architecture in about fifty lines of code.

[0065] The first NI neural network is typically a convolutional neural network, CNN (for “Convolutional Neural Network” in English).

[0066] This first neural network N1 aims to extract a feature map FM or "feature map" according to the usual English language terminology, from a digital image submitted to it.

[0067] The digital image can be represented in a CoxHoxWo space, with Ho, Wo representing, respectively, the height and width of the image, and Co representing the number of channels per pixel, typically a color encoding of the pixels.

[0068] The FM feature map generated by the first neural network can be represented in a CxWxH space. In general, the width W and height H of the feature map are reduced compared to the dimensions of the digital image. For example, W = W0 / 32 and H = H0 / 32 can be considered. The number of channels C can be larger. A typical value is C = 2048.

[0069] The second component C2 corresponds to an encoder-decoder network, which can be decomposed into a first subnetwork, or encoder network (or more simply encoder) NE, and a second subnetwork, or decoder network (or more simply decoder) ND.

[0070] The NE encoder network takes as input a vector E, constructed from the FM feature map and a spatial encoding of the PE positions.

[0071] The height and width of the FM feature map are thus transformed into a one-dimensional space corresponding to the input vector Eb. Since it is desired that the DETR transformer be invariant to permutations of the input space (i.e., to the order of requests), it is necessary to incorporate spatial data into the information managed by the encoder. Therefore, a positional PE encoding is inserted into the input vector Eb presented to the input layer of the encoder network NE.

[0072] The NE encoder network encodes and enriches the FM characteristic map in order to obtain an enriched Eo characteristic map at output.

[0073] The ND decoder network processes a set of object requests Qb Q2.. .QN in order to generate predictions Pb P2...PN respectively, using information on the digital image considered as captured by the output Eodu encoder network NE.

[0074] Each prediction is processed independently by a neural network from a set of N neural networks, in order to determine a (final) detection prediction from the output layer of the ND decoder. These neural networks are, in the original DETR architecture, feedforward networks (FFNs).

[0075] For each query of objects Qb Q2.. .QN, the transformer therefore generates a prediction. This prediction can be provided to a detection layer, N3, in order to generate a final prediction consisting of - either in a predicted bounding box and a class corresponding to the type of object detected (positive query), or, - either in a class corresponding to the non-detection of an object or an extremely low detection score (negative query).

[0076] This N3 subnetwork is typically a forward-propagating (FFN) network.

[0077] This N3 subnetwork is sometimes considered to be part of the C2 decoder. In the following, the network comprising a succession of layers including a self-attention layer, a cross-attention layer and an N3 detection layer will be called a "decoder".

[0078] The aforementioned article "End-to-End Object Detection with Transforms" provides further details on the architecture, learning process and inference process of the DETR detection transformer.

[0079] Since the publication of this article, much work has been undertaken to improve the DETR transformer. Much of this work aims to improve the performance of DETR, particularly in terms of convergence speed, and also to understand the role that object queries can play in the behavior of the ND decoder and to study how to use them in order to improve the overall performance of the transformer.

[0080] In particular, work consists of determining the content of object queries.

[0081] The very architecture of a DETR detection transformer requires that these Requests have common characteristics with the encoder input, and therefore contain a positional encoding. Generally speaking, then, object requests contain a position from which the ND decoder (in collaboration with the N3 detection layer) must attempt to predict an optimized bounding box and a class of a detected object.

[0082] Various works have been proposed to specify the position encodings of the queries allowing to improve the performance of the ND decoder in learning and / or in prediction (or inference).

[0083] For example, the query can be a feature vector, the coordinates of a bounding box (not optimized and therefore whose learning will consist of evolving the coordinates), a point coordinate (which can evolve in learning and whose dimensions of the bounding box are also sought), etc.

[0084] It should be noted that the proposed method can be adapted to different embodiments concerning the structure of object queries and is therefore independent of these queries.

[0085] In learning, we seek to optimize the parameters of the decoder so that it generates the correct predictions from object requests and the outputs of the encoder.

[0086] This learning process is based on object requests provided as input to the ND / N3 decoder. As will be seen in more detail later, this decoder consists of a succession of layers, including attention layers. These attention layers allow information sharing between the different object requests, influencing their convergence towards predictions along their path through the succession of layers of the decoder / detector.

[0087] The inventors have undertaken experimental studies to better understand the interactions between the different layers and object requests.

[0088] It follows that the best results are obtained through a combination of self-attention and cross-attention layers, as these two layers cooperate in the convergence of object predictions based on object queries. It also follows that a decrease in the number of object queries has a limited impact on the average accuracy of the prediction result: thus, a decrease from 300 to 150 queries resulted in a degradation of the average accuracy of 0.5 points.

[0089] The curve in [Fig.3] illustrates the evolution of the average precision (on the ordinate) as a function of the number of object queries (on the abscissa).

[0090] We observe that the average precision, or AP, decreases slightly as the number of object queries decreases until it reaches excessively low values. At this point, the average precision collapses. Therefore, when a certain threshold is not exceeded, decreasing the number of object queries does not significantly affect the average precision.

[0091] Studies show that it may be possible to manipulate a number of object queries, for example, to select the most relevant ones: excluding some of them will not, in itself, have a substantially negative impact on overall performance (less than 2% variation between 300 and 100). Furthermore, this negative impact can easily be offset by the fact that the correct selection of queries allows for a gain in average accuracy (AP).

[0092] It also appears from this study that not all queries are necessary for the proper functioning of the DETR model.

[0093] Figure 4 illustrates an ND decoder, according to one embodiment of the invention, during the learning phase.

[0094] This ND decoder comprises a succession of layers including a first self-attention layer, SL, a second cross-attention layer, CL, and a third detection layer N3, typically a forward-propagating network (FFN).

[0095] The operation of these attention layers, SL, CL, can conform to the mechanisms of a detection transformer DETR, as described in the seminal article or in the multiple subsequent developments. This operation was initially described in the article by Vaswani, Ashish, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is All You Need,” NIPS (2017).

[0096] Classically, each object request Qb Q2... QN forming a set Q of object requests, is processed by the succession of layers (SL, CL, N3) of the decoder to capture the information Eo provided by the encoder NE.

[0097] Object requests are first submitted to a first layer of self-attention, SL.

[0098] The outputs of this first SL layer feed an input Q (for "Query" in English) of the second cross-attention layer, CL.

[0099] The Eo information retrieved from the encoder is, for its part, provided to inputs K (for "Key" or key) and V (for "Value" or values) of this same cross-attention layer CL.

[0100] The resulting predictions from this cross-attention layer CL are then provided to a detection layer N3, which provides the final predictions Pb P2>... PN.

[0101] These final predictions can be used in the prediction phase, or provided to an LM learning mechanism, in the training or learning phase.

[0102] During the training phase, the predictions corresponding to each object request are compared to the GT (ground truth) labels of the training set. In [Fig. 4], the LM box represents this step of comparing the labels and the predictions from the last layer of the detector. Depending on the comparison result, a learning mechanism modifies the parameters of the different layers of the decoder (in particular the synaptic weights).

[0103] Various mechanisms have been proposed for this comparison step, and, more generally, for the learning phase. Therefore, different cost functions (or "loss functions") can be proposed.

[0104] In particular, the comparison may aim to form a one-to-one bipartite matching (“one-to-one assignment” in English).

[0105] This matching can be viewed as an assignment problem. This type of problem can be formalized as finding the optimal allocation of tasks to agents. Each agent can perform a single task at a given cost, and each task must be performed by a single agent. The assignments (i.e., the agent-task pairs) all have a defined cost. The goal is to minimize the total cost of the assignments in order to perform all the tasks.

[0106] More formally, the objective is to determine a matching of a size equal to the number of tasks, with minimum weight in a weighted bipartite graph. If there are as many agents as tasks, the goal is to determine a perfect matching of minimum weight in a weighted bipartite graph. The assignment problem can be solved in polynomial time using the Hungarian algorithm.

[0107] This type of problem is well described in the scientific literature, of which the Wikipedia page is a good starting point:

[0108] http s: / / en. wikipedia. org / wiki / As signment_problem

[0109] The bipartite matching between predictions Pi P2.. .PNet and GT labels can be implemented by a Hungarian algorithm. The Hungarian algorithm, classically used in DETR detection transformers, was proposed in the article by Harald W. Kuhn, "The Hungarian Method for the assignment problem" in Naval Research Logistic Quarterly, 2 (1955), pp. 83-97.

[0110] Furthermore, proposals have also been made to group object queries and to match each group of corresponding predictions to a label GT. We can refer for example to the article Chen, Qiang et al. “Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment” 2023 IEEE / CVF International Conference on Computer Vision (ICCV) (2022): 6610-6619.

[0111] In general, the cost function £, used during the learning phase, can be a linear combination of several cost functions. For example: — ^cls + -^box

[0112] with: £cls the cost function evaluating the classification, £box 'a cost function evaluating the prediction of a bounding box

[0113] The details of the calculation of cost functions are widely described in the scientific literature. They are also the subject of freely available computer programs in library form, for example in the Pytorch software library.

[0114] It is perfectly possible here to use the cost functions classically used for classic DETR models, namely: - For classification: a cost function of the "focal loss" type, or focal loss in French - For the prediction of the bounding box: a cost function based on GloU (for "Generalized Intersection over Union") and L1 (or absolute loss).

[0115] The "GloU" is a metric used to evaluate the similarity between two bounding boxes, improving the IoU (for "Intersection over Union") by also considering the minimum bounding area that covers both boxes. The L1 cost function measures the sum of the absolute values ​​of the differences between the predicted and actual values.

[0116] The proposed method can be applied to different learning schemes and to all cost functions. Since these aspects are accessible to those skilled in the art, they will not be further detailed in this description.

[0117] Similarly, the interactions between the different layers of the decoder, and between the decoder and the encoder, are readily accessible to those skilled in the art. Besides the seminal article on the DETR transformer, numerous subsequent publications describe the architecture and its many variants in greater detail and / or in a more didactic manner.

[0118] As previously mentioned, not all queries are necessary for the proper functioning of the DETR model. It can therefore be assumed that The set Q of object queries forms a partition of a subset QR of relevant queries and a subset Q1RR of irrelevant queries: 101191 2=^ ej

[0120] However, there are no metrics to identify whether an object query is relevant or not. In particular, because an object query encodes both position and image data, it does not seem possible to determine a sufficiently stable metric.

[0121] According to the proposed learning process, a routing mechanism is proposed allowing the model to learn a transmission of object requests among one or more layers of the succession of layers constituting the decoder.

[0122] In particular, according to the proposed method, additional layers are driven so that a portion of the outputs of at least one layer of the decoder are transmitted as input to the subsequent layer of said succession, so that the complementary portion bypasses at least one layer of said succession.

[0123] In other words, the proposed routing allows a given layer to transmit an object request either to the subsequent layer in the succession of layers (which corresponds to the normal operation of the DETR model), or to bypass one or more layers in the succession.

[0124] Thus, through learning, certain object requests may not be submitted to certain layers, such as the self-attention layer SL or the cross-attention layer CL.

[0125] These additional layers can be placed between each pair of successive layers of the decoder, that is to say in particular at the PI, P2, P3 interfaces shown in [Fig.4].

[0126] According to one embodiment, these additional layers comprise - a suitable RTR routing layer to determine a score for each object request, - a GTH grouping layer adapted to transmit the output corresponding to said request to the subsequent layer (according to the normal operation of the DETR model) or to bypass said at least one layer of the succession according to the score determined by the routing layer. - A suitable SCT reordering layer to reorder forecasts (which therefore come from several different previous layers) according to the ordering of object queries

[0127] This score can take any form (numeric, binary value...) allowing the correct routing of an object request.

[0128] According to one embodiment, the RTR routing layers are linear layers projecting object queries into a 2-dimensional space to determine a "score": score = linear{Q^ = (Wp GR^5*2

[0129] e Rdx2 is the learnable projection matrix, d being the encoding dimension (or "em bedding" in English). Nq is the number of object queries Q={qo , qi...qNq}. Typically Nq=300.

[0130] The WR matrix is ​​the only additional set of parameters added compared to the DETR models.

[0131] By applying max and sigmoid functions to these scores, one can obtain binarized oEE indices allowing the routing of object queries to bypass or not subsequent Transformer layers: has EE — maxÇsigmoid(score)) G

[0132] Consequently, the set of aEE indices (where EE stands for "Entry-Exit" in English) forms a partition of two disjoint subsets: represents the set of object query indices intended for the next layer of the Transformer (according to the succession of layers) and represents the set of object query indices intended to bypass the next layer of the Transformer.

[0133] We can denote Qs and QP the object queries corresponding, respectively, to the index sets and ^p.

[0134] The grouping layer, GTH (for "Gathering" in English), is a layer that allows the selection of object queries based on indices. According to one embodiment, we therefore have: Çs = Gt7i«?, <Ts) = Ç(fe}) = (¢0-5(0)^05(1),...^^(^)) e

[0135] Ns represents the number of queries for Qs objects selected to be passed to the next layer in the succession of layers of the Transformer.

[0136] The reordering layer, SCT (for "scattering"), is a layer that allows object queries to be correctly placed back into their initial positions. We can write: Q = Sct{Q s , = (¢0-5(0).¢05(1),-?»5(WQ5)) 6

[0137] The routing and grouping layers can be considered as a single module, i.e., co-located on the same interface PI, P2, P2, with the grouping layer immediately following the routing layer. From a point of view Technically, however, it may involve two distinct neuronal layers targeting different functions.

[0138] These layers can therefore allow different paths for object queries through the succession of layers of the decoder, these paths being learned during the training phase. Thus, object queries will be processed differently by the decoder depending on their relevance to the final prediction.

[0139] The SCT reordering layer is optional but allows for further improvement of the performance of the proposed DETR model.

[0140] Indeed, the GTH grouping layer modifies the arrangement of the queries with each other, since they are grouped into two subsets according to their destination (subsequent layer or bypass).

[0141] Experimental studies have been conducted to compare average accuracies with and without this SCT reordering layer: a much higher average accuracy (43.1) is shown with the addition of this reordering layer than without (19.1).

[0142] [Fig.5a] illustrates an embodiment in which routing, RTR, and grouping, GTH layers are located at the beginning of the succession (i.e. at the PI interface on [Fig.4]), and adapted to allow bypassing the self-attention layer SL.

[0143] Thus, a first part of the object requests are passed to the self-attention layer SL, and a second (complementary) part is passed to the cross-attention layer CL without going through the self-attention layer. The input to the cross-attention layer therefore consists of all the object requests (either processed by the self-attention layer or coming directly from the decoder input).

[0144] Preferably, an SCT reordering layer is provided to reorder the predictions from the SL self-attention and GTH grouping layers. This reordering layer is positioned just after the bypassed layer, i.e. immediately after the SL self-attention layer, which corresponds to the P2 interface in [Fig.4].

[0145] Self-attention layers facilitate query communication, enabling the elimination of duplicate predictions and the sharing of redundant data. To address this issue, a routing module is proposed that bypasses the self-attention layer. This allows object queries to be updated directly from feature maps (FMs), without sharing information between queries at different levels.

[0146] Figure 5b illustrates another embodiment in which RTR routing and GTH grouping layers are located immediately after the auto-seam layer attention SL, and adapted to allow bypassing of the cross attention CL and N3 detection layers.

[0147] “immediately after” should be understood here in accordance with the usual order of succession layers in a DETR model. This is the P2 interface on [Fig.4].

[0148] Preferably, an SCT reordering layer is provided to reorder the predictions from the GTH grouping layer and the last bypassed layer, i.e. immediately after the N3 detection layer, which corresponds to the P3 interface on the [Fig.4].

[0149] In general, cross-attention (CL) layers allow object queries to capture information from encoded feature maps, Eo (i.e., information passed by the encoder), thereby updating their content. However, while all object queries are updated, only a subset is supervised, either by the final loss or the auxiliary losses calculated in each stack. Consequently, some queries are over-supervised, i.e., they are supervised multiple times across different stacks. As illustrated in [Fig. 5b], the proposed routing mechanism allows queries to bypass the cross-attention layer, thus preventing them from updating their content based on the information passed by the encoder.

[0150] [Fig.5c] illustrates another embodiment in which RTR routing and GTH grouping layers are located at the beginning of the decoder layer succession, i.e. at the PI interface on [Fig.4], and adapted to allow bypassing of the entire succession (SL, CL, N3 layers), i.e. up to the P3 interface on [Fig.4].

[0151] Preferably, an SCT reordering layer is provided to reorder the predictions from the GTH grouping layer and the last bypassed layer, i.e. immediately after the N3 detection layer, which corresponds to the P3 interface on the [Fig.4].

[0152] As indicated in the above-cited article by Chen, F., Zhang, H., Hu, K., Huang, YK, Zhu, C., & Savvides, M. (2023), “Enhanced training of query-based object detection via selective query recollection”, object queries can move their associations between decoder layers, which underlines the importance of the decoder stack for the consistency of object query updates from self-attention and cross-attention modules.

[0153] These three examples illustrated by figures 5a, 5b, 5c are cumulative, that is to say that additional layers can be placed at different interfaces, and that at each interface, the mechanism can be implemented.

[0154] Thus, pairs of RTR routing and GTH grouping layers can be placed at interfaces PI, P2, each implementing a distinct routing, towards the

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167] interfaces P2 or P3. More specifically, according to the examples in figures 5a, 5b, 5c, the following workarounds are possible: PI P2 corresponding to the embodiment of [Fig.5a] P2 P3 corresponding to the embodiment of [Fig.5b] PI P3 corresponding to the embodiment of [Fig.5c] With these three routing mechanisms trained, we can predict the three possible routes combining. Thus, an input object request (PI) can be routed to the self-attention layer (SL), the cross-attention layer (P2), or the decoder output (P3). From the self-attention layer (P2), an object request can be routed to the cross-attention layer or the decoder output (P3). Experimental studies undertaken by the inventors highlight the advantages of the proposed routing mechanism. The following table compares the average AP (for "Average Precision" in English) for a DETR type model and the same model to which the proposed routing mechanism has been added. [Tables 1] Without routing With routing Deformable DETR 38.3 43.1 Conditional DETR 36.6 39.9 DAB-DETR 42.2 43.5 The “Deformable DETR” model is described in particular in the seminal article by X. Zhu, W. Su, L. Lu, B. Li, X. Wang and J. Dai, “Deformable DETR: Deformable Transformers for End-to-End Object Detection”, in International Conference on Learning Representations, 2020. The "Conditional DETR" model is described in particular in the seminal article of D. Meng, The DAB-DETR model is described in particular in the seminal article by S. Liu, F. Li, H. Zhang, X. Yang, X. Qi, H. Su, J. Zhu and L. Zhang, “DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR”, in ICLR, 2022 Regarding the different routing mechanisms, experimental studies show that the combination of mechanisms illustrated in Figures 5a, 5b and 5c presents the best results. The following table compares the average AP (for "Average Precision") for a DETR-type model and the same model with different Routing mechanisms have been added: those of figures 5a, 5b, 5c and a combination of the three.

[0168] [Tables2] DETR alone 38.3 With routing according to figure 5a 40.6 With routing according to figure 5b 32.5 With routing according to figure 5c 23.1 With a combination of these three routes 43.1

[0169] It therefore appears that a combination of the three types of routing produces the best levels of performance.

[0170] The proposed method is also applicable to other transformer designs for object detection in digital signals, for example in images or image sequences

[0171] Of course, the present invention is not limited to the examples and embodiment described and illustrated, but is defined by the claims. In particular, it is susceptible of numerous variations accessible to those skilled in the art.

Claims

Demands

1. A method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network (NI) adapted to determine features within said digital image, an encoder (NE) for generating information from said features and a decoder comprising a succession of layers including a self-attention layer (SL), a cross-attention layer (CL) and a detection layer (N3), said decoder being adapted to generate predictions (Pb P2.. .PN) from object queries (Qb Q2...Qn) as a function of information (Eo) provided by said encoder (NE), and in which additional layers are trained so that a portion of the outputs of at least one layer of said decoder are passed as input to the subsequent layer of said succession, such that the complementary portion bypasses at least one layer of said succession.

2. A learning method according to the preceding claim, wherein said additional layers comprise a routing layer (RTR) adapted to determine a score for each of said object requests, and a grouping layer (GHT) adapted to forward the corresponding output to the subsequent layer or bypass said at least one layer of said succession according to said score.

3. A learning method according to the preceding claim, wherein said additional layers comprise a reordering layer (SCT) adapted to reorder the predictions according to the ordering of object queries.

4. A learning method according to the preceding claim, wherein routing and grouping layers are located at the beginning of said sequence, and adapted to allow bypassing of said self-attention layer (SL).

5. A learning method according to any one of claims 3 or 4, wherein routing and grouping layers are located immediately after said self-attention layer (SL), and adapted to allow bypassing said cross-attention (CL) and detection (N3) layers.

6. A learning method according to any one of claims 3 to 5 wherein routing and grouping layers are located at the beginning of said sequence and adapted to allow bypassing of said sequence (SL, CL, N3).

7. A method for detecting objects within a digital image, said method comprising a learning method according to one of the preceding claims.

8. A method according to any one of the preceding claims in which said digital image is derived from an image stream generated by a video camera.

9. A computer program capable of being implemented on an access point to a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, carries out the steps of the method defined in claims 1 to 8.

10. Processing device comprising means for carrying out the process according to any one of claims 1 to 8.

Images