A detection transformer for object detection in digital images based on subgroup partitioning of the set of object requests.

Abstract

Method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network (N1) adapted to determine features within the image, an encoder (NE) to generate information from these features, and a decoder comprising a self-attention layer (SL), the decoder being adapted to generate predictions (P1, P2…PN) from object queries (Q1, Q2…QN) based on information (EO) provided by the encoder (NE), in which the queries are structured as subgroups (G1, G2…GK), the learning comprising a search for matching the predictions of each subgroup with the same training set. Figure for the abstract: Fig. 2

Description

Title of the invention: Detection transformer for object detection in digital images based on partitioning into subgroups of the set of object queries FIELD OF THE INVENTION

[0001] The present invention relates to the field of computer vision, and more specifically to the field of object detection in a digital image.

[0002] One of the goals of object detection in a digital image is to predict a set of bounding boxes in the image, each containing a detected object, and to assign to each box a class corresponding to the type of detected object.

[0003] Artificial intelligence (AI) has revolutionized the field of computer vision, enabling machines to perceive and understand the visual world at an unprecedented level.

[0004] Different technologies have been implemented to solve this general problem.

[0005] In 2014, regional convolutional neural networks or “RCNN” (for “Region-based Convolutional Neural Network”) were proposed. They consist of two main stages, and are therefore classified into a family of two-stage detectors (or “Two-stage-detectors” in English). The approach first calculates a list of proposals of areas of interest in the form of a bounding box in a stage called “region proposals”, then, in a second stage, these regions are transmitted to a convolutional neural network (or “CNN”) to locate and classify the objects.

[0006] In parallel, another category of one-stage detector algorithms is being developed. Among the best known is the “YOLO” architecture, which is presented in its first version in Redmon, J., Divvala, SK, Girshick, RB, & Farhadi, A. (2015). “You Only Look Once: Unified, Real-Time Object Detection”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779-788. Yolo has achieved very fast performance by removing the region proposal step and replacing it with a single network that handles end-to-end detection. The accuracy remained lower than the “Two-stage-detectors” methods, but with the evolution of “YOLO” to its fourth version in 2020, many improvements have been made to improve accuracy.

[0007] More recently, it has been proposed to use the transformer mechanism, already used in the separate field of speech recognition and machine translation, for object detection.

[0008] One of the first successful attempts to integrate transformers into computer vision was the Vision Transformer (ViT) model, introduced in Do-sovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit and Neil Houlsby. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” ArXiv abs / 2010.11929 (2020).

[0009] The ViT model treats an image as a sequence of non-overlapping pixel blocks (or "patches") and applies a standard transformer architecture to process these patches. This approach has demonstrated competitive performance on image classification tasks, even outperforming state-of-the-art CNNs in some cases.

[0010] Following the success of ViT, several other transformer-based models have been proposed for various computer vision tasks.

[0011] For example, the DETR model, introduced in Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. “End-to-End Object Detection with Transformers.” ArXiv abs / 2005.12872 (2020) exploits transformers for object detection and achieves competitive results compared to traditional object detection methods.

[0012] This DETR model (for “DEtection Transformer” in English) is based on an architecture of 4 neural networks: a first network processes the digital image to provide a map of characteristics, this map is provided to a second neural network, or encoder, whose output is provided to a third neural network, or decoder.

[0013] The decoder also takes object queries as input in order to predict, in collaboration with a fourth neural network, a result (bounding box associated with a class) for each query, using the information provided by the decoder.

[0014] Many developments of this DETR model have been proposed more recently, in particular to specify how to determine queries and their influence on predicted results.

[0015] For example, it has been proposed in Chen, Qiang, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng and Jingdong Wang. “Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment.” (2022) to present groups of queries during training in order to assign a single label to each group, which leads to as many positive queries as there are actually objects in the digital image.

[0016] However, despite the large number of groups used in learning, it turns out that only part of the information is actually used. Indeed, only a only one group of queries is used during the inference (or prediction) phase, thus under-exploiting the information from the learning phase.

[0017] There is therefore a need to improve the current proposals of the state of the art to maximize the potential of the query set, in order to improve performance in object detection in digital images. Summary of the invention

[0018] For these purposes, according to a first aspect, the present invention can be implemented by a method for learning a model for detecting objects in a digital image, said model being of the detection transformer type and comprising a first neural network adapted to determine characteristics within said digital image, an encoder for generating information from said characteristics and a decoder comprising a self-attention layer, said decoder being adapted to generate predictions from object queries as a function of information provided by said encoder, in which said object queries are structured in the form of subgroups forming a partition of all of said queries, said learning comprising a search for matching the predictions of each subgroup with the same training set

[0019] According to preferred embodiments, the invention comprises one or more of the following features which can be used separately or in partial combination with each other or in total combination with each other: - said self-attention layer is adapted to allow independent learning for each subgroup. - an attention mask is applied to said self-attention layer, said attention mask being intended to allow the transmission of information only within a subgroup. - a cost function is provided to optimize the diversification of said object requests. - said cost function is based on a supervised correlation matrix during said learning so that its values ​​are between two predefined limits.

[0020] Another aspect relates to a method for detecting objects within a digital image, said method comprising a learning method as previously described.

[0021] According to one embodiment, said digital image comes from a stream of images generated by a video camera.

[0022] Another aspect of the invention relates to a computer program capable of being implemented on an access point to a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, performs the steps as previously described.

[0023] Another aspect of the invention relates to a processing device comprising means for implementing the method as previously described.

[0024] Other characteristics and advantages of the invention will appear on reading the following description of a preferred embodiment of the invention, given by way of example and with reference to the appended drawings. BRIEF DESCRIPTION OF THE FIGURES

[0025] The attached drawings illustrate the invention: [Fig.l] illustrates an example of implementation of a method according to an embodiment of the invention;

[0026] [Fig.2] diagrams an architecture of a possible implementation of a detection transformer according to embodiments of the invention;

[0027] [Fig.3] illustrates a decoder associated with a detection layer, according to an embodiment of the invention, during the learning phase;

[0028] [Fig.4] illustrates an example of an attention matrix according to an embodiment of the invention.

[0029] DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0030] [Fig.l] illustrates an example of implementation of a method according to an embodiment of the invention.

[0031] This example illustrates a digital image IMG in which two objects are represented: a dog, 01 and a kennel, 02.

[0032] The method aims to detect these objects, that is to say on the one hand to determine a bounding box, respectively PI, P2, corresponding to each of these objects, and, for each object to determine an associated class.

[0033] Bounding boxes are geometric shapes (usually rectangular) containing the detected object. The method also aims to optimize these boxes, i.e. to find the minimal surface box enclosing the object.

[0034] The determination of the classes depends on the learning that has been carried out on a labeled training set. It therefore consists of assigning to the object detected in a bounding box a class corresponding to one of the labels learned during the learning phase.

[0035] In the illustrated example, it can be assumed that the training set included many images representing dogs, of different breeds, in different positions and scales. The neural network is then able to generalize the different representations associated with this same label "dog" so as to be able to associate a new representation of the “dog” class.

[0036] Such a mechanism can be used in different tasks of detecting objects in digital images. In particular, it can be applied to detection in sequences of digital images, or video.

[0037] These digital images can be acquired by appropriate sensors (photographic or video camera, possibly infrared camera, etc.), and processed by a processing platform comprising at least one microprocessor, one or more memories and possible additional electronic circuits, provided to, in collaboration, implement the proposed method.

[0038] In particular, the platform can process the images acquired in real time in order to enable the detection of objects in real time or near real time.

[0039] For example, transformer-based models have been used for tasks such as video understanding, scene understanding, and 3D point cloud processing.

[0040] Many uses are possible for mechanisms for detecting objects in images or in image streams, for example in the sports field, but also in any other field (video surveillance, etc.)

[0041] The detection of an object in an image or in a sequence of images (or videos) can make it possible to position a marker on the detected object, or any other information associated with the object and for example recoverable via a database.

[0042] Depending on the use cases, and / or the choices of a user, different types of information can thus be displayed superimposed on the image (or images) and positioned so that the user can clearly associate this information with the detected object.

[0043] Several objects can be processed in parallel: for example, in a race, or in a team sport match, the names of the different participants can be displayed superimposed and opposite their position on the image.

[0044] Also, a human-machine interface can be provided so that the user can select a particular object (player, horse, bicycle, car, etc.) and follow it throughout the event (race, match, etc.) by means of a particular marker allowing it to be distinguished from other objects.

[0045] Another example of use is the extraction of detected objects to feed virtual event models (races, matches, etc.), in particular for realistic video games, or for analysis or reporting videos based on the data thus collected.

[0046] Particularly interesting scenes can also be extracted to be broadcast again or modeled, or even marketed in other forms (NFT, etc.).

[0047] In the field of sports racing, for example, it is possible to detect runners,

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] bicycles, cars, or horses, crossing a finish line. For example, for a horse race, such a mechanism can make it possible to: - identify all the horses in a race in real time to specify to a user their positions on a video in a precise manner by integrating some metadata such as speed, name, age, race history, etc. via a cross-referencing with a database, - detect the starters in a race in order to determine if all the horses are ready to start, - detect anomalies during a race, for example if the right number of horses are on the track, - detect and count people present in the racecourse, - collect data on a horse's positioning on the racecourse throughout the race in order to increase knowledge on horse behavior, - use this knowledge to display a “ghost” horse during another race in order to allow a visual comparison between the behavior of the same horse during different races, - build statistical databases from a set of races in order to determine indicators on horses and jockeys, These examples can obviously be applied to other types of races: car races, bicycle races, running (100 meters, marathon, etc.), etc. In team sports (football, rugby, etc.), individual players can also be tracked, as mentioned above, for various applications. Other types of objects can also be detected (balls, etc.). A race ranking can be deduced automatically and immediately, thus facilitating race management for an operator. The proposed object detection method is based on the DETR model It is proposed to improve the DETR model learning process, in order to create a more efficient model allowing, in the prediction phase, better detection of objects in the submitted digital images. The proposed method is independent of the training set. It simply needs to associate a set of digital images with labels or "ground truth". These labels provide a basis for comparison for the predictions made by the transformer for each image. The errors resulting from these comparisons are used to iteratively modify the state of the transformer (the various internal parameters, or "synaptic weights") until convergence. The training set may be a publicly available base, such as example the Coco database distributed by Microsoft (for “Common Object in Context”). This database was described in the article Lin, Tsung-Yi, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar and C. Lawrence Zitnick. “Microsoft COCO: Common Objects in Context.” European Conference on Computer Vision (2014).

[0057] As mentioned previously, the DETR (DEtection Transformer) model was introduced by the article by Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. “End-to-End Object Detection with Transformers.” ArXiv abs / 2005.12872 (2020).

[0058] [Fig.2] illustrates an architecture of a possible implementation of a DETR detection transformer.

[0059] Such a DETR transformer can be broken down into three main components: - a first component Cl comprising a first neural network, NI, designed to determine a set of characteristics for each digital image submitted as input; - a second component C2 comprising a second neural network of the NE encoder - ND decoder type; - a third component C3 comprising a set of neural networks (detection layer) N3 intended to determine a detection prediction from the output layer of the decoder of the second neural network N2.

[0060] According to one embodiment, such a DETR transformer can be implemented on an artificial intelligence platform having the functionality of proposing classic neural network models, such as convolutional neural networks and encoder-decoder neural networks.

[0061] The article by Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andréas Kôpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai and Soumith Chintala. “PyTorch: An Imperative Style, High-Performance Deep Learning Library” Neural Information Processing Systems (2019), proposes a library providing components to easily implement a DETR architecture in about fifty lines of code.

[0062] The first NI neural network is typically a convolutional neural network, CNN (for “Convolutional Neural Network” in English).

[0063] This first NI neural network aims to extract a FM characteristic map or “feature map” according to the usual terminology in English, from a digital image submitted to it.

[0064] The digital image can be represented in a CoxHoxWo space, with Ho, Wo representing, respectively, the height and width of the image, and Co representing the number of channels per pixel, typically a coding of the color of the pixels.

[0065] The FM feature map generated by the first neural network can be represented in a CxWxH space. In general, the width W and the height H of the feature map are reduced, compared to the dimensions of the digital image. For example, W=W0 / 32 and H=Ho / 32 can be considered. The number of channels C can be larger. A typical value is C=2048.

[0066] The second component C2 corresponding to an encoder-decoder network, which can be broken down into a first sub-network, or encoder network (or more simply encoder) NE, and a second sub-network, or decoder network (or more simply decoder) ND.

[0067] The encoder network NE takes as input a vector E, constructed from the FM characteristic map and a spatial encoding of the PE positions.

[0068] The height and width of the FM feature map are therefore transformed into a one-dimensional space corresponding to the input vector Eb. Since it is desired that the transformer DETR be invariant to permutations of the input space (i.e. to the order of the requests), it is necessary to incorporate spatial data into the information managed by the encoder. Also, a positional encoding PE is inserted into the input vector Eb presented to the input layer of the encoder network NE.

[0069] The encoder network NE encodes and enriches the FM characteristic map in order to obtain an enriched characteristic map Eo at output.

[0070] The decoder network ND processes a set of object requests Qb Q2.. .QN in order to generate predictions Pb P2...PN respectively, using the information on the digital image considered as captured by the output Eo of the encoder network NE.

[0071] Each prediction is processed independently by a neural network from a set of N neural networks, in order to determine a (final) detection prediction from the output layer of the ND decoder. These neural networks are, in the original DETR architecture, feed forward networks (FFN).

[0072] For each object request Qb Q2.. .QN the transformer therefore generates a prediction. This prediction can be provided to a detection layer, N3 in order to generate a final prediction consisting - either into a predicted bounding box and a class corresponding to the type of object detected (positive query), or, - either in a class corresponding to the non-detection of an object or a score of extremely low detection (negative query).

[0073] This N3 subnetwork is typically a forward propagation network (FFN).

[0074] The aforementioned article "End-to-End Object Detection with Transformers" provides more details on the architecture, training method and inference method of the DETR detection transformer.

[0075] Since the publication of this article, much work has been undertaken to improve the DETR transformer. Much of this work aims to improve the performance of DETR, particularly in terms of convergence speed, and also to understand the role that object queries can play in the behavior of the ND decoder and to study how to use them to improve the overall performance of the transformer.

[0076] In particular, work consists of determining the content of object requests.

[0077] The very architecture of a DETR detection transformer requires that these queries have common features with the encoder input, and therefore contain an encoding of positions. Generally speaking, therefore, object queries contain a position from which the ND decoder (in collaboration with the N3 detection layer) must attempt to predict an optimized bounding box and a class of a detected object.

[0078] Various works have been proposed to specify the position encodings of the requests allowing to improve the performances of the ND decoder in learning and / or in prediction (or inference).

[0079] For example, the query can be a vector of characteristics, the coordinates of a bounding box (not optimized and therefore the learning of which will consist of evolving the coordinates), a point coordinate (which can evolve during learning and the dimensions of the bounding box of which are also sought), etc.

[0080] It should be noted that the proposed method can be adapted to different embodiments concerning the structure of object requests and is therefore independent of these requests.

[0081] In learning, we seek to optimize the decoder parameters so that it generates the correct predictions from the object requests and the encoder outputs.

[0082] It was found that learning is sensitive to the number of positive queries, i.e. those which correspond to the prediction of an object class and a bounding box, as opposed to negative queries which correspond to the prediction of a class corresponding to the non-detection of an object.

[0083] The following article, previously cited, has particularly studied this phenomenon: Chen, Qiang, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng and Jingdong Wang. “Group DETR: Fast DETR Training with Group-Wise One- to-Many Assignment.” (2022).

[0084] This paper finds that a large number of positive queries improves learning performance but that the number of queries in general improves performance only slightly. It would therefore be appropriate to increase the ratio of positive to negative queries.

[0085] As we also want to avoid assigning several positive predictions to the same label, which is typically done in “One-to-many” approaches, leading to the need for a post-processing filtering step to remove multi-predictions, the article proposes to group the object queries by assigning several positive queries per label but by decoupling these assignments into several independent groups, so that we only have the assignment of a positive query to a given label in each of the groups.

[0086] In inference, only a single group is kept, in order to eliminate redundancy of the learned characteristics.

[0087] The inventors found that, in doing so, all of the predictions made by the ND decoder during the learning phase are not taken into account. In particular, the impact of negative queries is not taken into account in the trained model, nor even studied in the scientific literature.

[0088] One of the goals is therefore to take advantage of negative queries in the learning process in order to optimize the model.

[0089] [Fig.3] illustrates an ND decoder associated with a detection layer N3, according to an embodiment of the invention, during the learning phase.

[0090] This ND decoder is mainly made up of a first self-attention layer, SL, and a second cross-attention layer, CL.

[0091] The operation of these attention layers, SL, CL can be consistent with the mechanisms of a detection transformer DETR, as described in the seminal article or in multiple subsequent developments. This operation was initially described in the article by Vaswani, Ashish, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. “Attention is All you Need”, NIPS (2017).

[0092] According to the proposed method, the object queries Qb Q2... QN are partitioned into K subgroups. In other words, each query is associated with one and only one subgroup Gi.

[0093] If we consider N object queries, we can therefore define the set of K subgroups Gi, G2...Gk with:

[0094] [Math.l] ^i = [ )^-1 ('

[0095] In the example of [Fig.3], sub-subgroups Gb G2 are represented, the first subgroup is made up of the queries Qi to Q; and the second subgroup G2 is made up of the queries Qi+i to QN.

[0096] Conventionally, each request for objects Qb Q2... QN is processed by the decoder ND then by the detection layer N3 to capture the information provided by the encoder NE.

[0097] According to the proposed method, during the learning phase, the predictions corresponding to each subgroup G; are compared with the set of GT labels (for “ground truth”) of the learning set.

[0098] This confrontation aims to form a one-to-one bipartite pairing (“one-to-one assignment” in English).

[0099] This matching can be seen as an assignment problem. This type of problem can be formalized as finding the best assignment of tasks to agents. Each agent can perform a single task for a given cost and each task must be performed by a single agent. The assignments (i.e., the agent-task pairs) all have a defined cost. The goal is to minimize the total cost of the assignments in order to perform all the tasks.

[0100] More formally, the objective is to determine a matching of a size equal to the number of tasks, of minimum weight in a valued bipartite graph. If there are as many agents as tasks, it is a question of determining a perfect matching of minimum weight in a valued bipartite graph. The assignment problem can be solved in polynomial time by the Hungarian algorithm.

[0101] This type of problem is well described in the scientific literature, of which the Wikipedia page is a good entry point:

[0102] https: / / en.wikipedia.org / wiki / Assignment_problem

[0103] The bipartite matching between Pi P2...PN predictions and GT labels can be implemented by a Hungarian algorithm. The Hungarian algorithm, classically used in DETR detection transformers, was proposed in the article by Harald W. Kuhn, "The Hungarian Method for the assignment problem" in Naval Research Logistic Quarterly, 2 (1955), pp. 83-97.

[0104] In other words, the approach is very different from that of the “Group DETR” technique explained previously in which the set of queries is artificially increased since here this set of queries is constant but the labels of the training set are compared with a plurality of subgroups independently of each other.

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] [YES]

[0112]

[0113]

[0114]

[0115]

[0116] This confrontation is carried out by an LM matching mechanism which aims to among the predictions produced at the output of the N3 detection layer, we search for those which are closest to the corresponding label, we use the selected predictions, corresponding to positive requests, to calculate an error according to a loss function (or "loss function" in English) and thus update the RN decoder network and the N3 detection layer. For each subgroup, we therefore have a set of predictions, each forming a tuple formed of a bounding box prediction and a class prediction cl (i varying in the interval corresponding to the subgroup concerned). Labels are also a set of tuples, made up of boxes in global b; and classes Cj (i varying on the training set). We seek to match each subgroup with the training set. More precisely, learning includes a search for matching predictions of each subgroup with the set of labels from the training set. In the Hungarian algorithm that can be used in learning, we therefore seek an optimal matching ^Gk for each subgroup Gk, on the set E of per possible mutations J being the number of queries in each subgroup). [Math.2] °Gk = argminE* Cmateh ( yy ) / a \ represents the matching cost between the label y; and the yq index prediction Each label y; can be seen as having two components, a class c; and a bounding box b; : y;=(Ci, b;). Similarly, each prediction can be written: a _ hGk\ yq^y-qi' Uqi ] We can then write the matching cost for a subgroup Gk as: [Math.3] = + X hox box(b(i}, bÿ)] In this equation, - M is the number of labels in the training set - Cls() and box() are cost functions for calculating a distance between, respectively, two classes and two bounding boxes. - ^cis and 4c.x are two coefficients, forming adjustable parameters and which can be predefined, of the algorithm.

[0117] This equation expressing the matching cost for a subgroup Gk '■'match clearly shows that each subgroup Gk is matched against the set of available labels, M, in order to increase the number of positive queries.

[0118] In this way, we perform a subgroup matching between the predictions of subgroup Gk and the set of duplicated ground truth labels for subgroup Gk.

[0119] According to one embodiment of the invention, it is sought to ensure learning independently for each subgroup Gk.

[0120] To do this, according to one embodiment, an attention mask AM is applied to the self-attention layer SL. This attention mask is intended to allow the transmission of information within a subgroup but not between distinct subgroups.

[0121] Such an attention mask AM can take the form of a matrix whose rows and columns form the N different object requests Qb Q2... Qn provided as input to the self-attention layer SL.

[0122] This attention matrix AM can be constituted so that each cell has a value allowing to block or allow the exchange of information between the query on the abscissa and the query on the ordinate, depending on whether or not the queries belong to the same subgroup.

[0123] [Fig.4] illustrates an example of an attention matrix for 4 subgroups, Gb G2, G3, G4.

[0124] The gray areas correspond to the cells associated with queries belonging to the same subgroups, while the white areas correspond to cells associated with queries belonging to distinct subgroups. The gray areas are associated with a numerical value (1) allowing, when calculating the outputs of the self-attention layer, the taking into account of queries from the same subgroup, and the white areas are associated with a numerical value (0) allowing the exclusion of the queries concerned in these same calculations.

[0125] According to one embodiment, the attention matrix AM is added to the attention generated by the similarity calculation between the query Q and the key K, generated by the object query, normalized by a scale factor.

[0126] This mechanism forces many queries to collectively identify a single object, thereby reducing the prevalence of negative queries to a collection of redundant queries that collectively increase confidence in the presence of objects. Facilitating robust learning, this procedure results in increased efficiency.

[0127] Nevertheless, during the optimization phase of the paired object queries, a predisposition to the homogenization of the characteristics appears. To exploit the potential for seamless convergence of features among these object queries, according to one embodiment, a diversification strategy is introduced. This strategic intervention, implemented in the query space, serves to increase the distinctiveness of the query.

[0128] To do this, a new cost function is proposed, which can be used in conjunction with the other cost functions during the learning phase, according to one embodiment of the invention.

[0129] This cost function is based on a correlation matrix corr(). This correlation matrix gives a correlation value between -1 and 1 for each prediction pair P; at the output of the detection transformer.

[0130] The correlation matrix can be supervised during the learning phase so that the values ​​(outside the diagonal) are always between two predefined limits ai, ai forming a constraint (with ai <a2).

[0131] To do this, we can define a cost function of the following form with the aim of optimizing the diversification of object requests:

[0132] [Math.4] ieOffDiag

[0133] Dec; represents the output of the neural network N3 and therefore the prediction that we seek to match with the GT labels during this learning phase.

[0134] As previously mentioned, the N3 subnetwork may be a forward propagation network (FFN).

[0135] We can then write

[0136] [Math.5] FF Ni softmaxi —7==---- x (FW V ) \ ' \ Fk /

[0137] F represents the feature map, and Wq, Wk and Wv are synaptic weight matrices. dk is the dimension of the key.

[0138] The correlation calculation can be carried out in accordance with known methods and for example made available on neural network platforms such as Pytorch.

[0139] Information is available for example on the following links:

[0140] https: / / pytorch.org / docs / stable / generated / torch.corrcoef.html

[0141] https: / / pytorch.org / docs / stable / generated / torch.cov.html

[0142] Other examples of calculating correlation terms are of course accessible to those skilled in the art.

[0143] The global cost function L, used during the learning phase, can be Yj=xcorr(Decb Dec;) -a^ ! DeCi = nonn a linear combination of this new cost function with the cost functions usually used

[0144] L = LDOg 4- LcZy +

[0145] with:

[0146] Lc / s the cost function evaluating the classification,

[0147] ^box the cost function evaluating the prediction of a bounding box

[0148] Lnog the cost function described above evaluating the diversification of object requests.

[0149] According to one embodiment, the learning can be subdivided into two phases: - In a first phase, conventional learning can be carried out on all of the subgroups, then - In a second phase, learning is implemented separately for each subgroup, using the cost function described previously. For each subgroup, different ab (^) values ​​are used.

[0150] According to one embodiment, a conventional non-maximum suppression (NMS) mechanism is used to counteract redundancy in object queries.

[0151] This redundancy provides an incentive to clearly separate predictions into two discernible groups: one consisting of related predictions, possibly forming multiple groups, and the other encompassing disparate predictions. Incorporating NMS in this context effectively mitigates redundancy issues and amplifies the effectiveness of our proposed approach. However, the conventional greedy NMS technique has detrimental effects on the detections themselves. This is mainly due to the standard NMS approach, which ranks predictions in descending order of their scores.

[0152]

[0153]

[0154] On the other hand, in the case of two subgroups, the method we propose involves the generation of a cost matrix (Cpred) using the object center (CCenterX) class (Cdass) and box size (Csize) parameters: Cpred Xbox(Ccenter4“Csize) + Xciass(Cciass) This approach facilitates the formation of groups containing identical boxes with class attributes, with weighting parameters Xbox = 5 and X _ O class respectively —

[0155] In the case of N groups, we partition the predictions into N groups using the top-N on the scores of each object query. Subsequently, the NMS is applied individually to each of these prediction groups, resulting in a more refined and efficient suppression process.

[0156] The proposed method, in its various embodiments, makes it possible to take advantage negative requests during the learning phase of a DETR detection transformer.

[0157] Through experimental studies, it has been considered that instances classified as negative based on their prediction scores often exhibit superior prediction coordinates when compared to the ground truth values ​​(labels). Interestingly, these instances outperform even the most reliable queries. Leveraging this insight, an approach is presented that exploits the potential of negative queries by strategically forming diverse query subgroups. This innovative strategy not only improves object detection in different scenarios but also provides the ability to detect objects across multiple instances.

[0158] The following table gives performance indicators for different transformers in DETR detection.

[0159] The “Deformable DETR” transformer (Deformable DETR) is an evolution of the DETR transformer described in the article Zhu, Xizhou, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang and Jifeng Dai. “Deformable DETR: Deformable Transformers for End-to-End Object Detection.” ArXiv abs / 2010.04159 (2020)

[0160] The “Conditional DETR” transformer is an evolution of the DETR transformer described in the article Meng, Depu, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun and Jingdong Wang. “Conditional DETR for Fast Training Convergence.” 2021IEEE / CVF International Conference on Computer Vision (ICCV) (2021): 3631-3640.

[0161] The proposed mechanisms can be implemented both with the original detection transformer and with its evolutions.

[0162] Each line, corresponding to a distinct implementation, includes a number of iterations allowing convergence to be achieved and average performance indicators AP (for “Average Performance”), globally (AP) for small detected objects (APS), for medium objects (APM) and for large objects (APL).

[0163] [Tables 1] Process Iterations AP APS APm apl DETR 500 42.0 62.4 45.8 61.1 Deformable DETR 12 38.3 58.0 41.7 51.4 Deformable DETR + propositions 12 39 58.4 42.3 51.9 Conditional DETR 12 36.6 57.3 39.4 53.4 Conditional DETR + propositions 12 37.1 58 39.6 54

[0164] The various mechanisms proposed therefore allow a substantial improvement object detection evaluation metrics based on the COCO training set used for testing.

[0165] The proposed method is further applicable to other transformer models for object detection in digital signals, for example in images or image sequences.

[0166] Of course, the present invention is not limited to the examples and the embodiment described and shown, but is defined by the claims. It is in particular susceptible of numerous variants accessible to those skilled in the art.

Claims

Claims

1. Method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network (NI) adapted to determine characteristics within said digital image, an encoder (NE) for generating information from said characteristics and a decoder comprising a self-attention layer (SL), said decoder being adapted to generate predictions (Pb P2.. .PN) from object queries (Qb Q2...QN) as a function of information (Eo) provided by said encoder (NE), in which said object queries are structured in the form of subgroups (Gb G2.. .Gk) forming a partition of all of said queries, said learning comprising a search for matching the predictions of each subgroup with the same training set.

2. Method according to the preceding claim, wherein said self-attention layer (SL) is adapted to allow independent learning for each subgroup.

3. Method according to the preceding claim, in which an attention mask (AM) is applied to said self-attention layer (SL), said attention mask being provided to allow the transmission of information only within a subgroup.

4. Method according to the preceding claim, in which a cost function is provided to optimize the diversification of said object requests.

5. Method according to the preceding claim, wherein said cost function is based on a supervised correlation matrix during said learning so that its values are between two predefined limits.

6. Method for detecting objects within a digital image, said method comprising a learning method according to one of the preceding claims.

7. Method according to one of the preceding claims in which said digital image comes from a stream of images generated by a video camera.

8. A computer program capable of being implemented on an access point to a wireless telecommunications network, the program comprising

9. Code instructions which, when executed by a processor, perform the steps of the method defined in claims 1 to 7. Processing device comprising means for implementing the method according to one of claims 1 to 7.

Images