A detection transformer for object detection in digital images based on subgroup partitioning of the set of object requests.

Abstract

Method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network (N1) adapted to determine features within the image, an encoder (NE) to generate information from these features, and a decoder comprising a self-attention layer (SL), the decoder being adapted to generate predictions (P1, P2…PN) from object queries (Q1, Q2…QN) based on information (EO) provided by the encoder (NE), in which the queries are structured as subgroups (G1, G2…GK), the learning comprising a search for matching the predictions of each subgroup with the same training set. Figure for the abstract: Fig. 2

Description

Title of the invention: Detection transformer for object detection in digital images based on subgroup partitioning of the set of object requests. FIELD OF THE INVENTION

[0001] The present invention relates to the field of computer vision, and more specifically to the field of object detection in a digital image.

[0002] One of the goals of object detection in a digital image is to predict a set of bounding boxes in the image, each containing a detected object, and to assign to each box a class corresponding to the type of object detected.

[0003] Artificial intelligence (AI) has revolutionized the field of computer vision, enabling machines to perceive and understand the visual world at an unprecedented level.

[0004] Various technologies have been implemented to solve this general problem.

[0005] In 2014, regional convolutional neural networks, or RCNNs (for "Region-based Convolutional Neural Network"), were proposed. They consist of two main stages and are therefore classified in a family of two-stage detectors. The approach first calculates a list of proposed regions of interest in the form of a bounding box in a stage called "region proposals," and then, in a second stage, these regions are passed to a convolutional neural network (or "CNN") to locate and classify the objects.

[0006] In parallel, another category of one-stage detector algorithms is being developed. Among the best known is the “YOLO” architecture, which was first presented in Redmon, J., Divvala, SK, Girshick, RB, & Farhadi, A. (2015). “You Only Look Once: Unified, Real-Time Object Detection”. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779-788. YOLO achieved very fast speeds by eliminating the region proposal step and replacing it with a single network that handles end-to-end detection. Accuracy remained lower than that of two-stage detector methods, but with the evolution of YOLO to its fourth version in 2020, numerous improvements were made to increase accuracy.

[0007] More recently, it has been proposed to use the transformer mechanism, already used in the separate field of speech recognition and machine translation, for object detection.

[0008] One of the first successful attempts to integrate transformers into computer vision was the Vision Transformer (ViT) model, introduced in Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit and Neil Houlsby. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” ArXiv abs / 2010.11929 (2020).

[0009] The ViT model treats an image as a sequence of non-overlapping pixel blocks (or "patches") and applies a standard transformer architecture to process these patches. This approach has demonstrated competitive performance on image classification tasks, even surpassing leading CNNs in some cases.

[0010] Following the success of ViT, several other transformer-based models have been proposed for various computer vision tasks.

[0011] For example, the DETR model, introduced in Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. “End-to-End Object Detection with Transformers.” ArXiv abs / 2005.12872 (2020) exploits transformers for object detection and obtains competitive results compared to traditional object detection methods.

[0012] This DETR model (for "DEtection Transformer" in English) is based on an architecture in 4 neural networks: a first network processes the digital image to provide a feature map, this map is provided to a second neural network, or encoder, whose output is provided to a third neural network, or decoder.

[0013] The decoder also takes object queries as input in order to predict, in collaboration with a fourth neural network, a result (bounding box associated with a class) for each query, using the information provided by the decoder.

[0014] Many developments of this DETR model have been proposed more recently, in particular to specify how to determine the queries and their influence on the predicted results.

[0015] For example, it was proposed in Chen, Qiang, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng and Jingdong Wang. “Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment.” (2022) to present groups of queries during training in order to assign a single label to each group, which leads to as many positive queries as there are actually objects in the digital image.

[0016] However, despite the large number of groups used in learning, it turns out that only a portion of the information is actually used. Indeed, only one only one group of queries is used during the inference (or prediction) phase, thereby under-exploiting the information from the learning phase.

[0017] There is therefore a need to improve current prior art proposals to maximize the potential of the query set, in order to improve performance in object detection in digital images. Summary of the invention

[0018] To this end, according to a first aspect, the present invention can be implemented by a method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network adapted to determine features within said digital image, an encoder for generating information from said features, and a decoder comprising a self-attention layer, said decoder being adapted to generate predictions from object queries based on information provided by said encoder, wherein said object queries are structured as subgroups forming a partition of the set of said queries, said learning comprising a search for matching the predictions of each subgroup with the same training set.

[0019] According to preferred embodiments, the invention comprises one or more of the following features which can be used separately or in partial combination with each other or in total combination with each other: - said self-attention layer is adapted to allow independent learning for each subgroup. - an attention mask is applied to said self-attention layer, said attention mask being designed to allow the transmission of information only within a subgroup. - a cost function is planned to optimize the diversification of said object queries. - said cost function is based on a correlation matrix supervised during said learning so that its values ​​are between two predefined bounds.

[0020] Another aspect relates to a method for detecting objects within a digital image, said method comprising a learning method as previously described.

[0021] According to one embodiment, said digital image is derived from a stream of images generated by a video camera.

[0022] Another aspect of the invention relates to a computer program suitable for implementation on an access point to a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, performs the steps as previously described.

[0023] Another aspect of the invention relates to a processing device comprising means for implementing the process as previously described.

[0024] Other features and advantages of the invention will become apparent from the following description of a preferred embodiment of the invention, given by way of example and with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE FIGURES

[0025] The attached drawings illustrate the invention: Figure [1] illustrates an example of the implementation of a process according to one embodiment of the invention;

[0026] [Fig.2] schematically illustrates an architecture of a possible implementation of a detection transformer according to embodiments of the invention;

[0027] [Fig.3] illustrates a decoder associated with a detection layer, according to an embodiment of the invention, during the learning phase;

[0028] Fig. 4 illustrates an example of an attention matrix according to one embodiment of the invention.

[0029] DETAILED DESCRIPTION OF EMBODIMENT METHODS OF THE INVENTION

[0030] Fig. 1 illustrates an example of the implementation of a process according to an embodiment of the invention.

[0031] This example illustrates a digital image IMG in which two objects are represented: a dog, 01 and a kennel, 02.

[0032] The method aims to detect these objects, that is to say on the one hand to determine a bounding box, respectively PI, P2, corresponding to each of these objects, and, for each object determine an associated class.

[0033] Bounding boxes are geometric shapes (generally rectangular) containing the detected object. The process also aims to optimize these boxes, that is, to find the box with the smallest surface area encompassing the object.

[0034] The determination of the classes depends on the learning that has been carried out on a labeled training set. It therefore consists of assigning to the object detected in a bounding box a class corresponding to one of the labels learned during the learning phase.

[0035] In the illustrated example, it can be assumed that the training set contained numerous images representing dogs of different breeds, in different positions and scales. The neural network is then able to generalize the different representations associated with this same label "dog" so as to be able to associate a new representation with the class "dog".

[0036] Such a mechanism can be used in various object detection tasks in digital images. In particular, it can be applied to detection in sequences of digital images, or video.

[0037] These digital images can be acquired by appropriate sensors (photographic or video camera, possibly infrared camera, etc.), and processed by a processing platform comprising at least one microprocessor, one or more memories and possible additional electronic circuits, designed to implement the proposed process in collaboration.

[0038] In particular, the platform can process images acquired continuously in order to enable real-time or near-real-time object detection.

[0039] For example, transformer-based models have been used for tasks such as video understanding, scene understanding, and 3D point cloud processing.

[0040] Many uses are possible for object detection mechanisms in images or image streams, for example in the sports field, but also in any other field (video surveillance, etc.)

[0041] The detection of an object in an image or in a sequence of images (or videos) can make it possible to position a marker on the detected object, or any other information associated with the object and, for example, retrievable via a database.

[0042] Depending on the use case, and / or the choices of a user, different types of information can thus be displayed superimposed on the image (or images) and positioned so that the user can properly associate this information with the detected object.

[0043] Several objects can be processed in parallel: for example in a race, or in a match of a team sport, the names of the different participants can thus be displayed superimposed and alongside their position on the image.

[0044] Also, a human-machine interface can be provided so that the user can select a particular object (player, horse, bicycle, car...) and follow it throughout the event (race, match...) by means of a particular marker allowing it to be distinguished from other objects.

[0045] Another example of use is the extraction of detected objects to feed models of virtual events (races, matches, etc.), particularly for realistic video games, or for analysis or reporting videos based on the data thus collected.

[0046] Particularly interesting scenes can also be extracted for rebroadcast or modeled, or even marketed in other forms (NFT....).

[0047] In the field of sports racing, for example, one can detect runners, bicycles, cars, or horses crossing a finish line.

[0048] For example, in a horse race, such a mechanism can: - to identify all horses in a race in real time to precisely specify their positions on a video to a user, integrating metadata such as speed, name, age, race history, etc., via cross-referencing with a database, - to identify the starters in a race in order to determine if all the horses are ready to start, - to detect anomalies during a race, for example, whether the correct number of horses are on the track, - to detect and count the people present in the racecourse, - to collect data on a horse's positioning on the racecourse throughout the race in order to increase knowledge about horse behavior, - to use this knowledge to display a "ghost" horse during another race in order to allow a visual comparison between the behaviors of the same horse during different races, - to build statistical databases from a set of races in order to determine indicators on horses, jockeys,

[0049] These examples can obviously be applied to other types of races: car races, bicycle races, foot races (100 meters, marathon...), etc.

[0050] In the context of team sports (football, rugby, etc.), it is also possible to track the different players as mentioned previously, for various applications. It is also possible to detect other types of objects (ball, etc.).

[0051] This allows for the automatic and immediate classification of a race, thus facilitating race management for an operator.

[0052] The proposed object detection method is based on the DETR model

[0053] It is proposed to improve the learning process of the DETR model, in order to constitute a more efficient model allowing, in the prediction phase, better detection of objects in the submitted digital images.

[0054] The proposed method is independent of the training set. The latter only has to associate a set of digital images with labels or "ground truth".

[0055] These labels provide a basis for comparison for the predictions established by the transformer for each image. The errors resulting from these comparisons allow the transformer's state (the various internal parameters, or "synaptic weights") to be iteratively modified until convergence.

[0056] The training set can be a publicly available database, such as the Coco database distributed by Microsoft (for "Common Object in Context"). This database was described in the article Lin, Tsung-Yi, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar and C. Lawrence Zitnick. “Microsoft COCO: Common Objects in Context.” European Conference on Computer Vision (2014).

[0057] As mentioned previously, the DETR model (for “DEtection Transformer” in English, i.e. Detection Transformer) was introduced by the article by Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. “End-to-End Object Detection with Transformers.” ArXiv abs / 2005.12872 (2020).

[0058] Figure 2 illustrates an architecture of a possible implementation of a transformer DETR detection.

[0059] Such a DETR transformer can be broken down into three main components: - a first component Cl comprising a first neural network, NI, intended to determine a set of characteristics for each digital image submitted as input; - a second component C2 comprising a second neural network of the type NE encoder - ND decoder; - a third component C3 comprising a set of neural networks (detection layer) N3 intended to determine a detection prediction from the output layer of the decoder of the second neural network N2.

[0060] According to one embodiment, such a DETR transformer can be implemented on an artificial intelligence platform having the functionality of proposing classical neural network models, such as convolutional neural networks and encoder-decoder neural networks.

[0061] The article by Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andréas Kôpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai and Soumith Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” published in Neural Information Processing Systems (2019), proposes a library providing components that allow for the easy implementation of a DETR architecture in approximately fifty lines of code.

[0062] The first NI neural network is typically a convolutional neural network, CNN (for “Convolutional Neural Network” in English).

[0063] This first NI neural network aims to extract an FM feature map or "feature map" according to the usual English language terminology, from a digital image submitted to it.

[0064] The digital image can be represented in a CoxHoxWo space, with Ho, Wo representing, respectively, the height and width of the image, and Co representing the number of channels per pixel, typically a color encoding of the pixels.

[0065] The FM feature map generated by the first neural network can be represented in a CxWxH space. In general, the width W and height H of the feature map are reduced compared to the dimensions of the digital image. For example, W = W0 / 32 and H = H0 / 32 can be considered. The number of channels C can be larger. A typical value is C = 2048.

[0066] The second component C2 corresponds to an encoder-decoder network, which can be decomposed into a first subnetwork, or encoder network (or more simply encoder) NE, and a second subnetwork, or decoder network (or more simply decoder) ND.

[0067] The NE encoder network takes as input a vector E, constructed from the FM feature map and a spatial encoding of the PE positions.

[0068] The height and width of the FM feature map are thus transformed into a one-dimensional space corresponding to the input vector Eb. Since it is desired that the DETR transformer be invariant to permutations of the input space (i.e., to the order of requests), it is necessary to incorporate spatial data into the information managed by the encoder. Therefore, a positional PE encoding is inserted into the input vector Eb presented to the input layer of the encoding network NE.

[0069] The NE encoder network encodes and enriches the FM characteristic map in order to obtain an enriched Eo characteristic map at output.

[0070] The ND decoder network processes a set of object requests Qb Q2.. .QN in order to generate predictions Pb P2...PN respectively, using information on the digital image considered as captured by the output Eodu encoder network NE.

[0071] Each prediction is processed independently by a neural network from a set of N neural networks, in order to determine a (final) detection prediction from the output layer of the ND decoder. These neural networks are, in the original DETR architecture, feedforward networks (FFN).

[0072] For each query of objects Qb Q2.. .QN, the transformer therefore generates a prediction. This prediction can be provided to a detection layer, N3, in order to generate a final prediction consisting of - either in a predicted bounding box and a class corresponding to the type of object detected (positive query), or, - either in a class corresponding to the non-detection of an object or an extremely low detection score (negative query).

[0073] This N3 subnetwork is typically a forward-propagating (FFN) network.

[0074] The aforementioned article "End-to-End Object Detection with Transformers" provides further details on the architecture, learning process and inference process of the DETR detection transformer.

[0075] Since the publication of this article, much work has been undertaken to improve the DETR transformer. A large amount of this work aims to improve the performance of DETR, particularly in terms of convergence speed, and also to understand the role that object queries can play in the behavior of the ND decoder and to study how to use them to improve the overall performance of the transformer.

[0076] In particular, work consists of determining the content of object queries.

[0077] The very architecture of a DETR detection transformer requires that these Requests have common characteristics with the encoder input, and therefore contain a positional encoding. Generally speaking, then, object requests contain a position from which the ND decoder (in collaboration with the N3 detection layer) must attempt to predict an optimized bounding box and a class of a detected object.

[0078] Various works have been proposed to specify the position encodings of the queries allowing to improve the performance of the ND decoder in learning and / or in prediction (or inference).

[0079] For example, the query can be a feature vector, the coordinates of a bounding box (not optimized and therefore whose learning will consist of evolving the coordinates), a point coordinate (which can evolve in learning and whose dimensions of the bounding box are also sought), etc.

[0080] It should be noted that the proposed method can be adapted to different embodiments concerning the structure of object queries and is therefore independent of these queries.

[0081] In learning, we seek to optimize the parameters of the decoder so that it generates the correct predictions from object requests and the outputs of the encoder.

[0082] It has been found that learning is sensitive to the number of positive queries, i.e. those that correspond to the prediction of an object class and a bounding box, as opposed to negative queries that correspond to the prediction of a class corresponding to the non-detection of an object.

[0083] The following article, cited above, has particularly studied this phenomenon: Chen, Qiang, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng and Jingdong Wang. “Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment.” (2022).

[0084] This article observes that a large number of positive queries improves performance in machine learning, but that the number of queries in general only slightly improves performance. It would therefore be advisable to increase the ratio between positive and negative queries.

[0085] As we also want to avoid assigning several positive predictions to the same label, which is typically done in "One-to-many" approaches, leading to the need for a post-processing filtering step to remove multi-predictions, the article proposes to group object queries by assigning several positive queries per label but decoupling these assignments into several independent groups, so that we only have the assignment of a positive query to a given label in each of the groups.

[0086] In inference, only a single group is retained, in order to eliminate redundancy of learned features.

[0087] The inventors have observed that, in doing so, all the predictions made by the ND decoder during the training phase are not taken into account. In particular, the impact of negative queries is not considered in the trained model, nor even studied in the scientific literature.

[0088] One of the goals is therefore to take advantage of negative queries in the learning process in order to optimize the model.

[0089] Figure 3 illustrates an ND decoder associated with an N3 detection layer, according to an embodiment of the invention, during the learning phase.

[0090] This ND decoder is mainly made up of a first layer of self attention, SL, and a second layer of cross attention, CL.

[0091] The operation of these attention layers, SL, CL, can conform to the mechanisms of a detection transformer DETR, as described in the seminal article or in the multiple subsequent developments. This operation was initially described in the article by Vaswani, Ashish, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is All You Need,” NIPS (2017).

[0092] According to the proposed method, object queries Qb Q2... QN are partitioned into K subgroups. In other words, each query is associated with one and only one subgroup Gb

[0093] If we consider N object queries, we can therefore define the set of K subgroups Gb G2...Gk with:

[0094] [Math.l] [W>.....«<]

[0095] In the example of [Fig.3], sub-subgroups Gb G2 are represented, the first subgroup consists of the requests Qi to Q; and the second subgroup G2 consists of the requests Qi+i to QN.

[0096] Classically, each request for objects Qb Q2... QNest is processed by the decoder ND and then by the detection layer N3 to capture the information provided by the encoder NE.

[0097] According to the proposed method, during the learning phase, the predictions corresponding to each subgroup G; are compared with the set of GT labels (for "ground truth") of the training set.

[0098] This confrontation aims to form a one-to-one bipartite matching (“one-to-one assignement” in English).

[0099] This matching can be viewed as an assignment problem. This type of problem can be formalized as finding the optimal allocation of tasks to agents. Each agent can perform a single task at a given cost, and each task must be performed by a single agent. The assignments (i.e., the agent-task pairs) all have a defined cost. The goal is to minimize the total cost of the assignments in order to perform all the tasks.

[0100] More formally, the objective is to determine a matching of a size equal to the number of tasks, with minimum weight in a weighted bipartite graph. If there are as many agents as tasks, the goal is to determine a perfect matching of minimum weight in a weighted bipartite graph. The assignment problem can be solved in polynomial time using the Hungarian algorithm.

[0101] This type of problem is well described in the scientific literature, of which the Wikipedia page is a good starting point:

[0102] https: / / en.wikipedia.org / wiki / Assignment_problem

[0103] The bipartite matching between predictions Pi P2.. .PNet and GT labels can be implemented by a Hungarian algorithm. The Hungarian algorithm, classically used in DETR detection transformers, was proposed in the article by Harald W. Kuhn, "The Hungarian Method for the assignment problem" in Naval Research Logistic Quarterly, 2 (1955), pp. 83-97.

[0104] In other words, the approach is very different from that of the "Group DETR" technique described earlier in which the set of queries is artificially enlarged, since here this set of queries is constant but the labels of the training set are confronted with a plurality of subgroups independently of each other.

[0105] This confrontation is carried out by an LM matching mechanism which aims to - among the predictions produced at the output of the N3 detection layer, we search for those which are closest to the corresponding label, - we use the selected predictions, corresponding to positive queries, to calculate an error as a function of a loss function and thus update the RN decoder network and the N3 detection layer.

[0106] For each subgroup, we therefore have a set of predictions, each forming a tuple consisting of a bounding box prediction and a class prediction Ci (i varying in the interval corresponding to the subgroup concerned).

[0107] The labels also constitute a set of tuples, formed of bounding boxes b; and of classes Cj (i varying on the training set).

[0108] We seek to match each subgroup with the training set. More precisely, the training includes a search for matching the predictions of each subgroup with the set of labels in the training set.

[0109] In the Hungarian algorithm, which can be used in learning, we therefore seek an optimal matching for each subgroup Gk, on the set E of PL K possible permutations'7 (Æ being the number of queries in each subgroup).

[0110] [Math.2] N ^Gk = argminE* ( yf % ) [YES] Match (y <f^ ) représente le coût d’appariement entre l’étiquette y; et la prédiction index y

[0112] Each label y; can be seen as having two components, a class Cj and a bounding box b; : y;=(Ci, b;).

[0113] Similarly, each prediction can be written

[0114] We can then write the matching cost for a subgroup Gk as:

[0115] [Math.3]

[0116] In this equation, - M is the number of labels in the training set - Cls() and box() are cost functions that allow you to calculate a distance between, respectively, two classes and two bounding boxes. - ^ds and ^box are two coefficients, forming adjustable and predefined parameters of the algorithm.

[0117] This equation expresses the matching cost for a subgroup Gk clearly shows that each subgroup Gk is matched with the set of available labels, M, in order to increase the number of positive queries.

[0118] In this way, we perform a subgroup matching between the predictions of subgroup Gk and the set of duplicated ground truth labels for subgroup Gk.

[0119] According to one embodiment of the invention, it is sought to ensure independent learning for each subgroup Gk.

[0120] To achieve this, according to one embodiment, an attention mask AM is applied to the self-attention layer SL. This attention mask is designed to allow the transmission of information within a subgroup but not between distinct subgroups.

[0121] Such an attention mask AM can take the form of a matrix whose rows and columns form the N different object queries Qh Q2... Qn provided as input to the self-attention layer SL.

[0122] This AM attention matrix can be constructed so that each cell has a value that allows the exchange of information between the query on the x-axis and the query on the y-axis to be blocked or allowed, depending on whether the queries belong to the same subgroup or not.

[0123] Fig. 4 illustrates an example of an attention matrix for 4 subgroups, G1, G2, G3, G4.

[0124] The shaded areas correspond to cells associated with queries belonging to the same subgroups, while the white areas correspond to cells associated with queries belonging to distinct subgroups. A numerical value (1) is assigned to the shaded areas to ensure that queries from the same subgroup are taken into account when calculating the outputs of the self-attention layer, and a numerical value (0) is assigned to the white areas to exclude the queries concerned from these same calculations.

[0125] According to one embodiment, the attention matrix AM is added to the attention generated by the similarity calculation between the query Q and the key K, generated by the object query, normalized by a scaling factor.

[0126] This mechanism forces numerous queries to collectively identify a single object, thereby reducing the prevalence of negative queries to a collection of redundant queries which, collectively, strengthen confidence in the presence of objects. Facilitating robust learning, this procedure generates increased efficiency.

[0127] However, during the optimization phase of queries for matched objects, a predisposition towards the homogenization of features emerges. To exploit the potential for a transparent convergence of features among these object queries, according to one embodiment, a diversification strategy is introduced. This strategic intervention, implemented in the query space, serves to increase the distinctiveness of the query.

[0128] To do this, a new cost function is proposed, which can be used jointly with other cost functions during the learning phase, according to one embodiment of the invention.

[0129] This cost function is based on a correlation matrix corr(). This correlation matrix gives a correlation value between -1 and 1 for each prediction pair P; at the output of the detection transformer.

[0130] The correlation matrix can be monitored during the learning phase so that the values ​​(off the diagonal) are always between two predefined bounds ai, ai forming a constraint (with ai <a2).

[0131] To achieve this, we can define a cost function \doq of the following form, the purpose of which is to optimize the diversification of object queries:

[0132] [Math.4] <v,: ÏJ^orr ( Decp Decy ) - a, 7«

[0133] Dec; represents the output of the N3 neural network and therefore the prediction that we seek to match with the GT labels during this learning phase.

[0134] As mentioned previously, the N3 subnet can be a forward-propagating (FFN) network.

[0135] We can then write

[0136] [Math.5] / / / \ Dec^normA FFN\ softmax\ ——-- x (FWV)

[0137] F represents the feature map, and Wq, Wk and Wv are synaptic weight matrices. dk is the key dimension.

[0138] The correlation calculation can be carried out in accordance with known methods and, for example, made available on neural network platforms such as Pytorch. + oi A-DOQ L-feOffDiag

[0139] Information is available, for example, at the following links:

[0140] https: / / pytorch.org / docs / stable / generated / torch.corrcoef.html

[0141] https: / / pytorch.org / docs / stable / generated / torch.cov.html

[0142] Other examples of calculating correlation terms are obviously accessible to a person skilled in the art.

[0143] The overall cost function A, used during the learning phase, can be a linear combination of this new cost function with the cost functions usually used

[0144] A = / ^doq + \is +

[0145] with:

[0146] is the cost function evaluating the classification,

[0147] A^ the cost function evaluating the prediction of a bounding box

[0148] ^doq the cost function described above evaluating query diversification of objects.

[0149] According to one embodiment, learning can be subdivided into two phases: - In the first phase, conventional learning can be carried out on all subgroups, then - In a second phase, a separate learning process is implemented for each subgroup, using the cost function described previously. Different ab values ​​are used for each subgroup.

[0150] According to one embodiment, a classic non-maximum suppression (NMS) mechanism to counterbalance redundancy in object queries.

[0151] This redundancy encourages a clear separation of the predictions into two distinct groups: one consisting of related predictions, possibly forming several groups, and the other encompassing disparate predictions. Incorporating NMS in this context effectively mitigates the redundancy problems and amplifies the effectiveness of the approach we propose. However, the conventional greedy NMS technique has detrimental effects on the detections themselves. This is primarily due to the standard NMS approach, which ranks predictions in descending order of their scores.

[0152] On the other hand, in the case of two subgroups, the method we propose involves the generation of a cost matrix (Cpred) using the object center (CCenterX) class (Cdass) and box size (Csize) parameters:

[0153] Cpred Z^boxCCcenter-tCsi / e) “t 7-class(Cclass)

[0154] This approach facilitates the formation of groups containing identical boxes with class attributes, with weighting parameters respectively Xbox = 5 and ^■class

[0155] In the case of N groups, we distribute the predictions into N groups using the top-N on the scores of each object query. Subsequently, the NMS is applied individually to each of these groups of predictions, resulting in a more refined and efficient suppression process.

[0156] The proposed method, in its various embodiments, makes it possible to take advantage of negative queries during the learning phase of a DETR detection transformer.

[0157] Experimental studies have shown that instances classified as negative based on their prediction scores often exhibit higher prediction coordinates when compared to the basic truth values ​​(labels). Interestingly, these instances outperform even the most reliable queries. Building on this idea, an approach is presented that exploits the potential of negative queries by strategically forming diverse query subgroups. This innovative strategy not only improves object detection in different scenarios but also enables object detection across multiple instances.

[0158] The following table gives performance indicators for different transformers in DETR detection.

[0159] The “Deformable DETR” transformer is an evolution of the DETR transformer described in the article Zhu, Xizhou, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang and Jifeng Dai. “Deformable DETR: Deformable Transformers for End-to-End Object Detection.” ArXiv abs / 2010.04159 (2020)

[0160] The “Conditional DETR” transformer is an evolution of the DETR transformer described in the article Meng, Depu, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun and Jingdong Wang. “Conditional DETR for Fast Training Convergence.” 2021 IEEE / CVF International Conference on Computer Vision (ICCV) (2021): 3631-3640.

[0161] The proposed mechanisms can be implemented both with the original detection transformer and with its evolutions.

[0162] Each line, corresponding to a distinct implementation, includes a number of iterations to achieve convergence and average performance indicators AP (for "Average Performance"), globally (AP) for small detected objects (APS), for medium objects (APM) and for large objects (APL).

[0163] [Tables 1] Process Iterations AP APS APm apl DETR 500 42.0 62.4 45.8 61.1 Deformable DETR 12 38.3 58.0 41.7 51.4 Deformable DETR + propositions 12 39 58.4 42.3 51.9 Conditional DETR 12 36.6 57.3 39.4 53.4 Conditional DETR + propositions 12 37.1 58 39.6 54

[0164] The various mechanisms proposed therefore allow a substantial improvement in the evaluation metrics of object detection based on the COCO training set used for the tests.

[0165] The proposed method is also applicable to other transformer designs for object detection in digital signals, for example in images or image sequences

[0166] Of course, the present invention is not limited to the examples and embodiment described and illustrated, but is defined by the claims. In particular, it is susceptible of numerous variations accessible to those skilled in the art.

Claims

Demands

1. A method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network (NI) adapted to determine features within said digital image, an encoder (NE) for generating information from said features, and a decoder comprising a self-attention layer (SL), said decoder being adapted to generate predictions (Pb P2.. .PN) from object queries (Qb Q2.. .QN) as a function of information (Eo) provided by said encoder (NE), wherein said object queries are structured as subgroups (Gi, G2...GK) forming a partition of the set of said queries, said learning comprising a search for matching the predictions of each subgroup with the same training set, in which said self-attention layer (SL) is adapted to allow independent learning for each subgroup, in which an attention mask (AM) is applied to said self-attention layer (SL), said attention mask being provided to allow the transmission of information only within a subgroup and in which a cost function is provided to optimize the diversification of said object queries.

2. A method according to the preceding claim, wherein said cost function is based on a correlation matrix supervised during said learning so that its values ​​are between two predefined bounds.

3. Method for detecting objects within a digital image, said method comprising a learning method according to one of the preceding claims.

4. A method according to any one of the preceding claims in which said digital image is derived from an image stream generated by a video camera.

5. A computer program capable of being implemented on an access point to a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, carries out the steps of the method defined in claims 1 to 4.

6. Processing device comprising means for implementing the process according to any one of claims 1 to 4.

Images