DETECTION TRANSFORMER FOR OBJECT DETECTION IN DIGITAL IMAGES BASED ON SUBGROUP PARTITIONING OF THE OBJECT SURVEY ARRANGEMENT

Description

FIELD OF INVENTION

[0001] The present invention relates to the field of computer vision, and more specifically to the field of object detection in a digital image.

[0002] One of the goals of object detection in a digital image is to predict a set of bounding boxes in the image, each containing a detected object, and to assign each box a class corresponding to the type of object detected.

[0003] Artificial intelligence (AI) has revolutionized the field of computer vision, enabling machines to perceive and understand the visual world at an unprecedented level.

[0004] Various technologies have been implemented to solve this general problem.

[0005] In 2014, regional convolutional neural networks, or RCNNs (for "Region-based Convolutional Neural Network"), were proposed. They consist of two main stages and are therefore classified as two-stage detectors. The approach first calculates a list of proposed regions of interest in the form of a bounding box in a stage called "region proposals." Then, in a second stage, these regions are fed into a convolutional neural network (or "CNN") to locate and classify the objects.

[0006] In parallel, another category of one-stage detector algorithms is being developed. Among the best known is the "YOLO" architecture, which was first presented in Redmon, J., Divvala, SK, Girshick, RB, & Farhadi, A. (2015). "You Only Look Once: Unified, Real-Time Object Detection". 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779-788. YOLO achieved very fast detection by eliminating the region proposal step and replacing it with a single network that handles detection end-to-end. While accuracy remained lower than two-stage detector methods, numerous improvements were made to YOLO's accuracy as it evolved to its fourth version in 2020.

[0007] More recently, it has been proposed to use the transformer mechanism, already used in the separate field of speech recognition and machine translation, for object detection.

[0008] One of the first successful attempts to integrate transformers into computer vision was the Vision Transformer (ViT) model, introduced in Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit and Neil Houlsby. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." ArXiv abs / 2010.11929 (2020).

[0009] The ViT model treats an image as a sequence of non-overlapping pixel blocks (or "patches") and applies a standard transformer architecture to process these patches. This approach has demonstrated competitive performance on image classification tasks, even outperforming leading CNNs in some cases.

[0010] Following the success of ViT, several other transformer-based models have been proposed for various computer vision tasks.

[0011] For example, the DETR model, introduced in Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. "End-to-End Object Detection with Transformers." ArXiv abs / 2005.12872 (2020) exploits transformers for object detection and obtains competitive results compared to traditional object detection methods.

[0012] This DETR model (for "DEtection Transformer" in English) is based on an architecture of 4 neural networks: a first network processes the digital image to provide a feature map, this map is provided to a second neural network, or encoder, whose output is provided to a third neural network, or decoder.

[0013] The decoder also takes object requests as input (" object queries " in order to predict, in collaboration with a fourth neural network, a result (bounding box associated with a class) for each query, using the information provided by the decoder.

[0014] Many developments of this DETR model have been proposed more recently, particularly to clarify how to determine queries and their influence on predicted results.

[0015] For example, it was proposed in Chen, Qiang, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng and Jingdong Wang. "Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment." (2022) to present groups of queries during training in order to assign a single label to each group, which leads to as many positive queries as there are actually objects in the digital image.

[0016] However, despite the large number of groups used in training, it turns out that only a portion of the information is actually used. Indeed, only one group of queries is used during the inference (or prediction) phase, thus underutilizing the information from the training phase.

[0017] There is therefore a need to improve current state-of-the-art proposals to maximize the potential of the query set, in order to improve performance in object detection in digital images. SUMMARY OF THE INVENTION

[0018] To this end, according to a first aspect, the present invention can be implemented by a method for learning an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network adapted to determine features within said digital image, an encoder for generating information from said features, and a decoder comprising a self-attention layer, said decoder being adapted to generate predictions from object queries based on information provided by said encoder, wherein said object queries are structured as subgroups forming a partition of the set of said queries, said learning comprising a search for matching the predictions of each subgroup with the same training set.

[0019] According to preferred embodiments, the invention comprises one or more of the following features which can be used separately or in partial combination with each other or in total combination with each other: The self-attention layer is adapted to allow independent learning for each subgroup. An attention mask is applied to this self-attention layer, designed to allow information transmission only within a single subgroup. A cost function is provided to optimize the diversification of object queries. This cost function is based on a correlation matrix supervised during the learning process so that its values ​​fall between two predefined bounds.

[0020] Another aspect concerns a method for detecting objects within a digital image, said method including a learning process as previously described.

[0021] According to one embodiment, the digital image is derived from a stream of images generated by a video camera.

[0022] Another aspect of the invention relates to a computer program suitable for implementation at an access point to a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, performs the steps as previously described.

[0023] Another aspect of the invention relates to a processing device comprising means for implementing the process as previously described.

[0024] Other features and advantages of the invention will become apparent from the following description of a preferred embodiment of the invention, given by way of example and with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE FIGURES

[0025] The attached drawings illustrate the invention: there figure 1 illustrates an example of implementing a process according to an embodiment of the invention; the figure 2 diagram illustrates an architecture of a possible implementation of a detection transformer according to embodiments of the invention; the figure 3 illustrates a decoder associated with a detection layer, according to an embodiment of the invention, during the learning phase; the figure 4 illustrates an example of an attention matrix according to one embodiment of the invention. DETAILED DESCRIPTION OF METHODS OF IMPLEMENTING THE INVENTION

[0026] There figure 1illustrates an example of the implementation of a process according to an embodiment of the invention.

[0027] This example illustrates a digital image (IMG) in which two objects are represented: a dog, O1 and a doghouse, O2.

[0028] The process aims to detect these objects, that is to say, on the one hand, to determine a bounding box, respectively P1, P2, corresponding to each of these objects, and, for each object, to determine an associated class.

[0029] The encompassing boxes, or "bounding box" In English, these are geometric shapes (usually rectangular) containing the detected object. The process also aims to optimize these boxes, that is, to find the box with the smallest surface area encompassing the object.

[0030] The determination of classes depends on the training that has been performed on a labeled training set. It therefore consists of assigning to the object detected in a bounding box a class corresponding to one of the labels learned during the training phase.

[0031] In the illustrated example, we can assume that the training set contained numerous images representing dogs of different breeds, in various positions and at different scales. The neural network is then able to generalize the different representations associated with the same label "dog" so as to be able to associate a new representation with the class "dog".

[0032] Such a mechanism can be used in various object detection tasks within digital images. In particular, it can be applied to detection in digital image or video sequences.

[0033] These digital images can be acquired by appropriate sensors (photographic or video camera, possibly infrared camera, etc.), and processed by a processing platform comprising at least one microprocessor, one or more memories and possible additional electronic circuits, designed to collaboratively implement the proposed process.

[0034] In particular, the platform can process images acquired continuously to enable real-time or near-real-time object detection.

[0035] For example, transformer-based models have been used for tasks such as video understanding, scene understanding, and 3D point cloud processing.

[0036] Object detection mechanisms in images or image streams have numerous potential uses, for example in the sports field, but also in any other field (video surveillance, etc.).

[0037] Detecting an object in an image or in a sequence of images (or videos) can allow a marker to be placed on the detected object, or any other information associated with the object and, for example, retrieved via a database.

[0038] Depending on the use case, and / or the choices of a user, different types of information can be displayed superimposed on the image (or images) and positioned so that the user can easily associate this information with the detected object.

[0039] Several objects can be processed in parallel: for example in a race, or in a team sport match, the names of the different participants can thus be displayed superimposed and alongside their position on the image.

[0040] Also, a human-machine interface can be provided so that the user can select a particular object (player, horse, bike, car...) and follow it throughout the event (race, match...) by means of a particular marker allowing it to be distinguished from other objects.

[0041] Another example of use is the extraction of detected objects to feed models of virtual events (races, matches, etc.), particularly for realistic video games, or for analysis or reporting videos based on the data thus collected.

[0042] Particularly interesting scenes can also be extracted for rebroadcast or modeled, or even marketed in other forms (NFT....).

[0043] In the field of sports racing, for example, one can detect runners, bicycles, cars, or horses crossing a finish line.

[0044] For example, in a horse race, such a mechanism can: identify all horses in a race in real time to specify to a user their positions on a video precisely by integrating some metadata such as speed, name, age, race history, etc.via cross-referencing with a database, detect the starters in a race to determine if all the horses are ready to start, detect anomalies during a race, for example if the correct number of horses are on the track, detect and count the people present in the racecourse, collect data on the positioning of a horse on the racecourse throughout the race in order to increase knowledge about the behavior of horses, use this knowledge to display a "ghost" horse during another race in order to allow a visual comparison between the behaviors of the same horse during different races, build statistical databases from a set of races in order to determine indicators on horses, on jockeys, .

[0045] These examples can obviously be applied to other types of races: car races, cycling races, running races (100 meters, marathon...), etc.

[0046] In team sports (football, rugby, etc.), it's also possible to track individual players as mentioned earlier, for various applications. It can also detect other types of objects (balls, etc.).

[0047] This allows for the automatic and immediate ranking of a trip, thus facilitating trip management for an operator.

[0048] The proposed object detection method is based on the DETR model

[0049] It is proposed to improve the learning process of the DETR model, in order to create a more efficient model allowing, in the prediction phase, better detection of objects in the submitted digital images.

[0050] The proposed method is independent of the training set. The latter only needs to associate a set of digital images with labels or "ground truth".

[0051] These labels provide a basis for comparison for the predictions made by the transformer for each image. The errors resulting from these comparisons allow the transformer's state (the various internal parameters, or "synaptic weights") to be iteratively modified until convergence is achieved.

[0052] The training set can be a publicly available database, such as the Coco database distributed by Microsoft (for "Common Object in Context"). This database was described in the article by Lin, Tsung-Yi, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C. Lawrence Zitnick. "Microsoft COCO: Common Objects in Context." European Conference on Computer Vision (2014).

[0053] As mentioned previously, the DETR model (for "DEtection Transformer" in English, i.e., Detection Transformer) was introduced by the article by Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. "End-to-End Object Detection with Transformers." ArXiv abs / 2005.12872 (2020).

[0054] There figure 2 illustrates an architecture of a possible implementation of a DETR detection transformer.

[0055] Such a DETR transformer can be broken down into three main components: a first component C1 comprising a first neural network, N1, intended to determine a set of features for each digital image submitted as input; a second component C2 comprising a second neural network of the type NE encoder - ND decoder; a third component C3 comprising a set of neural networks (detection layer) N3 intended to determine a detection prediction from the output layer of the decoder of the second neural network N2.

[0056] According to one embodiment, such a DETR transformer can be implemented on an artificial intelligence platform having the functionality of proposing classical neural network models, such as convolutional neural networks and encoder-decoder neural networks.

[0057] The article by Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai and Soumith Chintala, "PyTorch: An Imperative Style, High-Performance Deep Learning Library," published in Neural Information Processing Systems (2019), proposes a library providing components that allow for the easy implementation of a DETR architecture in approximately fifty lines of code.

[0058] The first neural network N1 is typically a convolutional neural network, CNN (for "Convolutional Neural Network").

[0059] This first neural network N1 aims to extract an FM feature map, or "feature map" according to the usual English terminology, from a digital image submitted to it.

[0060] The digital image can be represented in a C0 × H0 × W0 space, with H0, W0 representing, respectively, the height and width of the image, and C0 representing the number of channels per pixel, typically a color encoding of the pixels.

[0061] The FM feature map generated by the first neural network can be represented in a C×W×H space. Generally, the width W and height H of the feature map are reduced compared to the dimensions of the digital image. For example, W = W₀ / 32 and H = H₀ / 32. The number of channels C can be larger. A typical value is C = 2048.

[0062] The second component C2 corresponds to an encoder-decoder network, which can be broken down into a first subnetwork, or encoder network (or more simply encoder) NE, and a second subnetwork, or decoder network (or more simply decoder) ND.

[0063] The NE encoder network takes as input an EI vector constructed from the FM feature map and a spatial encoding of PE positions.

[0064] The height and width of the FM feature map are thus transformed into a one-dimensional space corresponding to the input vector EI. Since the DETR transformer is to be invariant to permutations of the input space (i.e., to the order of requests), it is necessary to incorporate spatial data into the information handled by the encoder. Therefore, a positional PE encoding is inserted into the input vector EI, which is presented to the input layer of the encoder network NE.

[0065] The NE encoder network encodes and enriches the FM characteristic map to obtain an EO enriched characteristic map at output.

[0066] The ND decoder network processes a set of object requests Q1, Q2...QN in order to generate predictions P1, P2...PN respectively, using information about the digital image considered as captured by the EO output of the NE encoder network.

[0067] Each prediction is processed independently by a neural network from a set of N neural networks, in order to determine a (final) detection prediction from the output layer of the ND decoder. These neural networks are, in the original DETR architecture, feedforward networks (FFNs).

[0068] For each object request Q1, Q2...QN, the transformer generates a prediction. This prediction can be provided to a detection layer, N3, to generate a final prediction consisting of either in a predicted bounding box and a class corresponding to the type of object detected (positive query), or, either in a class corresponding to the non-detection of an object or an extremely low detection score (negative query).

[0069] This N3 subnetwork is typically a forward-propagating (FFN) network.

[0070] The aforementioned article "End-to-End Object Detection with Transformers" provides further details on the architecture, learning process and inference process of the DETR detection transformer.

[0071] Since the publication of this article, much work has been undertaken to improve the DETR transformer. Much of this work aims to improve DETR's performance, particularly in terms of convergence speed, and also to understand the role that object queries can play in the behavior of the ND decoder and to study how to use them to improve the overall performance of the transformer.

[0072] In particular, work involves determining the content of object queries.

[0073] The very architecture of a DETR detection transformer requires that these requests share characteristics with the encoder input, and therefore contain positional encoding. Generally speaking, then, object requests contain a position from which the ND decoder (in conjunction with the N3 detection layer) must attempt to predict an optimized bounding box and a class for a detected object.

[0074] Various works have been proposed to specify the positional encodings of queries to improve the performance of the ND decoder in learning and / or prediction (or inference).

[0075] For example, the query can be a feature vector, the coordinates of a bounding box (not optimized and therefore whose learning will consist of evolving the coordinates), a point coordinate (which can evolve in learning and whose bounding box dimensions are also sought), etc.

[0076] It should be noted that the proposed method can be adapted to different implementations regarding the structure of object queries and is therefore independent of these queries.

[0077] In learning, we seek to optimize the decoder parameters so that it generates the correct predictions from object requests and encoder outputs.

[0078] It has been found that learning is sensitive to the number of positive queries, i.e., those that correspond to the prediction of an object class and a bounding box, as opposed to negative queries that correspond to the prediction of a class corresponding to the non-detection of an object.

[0079] The following article, previously cited, has particularly studied this phenomenon: Chen, Qiang, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng and Jingdong Wang. "Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment." (2022).

[0080] This article observes that a large number of positive queries improves performance in machine learning, but that the number of queries in general only slightly improves performance. Therefore, it would be advisable to increase the ratio between positive and negative queries.

[0081] Since we also want to avoid assigning several positive predictions to the same label, which is typically done in "One-to-many" approaches, leading to the need for a post-processing filtering step to remove multi-predictions, the article proposes to group object queries by assigning several positive queries per label but decoupling these assignments into several independent groups, so that we only have the assignment of a positive query to a given label in each of the groups.

[0082] In inference, only a single group is retained, in order to eliminate redundancy of learned features.

[0083] The inventors found that, in doing so, all the predictions made by the ND decoder during the training phase are not taken into account. In particular, the impact of negative queries is not considered in the trained model, nor has it been studied in the scientific literature.

[0084] One of the goals is therefore to take advantage of negative queries in the learning process in order to optimize the model.

[0085] There figure 3 illustrates an ND decoder associated with an N3 detection layer, according to an embodiment of the invention, during the learning phase.

[0086] This ND decoder is primarily composed of a first layer of self-attention (“ self attention (in English), SL, and a second layer of cross-attention (" "Cross-country skiing, be careful!" (in English), CL.

[0087] The functioning of these attention layers, SL and CL, can be consistent with the mechanisms of a detection transformer (DETR), as described in the seminal paper or in numerous subsequent developments. This functioning was initially described in the article by Vaswani, Ashish, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, "Attention is All you Need," NIPS (2017).

[0088] According to the proposed method, object queries Q1, Q2...QN are partitioned into K subgroups. In other words, each query is associated with one and only one subgroup Gi.

[0089] If we consider N object queries, we can therefore define the set of K subgroups G1, G2 ... GK with: G i = q i − 1 N K + 1 , … , q i N K

[0090] In the example of the figure 3, sub-subgroups G 1 , G 2 are represented, the first subgroup consists of queries Q 1 to Q i and the second subgroup G 2 consists of queries Q i+1 to QN .

[0091] Typically, each request for objects Q1, Q2...ON is processed by the ND decoder and then by the N3 detection layer to capture the information provided by the NE encoder.

[0092] According to the proposed method, during the learning phase, the predictions corresponding to each subgroup G i are compared with the set of GT labels (for "ground truth") of the training set.

[0093] This confrontation aims to form a one-to-one bipartite pairing (“one-to-one assignement” in English).

[0094] This matching can be viewed as an assignment problem. This type of problem can be formalized as finding the optimal allocation of tasks to agents. Each agent can perform a single task at a given cost, and each task must be performed by a single agent. The assignments (i.e., the agent-task pairs) all have a defined cost. The goal is to minimize the total cost of the assignments in order to complete all the tasks.

[0095] More formally, the goal is to find a matching group of the same size as the number of tasks, with minimum weight in a weighted bipartite graph. If there are as many agents as tasks, the goal is to find a perfect matching group with minimum weight in a weighted bipartite graph. The assignment problem can be solved in polynomial time using the Hungarian algorithm.

[0096] This type of problem is well described in the scientific literature, and the Wikipedia page is a good starting point: https: / / en.wikipedia.org / wiki / Assignment_problem

[0097] The bipartite matching between predictions P1, P2...PN and GT labels can be implemented by a Hungarian algorithm. The Hungarian algorithm, classically used in DETR detection transformers, was proposed in the article by Harald W. Kuhn, "The Hungarian Method for the assignment problem" in Naval Research Logistic Quarterly, 2 (1955), pp. 83-97.

[0098] In other words, the approach is very different from that of the "Group DETR" technique described earlier, in which the set of queries is artificially enlarged, since here this set of queries is constant but the labels of the training set are confronted with a plurality of subgroups independently of each other.

[0099] This confrontation is carried out by an LM matching mechanism which aims to among the predictions produced at the output of the N3 detection layer, we search for those that are closest to the corresponding label, we use the selected predictions, corresponding to positive queries, to calculate an error as a function of a loss function and thus update the RN decoder network and the N3 detection layer.

[0100] For each subgroup, we therefore have a set of predictions, each forming a tuple consisting of a bounding box prediction. b̂ i and a class prediction ĉ i (i varying in the interval corresponding to the subgroup concerned).

[0101] The labels also constitute a set of tuples, formed of bounding boxes bi and classes ci (i varying over the training set).

[0102] We aim to match each subgroup with the training set. More precisely, the training involves searching for a match between the predictions of each subgroup and the set of labels in the training set.

[0103] In the Hungarian algorithm, which can be used for machine learning, we therefore seek an optimal matching σ Gk for each subgroup G k, on the set ε of the N K possible permutations σ ( N K (where is the number of requests in each subgroup). σ Gk = argmin σ Gk ϵ E ∑ i = 1 N K C match y i y ^ q σ i

[0104] C match ( yi, ŷ q σ( i ) ) represents the matching cost between the label yi and the prediction ŷ q σ( i ) index q σ ( i ) Each label yi can be seen as having two components, a class ci and a bounding box bi: yi =(ci , bi ).

[0105] Similarly, each prediction can be written as: y ^ q i = c qi Gk b qi Gk

[0106] We can then write the matching cost for a subgroup G k as: C match Gk = ∑ i = 1 M λ cls . cls p σ G ^ k i , c qi Gk + λ box . box b i , b qi Gk

[0107] In this equation, M is the number of labels in the training set. Cls() and box() are cost functions that calculate a distance between, respectively, two classes and two bounding boxes. λ cls And λ box are two coefficients, forming adjustable and predefined parameters of the algorithm.

[0108] This equation expresses the matching cost C match Gk for a subgroup G k clearly shows that each subgroup G k is matched with the set of available labels, M, in order to increase the number of positive queries.

[0109] In this way, we perform a subgroup matching between the predictions of subgroup Gk and the set of duplicated ground truth labels for subgroup Gk.

[0110] According to one embodiment of the invention, we seek to ensure independent learning for each subgroup G k.

[0111] To achieve this, according to one embodiment, an attention mask AM is applied to the self-attention layer SL. This attention mask is designed to allow the transmission of information within a subgroup but not between distinct subgroups.

[0112] Such an attention mask AM can take the form of a matrix whose rows and columns form the N different object queries Q1, Q2...QN provided as input to the self-attention layer SL.

[0113] This AM attention matrix can be constructed so that each cell has a value that allows or blocks the exchange of information between the query on the x-axis and the query on the y-axis, depending on whether the queries belong to the same subgroup or not.

[0114] There figure 4 illustrates an example of an attention matrix for 4 subgroups, G1, G2, G3, G4.

[0115] The shaded areas correspond to cells associated with queries belonging to the same subgroups, while the white areas correspond to cells associated with queries belonging to different subgroups. The shaded areas are assigned a numerical value (1) to ensure that queries from the same subgroup are considered when calculating the outputs of the self-attention layer, and the white areas are assigned a numerical value (0) to exclude the queries in question from these calculations.

[0116] According to one embodiment, the attention matrix AM is added to the attention generated by the similarity calculation between the query Q and the key K, generated by the object query, normalized by a scaling factor.

[0117] This mechanism forces numerous queries to collectively identify a single object, thus reducing the prevalence of negative queries to a collection of redundant queries that, collectively, strengthen confidence in the presence of objects. Facilitating robust learning, this procedure leads to increased efficiency.

[0118] However, during the optimization phase of matched object queries, a tendency towards feature homogenization emerges. To exploit the potential for seamless feature convergence among these object queries, a diversification strategy is introduced according to a specific implementation. This strategic intervention, implemented within the query space, serves to enhance the distinctiveness of the query.

[0119] To achieve this, a new cost function is proposed, which can be used in conjunction with other cost functions during the learning phase, according to one embodiment of the invention.

[0120] This cost function is based on a correlation matrix corr(). This correlation matrix gives a correlation value between -1 and 1 for each prediction pair P i at the output of the detection transformer.

[0121] We can supervise the correlation matrix during the learning phase so that the values ​​(outside the diagonal) are always between two predefined bounds α 1 , α 1 forming a constraint (with α 1 <α 2 ).

[0122] To do this, we can define a cost function of the following form, with the aim of optimizing the diversification of object queries: L DOQ = ∑ i ∈ OffDiag ∑ j = 1 j ≠ i N i corr Dec i Dec j − α 1 2 + α 2 2

[0123] Deci represents the output of the N3 neural network and therefore the prediction that we are trying to match with the GT labels during this learning phase.

[0124] As mentioned previously, the N3 subnet can be a forward-propagating (FFN) network.

[0125] We can then write Dec i = norm FFN softmax Q i W q × FW k T d k × FW v

[0126] F represents the feature map, and Wq, Wk and Wv are synaptic weight matrices. dk is the key dimension.

[0127] Correlation calculation can be performed according to known methods and, for example, made available on neural network platforms such as Pytorch.

[0128] Information is available, for example, at the following links: https: / / pytorch.org / docs / stable / generated / torch.corrcoef.html https: / / pytorch.org / docs / stable / generated / torch.cov.html

[0129] Other examples of calculating correlation terms are obviously accessible to those skilled in the art.

[0130] The overall cost function The cost function used during the learning phase can be a linear combination of this new cost function with the cost functions usually used. L = L DOQ + L cls + L box with : the cost function evaluating the classification, the cost function evaluating the prediction of a bounding box the cost function described above evaluating the diversification of object queries.

[0131] According to one implementation method, learning can be subdivided into two phases: In the first phase, conventional learning can be performed on all subgroups. Then, in the second phase, learning is implemented separately for each subgroup, using the cost function described previously. Different values ​​of α₁, α₂ are used for each subgroup.

[0132] According to one embodiment, a classic non-maximum suppression (NMS) mechanism is used to counteract redundancy in object queries.

[0133] This redundancy necessitates a clear separation of predictions into two distinct groups: one consisting of related predictions, potentially forming several subgroups, and the other encompassing disparate predictions. Incorporating NMS in this context effectively mitigates redundancy issues and enhances the efficiency of our proposed approach. However, the conventional greedy NMS technique has detrimental effects on the detections themselves. This is primarily due to the standard NMS approach, which ranks predictions in descending order of their scores.

[0134] On the other hand, in the case of two subgroups, the method we propose involves generating a cost matrix (C pred) using the object center (C center), class (C class) and box size (C size) parameters: C pred = λ box C center + C size + λ class C class

[0135] This approach facilitates the formation of groups containing identical boxes with class attributes, with weighting parameters respectively λ box = 5 and λ class = 2.

[0136] In the case of N groups, we distribute the predictions into N groups using the top-N method on the scores of each object query. Subsequently, NMS is applied individually to each of these groups of predictions, resulting in a more refined and efficient deletion process.

[0137] The proposed method, in its various embodiments, makes it possible to take advantage of negative queries during the learning phase of a DETR detection transformer.

[0138] Experimental studies have shown that instances classified as negative based on their prediction scores often exhibit higher prediction coordinates compared to their basic truth values ​​(labels). Interestingly, these instances even outperform the most reliable queries. Building on this finding, an approach is presented that leverages the potential of negative queries by strategically forming diverse query subgroups. This innovative strategy not only improves object detection across various scenarios but also enables object detection across multiple instances.

[0139] The following table gives performance indicators for different transformers in DETR detection.

[0140] The "Deformable DETR" transformer is an evolution of the DETR transformer described in the article by Zhu, Xizhou, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang and Jifeng Dai. "Deformable DETR: Deformable Transformers for End-to-End Object Detection." ArXiv abs / 2010.04159 (2020)

[0141] The "Conditional DETR" transformer is an evolution of the DETR transformer described in the article by Meng, Depu, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun and Jingdong Wang. "Conditional DETR for Fast Training Convergence." 2021 IEEE / CVF International Conference on Computer Vision (ICCV) (2021): 3631-3640.

[0142] The proposed mechanisms can be implemented both with the original detection transformer and with its evolutions.

[0143] Each line, corresponding to a distinct implementation, includes a number of iterations to achieve convergence and average performance indicators AP (for "Average Performance"), globally (AP) for small detected objects (AP S), for medium objects (AP M) and for large objects (AP L). [Table 1] Process Iterations AP AP S AP M AP L DETR 500 42.0 62.4 45.8 61.1 Deformable DETR 12 38.3 58.0 41.7 51.4 Deformable DETR + proposals 12 39 58.4 42.3 51.9 Conditional DETR 12 36.6 57.3 39.4 53.4 Conditional DETR + proposals 12 37.1 58 39.6 54

[0144] The various mechanisms proposed therefore allow a substantial improvement in the evaluation metrics of object detection based on the COCO training set used for testing.

[0145] The proposed method is also applicable to other transformer designs for object detection in digital signals, for example in images or image sequences.

[0146] Of course, the present invention is not limited to the examples and embodiment described and illustrated, but is defined by the claims. In particular, it is susceptible of numerous variations accessible to those skilled in the art.

Claims

1. A method for training an object detection model in a digital image, said model being of the detection transformer type and comprising a first neural network (N1) adapted to determine features within said digital image, an encoder (NE) for generating information from said features, and a decoder comprising a self-attention layer (SL), said decoder being adapted to generate predictions (P1, P2, ..., PN) from object queries (Q1, Q(2),..., QN) based on information (EO) provided by said encoder (NE), wherein said object queries are structured in the form of subgroups (G1, G(2),..., GK) forming a partition of the set of said queries, said learning comprising a search for matching the predictions of each subgroup with the same training set, wherein said self-attention layer (SL) is adapted to allow independent learning for each subgroup, wherein an attention mask (AM) is applied to said self-attention layer (SL), said attention mask being provided to allow the transmission of information only within a subgroup and wherein a cost function is provided to optimise the diversification of said object queries.

2. A method according to the preceding claim, wherein said cost function is based on a supervised correlation matrix during said learning so that its values lie between two predefined bounds.

3. A method for detecting objects within a digital image, said method comprising a learning process according to one of the preceding claims.

4. A method according to one of the preceding claims, wherein said digital image is derived from a stream of images generated by a video camera.

5. A computer program capable of being implemented on an access point in a wireless telecommunications network, the program comprising code instructions which, when executed by a processor, perform the steps of the method defined in claims 1 to 4.

6. A processing device comprising means for implementing the method according to one of claims 1 to 4.