Device and method for detecting objects in aerial images

Abstract

Device and Method for Detecting Objects in Aerial Images. The present invention relates to a device for detecting a number N0 of objects in a current aerial image, the device comprising: a. at least one input interface configured to receive said current aerial image; b. at least one processor configured to: i. apply a learned function called the detection function to said current aerial image, so as to generate a detection result, said detection function being derived from a pre-trained model; c. at least one output interface configured to return said detection result. [Fig. 7]

Description

Title of the invention: Device and method for detecting objects in aerial images

[0001] The invention relates to the field of object detection in aerial images, in particular taken by aerial vectors of drone systems.

[0002] Drone systems are increasingly used in a wide variety of inspection and support missions. One potential application is assistance in firefighting operations, where images recorded by aerial vehicles can be used to detect and identify fire outbreaks.

[0003] Processing aerial images is difficult because these images exhibit both relatively low resolution and large scale variations. Furthermore, in the case of images taken by aerial vehicles, the challenge for real-time image processing lies in designing software that is both fast (and therefore resource-efficient) and accurate. Current methods are either designed for rapid inference, at the expense of detection accuracy, or designed to improve this accuracy, but consequently consume significant computing resources. There is also a need to develop a detection method suitable for edge computing. Moreover, a fast detection method could advantageously be implemented in parallel with other computing processes.

[0004] The invention presented aims to solve this problem. Description of the invention

[0005] To this end, the invention proposes a device and a method for detecting objects in aerial images in real time and with high precision.

[0006] A device is thus proposed for detecting a number N0 of objects in a current aerial image, the device comprising: - at least one input interface configured to receive the current aerial image; - at least one processor configured for: • Apply a learned function called a detection function to the current aerial image, so as to generate a detection result, the detection function being derived from a pre-trained model; - at least one output interface configured to return the detection result, the model comprising a set of floors including a first floor and a last floor, each floor returning as output a semantic feature map with a resolution higher than the semantic feature map returned by the corresponding lower floor, each floor above the first floor returning the corresponding semantic feature map from the semantic feature map returned by the corresponding lower floor,

[0007] at least one floor among the set of floors comprising a sequence consisting of a module comprising a Vision Transformer type sub-module followed by an attention module.

[0008] Thus, advantageously, such a model architecture makes it possible to obtain a detection result that is both accurate and fast, and allows the device according to the invention to be used in real-time applications. Another advantage is that such an architecture is scalable, so that it can be adapted to a wide variety of processor performance. For example, the device can be an emulated virtual machine activated remotely. Also, the model's stage set, returning semantic feature maps of different resolutions, makes it possible to generate a detection function capable of detecting objects of different sizes in the current aerial image.

[0009] Advantageously, the number No of detected objects is an integer. By integer, it is understood to be a zero number or a strictly positive integer.

[0010] In some embodiments, the model comprises:

[0011] - a feature extraction module configured to extract a plurality of current aerial image characteristic maps;

[0012] - a feature enhancement module comprising the set of stages and being configured to return semantic features from semantic feature maps as output, with at least one floor of the floor set receiving features directly or indirectly from the feature extraction module; and

[0013] - a prediction module configured to calculate at least one prediction on the basis semantic characteristics, the detection result being based on at least one prediction.

[0014] The term "directly" here means that the features are received by at least one stage, in operation, of the feature extraction module without intermediate layers or computation blocks. The term "indirectly" means that at least one stage is connected to the feature extraction module via intermediate computation layers or computation blocks.

[0015] In some embodiments, the first stage is connected to the feature extraction module only by convolution blocks and oversampling modules.

[0016] Thus, the architecture of the model used in the invention is simple and concise, without repeated layers, which enhances the performance of the device in operation, particularly in terms of inference speed due to reduced computational costs. In other words, the model is designed to perform fewer computational steps, and the detection function can therefore be implemented more efficiently by the device.

[0017] In certain embodiments, all stages of the stage set comprise a sequence consisting of a module including a Vision Transformer sub-module followed by an attention module. Integrating such a sequence into all stages of the feature enhancement module improves the accuracy of the detection result achieved by the device. Furthermore, it is more common and conventional to use convolution modules in the architecture of models dedicated to object detection, and much less natural to use Vision Transformer modules in these models.

[0018] In some embodiments, the at least one stage comprises a sequence of between one and three modules comprising a Vision Transformer type sub-module followed by an attention module.

[0019] Advantageously, the feature enhancement module comprises three stages.

[0020] Advantageously, the feature enhancement module comprises between three and five stages. A low number of stages allows for faster model inference. A high number of stages allows the model's performance to be adapted to detect large aerial images that may contain objects of varying sizes, ranging from small to large, relative to the dimensions of the aerial images.

[0021] In some embodiments, the feature extraction module consists of a set formed solely of convolution layers and residual blocks, each composed of a plurality of layers, the set comprising a total number of layers less than 50. In these embodiments, the feature extraction module does not include any other structures, in particular no Vision Transformer type sub-module or attention module.

[0022] Advantageously, the feature enrichment module includes a feature pyramid network neural network.

[0023] Advantageously, when the number No of detected objects is greater than or equal to 1, the detection result includes, for each detected object, a class of the object detected, a position information of the detected object in the current aerial image, and optionally a confidence score representative of the reliability of object detection.

[0024] Advantageously, when the number No of detected objects is greater than or equal to 1, the detection result further includes, for each object among the at least one object present, a confidence score representative of the object's detection reliability.

[0025] Advantageously, when the number No of detected objects is greater than or equal to 1, the detection result further includes, for each object among the at least one object present, a set of probabilities, each probability corresponding to the probability that the object belongs to a class among a set of classes.

[0026] Advantageously, when the number No of detected objects is greater than or equal to 1, the detection result includes, for each object among the at least one detected object, an object presence value between 0 and 1. The object presence value is also referred to as "objectness". When the object presence value is equal to 1, the corresponding object is detected with 100% reliability. When the object presence value is equal to 0, the corresponding object is detected with zero reliability.

[0027] In some embodiments, the Vision Transformer type submodule comprises a set of layer pairs, each pair comprising a first layer and a second layer, the module being configured to calculate, for each layer among the set of layer pairs, a set of attention scores for a set of windows forming between them a partition of an input image, the partition for a second layer of a layer pair being offset with respect to the partition for the first layer of said layer pair.

[0028] Thus, advantageously, by using offset partitions, the attention mechanism of the Transformer-type submodule applied to the input tokens (consisting of patches of the current aerial image) received as input can process neighboring patches, thereby efficiently capturing the local context, improving object recognition, and contributing to the acquisition of enriched features. In other words, a submodule of this type proves more accurate in a detection task, and consequently, the same is true for the detection function applied to the current aerial image.

[0029] In some embodiments, the Transformer type submodule is a Vision Transformer type module with offset windows.

[0030] In other embodiments, the Transformer-type submodule consists of an association of attention layers and a feedforward neural network.

[0031] In some embodiments, the attention module of at least one stage comprises a sequence of a first channel attention sub-module and a second spatial attention sub-module, the first channel attention sub-module being configured to apply a first attention algorithm to at least a part of the features from the module comprising the Transformer type sub-module, the second spatial attention sub-module being configured to apply a second attention algorithm.

[0032] Thus, the integration of such an attention module makes it possible to extract and retain important information both spatially and from the point of view of the channels and to attenuate (or even suppress) less important information from these two points of view.

[0033] In certain embodiments, the detection function is configured to perform fewer than 150 GigaFLOPS of floating-point operations per second. Compared to other models comprising a feature enhancement module (known as a "neck"), the model according to the invention makes it possible to generate a detection function performing a reduced number of operations, which is thus suitable for real-time applications.

[0034] In some embodiments, the model comprises a number N of layers.

[0035] In some embodiments, the model consists of a set of N layers arranged in a linear fashion.

[0036] In some embodiments, N is less than or equal to 100.

[0037] In some embodiments, the model comprises layers connected by non-linear paths.

[0038] In certain embodiments, at least one stage of the feature enhancement module comprises, downstream of said sequence, an additional module of the reparameterization type. Advantageously, the addition of such an additional module improves the inference speed and therefore the performance of the detection device, while maintaining the detection accuracy of the device that was determined during the training phase. The improvement in inference speed is linked to the reparameterization performed during the training phase. The model, in its version for the inference phase, has a reduced number of parameters and a reduced number of floating-point operations per second (FLOPS).

[0039] In certain embodiments, each stage of the feature enhancement module includes, downstream of said sequence, an additional module of the type of reconfiguration module.

[0040] In other embodiments, each stage of the feature enrichment module includes, downstream of said sequence, an Atrous convolution module.

[0041] In still other embodiments, at least one stage of the feature enrichment module comprises, downstream of said sequence, a Ghost module. Advantageously, a Ghost module makes it possible to increase the efficiency of the feature enrichment module by using a limited number of parameters while increasing the number of features output from the module.

[0042] In yet other embodiments, each stage of the feature enhancement module includes, downstream of said sequence, a Ghost module.

[0043] In certain embodiments, the prediction module includes at least one implicit addition block and / or one implicit multiplication block. Advantageously, such a block contributes to the feature enrichment task performed by the model as well as to improving the detection accuracy achieved by the detection function.

[0044] In some embodiments, the device is a computing device embedded in a drone.

[0045] Another aspect of the invention relates to a system for detecting a number N0 of objects in a current aerial image comprising:

[0046] - an on-board image sensor configured to acquire aerial images;

[0047] - a device for detecting a number N0 of objects in a current aerial image as described previously,

[0048] at least one input interface of said device being configured to receive aerial images, and the current aerial image being one among the aerial images.

[0049] In some embodiments, the system is an assembly embedded in a drone and comprising an image sensor and a computing device.

[0050] Another aspect of the invention relates to a computer-implemented method for detecting a number N0 of objects in a current aerial image implemented by a device as described above, the method comprising:

[0051] - a. (S10) receive said current aerial image;

[0052] - b. (S20) apply a learned function called a detection function to said current aerial image, so as to generate a detection result, said detection function being derived from a pre-trained model;

[0053] - c. (S30) return said detection result,

[0054] the model comprising a set of floors including a first floor and a last floor, each floor returning as output a semantic feature map with a resolution higher than the semantic feature map returned by the corresponding lower floor, each floor higher than the first floor returning the corresponding semantic feature map from the semantic feature map returned by the corresponding lower level,

[0055] at least one floor among the set of floors comprising a sequence consisting of a module comprising a Vision Transformer type sub-module followed by an attention module.

[0056] In some embodiments, the method further includes: d. triggering an action, for example an alert, based on at least one value of the detection result.

[0057] A computer program, implementing all or part of the method described above, installed on pre-existing equipment, is in itself advantageous.

[0058] Thus, the present invention also relates to a computer program comprising instructions for implementing the method described above, when this program is executed by a processor.

[0059] This program may use any programming language (for example, an object-oriented language or other), and may be in the form of interpretable source code, partially compiled code or fully compiled code.

[0060] Another aspect relates to a non-transient storage medium for a computer-executable program, comprising a set of data representing one or more programs, said one or more programs comprising instructions for, when said one or more programs are executed by a computer comprising a processing unit operationally coupled to memory means and an input / output interface module, to execute all or part of the method described above. Brief description of the drawings

[0061] Other features, details, and advantages of the invention will become apparent from the detailed description below. This description is purely illustrative and should be read in conjunction with the accompanying drawings, in which:

[0062] [Fig.1] schematically represents a situation in which one or more embodiments of the invention can be executed;

[0063] [Fig.2] represents an example of a set of components of a device command to detect a number No of objects in a current aerial image according to an embodiment of the invention;

[0064] [Fig.3] represents a flowchart with an example of a set of steps that can be executed to detect a number No of objects in a current aerial image according to an embodiment of the invention;

[0065] [Fig.4] schematically represents an architecture of a model to generate a function used by a device to detect a number No of objects according to an embodiment of the invention;

[0066] [Fig.5] represents an example of the architecture of a feature extraction module from the model of [Fig.4] which can be used in embodiments of the invention;

[0067] [Fig.6] schematically represents a network of the pyramid type of characteristics;

[0068] [Fig.7] represents an example of the architecture of a feature enrichment module of the model of [Fig.4] which can be used in embodiments of the invention;

[0069] [Fig.8] represents an example of offset partitions of the same image;

[0070] [Fig.9] schematically represents an architecture of a sub-module of the type Transformer with offset windows (Swin Transformer);

[0071] [Fig. 10] represents two successive Swin Transformer blocks in the offset window Transformer type submodule of [Fig.9];

[0072] [Fig. 11] represents an example of a module comprising a sub-module of the Transformer type with offset windows;

[0073] [Fig. 12] represents part of a RepConv type block, or reparameterization block, for training and for inference;

[0074] [Fig. 13] represents an example of a portion of the architecture of a feature enhancement module of the model of [Fig.4] according to an embodiment of the invention;

[0075] [Fig. 14] represents an example of the architecture of a model prediction module of [Fig. 4] that can be used in certain embodiments of the invention. Detailed description

[0076] A device 100 is proposed for detecting a number No of objects in an aerial image and an associated method 200 implemented by computer for detecting the presence of an object in a current aerial image.

[0077] Figure 1 schematically illustrates a situation in which Method 200 according to this disclosure can be implemented. More specifically, Figure 1 schematically represents elements relating to a firefighting intervention mission assisted by a drone system. A drone 10 is shown, which detects objects such as a fire or vegetation in the field of view of its onboard camera and records a corresponding image stream. The image stream is sent to a ground station associated with an operator, possibly via an intermediate operator manipulating a remote control device 300. The device 100 can be this ground station and implement method 200 according to the invention in order to detect fire outbreaks and specific objects, visualize the results associated with the detection, and, where appropriate, perform post-processing operations and trigger actions.

[0078] Advantageously, the drone 10 can include an on-board computing system which, alternatively to a ground station, can constitute the device 100 and implement the method 200.

[0079] Other applications of method 200 exist when an operation to detect objects in aerial images is envisaged.

[0080] In this application, a drone system is defined as a system comprising one or more autonomous aerial vehicles equipped with detection sensors and one or more ground stations for command and data collection from the detection sensors. Hereinafter, the autonomous aerial vehicles will be referred to as "drones." The drones together form a drone fleet. The ground station(s) remotely control the drones.

[0081] In some embodiments, one or more intermediate remote control devices remotely control the drone(s) and receive and transmit information to the ground station(s).

[0082] As seen above, drones are equipped with detection sensors such as cameras that detect and record images. Drones may include other detection sensors. Images detected and recorded by a drone 10 can be transmitted to the ground station(s) for processing.

[0083] Fig. 2 represents an example of device 100 in an embodiment of the invention and capable of implementing method 200 of detecting the presence of an object in an aerial image.

[0084] In this embodiment, the device 100 includes one or more memories 102, 103 for storing instructions enabling the implementation of the method 200 for detecting an object, the measurement data received, and temporary data for carrying out the different steps of the method 200.

[0085] The device 100 further comprises a circuit 101. This circuit may be, for example: - a processor capable of interpreting instructions in the form of a computer program (for example, a GPU graphics processor, or - an electronic board whose steps of the process of the invention are described in silicon, or - a programmable electronic chip such as an FPGA chip (for "Field-Programmable Gate Array" in English), such as a SoC (for "System On Chip") in English) or as an ASIC (for "Application Specified Integrated Circuit" in English).

[0086] SoCs or system on chip are embedded systems that integrate all the components of an electronic system into a single chip.

[0087] An ASIC is a specialized electronic circuit that combines features tailored to a specific application. ASICs are generally configured during manufacturing and can only be simulated by the user.

[0088] Field-Programmable Gate Array (FPGA) type programmable logic circuits are electronic circuits that can be reconfigured by the user.

[0089] This device 100 includes input and output interfaces 107 for receiving measurement data, in particular images from the drone 10 via a network 104, and for providing output data, in particular the results obtained from implementing method 200 according to the invention. Finally, the station 100 may include a screen and a keyboard to allow for easy interaction with a user. Of course, the keyboard is optional, particularly in the case of a station in the form of a touchscreen tablet, for example.

[0090] Depending on the embodiment, the device 100 may be a computer, a computer network, an electronic component, or another device comprising a processor operationally coupled to memory, as well as, depending on the chosen embodiment, a data storage unit, and other associated hardware elements such as a network interface and a media reader for reading and writing to removable storage media (not shown in the figure). The removable storage media may be, for example, a compact disc (CD), a digital video / multipurpose disc (DVD), a flash drive, a USB flash drive, etc.

[0091] Depending on the embodiment, the memory, data storage unit or removable storage medium contains instructions which, when executed by the control circuit 101, cause this control circuit 201 to perform or control the input interface and output interface 207, data storage in the memory(ies) 102, 103 and / or data processing parts of the implementation examples of the proposed method described herein.

[0092] In other embodiments and as mentioned above, method 200 can be implemented by a computing system embedded in the drone 10. In these embodiments, the embedded computing system directly receives the images recorded by the on-board sensor of the drone 10 and implements method 200 from the recorded images.

[0093] Method 200 will be described below with reference to [Fig. 3]. It is assumed that the drone 10 is in mission flight after taking off following an operator command, and is recording images. The images may be photographs or frames of a video stream. For example, the number of frames per second transmitted by drone 10 can be between 24 and 30 frames per second (fps). Drone 10 remains connected to device 100 (or an intermediate remote control device 300) with which it exchanges information and from which it receives instructions.

[0094] Method 200 implements a pre-trained machine learning model, which is advantageously implemented during an inference phase of this model. As a reminder, the inference phase of a model refers to the process during which the previously trained model is used to make predictions from new data that was not used during the model's training phase.

[0095] In a step S10, the device 100 receives a current image Io from a detection sensor of the drone 10.

[0096] In a step S20, a learned function called the detection function is applied to the current image Io by the control circuit 101 of the device 100. The detection function is configured to determine the presence, in the current image Io, of objects belonging to one class among a plurality of classes. Examples of classes are: a pedestrian, a set of flames, smoke, a land vehicle. Other examples of classes include: a bicycle, a car, a van, a truck, a tricycle, a bus, a motorcycle. For example, the plurality of classes comprises 10 classes. In another example, the plurality of classes comprises 12 classes. The detection function is derived from a previously trained model M, the architecture of which will be described below.As an example, model M can be a model built with the PyTorch interface, and the current image 10 can be converted into a string (for example, by Base64 conversion), which can then be converted into a tensor that can be used by the PyTorch model. The PyTorch interface is a programming interface for the PyTorch software library for deep learning, providing developers with conventions and functions for creating machine learning architectures, particularly those based on neural networks.

[0097] In a step S30, the detection function returns an output result R called the detection result. The detection function is configured to detect a number No of objects in the current image. No is a positive integer or zero. In other words, it is possible that when the detection function is applied to the current image Io, no objects will be detected in the current image Io. When the detection function determines the presence of a number No of objects greater than or equal to 1 in the current image Io, the detection result R includes a list of detected objects and, for each detected object, a data set including: information on The object's position B, the object's CL class, and optionally a confidence score representing the accuracy of the detection function's prediction of the object's presence. When the detection function does not determine the presence of any object in the current image Io, the detection result R can be a message indicating the absence of a detected object or an empty dataset.

[0098] In some embodiments, the position information of a detected object makes it possible to identify a boundary delimiting the position of the detected object on the current image Io. For example, the boundary is a bounding box. For example, the position information B of a detected object includes the coordinates in the current image Io of one vertex of the bounding box and those of the opposite vertex.

[0099] The model M from which the detection function is derived has an architecture schematically represented in [Fig. 4] and consists of a feature extraction module RI, a feature enrichment module R2, and a prediction module R3 connected in series. The architecture of model M is thus of the "backbone, neck, and head" type.

[0100] The feature extraction module RI consists of a first artificial neural network configured to receive the current image Io as input and return features as output. The feature enrichment module R2 consists of a second artificial neural network configured to receive features from the first artificial neural network RI as input and return new features called enriched features as output. The prediction module R3 consists of a third artificial neural network configured to receive enriched features from the second artificial neural network R2 as input and calculate at least one prediction.

[0101] In some embodiments, the first artificial neural network (ANN) comprises a set of convolutional layers and residual connections including residual blocks. Most of the convolutional layers are arranged linearly. The residual connections ensure better gradient flow, similarly to ResNet architectures. An example of an ANN that can be used within the scope of the invention is illustrated in [Fig. 5]. In this example, the ANN consists of 50 convolutional layers, each equipped with a batch normalization layer and a ReLU activation layer. A ReLU activation layer is a layer comprising a unit of linear rectification implementing a function that returns the value 0 for a negative variable value and the identity for a positive or zero variable value.

[0102] The list below gives a definition for each block shown in [Fig. 5]. In this figure, the formalism is as follows for the notation "3x3 conv, 32, / 1": 2D convolution layer with 32 independent kernel filters of size 3x3 and a step (in English "stride") of 1. The notation "Residual Block (64,128)x2" denotes the repetition of two structures each comprising a convolution layer comprising 64 filters and then 128 filters, followed by a residual block.

[0103] 3x3 conv: denotes a 3x3 convolutional module. Such a module applies a filter (kernel) of size 3x3 to an input feature map. The filter slides across the input map, calculating weighted sums of neighboring pixels. The resulting feature map output by this module detects spatial patterns and hierarchical representations.The use of such a module offers several advantages: it provides translation invariance; it allows for the capture of local spatial hierarchies, enabling the network to learn features at different scales; the use of smaller filters (compared to larger 5x5 or 7x7 filters) reduces the number of parameters; it introduces non-linearity through activation functions (such as ReLU), improving the expressive power of the model; more 3x3 layers can be stacked, allowing for the creation of deep networks; 3x3 convolutions can be performed in parallel; and 3x3 filters are effective in edge and gradient detection.

[0104] Residual block: A residual block consists of a skip connection that links the input of one layer to the output of a subsequent layer, bypassing certain intermediate layers. Such a block allows the network to learn the residual mapping (i.e., the difference between the desired output, subsequent layer, and the input) rather than directly learning the underlying mapping.Using such a block offers several advantages: a "skip" connection ensures that gradients can pass directly from output to input without fading / exploding, allowing for the formation of very deep networks; if a layer impairs performance, the "skip" connection allows it to be ignored during training, thus acting as implicit regularization; furthermore, the use of this type of architecture offers remarkable accuracy in various tasks, including image classification and object detection.

[0105] SPPCSPC (Spatial Pyramid Pooling Cross Stage Partial Connections): such a module combines the functionalities of a pyramid pooling module (SPP) and a partial network. Cross-stage partial networks (CSPNet); the pyramidal pooling module SPP pools features at multiple scales, enabling the capture of spatial hierarchies and multi-scale information. This is achieved by applying maximum grouping operations with different kernel sizes to the same input feature map, then concatenating the resulting grouped features. Therefore, regardless of the input map size, the SPP module generates a fixed-size output, simplifying the processing of features with varying spatial dimensions. A CSPNet-type network divides an input feature map into two parts.One part is processed by a series of convolutional layers, while the other is directly connected to the output, preserving the gradient flow and improving the network's learning capacity. The two parts are then concatenated, allowing the network to efficiently merge different levels of features. The SPPCSPC combination enriches the feature representation by capturing multi-scale information and maintaining the gradient flow, effectively improving both the network's speed and accuracy.

[0106] A SPPCSPC combination module offers a plurality of advantages: - Multi-scale feature extraction: The SPP pyramidal pooling component allows the network to extract features at multiple scales, which is essential for detecting objects of different sizes. This enables the capture of both fine and coarse image details.

[0107] - Gradient flow improvement: the staged partial network component Cross-connections in CSPNet ensure better gradient flow during backpropagation by dividing the feature map and allowing partial connections. This reduces the risk of gradient vanishing and enables more efficient training of deeper networks. - Reduced computational complexity: The CSPNet partial cross-stage network design reduces the network's computational complexity by dividing the input feature map and processing only a portion of it through convolutional layers. This makes the network more efficient without compromising performance. - Improved feature blending: By combining features processed by convolutional layers with unprocessed features, the CSPNet cross-stage partial network enables better feature blending. This results in richer feature representations that can improve object detection accuracy. - Flexibility: The fixed-size output of the SPP pyramidal pooling component allows for flexible processing of input feature cards of varying sizes. The SPPCSPC block can therefore adapt to different network architectures and input dimensions.

[0108] The first artificial neural network RI allows the extraction of a plurality of feature maps as it progresses through its layers, the features extracted from the first layers being so-called high-level features, such as edges, colors or patterns, and the features extracted from the deepest layers being semantic features.

[0109] In preferred embodiments, the RI feature extraction module comprises a set of layers with a number of layers less than or equal to 50. Thus, the RI feature extraction module is small in size, which is advantageous for real-time M-model applications.

[0110] In preferred embodiments, the feature enrichment module (second artificial neural network) R2 is of the Feature Pyramid Network (FPN) type and comprises an integer number N of stages.

[0111] As will be described later, the design of the architecture of the second artificial neural network R2 (the "neck") plays a crucial role in the trade-off between detection accuracy and the inference speed of the M model. Prior art "neck" module architectures include a high number of layers to improve detection accuracy at the cost of slow inference speed, making them unsuitable for real-time applications, even when using powerful graphics processors. The structure of the second artificial neural network R2 according to the invention has, compared to prior art solutions, a simpler and more concise architecture (in other words, with a reduced number of computational layers and computing blocks).

[0112] Figure 6 schematically illustrates an FPN feature pyramid network. A feature pyramid network consists of three main elements: a bottom-up pathway (BU) configured to decrease the spatial resolution of input data while providing output feature maps, such as maps C1 and C2 in Figure 6, with increased semantic richness; a top-down pathway (TD) equipped with lateral connections (LC) configured to increase the resolution of the semantic feature maps from the bottom-up pathway; and the lateral connections (LC) used to fuse the output data of the bottom-up pathway (BU) with that of the top-down pathway (TD). Thus, an FPN feature pyramid network results in a pyramid of multi-scale feature cards with enhanced semantic richness at each level.

[0113] Alternatively, the feature enrichment module can be of the type Path Aggregation Network (PANet). This is a variant of a Feature Pyramid Network (FPN) type network, comprising an additional upstream path merging each feature map from a tier of the TD downstream path with the feature map of the lower (i.e., higher resolution) tier.

[0114] In other embodiments, the feature enrichment module may have a network structure known as a bidirectional feature pyramid network (BiFPN). A bidirectional feature pyramid network is similar to a PANet path aggregation network, where each input feature of a given resolution feature map has a weight intended to take into account the importance of one input feature relative to another.

[0115] In these embodiments where the feature enrichment module R2 is of the feature pyramid network (FPN) type, at least one stage of the feature pyramid network receives features as input directly or indirectly from a layer C2, C3 of the feature extraction module (first artificial neural network) RI. The merging of the output data from the upstream channel with those from the downstream channel is performed by receiving, as input to stages other than the stage corresponding to the lowest spatial resolution, in the feature pyramid network, the output data from the lower-resolution stage. For example, the output of stage i of the network can be obtained by: p?'^ - Conv(I^^ResizeiP^) ), where Resize is a scaling operation (upsampling or downsampling) to adjust the resolutions of the two terms.

[0116] In some preferred embodiments, the number of floors N is equal to three floors. In other embodiments, the number of floors N is greater than three floors.

[0117] In preferred embodiments, at least one stage of the second artificial neural network R2 comprises a sequence of modules consisting of a module T including a Vision Transformer sub-module and an attention module A. Such a combination improves the performance of the second artificial neural network R2. The attention module A is configured to receive feature maps generated by the module T. The feature maps generated by the module T are tensors having parameters a width, a height, and a number of channels. In one example, feature maps are received in batches.

[0118] In preferred embodiments, each stage of the second artificial neural network R2 comprises such a sequence.

[0119] A description of a Vision Transformer-type submodule architecture can be found in the article "An image is worth 16x16 words: Transformers for image recognition at scale", Dosovitskiy A. et al., ICLR, (2021). A Vision Transformer-type architecture is an architecture adapted from Transformer-type architectures for processing two-dimensional images, using as input elements portions of an input image to be processed that form a partition of that input image.

[0120] More specifically, a Vision Transformer architecture partitions an input image to be processed into input elements, which are transformed into one-dimensional vectors. Positional encodings are then applied to enrich the one-dimensional vectors. The one-dimensional vectors enriched with positional encodings are then fed into a Transformer architecture.

[0121] As a reminder, a Transformer-type architecture comprises a sequence of modules, each module including a multi-head self-attention (MSA) module and a feed-forward neural network. The multi-head self-attention module weighs the relevance of each input token (element) against each other input token (element) and processes information accordingly to produce the output. Each output element from this sequence of modules is taken sequentially and passes through a classification module.

[0122] In this disclosure, the term "attention module" refers to a module configured to implement an attention mechanism, as known in the field of machine learning. A definition of attention in the field of machine learning can be found at the following link: https: / / en.wikipedia.org / wiki / Attention_(machine_learning). Attention Module A is configured to generate enriched feature maps by applying one or more attention mechanisms to feature maps received from Module T, which includes a Vision Transformer sub-module.

[0123] Figure 7 shows an example of the architecture of the feature enrichment module (second artificial neural network) R2 that can be used according to an embodiment of the object detection method 200. In this figure, the second artificial neural network R2 is a three-stage feature pyramid network (E1, E2, E3). Stage E1 receives as input CB data from a combination of feature maps C1, C2, and C3 from the deepest layers of the first artificial neural network RI shown. [Fig. 5]. Stage E3 receives as input, in addition to the data from stage E2, the feature map C3 from the deepest layer of the first artificial neural network RI. An example of combining the feature maps C1, C2, and C3 from the RI feature extraction module to obtain the CB data will be described later.

[0124] In the example in [Fig. 7], an MP block represents a so-called max pooling block. As a reminder, max pooling is a subsampling operation that reduces the size of feature maps. It consists of dragging a filter (usually a small window) over the feature map and selecting the maximum value in each region covered by the filter. Instead of using all the precise features generated by the previous convolutional layers, max pooling summarizes the most important features in a local region. This synthesis reduces the computational load and the number of parameters in the network. This technique has several advantages:

[0125] - Invariance with respect to changes in scale and orientation: the technique Maximum pooling allows for the detection of features that remain invariant to changes in scale and orientation. For example, if an object is rotated or scaled, the same main features will still be captured by the maximum pooling operation. Thus, by summarizing features, the maximum pooling technique makes the model more robust to variations in feature position within the input image and ensures that the network focuses on essential patterns rather than precise spatial locations.

[0126] - Minimizing overfitting: By reducing the By spatially resolving feature maps, the maximum pooling technique prevents overfitting of a model. It encourages the model to learn more general features rather than memorizing specific details from the training data.

[0127] - Computational efficiency: the maximum pooling technique reduces the The complexity of calculations is reduced by minimizing the sampling of feature maps. This allows subsequent layers to operate on smaller representations, which speeds up training and inference.

[0128] - Increased receptive field: the technique of Pooling the maximum potential increases the receptive field of neurons in deeper layers. Neurons can "see" a broader context, which is beneficial for capturing global patterns.

[0129] On [Fig.7] also, the modules marked “CONV” denote convolution layers with kernels of size 1x1 or 3x3.

[0130] In certain embodiments, the Vision Transformer T type submodule comprises a set of layer pairs, each pair comprising a first layer and a last layer. The Vision Transformer T type submodule is then configured to calculate, for each layer among the set of layer pairs, a set of attention scores for a set of windows forming a partition of an input image, the partition for a second layer of a layer pair being offset with respect to the partition for the first layer of said layer pair.

[0131] Figure 8 shows an example of two partitions P1 and P1+1 corresponding to two successive layers. Each of the partitions P1 and P1+1 comprises a set of windows W, each containing a set of patches P. The windows W of the partition P1+1 are offset relative to the windows W of the partition P1. The attention module receiving the partition P1 or P1+1 calculates an attention score for each window W. The attention scores calculated on the partition P1+1 cross the boundaries of the windows W of the partition P1, which has the effect of creating connections between the windows of the partition P1 in the attention scores calculated on the partition P1+1 and thus improving the model.

[0132] According to an example, the Vision Transformer type submodule can be a submodule of the Vision Transformer type with offset windows (in English “Swin Transformer” Swin TR).

[0133] An example of a Vision Transformer type submodule with offset windows Swin TR is illustrated in [Fig.9]. This module comprises the succession of a first submodule SMI, a second submodule SM2, a third submodule SM3 and a fourth submodule SM4.

[0134] The first submodule SMI comprises a linear embedding layer represented by the rectangle with zigzags and two blocks referred to herein as offset window blocks represented by the hatched rectangle with diagonal lines and the multiplicative coefficient x2. The second submodule SM2 comprises a patch fusion layer represented by the dotted-filled rectangle and two offset window blocks represented by the hatched rectangle with diagonal lines and the multiplicative coefficient x2. The third submodule SM3 comprises a patch fusion layer and six offset window blocks represented by the hatched rectangle with diagonal lines and the multiplicative coefficient x6. The fourth submodule SM4 comprises a patch fusion layer and two offset window blocks represented by the hatched rectangle with diagonal lines and the multiplicative coefficient x2.By patch fusion layer, it is understood, in . the present application, a layer receiving as input a set of NxM patches each comprising a determined number of features, this layer returning as output a reduced number of patches.

[0135] For example, such a layer can concatenate the features of groups of patches while modifying the dimensionality of the patch features. If an input patch comprises C features, a patch merging layer concatenating the features of groups of 2x2 patches will decrease the number of tokens by a factor of 2x2 and may return 2C features per group of 2x2 merged patches.

[0136] Figure 10 shows the structure of a succession of two staggered-window blocks. The first staggered-window block comprises a classical self-attention multihead module (MSA), followed by a multilayer perceptron module (MLP). A normalization layer (LN) is applied to both the MSA and the MLP, and a residual connection is applied to each of the MSA and MLP modules. The second staggered-window block comprises, instead of a classical MSA, a staggered-window-based module called S-MSA.The set of features received by the first offset window block is denoted Z1', the set of features calculated by the first offset window block is denoted z* and constitutes the input data of the second offset window block, the set of features calculated by the second Swin Transformer block is denoted ^+1. For example, the image partitions received by the first offset window block and by the second offset window block can correspond to the partitions PI and P1+1 described in [Fig.8].

[0137] Fig. 11 illustrates an example of a T module comprising a Vision Transformer Swin TR type sub-module with offset windows that can be integrated into at least one stage of the feature enhancement module R2.

[0138] In this example, the T module comprises three parallel branches. A first branch, shown on the left of [Fig. 11], comprises a convolution block. A second branch, shown in the center of [Fig. 11], comprises a convolution block and a set of Bottleneck BN modules. A third branch, shown on the right of [Fig. 11], comprises a convolution block followed by the Vision Transformer submodule with offset windows. The outputs of the first, second, and third branches are joined and concatenated by a concatenation block, the output of which is injected into a convolution block. The output data of the T module is the data from the last convolution block.

[0139] The use of a Vision Transformer with offset windows in the feature enhancement module offers several advantages: by using offset partitions, the attention mechanism applied to the input tokens (which constitute the image patches) can process neighboring patches, thus efficiently capturing the local context and improving object recognition. In other words, it is possible to efficiently capture contextual information with a Transformer with offset windows. This understanding of the local context facilitates object recognition in cluttered scenes; moreover, Transformer with offset windows performs better, in terms of accuracy, than traditional core networks (such as ResNet and EfficientNet).Their hierarchical design allows them to learn discriminating features for object detection tasks; also, despite the large size of their model, offset window Transformer modules remain computationally efficient due to their restricted field of attention, which makes them practical for real-world applications; moreover, they offer design flexibility that allows for a balance between accuracy and computational cost.

[0140] According to embodiments that can be combined with the preceding embodiments, the attention module A comprises a sequence consisting of a first channel attention submodule and a second spatial attention submodule. The first channel attention submodule is configured to apply a first attention mechanism to at least some of the features from the module T comprising the Transformer-type submodule. The second spatial attention submodule is configured to apply a second attention mechanism to the result of the first channel attention submodule. Such an attention module A makes it possible to extract and retain important information both spatially and from the channel perspective and to attenuate (or even suppress) less important information from both perspectives.

[0141] For example, the attention module A can be a convolutional block attention module (CBAM). A description of such a module can be found in the article "CBAM: Convolutional Block Attention Module" by WOO et al., in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 3–19. More specifically, a CBAM receives as input a feature map F belonging to the space and sequentially returns a first attention map Mc belonging to the space pCxlxl, obtained by applying a first channel attention module to the feature map F, and an intermediate feature map F' obtained by the element-by-element product of F and the first attention map Mc; and then a spatial attention map. two-dimensional Ms belonging to space and obtained by applying a second submodule of spatial attention to the intermediate feature map F', as well as a final feature map F” obtained by the element-by-element product of F' by the two-dimensional spatial attention map Ms.

[0142] The first channel attention submodule of a CB AM convolutional block attention module advantageously uses a combination of maximum pooling and average pooling operations, so as to obtain a first category of descriptors resulting from maximum pooling and a second category of descriptors resulting from average pooling. The first and second categories are injected into a multi-layer perceptron network, whose two outputs are then summed element by element. For an input feature map F, the attention map resulting from the first channel attention submodule can be written as: Mc = <t(MLP ( AvgPool ( F ) ) + MLP ( MaxPool ( F ) ) ) avec o la fonction sigmoide.

[0143] Le deuxième sous-module d’attention spatiale utilise également avantageusement des opérations de mise en commun du maximum et de mise en commun de la moyenne, ainsi que des opérateurs de convolution. Pour une carte de caractéristiques d’entrée F, la carte d’attention issue du deuxième sous-module d’attention spatiale peut s’écrire par exemple : MS(F) = a(fx\AvgPool(F) + MaxPool(F) ) )' with f7x7 a convolution operator of filter size 7x7.

[0144] Advantageously, each stage of the second artificial neural network R2 comprises, downstream of the sequence of module T, which includes a Transformer-type sub-module, and attention module A, an additional module RC called the reparameterization module. The additional module RC is configured to switch from a branching structure dedicated to a training phase to a direct-action structure dedicated to an inference phase. The addition of such a module improves the performance (i.e., the inference time) of model M while maintaining detection accuracy.

[0145] For example, the additional RC module can be of type RepVGG (or RepConv). A RepVGG module is a stack of layers that has a first configuration dedicated to its training, and a second configuration challenged during its inference phase, so that the module is made more accurate thanks to its training configuration, and faster thanks to its lighter inference configuration. In [Fig. 12], the Ltr_i layer represents the structure of a layer of the first configuration and includes a 3x3 conv block, a 1x1 conv block and an identity module IDs, each receiving the input features. The output features are a combination of the processing of the input features by each of these blocks. The Linf_i layer has the structure of a layer in the second configuration that includes only a 3x3 conv block. Such a block is therefore more accurate than a traditional convolution block, but faster than a residual block.

[0146] According to some embodiments, the second artificial neural network R2 can include, upstream of a feature pyramid network, so-called additional blocks BAI and BA2. For example, an additional block BAI comprises a sequence of convolutional layers, and an additional block BA2 comprises a sequence of convolutional layers and an upsampling module. Integrating such additional blocks into the feature enrichment module R2 reduces its complexity and computational resource requirements, resulting in faster processing compared to prior art feature enrichment modules (known as "necks"). An example of the arrangement of such additional blocks BAI and BA2 is shown in [Fig. 13].In this example, the additional blocks are combined with the deepest layers Cl, C2 and C3 of the first neural network and the CB data are the output data of one of the additional blocks BAL.

[0147] In the example in [Fig. 13], the modules labeled "Upsample" are modules configured to perform upsampling of a received input feature map. In the context of neural networks, upsampling in the network incorporates "Upsample" type blocks into the layers themselves, enabling the network to learn to efficiently increase the resolution of the input data during the training phase.

[0148] Oversampling blocks are generally used to recover spatial information lost during undersampling (for example, during maximum pooling or strided convolutions). These blocks increase the resolution of feature maps, allowing for finer localization and better delimitation of objects.

[0149] In other words, in the example of [Fig.13], the first stage El is connected to the RI feature extraction module only by convolution blocks and upsampling modules.

[0150] The extraction module (third neural network) R3 receives as input the enriched features CE from the second neural network R2 and generates, for each detected object, a set of information including: a class CL, position information B and a binary value O representing the detection result (affirmative in case of detection, negative in case of non-detection).

[0151] Advantageously, the prediction module (third artificial neural network) R3 has a branching structure, each branch delivering output data, within a class, location information, and a confidence score. According to an example, the location information is the coordinates, in the current image Io, of vertices of bounding rectangles surrounding the detected objects ("bounding box" in English).

[0152] An example architecture for the third artificial neural network R3 is shown in [Fig. 14]. In this example, the third artificial neural network R3 comprises two main branches, one of which is divided into two sub-branches. Each of the main branches comprises a 3x3 convolutional layer 11 as previously defined in the architecture of the first artificial neural network RI in [Fig. 5]. Then, each of the branches comprises a sequence of layer blocks 12, 13, 14, and 15.

[0153] Blocks in layers 12 and 14 are implicit representation blocks. A description of this type of block can be found in the article "Fou only learn one representation: Unified network for multiple tasks.", by Wang, CY et al., arXiv preprint arXiv:2105.04206 (2021).

[0154] More specifically, layer block 12 is a layer block called the implicit addition block and layer block 14 is a layer block called the implicit multiplication block. These layer blocks exhibit a simple architecture and processing efficiency.

[0155] The implicit addition block 12 adds a trained implicit tensor to the input feature map it receives as input. This implicit tensor is trained along with the network and acts as an additive bias to the input features. The implicit tensor generally has the same shape as the input feature map and is initialized either randomly, for example with additive Gaussian noise, or with a specific strategy. Initialization with Gaussian noise helps avoid overfitting the model and improves the model's generalization to a wider range of input data. During training, the layer block 12 learns to adjust the implicit tensor to optimize detection performance.The 12-layer block offers several advantages: By adding a learned tensor to the input features, the 12-layer block can help refine and improve feature representation, making it easier for subsequent layers to detect objects; since the 12-layer block only involves one addition per element, it introduces negligible computational overhead, making it efficient; furthermore, the 12-layer block can provide additional bias to the learning process, which can improve the model convergence rate during training.

[0156] The implicit multiplication block 14 involves element-by-element multiplication of an input feature map with an implicit tensor that has been trained. The implicit multiplication block 14 can be initialized randomly, for example with Gaussian multiplicative noise. Initializing with Gaussian noise helps avoid overfitting the model and improves its generalization to a wider range of input data. This tensor is also trained along with the network and acts as a multiplicative factor for the input features. The implicit tensor is trained to accentuate or attenuate certain features of the input feature map.The 14-layer block offers a plurality of advantages: by applying a trained multiplicative tensor, the 14-layer block can adapt the scale of features, which can help to highlight important features and suppress less important ones; furthermore, the 14-layer block only involves one multiplication per element, which is computationally inexpensive; moreover, the 14-layer block can improve feature discrimination by scaling them in a trained manner, which can improve the performance of the R3 prediction module.

[0157] The layer block 13 is called a 1x1 convolution block. Such a layer block applies a 1x1 filter to an input feature map by linearly projecting the feature map onto a channel dimension. Such a layer block is used to reduce the dimensionality of the input features and to transform these features.A 1x1 convolution block offers several advantages: it reduces the number of channels while preserving essential information; the use of feature projection allows for the creation of efficient embeddings; it enables channel pooling and aggregation of information from all channels; when used with an activation function (such as a ReLU function), it introduces non-linearity; it allows for managing the overall network complexity by adjusting the number of channels; and it offers flexibility in network design by allowing the number of channels in the network to be increased or decreased.

[0158] The use of the third artificial neural network R3 coupled with that of the enriched features CE from the second artificial neural network R2 according to the invention makes the architecture of the model M robust and efficient, and suitable for real-time applications.

[0159] The embodiments of device 100 and method 200 may be supplemented or modified as described below.

[0160] Thus, advantageously, the detection result R can undergo post-processing aimed at its refinement. For example, a confidence threshold can be applied to the predictions returned by the model M, so as to eliminate score detections low confidence. In another example, a non-maximum suppression (NMS) technique can be used when bounding rectangles overlap, thus eliminating duplicate detections. In yet another example, the bounding rectangles can be resized to the size of the initial current image Io.

[0161] The detection result R obtained in step S30 of method 200 for detecting an object can be advantageously used in various ways.

[0162] Thus, depending on the value of the detection result, an action, such as an alert, can be triggered. For example, an action can be triggered when the detection result R includes the detection of smoke or a set of flames.

[0163] Advantageously, in step S40, the detection result R can be displayed on a display device at station 100 for visualization. For example, for each detected object, a corresponding bounding rectangle can be displayed, positioned according to the coordinates of the vertices, for example, with the determined class and optionally the confidence score. Such a display allows an operator to assess the situation and decide on actions to be taken in real time.

[0164] In certain embodiments, the detection result R can be transmitted to an automated system capable of making decisions on actions to be taken, thereby minimizing human intervention. Thus, another advantage of the invention is to minimize human error, reduce operational costs, and provide accurate, real-time information that facilitates decision-making through the use of drone technology.

[0165] Table 1 presents detection performance results using the detection function from model M, consisting of the first, second, and third artificial neural networks RI, R2, and R3 shown in Figures 5, 7, and 14, applied to images from the VisDrone2019-DET database. Detection accuracy is evaluated using the metrics mAP50, mAP75, and mAP50:95 (mean average accuracy), and detection speed is evaluated by estimating the inference time with two different graphics processors.

[0166] The mAP metric is based on precision and recall metrics, which allow for the consideration of several object classes, and on the definition of positive predictions using the intersection over the union (loU). The intersection over the union is the ratio between the intersection area and the union area of ​​the predicted bounding rectangle and the ground truth bounding rectangle. The intersection over the union measures the overlap between the bounding rectangles of ground truth and prediction.

[0167] The Microsoft COCO convention is used for calculating the AP metric. This convention uses a 101-point interpolation, that is, it calculates The accuracy for 101 recall thresholds ranges from 0 to 1 in increments of 0.01. Furthermore, the mAP metric is obtained by calculating an average of several values ​​across multiple intersections on the union, with the exception of the AP50 metric, which corresponds to the AP for an intersection threshold on the union of 0.5. The calculation of the mAP metric according to the COCO convention includes the following steps:

[0168] 1. For each class, calculate the accuracy-recall curve by varying the confidence threshold of the model's predictions.

[0169] 2. Calculation of the AP metric of each class using the thresholds of 101 recalls.

[0170] 3. Calculation of the AP metric at different intersection thresholds on the union (loU), typically from 0.5 to 0.95 with a step of 0.05. A higher intersection threshold on the union requires a more precise prediction to be considered a true positive.

[0171] 4. For each intersection threshold, calculate the average of the AP values ​​in all classes.

[0172] 5. Calculation of the overall mAP metric by calculating the average of the AP values calculated for each intersection threshold on the union.

[0173] Reference 9th corresponds to the integration into model M of a single Swin Transformer sequence, CB AM, at the lowest stage of the feature enrichment module R2. Reference 1Oth corresponds to the integration into model M of a Swin Transformer sequence, CB AM, at each stage of the feature enrichment module R2.

[0174] The notations L (for "large"), M (for "medium"), and S (for "small") correspond to different versions of the M model, exhibiting, in that order, a decreasing depth related to the number of filters in each of the convolution layers of each of these versions. For example, a given layer of the L model containing 256 filters corresponds to a given layer of the M model containing 160 filters, and a given layer of the S model containing 120 filters. The order of magnitude of their GFLOPS (Giga floating-point operations of addition and multiplication) is approximately 150, 100, and 50, respectively, from the L model to the S model, via the M model.

[0175] It can be observed that integrating a Swin Transformer, CB AM sequence at each stage in the model improves the value of the mAP50:95 metric compared to integrating only at the lowest stage of such a sequence, which is very difficult to obtain from the VisDrone2019-DET database. Furthermore, the M model according to the invention provides a better value for the m50:95 metric compared to prior art models, such as the EdgeYOLO model, for which this metric is equal to 18.7.

[0176] [Tables] Model mAP50 mAP75 mAP50:95 Inference time NVIDIA-A40 ) (ms) Inference time NVIDIA-T120 0) (ms) L 9th 35.3 19.8 19.8 8.16 43.8 lOth 35.7 20.1 19.9 8.35 45.1 M 9th 34.0 17.6 18.4 7.4 26.0 lOth 32.9 17.3 17.9 9.1 28.8 S 9th 26.0 13.0 13.6 7.95 19.1 lOth 26.8 13.6 14.1 8.22 20.2

[0177] Moreover, the M model, in version L, has an inference time of approximately 45 ms per video frame.

[0178] The model M is trained prior to the implementation of the object detection method 200 during a training phase. The steps of the training phase will be described below.

[0179] According to one example, the M model can be trained from training data consisting of images annotated with bounding rectangle coordinates associated with a class. For example, the VisDrone2019-DET database can be used. This database includes images recorded by drones and depicting people and vehicles under a wide variety of conditions, which have been manually annotated. Furthermore, the images in this database exhibit varied resolutions, scales, and aspect ratios suitable for training models dedicated to image detection tasks.

[0180] Prior to their use for the training phase, the training data undergoes pre-processing including: scaling (so as to achieve an image resolution expected by the model M, for example 640x640 pixels), normalization of the pixel values ​​of the images on the same scale to ensure the consistency and numerical stability of the input data, for example between 0 and 1, an augmentation step of the training data aimed at optimizing the generalization of the model M (for example the Enhanced-Mosaic and Mixup augmentation techniques, described in the article "mixup: Beyond Empirical Risk Minimization", by Zhang et al.).

[0181] During the training phase, a cost function is minimized using the gradient descent technique. The cost function can advantageously be a combination of several cost functions, for example: a first cost function measuring the accuracy of location predictions, particularly coordinate predictions. of bounding rectangles; a second cost function measuring the accuracy of confidence scores; and a third cost function measuring the accuracy of class predictions.

[0182] The training phase consists of a series of iterations on the training dataset, ending after a predetermined number of iterations or when a criterion is met. The criterion may relate to a metric evaluating the model's performance at the end of the training phase. An iteration includes a forward propagation phase aimed at obtaining the model's predictions with the current weights, a step of calculating the cost function based on the predictions and the actual annotations, and a backpropagation step during which the model's weights are updated using the gradient descent technique.

[0183] Advantageously, the training phase can be staggered, that is, it comprises several steps, each of which uses a specific cost function. For example, a loU-type cost function can be used in the first step. A description of such a cost function can be found in “Generalized intersection over union: A metric and a loss for bounding box regression”, Rezatofighi, H., in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 658-666) (2019). Then, a “Hybrid-Random Loss” type cost function can be used in subsequent steps. Finally, at the end of the training phase, a “doU loss” type cost function (complemented to loU loss) can be used to accelerate model convergence.

[0184] Hyperparameters can be adjusted, such as the learning rate, batch size, and regularization strength.

[0185] The training phase can be finalized by a validation phase on a database called validation data.

[0186] A model evaluation phase can be performed, consisting of determining a plurality of metrics to measure the quality of the model. For example, the metrics of average precision and mean average precision and intersection on union can be calculated.

[0187] When the training phase is complete, the model parameters (weights) are saved, with a view to the inference (i.e. use) phase of the model.

[0188] Of course, the present invention is not limited to the embodiments described above by way of example; it extends to other variants. Other embodiments are possible.

[0189] Depending on the embodiment chosen, certain acts, actions, events or functions of each of the methods described in this document may be performed or occur in a different order than that in which they have been described, or may be added, merged, or not performed or not occur, as the case may be. Furthermore, in some embodiments, certain acts, actions, or events are performed or occur concurrently and not successively.

[0190] Although described through a number of detailed embodiment examples, the proposed method and the equipment / device for implementing the method include various variants, modifications, and improvements that will be obvious to those skilled in the art, it being understood that these various variants, modifications, and improvements form part of the scope of the invention, as defined by the following claims.Furthermore, the various aspects and features described above can be implemented together, separately, or substituted for one another, and all the different combinations and subcombinations of these aspects and features form part of the scope of the invention. In addition, some of the systems and equipment described above may not incorporate all the modules and functions described for the preferred embodiments. Industrial application

[0191] As mentioned above, the invention may be applicable in particular to the use of drone systems in assisting missions such as firefighting operations. The invention may also find applications in other fields, such as rescue operations in natural areas, forests or mountains, or natural disaster scenes, industrial environment monitoring, and vehicle identification.

Claims

Demands

1. Device (100) for detecting a number (No) of objects in a current aerial image (Io), the device (100) comprising: a. At least one input interface configured to receive said current aerial image (Io); b. At least one processor configured to: i. Apply a learned function called the detection function to said current aerial image (Io), so as to generate a detection result (R), said detection function being derived from a pre-trained model (M); c.At least one output interface configured to return said detection result (R), the model (M) comprising a set of stages including a first stage and a last stage, each stage returning as output a semantic feature map of higher resolution than the semantic feature map returned by the corresponding lower stage, each stage above the first stage returning the corresponding semantic feature map from the semantic feature map returned by the corresponding lower stage, at least one stage among the set of stages comprising a sequence consisting of a module (T) comprising a Vision Transformer type submodule followed by an attention module (A).

2. Device (100) according to claim 1, characterized in that the model (M) comprises: - a feature extraction module (RI) configured to extract a plurality of feature maps (C1, C2, C3) from the current aerial image (Io); - a feature enrichment module (R2) comprising the set of stages and configured to return as output semantic features (CE) from the semantic feature maps, at least one stage of the set of stages receiving features directly or indirectly from the feature extraction module (RI); and - a prediction module (R3) configured to calculate at least one prediction based on semantic features (CE), the detection result (R) being based on at least one prediction.

3. Device (100) according to claim 2, characterized in that the first stage is connected to the feature extraction (RI) module only by convolution layer blocks and oversampling modules.

4. Device (100) according to any one of the preceding claims, characterized in that all stages comprise a sequence consisting of a module (T) comprising a Vision Transformer type sub-module followed by an attention module (A).

5. Device (100) according to any one of the preceding claims, characterized in that the feature extraction (RI) module consists of an assembly formed solely of convolution layers and residual blocks each composed of a plurality of layers, said assembly comprising a total number of layers less than 50.

6. Device (100) according to any one of the preceding claims, characterized in that, when the number (No) of detected objects is greater than or equal to 1, the detection result (R) includes, for each detected object, a class (CL) of the detected object, a position information (B) of the detected object in the current aerial image (Io), and optionally a confidence score representative of the object's detection reliability.

7. Device (100) according to any one of the preceding claims, characterized in that the Vision Transformer type submodule comprises a set of layer pairs, each pair comprising a first layer and a second layer, the module being configured to calculate, for each layer among the set of layer pairs, a set of attention scores for a set of windows (W) forming between them a partition of an input image, the partition for a second layer of a layer pair being offset from the partition for the first layer of said layer pair.

8. Device (100) according to claim 7, characterized in that the Vision Transformer type sub-module is a Vision Transformer type module with offset windows (SwinTR).

9. Device (100) according to any one of the preceding claims, characterized in that the attention module (A) of at least one stage comprises a sequence of a first channel attention sub-module and a second spatial attention sub-module, the first channel attention sub-module being configured to apply a first attention mechanism to at least a part of the features from the module (T) comprising the Vision Transformer type sub-module (T), the second spatial attention sub-module being configured to apply a second attention algorithm.

10. Device (100) according to any one of the preceding claims, characterized in that at least one stage of the feature enhancement module (R2) comprises, downstream of said sequence, an additional module (RC) of the type of reparameterization module.

11. Device (100) according to any one of the preceding claims, characterized in that the prediction module (R3) comprises at least one implicit addition block and / or one implicit multiplication block.

12. System for detecting a number (No) of objects in a current aerial image (Io) comprising: a. an on-board image sensor configured to acquire aerial images; b. a device (100) for detecting the number (No) of objects in a current aerial image (Io) according to any one of claims 1 to 11, at least one input interface of said device being configured to receive aerial images, and the current aerial image (Io) being one of the aerial images.

13. A computer-implemented method (200) for detecting a number (No) of objects in a current aerial image (Io) implemented by a device (100) according to any one of claims 1 to 11, said method (200) comprising: a. (S10) receiving said current aerial image (Io); b. (S20) applying a learned function called the detection function to said current aerial image (Io), so that to generate a detection result (R), said detection function being from a pre-trained model (M); c. (S30) return said detection result (R), the model (M) comprising a set of stages including a first stage and a last stage, each stage returning as output a semantic feature map of higher resolution than the semantic feature map returned by the corresponding lower stage, each stage above the first stage returning the corresponding semantic feature map from the semantic feature map returned by the corresponding lower stage, at least one stage among the set of stages comprising a sequence composed of a module (T) including a Vision Transformer type sub-module and an attention module (A).

14. Method (200) according to the preceding claim further comprising: a. triggering an action, for example an alert, based on at least one value of the detection result (R).

15. Product computer program comprising instructions for implementing the method according to any one of claims 13 to 14, when this program is executed by a processor.

16. A non-transient, computer-readable recording medium on which a program is recorded for the implementation of the method according to any one of claims 13 to 14 when that program is executed by a processor.

Images