Recording medium for detecting at least one object

Abstract

Recording medium for detecting at least one object. The invention relates to a processor-readable recording medium on which is recorded a computer program comprising instructions for implementing, when the program is executed by a processor of an electronic device, a method for detecting an object, said method comprising: Obtaining multimodal images representing the same real and / or virtual spatiotemporal scene; Obtaining a global image from the multimodal images by applying an attention model; Detecting an object present in the scene by inference from an artificial intelligence model taking as input the global image and / or a tensor representing the global image. The invention also relates to the corresponding method, electronic device, and computer program product. Figure for the abstract: Fig. 4

Description

Title of the invention: Recording medium for detecting at least one object 1. Technical field

[0001] The present application relates to the field of automatic object detection (sometimes called "Computer Vision") in a real or virtual scene, such as detection techniques implementing neural network architectures for deep learning (or "Deep Learning" in English).

[0002] This application relates in particular to a method for detecting at least one object, as well as a corresponding electronic device, system, computer program product, and information storage medium. 2. Prior art

[0003] For several decades, we have witnessed a rise in automated systems in various fields such as industry, services, transport, agriculture and, more recently, in the private or public sphere (smart home, smart office, smart city). These automated systems often rely on automatic detection of objects present in their environment.

[0004] This automatic detection is often performed using so-called "computer vision" techniques, which implement automatic image analysis of an environment via "detector" or "classifier" neural networks (sometimes called automatic object recognition). When applied to images, "classifier" or "detector" neural networks make it possible to assign (during an inference phase) a probability for an image (or portion of images) to represent a certain class of objects, from among a plurality of candidate object classes. Detector neural networks also make it possible to specify where, within the image, the part of the image (often called the "bounding box") that contains this class of object is located.The inference phase is preceded by a learning phase during which the neural network learns to distinguish the characteristics of image parts corresponding to these candidate object classes from examples. For example, in the case of supervised learning of a "detector" type neural network, the examples provided are pairs, each associating an image with a list of bounding boxes within that image, each containing an object (called the object of interest) to which an object class has already been assigned. The reliability of the results obtained depends greatly on the training performed, particularly on the number and diversity of the examples provided. Good training often requires providing the neural network with thousands, or even millions, of examples, which... assumes a significant amount of time for collecting and annotating (often manually) these examples, as well as significant memory and processing capacity and processing time for these examples for the neural network being trained.

[0005] Learning a neural network is therefore a very time- and resource-intensive task.

[0006] However, the reliability of detection systems based on such neural network architectures can also vary greatly depending on the context in which these images are captured.

[0007] Thus, it may be difficult, or even impossible, to automatically and accurately detect objects in color images taken by a camera (such as a webcam) in the dark, whereas such objects will be more easily detectable on an image acquired, under the same conditions of darkness, but with an infrared camera.

[0008] Solutions based on the use of multimodal images have been developed. These solutions make it possible to take advantage of the capture capabilities of the different modalities used for better adaptation to different capture contexts. However, there is a need for an object detection solution that further improves the reliability of automatic object detection systems.

[0009] The purpose of this application is to propose improvements to at least some of the drawbacks of the prior art. 3. Description of the invention

[0010] The present application aims to improve the situation by means of an object detection method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0011] As described in more detail below, multimodal images are understood to be images acquired from diverse imaging sources (i.e., different types of sensors) and representing the same portion of a scene. The diversity of imaging sources allows the multimodal images to complement each other with regard to the representation (description) of the common portion of the scene.

[0012] The object detection method can be implemented at least partially in an electronic device.

[0013] Depending on the implementations of the solution described in this application, it may involve various objects, connected or not, such as vehicles, people, animals, robots, and / or machines.

[0014] For the sake of brevity, the term "image" will be used in the remainder of the application to refer either to an image itself, or to a tensor of the same dimensions as this image (in terms of pixels) representing it.

[0015] In certain embodiments, said artificial intelligence model is adapted to object detection in a single-mode image.

[0016] In certain embodiments, said global image is obtained by applying at least one first attention model to an image resulting from a fusion of said multimodal images.

[0017] In some embodiments, at least one of the first of the merged multimodal images is pre-filtered by applying at least one second attention model.

[0018] In certain embodiments, at least one second of the merged multimodal images is pre-filtered by applying said second attention model.

[0019] In certain embodiments, the set of said merged multimodal images is pre-filtered by application of said second attention model.

[0020] In certain embodiments, at least one second of the merged multimodal images is pre-filtered by applying at least one third attention model, different from said second attention model.

[0021] In some embodiments, the method includes obtaining an overall image from each video channel of said multimodal images.

[0022] In certain embodiments, the method includes a joint rendering of at least one geometric shape encompassing said at least one object detected in said global image and of a label representative of the classification of said object detected by said artificial intelligence model.

[0023] In certain embodiments, the method includes joint learning of said attention modules and said artificial intelligence model.

[0024] The features, presented individually in this application in connection with certain embodiments of the process of this application, can be combined with each other according to other embodiments of this process.

[0025] According to another aspect, the present application also relates to an electronic device adapted to implement the process of the present application in any of its embodiments.

[0026] For example, the present application thus relates to an electronic device comprising at least one processor configured for object detection comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0027] According to another aspect, the present application also relates to a system comprising at least one electronic device adapted to implement the process of the present application in any of its embodiments.

[0028] For example, the present application thus relates to a system comprising at least one electronic device comprising at least one processor configured to: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0029] The present application also relates to a computer program product comprising instructions for implementing the various embodiments of the above process, when the program is executed by a processor, and an information recording (information storage) medium, readable by a processor of an electronic device and on which the computer program is recorded.

[0030] For example, the present application relates to a computer program product comprising instructions for implementing, when the program is executed by a processor of an electronic device, an object detection method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0031] For example, the present application also relates to a recording medium readable by a processor of an electronic device and on which is recorded a computer program comprising instructions for implementation, when the program is executed by the processor, an object detection method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0032] The programs mentioned above may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0033] The information storage media referred to in this application may be any entity or device capable of storing the program. For example, a medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means.

[0034] Such a means of storage can, for example, be a hard drive, a flash memory, etc.

[0035] On the other hand, an information medium can be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, by radio, or by other means. A program according to the invention can, in particular, be downloaded onto an Internet-type network.

[0036] Alternatively, an information carrier may be an integrated circuit in which a program is incorporated, the circuit being adapted to execute or to be used in the execution of any of the embodiments of the method which is the subject of this patent application. 4. Brief description of the drawings

[0037] Other features and advantages of the invention will become more apparent upon reading the following description of particular embodiments, given by way of simple illustrative and non-limiting examples, and the accompanying drawings, among which:

[0038] Figure 1 presents a simplified view of a system, cited by way of example, in which at least some embodiments of the method of the present application can be implemented.

[0039] Figure 2 presents a simplified view of a device adapted to implement at least certain embodiments of the process of the present application.

[0040] Figure 3 presents an overview of the process of the present application, in some of its embodiments.

[0041] Figure 4 presents an example of a logical architecture of a set of modules adapted to implement at least partially the process of the present application.

[0042] Figure [Fig. 5] presents, in a synthetic way, the layers of the modules presented in [Fig. 4],

[0043] Fig. 6 presents in more detail the layers of the modules presented in Fig. 4, in an example of implementation.

[0044] Figure 7 presents multimodal images acquired in an example implementation.

[0045] Fig. 8 presents the merged image obtained from the multimodal images acquired from Fig. 7 and a rendering of encompassing geometric shapes and labels of detected and classified objects, from this merged image, in this implementation example. 5. Description of the implementation methods

[0046] The present application proposes a solution based on an analysis of a plurality of multimodal images, each representing the same spatio-temporal scene, to detect objects in that scene.

[0047] In the detailed examples below, the scene to be monitored may be a portion of an outdoor urban environment, for example a street intersection, in which various objects (such as pedestrians and vehicles) are moving or stationary (as in the multimodal images illustrated in [Fig. 7]). It is clear, however, that certain embodiments of the method that is the subject of this application can be implemented in other environments (in particular for other types of scenes to be monitored (for example, indoor scenes such as a portion of a factory, shop or dwelling, outdoor scenes such as a garden or a highway)) for the detection of various objects (robots, vehicles, human beings, animals, etc.).

[0048] By "images representing the same spatio-temporal scene", we mean images comprising visual elements representative of a common portion of a real or virtual environment.

[0049] This may include, for example, images captured from the same viewpoint, or from similar viewpoints, with identical or at least partially superimposable shooting angles (one being, for example, wider than the other), these captured images each representing the same common portion (or scene) of this environment.

[0050] These images are images captured within a common time interval, for example, images captured simultaneously or almost simultaneously. The size of the common time interval may depend on the embodiment, and in particular depending on the nature of the objects whose presence we want to capture (for example, their average speed of movement). For example, a time interval of a few milliseconds could be chosen in embodiments adapted to the detection of vehicles and pedestrians.

[0051] Multimodal images are images acquired by different types of sensors (implementing different acquisition modalities) and used for the same processing.

[0052] For example, the plurality of multimodal images may include color images, and / or black and white images, and / or infrared and / or thermal images, and / or depth or reflectance maps, and / or radar images, and / or stereovision images, and / or maps obtained from point clouds acquired by distance sensors (such as LIDAR (for "Light Detection And Ranging")).

[0053] The present application proposes to apply to at least some of these images, and / or to their fusion, an attention model (preceded and / or followed, optionally, by the application of at least one convolution), before a phase of object detection from the resulting image by a "detector" type neural network taking as input a monomodal image (or a tensor representative of such a monomodal image).

[0054] Such a solution thus makes it possible to benefit from the advantages of attention models in terms of image feature correlation while allowing the use of a "detector" type neural network, provided by third parties, and in particular, pre-trained detectors, for example with a large training database. The detector can notably be a model adapted to the detection of objects in 2D images.

[0055] The detector can, according to the embodiments, be designed specifically for images (for example with a deep learning model based on convolution layers) or be more general (i.e. not limited to images), for example with a deep learning model based on layers of "Transformers".

[0056] The use of a deep learning model designed for images can limit the amount of data used for training.

[0057] The use of a more general deep learning model (therefore without inductive bias) can lead to better performance.

[0058] The detector used may, in particular, use a model known to have good performance in terms of relevance as well as execution speed, such as a YOLO© model (for "You Only Look Once" in English) described by Redmon et al. in the article "You Only Look Once: Unified, Real-Through Object Detection") or an RT-DETR© model (for "Real-Time Detection") or an RT-DETR© model (for "Real-Time Detection"). (Transformer in English). The solution in this application thus presents many advantages compared to other existing multimodal image fusion techniques.

[0059] In the field of multimodal image computer vision, different types of fusion can be distinguished: upstream fusion (or "early fusion" in English), middle fusion (or "middle fusion" in English) and downstream fusion (or "late fusion" in English):

[0060] Upstream fusion consists of merging the different multimodal images acquired before providing the result of this fusion to a neural network (of detector type for example) with a single input (only one monomodal image being processed at a time by this neural network).

[0061] Upstream fusion has the advantage of allowing the reuse of neural network models designed for single-modal images. It also has the advantage of learning the relationships between multimodal images from their low-level features. However, since fusion precedes processing by the neural network, this technique does not allow the neural network to separately distinguish the intrinsic features of each of the multimodal images, which can impair the accuracy of object detection, for example.

[0062] In the downstream fusion, each image is processed separately by a different unimodal neural network. The position and / or object class proposals (and associated probabilities) provided by the different unimodal neural networks are merged to produce a single proposal.

[0063] Downstream fusion allows the intrinsic features of each of the multimodal images to be extracted via each of the neural networks. However, it does not take advantage of the learning capabilities of the neural networks to relate (correlate) the features of different images.

[0064] Mid-frame (or “intermediate”) fusion requires a multi-input neural network (detector, classifier, etc.) with one input for each of the multimodal images, whose architecture has been specialized for processing multimodal images (the architecture may depend in particular on the nature of the modalities and / or the number of multimodal images to be processed for the same spatiotemporal scene). It takes place within the “feature extractor” (or “backbone”) of this neural network and consists of merging (by addition(s) or concatenation(s), for example) the intermediate features, in the latent space, of each of the multimodal images.

[0065] By latent space (also called "latent feature space" or "integration space"), we mean an abstract multidimensional space that encodes an internal representation (the intermediate or latent features) of each The internal representation of an image is the input to the model; the distance between the internal representations, in this latent space, of two images is a function of the similarity between those images. The internal representation of an image can, for example, be obtained from the output of layers of a neural network (such as a combination of convolution and polling layers) taking that image as input. In particular, the internal representation of the images can be of lower dimension than the input images (or a tensor representing them).

[0066] Among the various types of fusion introduced, middle fusion is the most widely used because it generally provides the best performance. It allows both the retrieval of the high-level features of each of the multimodal images (in the input layers of the neural network) and the pooling of their latent spaces (taking into account their low-level features). However, implementing this type of network is more complex, as it requires creating an architecture and a model dedicated to the different modalities of the images to be processed.

[0067] In the fusion process, the linking, in the common latent space, of latent features relating to different multimodal images (for example, during the latent feature extraction phase) sometimes involves attention mechanisms. Attention mechanisms are machine learning methods that determine the relative importance of each element in a set of elements with respect to the other elements in that set.

[0068] Attention mechanisms allow the neural network model used, during fusion in the middle, to identify and prioritize certain information present in the latent space and useful for the desired outcome (here, the location and class of objects present in the multimodal images), thus improving the performance of multimodal image fusion models in terms of the relevance of the results. In particular, they can contribute to a better understanding of the overall context of the information to be processed and make the systems more robust to noise and adaptable.

[0069] The present application proposes a solution allowing the benefits of attention mechanisms to be taken advantage of, without resorting to fusion in the middle, in order to allow in particular the reuse of state-of-the-art neural network models (classifier and / or detector for example) designed for single-modal images.

[0070] Thus, unlike fusion in the middle, the solution of the present application makes it possible to use neural network models that process single-modal images, and in particular existing models from the literature (for example, models recognized for their effectiveness).

[0071] Moreover, unlike upstream or downstream merging, the solution in the present application allows, at least in certain embodiments, for a correlation of the intrinsic characteristics of the different multimodal images. through the use of attention mechanisms, prior to the use of these neural networks for single-modal image processing.

[0072] This connection can prove important when the architecture of the neural network is complex, or considered as a black box.

[0073] The present application is now described in more detail in relation to [Fig.1].

[0074] Figure 1 represents a telecommunications system 100 in which the method of the present application can be implemented, in at least some of its embodiments. The system 100 comprises one or more electronic devices, at least some of which can communicate with each other via one or more communication networks 120, possibly interconnected, such as a local area network (LAN) and / or a wide area network (WAN). For example, the network may include a corporate or home LAN and / or a WAN of the internet, cellular, GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), Wi-Fi (Wireless), etc. type.

[0075] As illustrated in [Fig.1], the system 100 may also include several electronic devices, such as a communication terminal 110 (such as a computer, for example a laptop, a smartphone, a tablet), sensors 130, and / or a server 140, for example a server adapted to process (analyze for example) data obtained via at least one of the electronic devices of the system 100, a storage device 150, for example a storage device adapted to store data obtained via at least one of the electronic devices of the system 100. The system may also include management and / or network interconnection elements (not shown).

[0076] In particular, some of the sensors 130 can be adapted to provide images in various forms (or modalities). These may include, for example, cameras (such as an optical camera 132, an infrared camera 134, a thermal camera 136), radar, or distance sensors 138).

[0077] Alternatively, the sensors 130 illustrated in [Fig.1] can be integrated into the device 110.

[0078] Fig. 2 illustrates a simplified structure of an electronic device 200 of the system 100, for example terminal 110 or server 140 of Fig. 1, adapted to implement the principles of the present application.

[0079] The device 200 includes, in particular, at least one memory M 210. The device 200 may, in particular, include a buffer memory, volatile memory, for example of the RAM (Random Access Memory) type, and / or non-volatile memory (for example of the ROM (Read-Only Memory) type). (Only Memory, according to English terminology). The device 200 may also include a processing unit UT 220, equipped, for example, with at least one processor P 222, and controlled by a computer program PG 212 stored in memory M 210. At initialization, the code instructions of the computer program PG are, for example, loaded into RAM before being executed by the processor P. Said at least one processor P 222 of the processing unit UT 220 may, in particular, implement, individually or collectively, any of the embodiments of the method of the present application (described in particular in relation to [Fig. 3]), according to the instructions of the computer program PG.

[0080] The device may also include, or be coupled to, at least one FO 230 input / output module, such as a COM communication module 232, enabling, for example, the device 200 to communicate with other devices (for example, sensors 130) of the system 100, via wired or wireless communication interfaces, and / or such as an ACQ acquisition module 234 enabling the device to acquire by itself data (in particular images) representative of an environment to be monitored.

[0081] It is noted that the device 200 can for example acquire by itself via the acquisition module 234 certain multimodal images and receive on the other hand via the communication module 232 other images from a sensor 130 of the system 100 (or information (such as point clouds) allowing to construct other images, these images (or information) can optionally be time-stamped, so as for example to be temporally synchronizable with each other.

[0082] An I / O input / output module 230 of the device 200 can be an interface module with a user of the device (also referred to more simply in this application as a "user interface").

[0083] The user interface of the device means, for example, an interface integrated into the device 200, or a part of a third-party device connected to that device by wired or wireless communication means. For example, it could be a secondary display of the device or a set of speakers connected wirelessly to the device.

[0084] A user interface may, in particular, be an "output" user interface adapted for rendering (or controlling the rendering) of an output element of a computer application used by the device 200, for example, an application running at least partially on the device 200 or an "online" application running at least partially remotely, for example, on the server 140 of the system 100. Examples of output user interfaces of the device include one or more screens, in particular at least one graphics screen (touchscreen, for example), a or several speakers, a connected headset. The interface of the device 200 can, for example, be adapted to the rendering of the images illustrated by one of Figures 4, 5, 7 and 8.

[0085] By rendering, we mean here a reproduction (or "output" according to English terminology) on at least one user interface, in any form whatsoever, for example including textual, audio and / or video components, or a combination of such components.

[0086] Furthermore, a user interface can be an "input" user interface adapted for acquiring a command from a user of the device 200. This may include an action to be performed in connection with a returned item, and / or a command to be transmitted to a computer application used by the device 200, for example an application running at least partially on the device 200 or an "online" application running at least partially remotely, for example on the server 140 of the system 100. Examples of input user interfaces of the device 200 include audio acquisition means (microphone for example), and / or video acquisition means (such as cameras), radar acquisition means (such as the LIDAR 138 already mentioned), a keyboard, a mouse.

[0087] As indicated above, the device may include or be coupled (via its communication means) to at least one acquisition module, such as at least one sensor (hardware and / or software) coupled to said device (for example, sensors 132, 134, 136 and 138 already introduced), enabling the acquisition of multimodal images of the scene to be monitored. Examples of sensors include an optical camera, enabling the acquisition of so-called "visible" images (color or black and white), an infrared camera, and distance sensors (such as a LIDAR (Light Detection and Ranging) sensor or a sonar).

[0088] The device may also include or be coupled with other types of sensors, capable for example of communicating additional information on the physical environment to be monitored (positioning module providing GPS coordinates for example) and / or on at least one capture module (for example a position and / or an orientation of a camera monitoring the scene).

[0089] Information acquired via the acquisition and input / output modules can, for example, be transmitted to the processing module 210.

[0090] For example, said at least one processor of the device 200 may in particular be adapted for object detection comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0091] Some of the above input / output modules are optional and may therefore be absent from the device 200 in certain embodiments. In particular, while the present application is sometimes detailed in relation to a device communicating with at least one other device of the system 100, the method can also be implemented locally by the device 200, without requiring communication with another device, in certain embodiments. For example, in certain embodiments, the device can use images / information acquired by sensors internal to the device, perform local object detection in these images, and display the result of this detection locally on a screen local to the device.

[0092] Thus, at least some embodiments of the method described in this application offer a solution that can help to locally analyze temporally and spatially correlated multimodal images, without requiring these images to be sent to a remote server or the cloud. Such embodiments can help to preserve the confidentiality of the processed data (particularly when it comes to detecting people or objects that allow for their identification). Such embodiments may also not require internet access and are therefore suitable for mobile situations (such as crossing a "white zone" without network coverage, or roaming in a geographical area where network access may entail significant costs for a user).

[0093] In some of its embodiments, on the contrary, the method can be implemented in a distributed manner between at least two devices 110, 130, 140 and / or 150 of the system 100. For example, the device 200 can obtain, via a communication module of the device, information acquired by sensors 130 external to the device and temporarily stored in the storage device 140 (before being transmitted to the device 200), perform a detection based on artificial intelligence modules previously downloaded (after their training for example) from the server 140 and return the results of this detection locally and / or to a third device via a communication module of the device.

[0094] The term "module" or the term "component" or "element" of the device refers here to a hardware element, in particular a wired one, or a software element, or a combination of at least one hardware element and at least one software element. The method according to the invention can therefore be implemented in various ways, in particular in wired and / or software form.

[0095] The process of the present application is now presented in relation to Figures 3 to 5.

[0096] Figure 3 illustrates certain embodiments of method 300 of this application, based on neural network inference, the logical architecture of which is illustrated in Figure 4. Figures 5 and 6 illustrate in more detail (in white box) a module (MEFA module) introduced below in connection with Figure 4.

[0097] As illustrated in [Fig. 4], according to the present application, a module 420 (hereinafter referred to as “MEFA” (for “Module for “Early Fusion” with Attention”) comprising, in particular, convolution layers and attention layers, is used upstream of a generic deep learning model 440 for vision systems, designed for processing a single image (e.g., 2D) at a time. This model can be a detection module (or, alternatively, a classification module).

[0098] This MEFA module 420 takes as input several images 410, 412, 414 (for example, 2D images) of the same scene under different modalities (visible, infrared, etc.). The output of this MEFA module is an intermediate image 430, or a tensor of size (w, h, x) representing it, where w and h represent the spatial dimensions (width and height respectively), in pixels for example, of the intermediate image, and where x represents a configurable number of channels, adapted to the use of the generic model. The intermediate image output of the module 420 can, in particular, have the same spatial dimensions as the multimodal images input to the module 420.

[0099] The generic model can be a one-stage or two-stage detection model (or alternatively a model dedicated to another computer vision task than detection, such as classification).

[0100] The process 300 can for example be implemented by the electronic device 200 illustrated in [Fig.2].

[0101] As illustrated in [Fig.3], the method 300 can include a 310 acquisition of a multimodal plurality of images. The 310 acquisition of the images can, for example, include receiving on a communication interface of the device at least one of these images, or a tensor representing one of these images, and / or a read access to a local or remote storage area of ​​at least one of these images, or their tensor (for example, a database as illustrated by element 150 of [Fig.1]), or a read access to a buffer accessed for writing by at least one of the sensors monitoring the scene.

[0102] Optionally, obtaining an image 310 may further include obtaining additional data (or metadata) associated with that image, such as a date and / or time of capture, an identifier of the camera that acquired the image, information relating to the positioning and / or capture of a camera that acquired this image, an image identifier, a resolution used for image acquisition, etc.

[0103] As illustrated in [Fig. 3], the process may include preprocessing 320 of the raw data corresponding to the obtained images. This preprocessing 320 may optionally include applying at least one image processing operation to at least one of the obtained images to "homogenize" all the obtained images before their subsequent use. Indeed, some images may exhibit differences in shooting style or image resolution. This may involve, for example, cropping some images taken with a wider shot than others, to retain only portions relating to a scene of interest in the wider-angle images. It may also involve spatial and / or component-number homogenization of the images.For example, 320 preprocessing might involve modifying the image resolution to obtain images of the same size (in terms of the number and distribution of pixels), or filtering and / or transforming these images to obtain the same number of components for each image. For example, color images obtained by an optical camera might have three components: R, G, and B, corresponding to the video channels (red, green, blue) (or Red, Green, Blue (RGB) in English terminology). Grayscale images obtained by an infrared camera can be grouped into batches of images (3, for example), spaced temporally (within the time interval common to all the multimodal images processed together). LiDAR images can be constructed by projecting the point cloud onto each component of the color images.

[0104] In some embodiments, the preprocessing may include correcting at least one of the images. This may involve, for example, filtering the images to correct certain isolated pixels (in terms of color, for example) based on the surrounding pixels, or conversely, padding to fill, for example, "missing" pixels in the resulting images.

[0105] Image preprocessing can further enable the production of images 410, 412, 414 (Figures 4, 5 and 6) whose size is consistent with the expected size of the input data of the MEFA module described below and illustrated in Figures 4, 5 and 6. The expected size of the input data of the MEFA module may, in certain embodiments, depend on the expected size of the input data of the detector 440 during the detection (350) of the detected objects. In particular, in certain embodiments where the MEFA module produces output data 430 of the same size as its own input data, the expected size of the input data of the MEFA module may be identical to the expected size of the input data of the detector 440. For example, if the detector expects as input a tensor of size (w, h, x), the sizes of the images obtained can be modified to obtain images of dimension w, h for each of the x channels. The values ​​w, h and / or x can for example be parameterizable data (step 302), accessible by reading a memory area (such as a configuration file for example).

[0106] For example, in the case of using a MEFA module that produces an output image with the same dimensions as the input images of that module, the resulting images can be modified in terms of scale and recentered to have an input resolution of the MEFA model identical to the resolution of the generic detection model. Consequently, the output image of the MEFA module, concatenating the input images, will also have the resolution expected by the model.

[0107] For example, in the case of using a MEFA module producing an output image of the same dimensions as the input images of this module, and using, for detection, a model designed to process images with a resolution of 640 x 640 pixels, such as a YOLO © model like YOLOv8, the images obtained can be modified, in terms of scale and recentered, to have a resolution of 640 x 640 pixels as input to the MEFA model.

[0108] As illustrated, the process 300 can also include a processing 330 of the images pre-processed by the MEFA module 420. This processing includes in particular an application 331 of at least one convolution and an application 332 of an attention mechanism (“local”) to at least one of the images (for example to the set of images once spatially unified).

[0109] The convolution application can allow for the creation of a better representation of the latent space.

[0110] Attention mechanisms serve as a guide to select the information needed by the detector used downstream of the MEFA model.

[0111] The local attention mechanism, when applied to a representation of an image, selects the information necessary for the detector in that representation (independently of the other images in the plurality of multimodal images). For example, applying a local attention mechanism to an image (“visible”) acquired by an optical camera can make it possible to focus object detection on certain color, shape, and texture information in portions of that “visible” image.

[0112] These convolution and / or attention mechanism applications 331, 332 on at least one of the images may be optional in certain embodiments.

[0113] The method 300 can also include a fusion 333 of the features of the different multimodal images. This can, for example, involve a fusion by addition (at the pixel level, for example) of the image features. Such an embodiment offers advantages in terms of memory usage and time processing. This could, for example, involve image fusion by concatenation (at the level of each channel, for example). Such a method can preserve all image characteristics without loss of information, and therefore offer advantages in terms of detection reliability.

[0114] As illustrated, the process 300 may include an application 334 of at least one convolution and an application 335 of an attention mechanism (“global”) to the merged features.

[0115] These convolution and / or global attention mechanism applications 334, 335 on the result of the fusion may be optional in certain embodiments.

[0116] The application 338 of a global attention mechanism can make it possible to identify, in the merged intermediate features, relationships between features originating from at least two of the multimodal images (and in particular between features some of which have been considered important by a local attention mechanism). These relationships can make it possible to emphasize, and / or retain or not, features of the multimodal images.

[0117] For example, the application of a global attention mechanism can make it possible to fuse shape information from a visible image with depth information from a LiDAR image, thus facilitating the detection of an object in the "pedestrian" class. In one application example, a life-size portrait of a person will not be considered a pedestrian because the depth information provided by the LiDAR indicates that it is a flat object.

[0118] The MEFA module produces as output an intermediate image 430, or the tensor representing it, whose size can be identical, for example, to the size of each of the input images of this module.

[0119] The method 300 may include obtaining 350 at least one position and / or at least one class 450 of objects from the tensor obtained. As illustrated in Figures 4 to 6, this obtaining of position(s) and / or class(es) may include an inference of a detector 440, after supplying this detector with the intermediate image 430 obtained via the MEFA module, or the tensor representing it, to obtain at the output of this detector 440 at least one position and / or at least one class (element 450) of at least one object detected in this intermediate image 430.

[0120] Optionally, the method may include, prior to the implementation of the detector 440, an adaptation 340 of the dimensions of the intermediate image, or of the tensor representing it, to obtain an image, or a tensor, of dimensions adapted to the detector 440 (Figures 4 to 6) in charge of the detection.

[0121] For example, adaptation 340 may include applying a convolution to the merged image (or the image resulting from the application of the global attention model) so as to adapt the dimension, in terms of the number of channels, of the output image of the MEFA 420 model, and to make it directly usable by the 440 detector used.

[0122] Thus, as explained, the tensor once adapted, may be of size (w, h, x), where (w, h) represents the expected spatial dimension (w: width, h: height), in pixels, for an input image of detector 440 used during the acquisition step 350 described above, and x represents the number of channels expected for this input image of the generic model of detector 440.

[0123] In some embodiments, the method may include a 360 rendering of at least one indication relating to an object detected on a user interface of said device 200.

[0124] For example, as illustrated in Figures 3, 7, and 8, in certain embodiments, the 360 ​​rendering may include a rendering 362 of at least one of the multimodal images 410, 412, 414, preprocessed or not, and / or a rendering 364 of an overall (intermediate) image 430 constructed from the merged and possibly post-processed features of the multimodal images. The 360 ​​rendering may further include a rendering 366 of a copy 700 of at least one (410) of the multimodal images, on which has been inserted at least one encompassing geometric shape representative of the location, in that image, of a detected object, in association with a rendering 368 of an alphanumeric (for example, a label) and / or graphic indication representative of the class associated by the artificial intelligence model with the detected object.

[0125] In other embodiments, only the representative indication of the class associated by the artificial intelligence model with the detected object can be rendered.

[0126] As illustrated in [Fig. 3], the MEFA 420 module may have been previously configured (for example, during an initialization phase) so that its output is adapted to the input of the generic model (in terms of input and / or output image resolution and the number of input and / or output channels). A default value for the number of output channels of the MEFA model may also be defined (making this configuration optional).

[0127] Similarly, the MEFA 420 module may have been previously configured (for example, during the initialization phase) to set a number of multimodal images (associated with the same spatiotemporal scene) to be taken into account in the MEFA module. A default number of multimodal images as input to the MEFA model may be defined (for example, 3), making this parameterization optional.

[0128] It is noted that method 300 may include, prior to the inference of the models used (for example, after setting the number of input images, the image resolution and / or the number of input and / or output channels of the model), joint training of the MEFA module (in particular the mechanisms local and / or global attention) and the detector-type neural network model used. Indeed, as mentioned above, attentional mechanisms are used to identify elements useful to the detector. Joint training of the MEFA 420 module and the 440 detector can therefore help improve the relevance of the attentional mechanisms.

[0129] The internal architecture of the MEFA module is now presented in more detail, in connection with Figures 5 and 6.

[0130] As illustrated in [Fig. 5], the MEFA module 420 can include at least one series 4210, 4212, 4214 of one or more convolutional layers, each series being intended for processing one of the multimodal images input to the MEFA model. The output of each of these series of convolutional layers can be provided to a local attention model 422 (implementing a local attention mechanism). This local attention model can vary according to the image modalities.

[0131] According to the embodiments, these convolutions can be the same for all multimodal images or vary according to the images (in particular according to the modalities of the images), for example in terms of weight, the output tensors being for example all of the same dimension.

[0132] The local attention model may include, in particular, a layer for merging data relating to each of the images (i.e., the data from the convolution layers).

[0133] It is noted that these convolution series and local attention models may be optional in certain embodiments.

[0134] The merged data can be provided to at least one convolution layer 423 and then to a global attention model 424 (implementing a global attention mechanism).

[0135] It is noted that this at least one convolution layer and this global attention model may be optional(s) in certain embodiments.

[0136] In at least some embodiments, the MEFA module may include either local attention models (and associated convolution layers as described above), or a global attention model (and at least one associated convolution layer as described above), or both local attention models and a global attention model (and their convolution layers), as illustrated in Figures 5 and 6.

[0137] The MEFA module may also optionally include at least one 425 convolution layer, allowing the size of the data from the global attention model, if it exists, or failing that, from the fusion, to be reduced in particular to adapt it to the image size for which the detector was designed.

[0138] Figure 6 represents the architecture used for the experiment, the results of which are detailed later. The obtained monomodal images 410, 412, 414 are provided as input to convolution blocks 4210, 4212, 4214, whose outputs are provided to local attention blocks 4220, 4222, 4224. For the experiment, the convolution blocks are "Inception Block version A" type blocks of the model known as "Inception V3" (Szegedy et al., "Rethinking the Inception Architecture for Computer Vision."). The outputs of the local attention blocks are processed by merging (multiplication operations 4225, 4226, 4227 followed by concatenation 4228).The result of the fusion is provided as input to a convolution block 423 (of type "Inception Block version A" in the experiment), the output of the convolution itself being provided to global attention blocks 4240, 4242, 4244, whose outputs are merged (multiplication operations 610, 612, 614 and concatenation 616).

[0139] The solution to the present application may help to detect objects more effectively than some prior art solutions, by exploiting the different types of images provided by the sensors of system 100.

[0140] The process of the present application can be implemented, in at least some of its embodiments, in various technical fields, using multimodal images coupled with deep learning models, whether these are designed for images or more general.

[0141] For example, the method can be implemented in the field of medical imaging with multimodal images of the Magnetic Resonance Imaging (MRI) and PET (Positron Emission Tomography) types. It can also be implemented in the field of satellite imaging, with multimodal images of the visible, infrared, and hyperspectral types. It can also be implemented for monitoring systems and / or for automated driving. In particular, this application may find applications in fields where processing time constraints are significant, such as driver assistance systems, for example, autonomous driving (AD) or autonomous driving assisting systems (ADAS). Experimentation

[0142] The experimental results of the MEFA module used by the process of the present application are presented below.

[0143] For this experiment, the learning was carried out on 10000 images, and 100000 bounding boxes.

[0144] In the experiment, the objective of the MEFA model is to detect pedestrians and vehicles present in a scene (represented by three multimodal images, with respective modalities "visible", "infrared", and "LIDAR", provided as inputs to the model) and to indicate the coordinates, in at least one of these images, of bounding boxes representing the detected pedestrians and vehicles. The bounding boxes are labeled according to the detection result (position and / or class). The cameras are placed inside the car, and the scene captured by the images from these cameras is a scene within the field of vision of a driver while driving their vehicle. Since the LIDAR sensor provides a point cloud (3D), the points of the cloud were projected onto the coordinates (in pixels) of the visible image in order to produce a 2D image superimposed on this visible image (same objects in the same locations).Alternatively, the points in the cloud could have been projected onto the coordinates of an image acquired by another sensor, for example, an infrared image.

[0145] The database used for training is the DENSE © database, composed of images under three different modalities: visible, infrared and LIDAR.

[0146] To verify the quality of the detection performance of the tested models (with and without coupling to the MEFA module), the following protocol was implemented:

[0147] - Search for the best hyperparameters of the tested generic model, trained on each of the modalities

[0148] - Search for the best hyperparameters of the tested generic model, trained on concatenations of three multimodal images of the same spatio-temporal scene, of respective modalities "visible", "infrared" and "LIDAR".

[0149] - Use the 5-fold cross-validation (k-folding) method (5 folds in English). Such a method creates 5 training and validation subsets from the same database so that the model's performance can be estimated multiple times, thus obtaining a more robust estimate (with mean and variance) of the model's performance.

[0150] In the experiment, we have: • Trained the tested single-modal models for each of the modalities, • Trained the generic model tested on the concatenation of the 3 modalities, • Trained the generic model tested combined with the MEFA module using the hyperparameters of the generic model trained on the concatenation of the 3 modalities.

[0151] The evaluation of the models is carried out using the "mean of the mean forecast" (i.e. the metric known as "Mean Average Accuracy" (mAP) in English) with an interval on the union equal to 0.5 and a confidence parameter of 0.1.

[0152] A first experiment is carried out on an embodiment where the MEFA module is in the generic YOLOv8 model (as a detector).

[0153] In this embodiment, the convolution layers can for example be layers of type "Inception Block version A" and the attention model used is the local and global attention model of Chaturvedi et al., 2022 ("Pay attention to adverse weather: Weather-aware attention based object detection. - 2022 - 26th International Conference on Pattern Recognition (ICPR), pages 4573-4579).

[0154] Table 1 below summarizes the results obtained with the generic YOLOv8 model in Nano size.

[0155] [Tables 1] Average mAP model across 5 plies Standard Deviation Best result YOLOv8 Nano Visible 63.53% 0.32% 64.04% YOLOv8 Nano Infrared 61.35% 1.95% 62.79% YOLOv8 Nano LID AR 66.18% 1.59% 68.24% YOLOv8 Nano Concatenation 69.03% 0.64% 69.65% MEFA + YOLOv8 Nano 70.68% 2.69% 73.61%

[0156] For this experiment, the YOLOv8 Nano Visible / Infrared / LIDAR models are the state-of-the-art YOLOv8 models trained on a 3-channel image.

[0157] The YOLOv8 Nano Concatenation model corresponds to a model adapted to support an input resulting from the concatenation of the three modalities (visible, infrared and LIDAR), i.e. a 9-channel “image”.

[0158] The MEFA model combined with the generic YOLOv8 model corresponds to the MEFA module without the optional convolution layer after merging (and / or the application of convolution followed by a global attention model). This implies that the YOLOv8 model has been adapted to support inputs corresponding to the output of the MEFA module, namely an “image” composed of 256 channels.

[0159] The results of this experiment prove that the MEFA model combined with the generic YOLOv8 model surpasses in accuracy the YOLOv8 model adapted for concatenation.

[0160] According to a second example, the MEFA model is tested in conjunction with the generic RT-DETR vl model (Zhao et al., “DETRs Beat YOLOs on Real-Time Object Detection”) pre-trained on the COCO basis. Table 2 below presents the preliminary results obtained on the MEFA + RT-DETR model.

[0161] [Tables2] Average mAP model over 5 plies Standard Deviation Best result RT-DETRvl Concentration 72.55% 0.65% 73.12% MEFA + RTDETR vl 73.68% 0.58% 74.23%

[0162] The RT-DETRvl Concatenation model corresponds to a model adapted to support an input resulting from the concatenation of the three modalities (visible, infrared and LIDAR), i.e. a 9-channel “image”.

[0163] The MEFA model combined with the generic RT-DETRvl model corresponds to the MEFA module without the optional convolution layer after merging (and / or the application of convolution followed by a global attention model). This implies that the YOLOv8 model has been adapted to support inputs corresponding to the output of the MEFA module, namely an “image” composed of 256 channels.

[0164] The results of this experiment prove that the MEFA model combined with the generic RT-DETRvl model surpasses in accuracy the RT-DETRvl model adapted for concatenation.

Claims

Demands

1. A processor-readable recording medium on which is recorded a computer program comprising instructions for the implementation, when the program is executed by a processor of an electronic device, of an object detection method, said method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detecting at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

2. Recording medium according to claim 1 wherein said artificial intelligence model is adapted for object detection in a single-mode image.

3. Recording medium according to claim 1 or 2 wherein said global image is obtained by applying at least one first attention model to an image resulting from a fusion of said multimodal images.

4. Recording medium according to any one of claims 1 to 3 wherein at least a first of the merged multimodal images is pre-filtered by application of at least a second attention model.

5. Recording medium according to any one of claims 1 to 4 wherein at least one second of the merged multimodal images is pre-filtered by application of said second attention model.

6. Recording medium according to any one of claims 1 to 5 wherein the set of said merged multimodal images is pre-filtered by application of said second attention model.

7. Recording medium according to any one of claims 1 to 4 wherein at least one second of the merged multimodal images is pre-filtered by applying at least one third attention model, different from said second attention model.

8. Recording medium according to any one of claims 1 to 7 wherein the method comprises obtaining an overall image from each video channel of said multimodal images.

9. Recording medium according to any one of claims 1 to 8 wherein the method comprises a joint rendering of at least one geometric shape encompassing said at least one object detected in said global image and of a label representative of the classification of said object detected by said artificial intelligence model.

10. Recording medium according to any one of claims 1 to 9 wherein the method comprises joint learning of said attention modules and said artificial intelligence model.