Method and device for learning an image restoration model for a vision system.

Abstract

Method or device for training an image restoration model for a vehicle-mounted vision system. Specifically, the method comprises receiving initial representative training image data (31) acquired by a camera of the vision system, initial training of the image restoration model from portions of the training images obtained by clipping them, extraction of bounding boxes associated with training objects from the training images by an object detection model integrated into the image restoration model, and at least one further training of the image restoration model from the bounding boxes. Figure for the abstract: Figure 3

Description

Title of the invention: Method and device for learning an image restoration model for a vision system. technical field

[0001] The present invention relates to a method for training an image restoration model usable for object detection and image segmentation for an embedded vision system, for example in a motor vehicle, from training data acquired by the vision system, and more particularly from images acquired by a camera of the vision system. The present invention also relates to methods and devices for acquiring training data for an image restoration model and / or an object detection model in an image for an embedded vision system in a vehicle. Technological background

[0002] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0004] The ADAS systems installed in a vehicle are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras make it possible, in particular, to detect and locate other road users or any obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle's speed; - to act on the braking system in case of risk of impact with an object.

[0005] The quality of the training of an image restoration, object detection, and segmentation model—that is, identification of its type, for example—is, however, very important. Indeed, objects have very different appearances. based on their position in a three-dimensional scene unfolding around a vehicle. The quality of the data emitted by the vision system therefore determines the proper functioning of the driver assistance devices that use this data.

[0006] Numerous training databases are available for learning different models from image series containing objects, but these images are most often acquired by pinhole cameras, meaning that these images do not contain distortion or that the distortion is negligible. An object observed by this type of camera therefore appears similar regardless of its position within an image. However, to allow observation of a wide three-dimensional scene and thus understand the environment of a vehicle, onboard vision systems sometimes include wide-angle cameras particularly suited to this type of observation, but which then produce images with significant distortion. A specific database is therefore necessary to learn an image restoration model, this database containing images similar to those acquired by a camera of an onboard vision system in the vehicle. Summary of the present invention

[0007] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0008] Another object of the present invention is to improve the relevance of the training data of an object detection model for a vision system embedded in a vehicle

[0009] Another object of the present invention is to improve road safety, in particular by improving the reliability of AD AS systems powered by data obtained from a vision system on board the vehicle.

[0010] According to a first aspect, the present invention relates to a method for learning an image restoration model for a vision system embedded in a vehicle, the vision system comprising at least one camera arranged to acquire an image of a three-dimensional scene of an environment external to the vehicle, the method being implemented by a processor and being characterized in that it comprises the following steps: - reception of initial data representing training images acquired by the camera; - initial learning of the image restoration model from parts of the first training images, the parts being obtained by cutting the first training images; - Extraction of bounding boxes associated with training objects from training images by an object detection model integrated into the image restoration model, the object detection model detecting pixels in a training image corresponding to a training object among the training objects and defining a bounding box in the training image around the pixels; - new learning of the image restoration model from bounding boxes.

[0011] Such a method then makes it possible to learn or train an image restoration model associated with a vision system embedded in a vehicle from data acquired by the vision system itself. This data is then perfectly representative of data acquired for objects actually belonging to those of a road environment traversed by the vehicle.

[0012] According to one variant, the method further includes a step of encoding pixel positions of a training image in a reference frame associated with the training image, a position of a bounding box being defined in this reference frame.

[0013] According to another embodiment, the method further comprises generating a training database from the training images and bounding boxes, each element of the training database being associated with a training object among the training objects and comprising second data representing: • a type of object to which the learning object corresponds, • the bounding box associated with the learning object, and • a first determined distance between an optical center of the training image and a center of the bounding box associated with the training object.

[0014] According to a further variant, the process further comprises a phase of quantitative evaluation of elements of the database comprising the following steps: - reception of fourth data points representative of a type of target object, - determination of a distribution of elements from the database comprising an object type corresponding to the target object type as a function of the first distance, - determination of a score by comparison of said distribution to a target distribution.

[0015] According to yet another variant of the process, the second data are representative of a type of road environment, the type of road environment being determined from the first or second training image.

[0016] According to another variant of the method, the second data represent a second distance separating the learning object from the vehicle, with third data representing the second distance being received from a telemetry system on board the vehicle.

[0017] According to yet another variant of the method, pixels contained within a bounding box correspond to a single learning object.

[0018] According to a further variant of the method, the initial learning and relearning respectively comprise the following steps: - Restoration of a fragment of a training image from pixels contained within, respectively, the portion of the training image or the bounding box, - Determination of a photometric error by comparing pixels of the restored fragment to pixels of, respectively, the portion of the training image or the bounding box, and - learning the image restoration model by minimizing photometric error.

[0019] According to a second aspect, the present invention relates to a device for learning an image restoration model for a vision system embedded in a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the process according to the first aspect of the present invention.

[0020] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0021] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0022] Such a computer program may use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0023] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0024] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0025] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or Hertzian radio, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded onto an Internet-type network.

[0026] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0027] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 6, in which:

[0028] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a a particular and non-limiting example of a realization of the present invention;

[0029] [Fig.2] illustrates a flowchart of the different stages of a learning process of an image restoration model for a vision system embedded in the vehicle of [Fig.1], according to a particular and non-limiting embodiment of the present invention;

[0030] [Fig.3] schematically illustrates an image acquired by a camera of the system of vision of [Fig.1] and comprising an object, according to a particular and non-limiting embodiment of the present invention;

[0031] [Fig.4] illustrates a diagram representing the distribution of a number of elements of the distance class learning database, according to a particular and non-limiting embodiment of the present invention;

[0032] [Fig.5] schematically illustrates a device configured for learning a image restoration model for a vehicle-mounted vision system [Fig. 1], according to a particular and non-limiting embodiment of the present invention; and

[0033] [Fig.6] illustrates a flowchart of the different stages of a learning process of an image restoration model from an image acquired by a camera of the vision system of the [Fig.1], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0034] A method and device for learning an image restoration model for a vision system embedded in a vehicle will now be described in what follows with joint reference to Figures 1 to 6. The same elements are identified with the same reference signs throughout the description that follows.

[0035] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0036] Fig. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention.

[0037] The vehicle 10 is located in an environment 1 corresponding, for example, to a road environment consisting of a network of roads accessible to the vehicle 10.

[0038] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, or a motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is to say, a vehicle operating according to a predetermined level of autonomy or under the total supervision of the driver.

[0039] The vehicle 10 advantageously comprises at least one onboard camera 11, configured to acquire images of a three-dimensional scene unfolding in the environment of the vehicle 10 from a common viewing position. The camera 11 forms a monocular vision system if used alone, as illustrated in [Fig. 1]. However, the present invention is not limited to a monocular vision system comprising a single camera but extends to any vision system comprising at least one camera, for example, 1, 2, 3, or 5 cameras.

[0040] The camera 11 has intrinsic parameters, including: - a focal length, - distortions which are due to imperfections in the optical system of camera 11, - a direction of the optical axis of camera 11, and - a resolution.

[0041] The intrinsic parameters characterize the transformation which associates, for an image point, hereafter called "point", its three-dimensional coordinates in the camera 11 reference frame with the pixel coordinates in an image acquired by the camera 11. These parameters do not change if the camera is moved.

[0042] According to one particular embodiment, the camera 11 is calibrated, for example, by following a calibration method known to those skilled in the art such as that presented in the document "Single View Point Omnidirectional Camera Calibration from Planar Grids", written by Christopher Mei and Patrick Rives and published in April 2007, hereafter referred to as the Mei model, the intrinsic parameters of the camera 11 being then known.

[0043] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most significant effects, namely radial, decentering, and prismatic distortions, induced by defects in lens curvature, parallelism, and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, meaning that distortions are not taken into account, and their correction is addressed during image acquisition or calibration.

[0044] The camera 11 is arranged so as to acquire an image of a three-dimensional scene from a defined viewpoint, the viewpoint is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10 as illustrated in [Fig.1].

[0045] The camera 11, for example, acquires images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering an acquisition field 12. An object 13 is placed in the acquisition field 12 of the camera 11, defining an occlusion field 14 for the vision system, an object present in the occlusion field 14 not being observable by the camera 11 from its current observation position.

[0046] It is evident that it is possible to use such a vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently, the invention not being limited to the observation of a three-dimensional scene taking place in front of the vehicle 10 carrying the camera 11.

[0047] According to one particular embodiment, the camera 11 is a wide-angle camera, a wide-angle camera being, for example, equipped with a lens designed to acquire a representative image of a three-dimensional scene seen over a wider field of view than a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene unfolding in front of or around the camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle α of the field of view of the camera 11 is, for example, equal to 120°, 145°, 180°, or 360°, whereas a standard camera offers, for example, a field of view open at an angle of 45° or less.Such a camera 11 corresponds, for example, to a camera equipped with mirrors or to a "fisheye" camera. Wide-angle lenses have a shorter focal length. Compared to standard lenses, this makes them suitable for capturing images of landscapes, architecture, road intersections, or any other subject requiring a wide perspective. Wide-angle cameras are, for example, used to capture immersive and dynamic images with an extended depth of field.

[0048] An image acquired by the camera 11 at a given acquisition time is in the form of data representing pixels characterized by: - ​​coordinates in the image; and - data relating to the colours and brightness of objects in the observed scene in the form of, for example, RGB colourimetric values ​​(from the English "Red Green Blue", in French "Rouge Vert Bleu") or HSL (Tone, Saturation, Luminosity).

[0049] Each pixel of the acquired image represents an object in the three-dimensional scene present in the field of view of the camera 11. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a point of light resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are emitted or reflected, which are captured by a photosensitive sensor of the camera 11 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image that faithfully represents the physical object observed by the camera 11. An image point described above is therefore a point on the surface of an object in the three-dimensional scene.

[0050] When the vehicle 10 is in motion, then two images acquired by the camera 11 at two distinct time instants represent views of the same three-dimensional scene taken from different viewpoints or observation positions, the observation positions of the camera 11 being distinct. On this three-dimensional scene there are, for example: - buildings; - road infrastructure; - other users or stationary objects, for example a parked vehicle; and / or - other users or moving objects, for example another vehicle, a cyclist or a moving pedestrian.

[0051] According to a particular embodiment, an image acquired by the camera 11 includes a distortion equal to 0.5%, 0.8%, or greater than 1%. The measurement of such distortion corresponds to determining a ratio between: - the maximum spacing of a pixel in the image of a straight line in the first three-dimensional scene whose image is a line touching the longest edge of the first image, either in the center of the image edge, or at the corners of the image edge, and - the length of this edge.

[0052] In the world of photography, distortion is commonly considered to be: • negligible if it is less than 0.3%, • not very sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • problematic if it is greater than or equal to 1%.

[0053] Barrel distortion is characterized by a positive percentage, while crescent distortion is characterized by a negative percentage.

[0054] Each image acquired by the camera 11 is for example sent to a processor, for example a computer of a device equipping the vehicle 10, or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10. It is then used during the implementation of a method for determining the depth of one of its pixels and / or during the implementation of a method for learning a depth prediction model associated with this camera of the vision system.

[0055] A process for learning an image restoration model for a vision system embedded in the vehicle 10 is advantageously implemented by the vehicle 10, i.e. by a computer or a combination of computers of the embedded system of the vehicle 10, for example by the computer or computers in charge of the vision system of the vehicle 10.

[0056] In a first operation, initial data are received and are representative of learning frames acquired by the camera 11. These images are for example recorded in a device embedded in the vehicle 10 and comprising a memory accessible by a computer implementing the process.

[0057] According to a particular embodiment, the positions of the pixels in the training images are not expressed in Cartesian coordinates in the current image 31 but are encoded in another reference frame associated with the current image 31.

[0058] Position coding is performed by transforming the position into a combined function using a sinusoidal curve and a cosine curve, following the sub-operations below: • In a first sub-operation, the x-coordinate in the horizontal direction and the y-coordinate in the vertical direction in pixel coordinates (0,1,2,3,4,5,6...) are normalized in a range 0 to 2n. • In a second sub-operation, a denominator that increases in pairs is calculated, the denominator thus describes the following series: 1, 1, 2, 2, 3, 3... with the magnitudes greater than 1 to decrease the magnitude of the coordinate value, and preferably with an exponential increase. • In a third sub-operation, the values ​​of the abscissa x and ordinate y are divided by the denominator. • In a fourth sub-operation, a sine function is applied to the digits in odd positions and a cosine function to the digits in even positions of the x-coordinate and y-coordinate. The result forms two matrices with the height and width of the characteristic map respectively for x and y. • In a fifth sub-operation, the two position coding matrices are summed across all channels of both feature maps.

[0059] Such position coding subsequently allows the position of the pixels in the original training image to be preserved and the level of distortion associated with them to be evaluated. This is particularly important when the training images exhibit strong distortion, such as when images are acquired by a wide-angle camera. Indeed, many image restoration and object detection models are developed for pinhole cameras without image distortion. Although there is a means of image enhancement, such as image stretching for object detection, this distortion is uniform across the entire image, whereas for a wide-angle camera, for example, one calibrated with the Mei model, the level of distortion varies across the image depending on the distance from the optical center.

[0060] Using the received training images, the image restoration model is initially learned from parts of the training images, the parts being obtained by cutting the training images. This phase is also called the initial training phase.

[0061] The image restoration model is capable of restoring or reconstructing an entire image fragment from pixels of that fragment. Such an image restoration model is known to those skilled in the art and is presented, for example, in the paper "Masked Autoencoders Are Scalable Vision Leamers" by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick, published in November 2021.

[0062] Initially, in the absence of a training database, it is possible to train this image restoration model from images acquired by camera 11 and more precisely from fragments of these images.

[0063] Ideally, each fragment should comprise pixels corresponding to an object present in a three-dimensional scene observed by the camera 11. However, During the initialization of this learning, an object detection model integrated into the image restoration model is unable to accurately detect objects in an image fragment.

[0064] Thus, to ensure that a fragment contains pixels corresponding to only one object, this invention proposes using image fragments or cropped images by dividing the original image—that is, the image received from the camera 11 without cropping and therefore in its entirety—into non-overlapping areas. All these areas or fragments function as input images for the object detection model. The advantage of the cropped image is that it contains fewer objects than the entire image. To avoid an abundance of objectless images, such as images representing the sky, trees in leaf, the road surface, etc., it is necessary to limit image recording in the city center, where each image represents numerous static and / or dynamic objects.Optionally, it is possible to avoid areas that are often devoid of objects. For example, if a video, consisting of a series of images acquired consecutively by camera 11, frequently shows the sky at the top center of the frame, this area can be avoided for cropping in all the images of that video. Since this operation is performed on the video as a whole and not frame by frame, the time required for this manual operation is acceptable. The cropped image size can be chosen experimentally by observing the images with human operators or by the size of a typical object, for example, a vehicle at an average distance, for example, 50 meters, in the middle of the line from the optical center to the edge of the image.

[0065] Indeed, if the image restoration model is not yet trained, it is not possible to train this model with the original images because they often contain a plurality of objects. To begin training, the current image is cropped into several areas, each area thus containing a single object and functioning as an image within the training dataset. Preserving the position encoding of the area in the original image represents information about the level of distortion in that area.

[0066] The learning phases are self-supervised, meaning they are performed using data acquired by the vision system itself, i.e., images acquired by camera 11. This self-supervision is not a modeling method, but rather a training method for image restoration and / or object detection models. These methods can be applied to different models as soon as compatibility is ensured. Two types of methods are commonly used: generative, such as the one presented in the document "Masked Autoencoders Are Scalable Vision Learners," hereafter referred to as the MAE method, and contrastive, as presented in the document "Bootstrap your own latent: A new approach to self-supervised learning" by Jean-Bastien Grill and Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Cari Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos and Michal Valko and published in June 2020.

[0067] It should be noted that this invention aims in particular to adapt the generative means to the camera exhibiting a strong image distortion.

[0068] The MAE method defines the proxy task as the restoration of a current image. The current image is divided into several rectangular areas (16, 25, 36, etc.), and more than 75% of the areas are masked. The model to be trained receives the visible areas or patches as input data, and the output data includes encodings or matrix representations for the visible areas: each area has a representation. The MAE method uses these representations, and a representation dedicated to the masked areas, as input data to restore the current image by means of a decoding model called a "decoder." A loss function determines a squared and averaged photometric error across all pixels in the current image. It is proven that the MAE method is capable of training object detection models in an image and their segmentation, that is, the identification of an object type.The learning phase then includes the following operations.

[0069] In a second operation, a fragment of a training image is constructed or restored from pixels included in a fragment of the training image, here a part obtained by cropping the training image.

[0070] In a third operation, a photometric error is determined by comparing the restored fragment, i.e., the fragment reconstructed during the second operation, to the fragment of the training image. A photometric error determined by comparing two fragments refers to the difference between the values ​​associated with pixels, such as the luminance or color values ​​of corresponding pixels in these two images.

[0071] It should be noted that each fragment is aligned, the fragment comprising the pixels within the current bounding box used for generating the second image having the same coordinates in each image as well as the same scale. Thus, the second image has the same dimensions as the current image.

[0072] For each pixel, the difference in luminance or color value, depending on the RGB channels, for example, is determined. If the images are in grayscale, the grayscale levels are simply compared. If they are in color, the values ​​of each channel (red, green, blue), for example, are compared.

[0073] The photometric error is then determined using metrics such as: • the mean squared error, also called "MSE" (from the English "Mean Squared Error"), corresponding to the average of the squares of the differences between the corresponding pixel values, • the mean absolute difference, also called "MAD" (from the English "Mean Absolute Difference"), corresponding to the average of the absolute differences between the pixels, or • the peak signal-to-noise ratio, also called "PSNR" (from the English "Peak Signal-to-Noise Ratio"), which allows for the evaluation of image quality based on the difference between images.

[0074] The photometric error summarizes the differences across all pixels of the image. The higher this number, the greater the difference between the images.

[0075] In a fourth operation, the image restoration model is learned by minimizing the photometric error.

[0076] The second, third and fourth operations, which constitute the initial learning phase, are then repeated for each fragment of each training image.

[0077] In a fifth operation, bounding boxes associated with training objects are determined from the training images by the previously learned object detection model, the object detection model detecting pixels in a training image corresponding to a training object and determining a bounding box in this training image around these pixels.

[0078] The detection of an object in an image and the determination of a bounding box are known to those skilled in the art, these operations being carried out for example using algorithms such as Yolo7®, centemet® or vfnet® presented respectively in the following documents: • “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors”, written by Chien-Yao Wang, Alexey Bochkovskiy and Hong-Yuan Mark Liao and published in July 2022, • “CenterNet: Keypoint Triplets for Object Detection,” authored by Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian, ​​and published in April 2019, and • “VarifocalNet: An loU-aware Dense Object Detector”, written by Haoyang Zhang, Ying Wang, Feras Dayoub and Niko Sünderhauf and published in August 2020.

[0079] Note that the document "Masked Autoencoders Are Scalable Vision Learners" also presents an object detection model associated with the image restoration model.

[0080] According to a particular embodiment, bounding boxes are subsequently considered only if the training object is fully visible in the training image. Fully visible here means that no other object partially obscures the training object; that is, the training object is not occluded by another object, or that no part of the training object is located outside the field of view of the camera 11. This allows only one element of a database to be subsequently saved in which the training object is fully visible.

[0081] According to a particular embodiment, the current pixels correspond to a single training object. Indeed, if current pixels belonged to several objects, the amalgamation between these objects would hinder any subsequent learning.

[0082] A new learning phase then begins, which is carried out using fragments of the training images corresponding to bounding boxes.

[0083] In a sixth operation, each fragment of a training image is restored from pixels including the bounding box and corresponding to a training object.

[0084] In a seventh operation, a photometric error is determined by comparing the restored fragment, i.e. reconstructed during the sixth operation, to the fragment of the training image.

[0085] In an eighth operation, the image restoration model is learned by minimizing the photometric error.

[0086] The sixth, seventh and eighth operations are similar respectively to the second, third and fourth operations, and this time constitute the new learning phase, and are then repeated for each bounding box of each learning image.

[0087] Figure 3 schematically illustrates the training image 31 comprising pixels corresponding to a training object, according to a particular and non-limiting embodiment of the present invention. The pixels corresponding to the training object are called current pixels and form a set 'E' of pixels. The bounding box 'BE' is defined and encompasses the set of current pixels.

[0088] According to a particular embodiment, in a ninth operation, a type of learning object is determined from the current pixels.

[0089] According to a particular embodiment, an object type belongs to a set of object types including objects commonly encountered in a road environment such as: - static objects such as: • a building, • a road sign, • a sidewalk, • ground markings, and - dynamic objects such as: • a vehicle, • a bicycle, • a pedestrian, • a wheelchair.

[0090] It should be noted that this list is not exhaustive; similarly, an element of this list is, according to another embodiment, divided into several sub-elements. Thus, it is possible to differentiate different types of vehicles, for example: • a truck, with or without a trailer, • a car, • a bus, • a motorcycle.

[0091] The segmentation or identification of the type of object is known to the person skilled in the art, the image restoration model presented in the document "Masked Autoencoders Are Scalable Vision Leamers" being, for example, equipped with this type of tool.

[0092] According to another particular embodiment, in a tenth (optional) operation, an environment type is determined from the current pixels. Each element of the training database then includes data representative of a road environment type. A road environment type belongs, for example, to a set of road environment types including: • a parking lot, • a bus stop, • a train station, • a city center within an urban area, • an urban area (excluding the city center), • a residential area, • a departmental or national road in a rural area, • a restricted-access road, • a motorway, • a roundabout, • an intersection, • a level crossing, and • a wooded area, with or without leaves from the trees.

[0093] Thus, a fragment of a training image includes at a minimum pixels associated with the training object and, where applicable, pixels corresponding to the immediate environment of the training object.

[0094] In an eleventh operation, a first distance 'D' between an optical center 'Co' of the current image 31 and a center of the bounding box 'CBE' is determined.

[0095] A first distance corresponds to a distance expressed in pixel units and specifying the distance separating the position in an image from the center of the first pixels relative to the optical center of that image. This data makes it possible to quantify a level of distortion according to which the first pixels represent the first object, which is all the more true when the camera 11 generates strong distortions in the images it acquires, as is the case for a wide-angle camera, for example.

[0096] The positions of the centers 'Co' and 'CBE', as well as all pixel coordinates in general, are defined in the reference frame of the current image 31, including when the current pixels are subsequently extracted from the current image 31. Indeed, the distortion of an image acquired by the camera 11 is a function of the distance to the optical center. The appearance of a certain object in the image, namely its shape and size, depends on this distance. The accumulation of bounding box cropped images obtained via the object detection model amounts to creating a database similar to a database like ImageNet®, presented in the document "ImageNet Large Scale Visual Recognition Challenge", written by Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C.Berg and Li Fei-Fei published their work in September 2014, as these images contain only one object, allowing for the training of the image restoration model. Accumulating images of a certain type of object at different locations within the image provides sufficient variation in its appearance depending on the distance from the optical center.

[0097] The first distance 'D' between the center of the bounding box 'CBE' and the optical center 'Co' of the current image indicates the level of distortion of the appearance of the object in the image 31: the closer the current pixels are to the optical center 'Co', i.e. the smaller the first distance 'D', the less distortion there is, while the further the current pixels are from the optical center 'Co', i.e. the larger the first distance 'D', the greater the distortion there is. To be able to visualize the different levels of distortion of a certain type of object, the first distance 'D' is to be classified among several intervals, for example five (5) intervals uniformly, 0 - 20%, 21% - 40%, 41% - 60%, 61% - 80%, 81% - 100% of the maximum distance corresponding to half the diagonal of the current image 31 if the optical center 'Co' is at the center of the current image 31.In this case, the optical center 'Co' has abscissa 'xCo' with value L / 2 with 'L' the width of the current image 31 and ordinate 'yco' with value H / 2 with 'H' the height of the current image 31, the origin 'O' of the current image 31 being positioned at the top at . Left of the current image 31. It is not advisable to choose too few intervals, for example three (3), because these intervals would not accurately represent the different levels of distortion. However, it is not feasible to choose too many intervals, for example more than thirty (30), because it would be difficult to accumulate enough data for each interval.

[0098] The first distance 'D' associated with the learning object, i.e. the distance separating the optical center of the current image 'Co' from the center of the bounding box 'CBe', is determined by the following function: [Math.l] = | (xm yco}, (xBE,yBE] | = ^(xco-xBE)~^ (yœ-yBEŸ

[0099] With: * the first distance, • xco is the abscissa of the optical center 'Co', • yCo is the ordinate of the optical center 'Co', • xbe the ordinate of the center of the current image 'CBE', and • ^be the ordinate of the center of the current image 'CBE'.

[0100] Note that adapting the position coding allows the position coding in the training image 31 to be retained with its resolution for the pixels present in an image cropped by the bounding box 'BE'. These operations provide information on the object's appearance as a function of the position of the bounding box 'BE' in the current image 31.

[0101] In a twelfth (optional) operation, third data representing a second distance separating the learning object from the vehicle 10 are received from a telemetry system on board the vehicle 10. Such third data are, for example, received from a LiDAR® system or another telemetry system allowing the estimation of a distance separating the vehicle 10 or the camera 11 from the current object observed by the camera 11.

[0102] Thus, a low-resolution LiDAR can be installed on the vehicle 10, which is used to measure the distance of the current object but not to determine a distance or depth for each pixel of the current image 31.

[0103] In a thirteenth operation, a training database is generated from the training images and bounding boxes, each element of the database being associated with a training object among the training objects detected during the fifth operation.

[0104] Each element of the training database then includes secondary data representing: • a type of object to which the learning object corresponds, • a type of environment in which the learning object is situated (if the tenth operation is implemented), • the bounding box associated with the learning object, that is, the pixels corresponding to the learning object and its immediate environment, • a first determined distance between an optical center of the training image and a center of the bounding box associated with the training object, and • a distance separating the learning object from the vehicle or vision system (if the twelfth operation is implemented).

[0105] The objective of this training database is to be homogeneous, that is, to understand elements of all types of objects and environments, but also, for each type of object, to have bounding boxes representing it arranged homogeneously throughout the training images. The same applies to first and second distances; elements of the same type of object must exhibit different first and second distances.

[0106] To achieve this objective, training data are accumulated, for example by following certain rules in order to obtain first distance distributions for a homogeneous type of object or following a defined law. The same applies to second distances.

[0107] In a quantitative evaluation phase of database elements, a distance distribution is determined from initial distances of a set of elements having the same type of training object. In other words, the elements of the training database exhibiting a target object type are selected, then the initial distances associated with each of these elements are extracted, and an initial distance distribution is determined.

[0108] Thus, in a fourteenth operation, fourth data representing a type of target object are received, the type of target object being, for example, entered by a user from a human-machine interface.

[0109] In a fifteenth operation, a distribution of database elements comprising an object type corresponding to said target object type is determined as a function of the first distance.

[0110] According to one embodiment, the distribution of distances is determined by distance classes. The distance classes are, for example, of equal width or of different widths.

[0111] Figure 4 illustrates an example of a distribution of distances by classes in the form of a histogram. Each bar 41, 42, 43, 44, 45 of the histogram represents the ratio between the number of elements of the defined learning object type included in a class and the total number of elements of the defined learning object type. For example: - the first class, represented by a first bar 41, includes all the elements of the second set of elements whose first distance is between 0 and db - the second class represented by a second bar 42 includes all the elements of the second set of elements whose first distance is between di and d2, - the third class, represented by a third bar 43, comprises all the elements of the second set of elements whose first distance is between d2 and d3, - the fourth class, represented by a fourth bar 44, comprises all the elements of the second set of elements whose first distance is between d3 and d4, and - the fifth class represented by a fifth bar 45 includes all the elements of the second set of elements whose first distance is greater than d4.

[0112] The x-axis represents the first distance 'D', for example expressed in pixels (px), while the y-axis represents the ratio or frequency P that each class represents. The ratio is, for example, expressed as a percentage (%).

[0113] According to a particular embodiment, each class has a ratio between a minimum threshold Smin and a maximum threshold Smax. Such limits ensure that each class is represented homogeneously. Thus, for each distance range, a similar number of elements corresponding to the defined type of training object are recorded in the training database.

[0114] The minimum and maximum thresholds are, for example, defined as a function of the total number of elements in the second set of elements and the total number of distance classes; for example, the minimum threshold is determined by the following function: Smin = 2 x NT / (3 x IT) Where: • Smin is the minimum threshold, • Nt the total number of elements in the second set of elements, and • IT the number of intervals, and the maximum threshold Smax is determined by the following function: S - 9 x S ^max — æ umm-

[0115] By selecting the relevant training objects to constitute the training database, that is, the elements whose arguments allow to obtain a defined distribution, we obtain a distribution of elements from the database corresponding to the same type of object according to the first homogeneous distance, for example.

[0116] According to the particular embodiment example in which each element of the training database includes data representative of a type of road environment, the distribution is determined for the type of training object defined but also for a type of road environment defined.

[0117] In a sixteenth operation, a score is determined by comparing the previously determined distribution to a target distribution, the score being for example representative of the homogeneity of the frequency associated with the values ​​of first distances or classes.

[0118] According to one variant, in a seventeenth operation, a distribution of second distances or separations is determined. The separations are grouped into classes according to a measurement range, which is, for example, between one and one hundred meters (1-100m). The class boundaries are then defined, for example, every ten meters (10m). As with the distance classes, the class widths are not, however, necessarily equal. The probability associated with a separation class is defined by the following function: [Math.2]

[0119] With: • P the frequency associated with a distance class ij, • the object index characterizing the type of learning object defined, • N indicates the number of elements in the database that correspond to the type of learning object defined in the distance class (index ij) and in the entire database (total index).

[0120] The probability distributions of each type of learning object defined in the defined environment as a function of the first distance and / or as a function of the distance where applicable are ideally homogeneous, that is to say that the bars of the histograms are almost of the same height.

[0121] It should be noted that if the accumulation of data for a defined type of training object is too expensive, the tolerance on the probability distribution defines a maximum frequency associated with a first class that does not exceed twice a minimum frequency associated with a second class. The minimum frequency of an interval is defined by the following function:

[0122] [Math.3] / P ■ = h mm lmal. /

[0123] With: • min the minimum frequency, and • i>mix the total number of classes.

[0124] Thus, for ten (10) intervals for example, the minimum frequency is equal to about 6.7% and the maximum frequency is equal to about 13.3%.

[0125] According to a particular embodiment, the current element is recorded if the total number of elements in the second set of elements is less than a total threshold. Such a condition thus prevents overloading or saturating the database and avoids using excessive memory capacity to store the training database.

[0126] Thus, the detection of training objects in a training image functions as a means of data accumulation. The continuous learning of the image restoration model and / or the object detection model, and the updating of the training database, are then carried out progressively until satisfactory performance of the image restoration and / or object detection model is achieved in an image. In other words, the training phase of the image restoration and / or object detection model is performed with the training database, and, for example, training is performed again when this database is enriched. Thus, this training phase takes place at different times, for example, when the database is fully populated and / or when the database is enriched.

[0127] The image restoration model training is repeated, in particular, to improve the reliability of the image restoration model and also to obtain better dataset statistics. The accumulation and evaluation of the dataset is described as follows: • Define the desired object types to detect and the relevant environments. • To limit the amount of data, only begin recording when both the previously defined objects and environment are present. Data accumulation for objects is independent of data accumulation for environments. For example, an image of a car taken on a highway is equivalent to another image of the car in the city center. Conversely, all images taken on the highway are equivalent for that same environment, regardless of the object they contain. If recording is repeated for a specific object or environment, We need to find the combination between the object with the fewest images and the environment with the fewest images. • Sufficient examples must be accumulated for each type of object, for example, one thousand (1000). Each type of object can contain different shapes; for example, there are different categories of "passenger car" depending on size. All these categories constitute one thousand (1000) examples for the "passenger car" type, as they have similar geometric characteristics. The number of images of a certain type of object can be found using the object detection model: the number of detections of the bounding box for that type of object. The threshold for the number of images must be determined by training the model: if performance is poor despite a good probability distribution of appearances, this threshold must be increased.

[0128] This training data is thus used to adjust the parameters of the image restoration model which is provided with an object detection model and / or the object detection model in an image, associated with the vision system on board the vehicle, thus making reconstructions and object detections more reliable and increasing the robustness of each model.

[0129] If an AD AS uses data from the onboard vision system When an object, such as another road user, is used as input data to determine its presence, the ADAS (Advanced Driver Assistance System) can then accurately consider and process this object. For example, if the ADAS's function is to warn the vehicle's driver of the presence and position of another road user, the driver receives accurate information and reacts accordingly. This improves the safety of all road users.

[0130] Figure [Fig. 2] illustrates a flowchart of the different stages of a process 2 learning an image restoration model for a vehicle-mounted vision system comprising a camera, for example the camera 11 mounted in the vehicle 10 of [Fig. 1], according to a particular and non-limiting embodiment of the present invention. The method 2 is implemented, for example, by a device mounted in the vehicle 10 or by the device 5 of [Fig. 5].

[0131] In a first step 21, initial representative data from learning frames acquired by the camera 11 are received.

[0132] In a second step 22, the image restoration model is initially learned from parts of the training images obtained by cutting the training images;

[0133] In a third step 23, bounding boxes associated with training objects are extracted from the training images by an object detection model integrated into the image restoration model, the object detection model detecting pixels in a training image corresponding to an object training among the training objects and determining a bounding box in the training image around these pixels.

[0134] In a fourth step 24, the image restoration model is again learned from the bounding boxes.

[0135] According to one variant, the variants and examples of the operations described in relation to Figures 1, 3 and 4 apply to the steps of process 2 of [Fig.2].

[0136] Figure 6 illustrates a flowchart of the different stages of a learning process 6 for an image restoration model for a vehicle-mounted vision system comprising a camera, for example the camera 11 mounted in the vehicle 10 of Figure 1, according to a particular and non-limiting embodiment of the present invention. The learning process 6 is, for example, implemented by a device mounted in the vehicle 10 or by the device 5 of Figure 5.

[0137] According to a particular embodiment, this learning method 6 is included in the method 2 for learning an image restoration model for a vision system embedded in a vehicle, for example in the vehicle 10.

[0138] The input data of this learning process 6 then corresponds to a fragment of image acquired by a camera mounted in a vehicle, obtained by cropping or cutting this image in particular, but not exclusively, by defining a bounding box around pixels corresponding to an object located in a three-dimensional scene observed by the camera.

[0139] In a first learning step 'a', a restored image is constructed by the image restoration model from the image fragment, in particular from the pixels corresponding to the object.

[0140] In a second learning step 'b', a photometric error is determined by comparing the restored image to the image fragment.

[0141] In a third learning step 'c', the image restoration model is learned by minimizing the photometric error.

[0142] Such a learning process 6 is thus implemented for a set of image fragments contained in a training database or implemented iteratively following the addition of elements in this training database.

[0143] According to one variant, the variants and examples of the operations described in relation to Figures 1, 3 and 4 apply to the process steps 6 of [Fig.6].

[0144] Figure 5 schematically illustrates a device 5 configured for learning an image restoration model for a vision system embedded in a vehicle, for example in the vehicle 10 of Figure 1, according to a particular and non-limiting embodiment of the present invention. The device 5 corresponds for example to a device embedded in the first vehicle 10, for example a computer.

[0145] The device 5 is, for example, configured to carry out the operations described opposite Figures 1, 3, and 4 and / or the steps described opposite Figures 2 and 6. Examples of such a device 5 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of the device 5, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. The device 5 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0146] The device 5 comprises one (or more) processor(s) 50 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 5. The processor 50 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 5 further comprises at least one memory 51, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0147] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 51.

[0148] According to various particular and non-limiting embodiments, the device 5 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0149] According to a particular and non-limiting embodiment, the device 5 includes a block 52 of interface elements for communicating with external devices. The interface elements of the block 52 include one or more of the following interfaces: - Radio frequency (RF) interface, for example, Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology narrow), or LoRa in the 868 MHz frequency band, LTE (from the English "Long-Term Evolution" or in French "Evolution à long terme"), LTE-Advanced (or in French LTE-avancé); - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HD MI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0150] According to another particular and non-limiting embodiment, the device 5 includes a communication interface 53 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 530. The communication interface 53 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 530. The communication interface 53 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0151] According to a particular and non-limiting embodiment, the device 5 can provide output signals to one or more external devices, such as a display screen 540, touch or not, one or more loudspeakers 550 and / or other peripherals 560 (projection system) via the output interfaces 54, 55, 56 respectively. According to a variant, one or more of the external devices is integrated into the device 5.

[0152] Of course, the present invention is not limited to the embodiments described above but extends to a method for acquiring a training database for an image restoration model for a vehicle-mounted vision system, the training database being constructed from images acquired by a vehicle-mounted camera, and more specifically from fragments of these images, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0153] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 5 of [Fig.5].

Claims

Demands

1. Method for learning an image restoration model for a vision system embedded in a vehicle (10), the vision system comprising at least one camera (11) arranged to acquire an image of a three-dimensional scene of an environment outside the vehicle (10), said method being implemented by a processor and being characterized in that it comprises the following steps: - receiving (21) first representative data of training images acquired by said camera (11); - initial learning (22) of the image restoration model from parts of the training images, said parts being obtained by cutting said training images;- extraction (23) of bounding boxes associated with training objects from the training images by an object detection model integrated into said image restoration model, said object detection model detecting pixels in a training image corresponding to a training object among said training objects and determining a bounding box in said training image around said pixels; - further training (24) of the image restoration model from the bounding boxes.

2. A method according to claim 1, further comprising a step of encoding pixel positions of a training image in a reference frame associated with said training image, a position of a bounding box being defined in said reference frame.

3. A method according to claim 1 or 2, further comprising the generation of a training database from the training images and bounding boxes, each element of said database being associated with a training object among said training objects and comprising second data representing: • a type of object to which said training object corresponds, • said bounding box associated with said training object, and • a first determined distance between an optical center of said training image and a center of the bounding box associated with said training object.

4. A method according to claim 3, further comprising a phase of quantitative evaluation of database elements comprising the following steps: - receiving fourth data representative of a target object type, - determining a distribution of database elements comprising an object type corresponding to said target object type as a function of the first distance, - determining a score by comparing said distribution to a target distribution.

5. A method according to claim 3 or 4, wherein the second data are representative of a type of road environment, said type of road environment being determined from the first or second training image.

6. A method according to any one of claims 3 to 5, wherein the second data represent a second distance separating said learning object from the vehicle (10), the third data representing said second distance being received from a telemetry system on board the vehicle (10).

7. A method according to any one of claims 1 to 6, wherein pixels contained within a bounding box correspond to a single training object.

8. A method according to any one of claims 1 to 7, wherein the initial training (22) and retraining (24) respectively comprise the following steps: - restoration (a) of a fragment of a training image from pixels included in respectively said part of the training image or said bounding box, - determination (b) of a photometric error by comparing pixels of said restored fragment to pixels of respectively said part of the training image or said bounding box, and - training (c) of said image restoration model by minimizing said photometric error.

9. Device (5) for acquiring a training database of an object detection model for a vehicle-mounted vision system (10), said device (5) comprising a memory (51) associated with at least one processor (50) configured for the

10. implementation of the steps of the process according to any one of claims 1 to 8. Vehicle (10) comprising the device (5) according to claim 9.

Images