Method and device for learning a depth prediction model capable of processing dynamic objects present in a scene.

Abstract

Method or device for learning a depth prediction model associated with a monocular vision system. The method comprises a first learning phase (25) of the depth prediction model from images acquired (21) by the vision system's camera, in which pixels corresponding to dynamic objects are excluded through the determination of bounding boxes (22), and a second learning phase (28) of the depth prediction model from synthetic images (26) representing dynamic objects. The pixels of the synthetic images are associated with target depths determined by the image generation model that generated the synthetic images, thus including annotated data. Figure 2 (for the abstract)

Description

Title of the invention: Method and device for learning a depth prediction model capable of processing dynamic objects present in a scene. technical field

[0001] The present invention relates to methods and devices for learning a depth prediction model associated with a vision system embedded in a vehicle, for example, in a motor vehicle. The present invention also relates to a method and device for determining depth and / or measuring the distance separating a static or dynamic object from a vehicle equipped with a vision system. Technological background

[0002] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0004] The ADAS systems installed in a vehicle are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras make it possible, in particular, to detect and locate other road users or any obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle's speed; - to act on the braking system in case of risk of impact with an object.

[0005] The position of another user or an obstacle in a three-dimensional scene is, for example, determined by a vision system comprising a prediction model for a depth associated with a pixel or a distance separating the vision system from an object in a three-dimensional scene. However, the accuracy of a predicted depth or distance for a dynamic object, that is, an object in movement in the three-dimensional scene is generally not satisfactory for a vision system comprising only a single camera, also called a monocular vision system.

[0006] Indeed, a monocular vision system uses several consecutive images to predict depths associated with pixels in one of the images. This monocular vision system has the drawback of not being able to accurately predict depths associated with pixels in an image when objects in the observed three-dimensional scene are moving within that same scene, because the depth is obtained by comparing the positions of pixels corresponding to the same object in the three-dimensional scene in different images. However, if the observed object is moving, then the pixels associated with that object are displaced between two images acquired at different times, and this displacement is due both to the movement of the monocular vision system mounted in the vehicle and to the inherent movement of the dynamic object within the three-dimensional scene.A depth prediction model is therefore unable to accurately determine the position of this dynamic object.

[0007] Moreover, the presence of a dynamic object in a learning phase then disrupts the learning of the depth prediction model, that is to say that the quality of the learning of the depth prediction model is degraded when pixels corresponding to dynamic objects are taken into consideration during the learning phase.

[0008] The quality of the training of the depth or distance prediction model is, however, very important. Indeed, the depths or distances predicted by the model represent, for example, the distances to other road users or obstacles present in the road environment of the vehicle equipped with the vision system, whose output data feeds ADAS (Advanced Driver Assistance Systems). The proper functioning of the driver assistance devices using this data therefore depends on the quality of the data emitted by the vision system. Summary of the present invention

[0009] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0010] Another object of the present invention is to improve the learning phase of a depth prediction model from images acquired by a vision system and representing objects of a three-dimensional scene moving in that same scene, the depth prediction model being implemented in particular by a neural network and associated with a vision system embedded in a vehicle.

[0011] Another object of the present invention is to improve road safety, in particular by improving the reliability of AD AS systems powered by data obtained from a camera of a vision system.

[0012] According to a first aspect, the present invention relates to a method for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, the vision system comprising a camera arranged to acquire an image of a three-dimensional scene taking place near the vehicle, the method being implemented by at least one processor, and being characterized in that it comprises two learning phases, • an initial learning phase comprising the following steps: - reception of representative data from a first image and a second image acquired by the camera at two distinct acquisition times; - determination of bounding boxes associated with objects in the three-dimensional scene in the first and second images and of an object type associated with each bounding box, - prediction of depths associated with pixels of the first image not included in a bounding box whose associated object type corresponds to a target object type, called first pixels, and of depths associated with pixels of the second image not included in a bounding box whose associated object type corresponds to said target object type, called second pixels, by the depth prediction model; - generation of a third image from the first pixels, the depths associated with the first pixels and a movement of the vehicle between the two temporal moments of acquisition and of a fourth image from the second pixels, the depths associated with the second pixels and said movement of the vehicle; - initial training of the depth prediction model by minimizing a first loss error determined from a first intermediate error determined by comparing the third image to the second image and a second intermediate error determined by comparing the fourth image to the first image; and • a second learning phase comprising the following steps: - reception of a synthetic image and target depths associated with pixels of the synthetic image representing an object corresponding to the type of target object, said synthetic image and said target depths being generated by a synthetic image generation model; - determination of depths associated with said pixels of the synthetic image by the depth prediction model; - second learning of the depth prediction model by minimizing a second loss error determined by comparing the predicted depths to the target depths associated with said pixels of the synthetic image.

[0013] The method advantageously allows the first loss error to be determined only for pixels that do not correspond to an object of a certain type, for example, a dynamic object. The presence of a dynamic object in an image acquired by the vision system then does not impact the training of the depth prediction model, as a portion of the image containing pixels corresponding to this dynamic object is excluded when determining the first loss error. It is therefore possible to learn a depth prediction model associated with the vision system even when images acquired by this vision system include pixels representing a dynamic object present in the observed three-dimensional scene.

[0014] The second learning phase then trains the depth prediction model using annotated synthetic images, these synthetic images representing objects of the type of object not previously considered in the first learning phase. This second learning phase enables the depth prediction model to also predict accurate depths for objects of said type of object, for example, for dynamic objects.

[0015] The depth prediction model is then reliable for pixels corresponding to both static and dynamic objects present in the observed three-dimensional scene.

[0016] According to a variant of the method, the third image, respectively fourth image, is generated by the following function: ^ = ^(^1^(^^(^))]) With : • P^ the coordinates of a pixel in the third image, respectively fourth picture, • A function to convert from homogeneous coordinates to pixel coordinates removing one dimension from a vector, • A camera-associated direction prediction model, • Let T be a matrix comprising the extrinsic parameters, and • 0 a projection function in the three-dimensional scene of a pixel Pt in based on its coordinates in the first image, respectively second image, and of the depth which is associated with it.

[0017] According to another variant of the method, the first and second intermediate errors include a photometric error determined by the following function: U{p} = EJ ( 1-a) • \l{p)-î(p) | +a■ ( 1 -±SSIM(l(p), ï(p) ) ) ] With: • L*{p} is the photometric error associated with a pixel p defined by its coordinates in the first image, respectively the second image, • l(p) a value of the pixelp in the first image, respectively second image, * l(p) a value of pixelp in the fourth image, respectively third image, • SSIM is a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of road environment.

[0018] According to a further embodiment of the method, the first and second intermediate errors include a reconstruction error determined by the following function: With : • ^smooth(P) a reconstruction error for a pixelp of the third image, respectively fourth image, • a depth associated with the pixelp, • VF is a parameter matrix, • 0 the order of a smoothing gradient, • an L1 norm of the second-order depth gradients is calculated with VF = 1, et0 = 2, • x and V' are the dimensions of the third image, and respectively the fourth image, • P is a hyperparameter dependent on the road environment in which the vehicle is traveling, and • It(p) a value of pixel p in the third image, respectively fourth image.

[0019] According to yet another variant of the method, the first loss error is determined by the following function: With : • The first mistake of loss, • L^p) the first intermediate error for a pixelp of the first image, and • L2(p) the second intermediate error for a pixel p of the second image corresponding to the pixel p of the first image.

[0020] According to yet another variant of the method, a first intermediate error, respectively second intermediate error, associated with a pixel of the first image, respectively second image, contained in a bounding box corresponding to a type of target object is zero.

[0021] According to an additional variant of the method, the type of target object belongs to a set of target objects comprising: - a car, - a truck, - a coach, - a motorcycle, and - a bicycle.

[0022] Such target objects are here objects whose own velocity in the three-dimensional scene is not negligible and for which the first learning phase is not suitable, the own motion of an object of this type of target object affecting the first loss error if pixels associated with this object are taken into consideration when calculating the first loss error.

[0023] According to a second aspect, the present invention relates to a device configured to learn a depth prediction model by a vision system embedded in a vehicle, the device comprising a memory associated with at least one processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0024] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0025] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0026] Such a computer program may use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0027] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0028] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may include a means storage, such as a ROM, CD-ROM or microelectronic circuit type ROM, or a magnetic recording medium or a hard disk drive.

[0029] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0030] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0031] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 3, in which:

[0032] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a a particular and non-limiting example of a realization of the present invention;

[0033] [Fig.2] illustrates a flowchart of the different stages of a method for learning a depth prediction model associated with the vision system of [Fig.1], according to a particular and non-limiting embodiment of the present invention; and

[0034] [Fig.3] schematically illustrates a device configured to learn a depth prediction model associated with the vision system on board the vehicle of [Fig.1], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0035] A method and device for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle will now be described in what follows with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the description that follows.

[0036] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0037] Fig. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention.

[0038] The vehicle 10 is located in an environment 1 corresponding, for example, to a road environment consisting of a network of roads accessible to the vehicle 10.

[0039] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, or a motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is to say, a vehicle operating according to a predetermined level of autonomy or under the total supervision of the driver.

[0040] The vehicle 10 advantageously comprises at least one onboard camera 11, configured to acquire images of a three-dimensional scene unfolding in the environment of the vehicle 10 from a common viewing position. The camera 11 forms a monocular vision system if used alone, as illustrated in [Fig. 1]. However, the present invention is not limited to a monocular vision system comprising a single camera but extends to any vision system comprising at least one camera, for example, 1, 2, 3, or 5 cameras.

[0041] The camera 11 has intrinsic parameters, including: - a focal length, - distortions which are due to imperfections in the optical system of camera 11, - a direction of the optical axis of camera 11, and - a resolution.

[0042] The intrinsic parameters characterize the transformation which associates, for an image point, hereafter called "point", its three-dimensional coordinates in the camera 11 reference frame with the pixel coordinates in an image acquired by the camera 11. These parameters do not change if the camera is moved.

[0043] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most significant effects, namely radial, decentering, and prismatic distortions, induced by defects in lens curvature, parallelism, and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, meaning that distortions are not taken into account, and their correction is addressed during image acquisition or calibration.

[0044] The camera 11 is arranged so as to acquire an image of a three-dimensional scene from a defined viewpoint; the viewpoint is, for example, located on or in the left rearview mirror of vehicle 10 or at the top of the windscreen of vehicle 10 as illustrated in [Fig.1].

[0045] The camera 11, for example, acquires images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering an acquisition field 12. An object 13 is placed in the acquisition field 12 of the camera 11, defining an occlusion field 14 for the vision system, an object present in the occlusion field 14 not being observable by the camera 11 from its current observation position.

[0046] It is evident that it is possible to use such a vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently, the invention not being limited to the observation of a three-dimensional scene taking place in front of the vehicle 10 carrying the camera 11.

[0047] According to one particular embodiment, the camera 11 is a wide-angle camera, a wide-angle camera being, for example, equipped with a lens designed to acquire a representative image of a three-dimensional scene seen over a wider field of view than a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene unfolding in front of or around the camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle α of the field of view of the camera 11 is, for example, equal to 120°, 145°, 180°, or 360°, whereas a standard camera offers, for example, a field of view open at an angle of 45° or less.Such a camera 11 corresponds, for example, to a camera equipped with mirrors or a "fisheye" camera. Wide-angle lenses have a shorter focal length compared to standard lenses, making them suitable for capturing images of landscapes, architecture, road intersections, or any other subject requiring a wide perspective. Wide-angle cameras are, for example, used to capture immersive and dynamic images with an extended depth of field.

[0048] An image acquired by the camera 11 at a given acquisition time is in the form of data representing pixels characterized by: - ​​coordinates in the image; and - data relating to the colours and brightness of objects in the observed scene in the form of, for example, RGB colourimetric values ​​(from the English "Red Green Blue", in French "Rouge Vert Bleu") or HSL (Tone, Saturation, Luminosity).

[0049] Each pixel of the acquired image represents an object in the three-dimensional scene present in the field of view of the camera 11. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a point of light resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are emitted or reflected, which are captured by a photosensitive sensor of the camera 11 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image that faithfully represents the physical object observed by the camera 11. An image point described above is therefore a point on the surface of an object in the three-dimensional scene.

[0050] When the vehicle 10 is in motion, then two images acquired by the camera 11 at two distinct time instants represent views of the same three-dimensional scene taken from different viewpoints or observation positions, the observation positions of the camera 11 being distinct. On this three-dimensional scene there are, for example: - buildings; - road infrastructure; - other users or stationary objects, for example a parked vehicle; and / or - other users or moving objects, for example another vehicle, a cyclist or a moving pedestrian.

[0051] According to a particular embodiment, an image acquired by the camera 11 includes a distortion equal to 0.5%, 0.8%, or greater than 1%. The measurement of such distortion corresponds to determining a ratio between: - the maximum spacing of a pixel in the image of a straight line in the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the image edge or at the corners of the image edge, and - the length of this edge.

[0052] In the world of photography, distortion is commonly considered to be: • negligible if it is less than 0.3%, • slightly sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • bothersome if it is greater than or equal to 1% or more.

[0053] Barrel distortion is characterized by a positive percentage, while crescent distortion is characterized by a negative percentage.

[0054] Each image acquired by the camera 11 is for example sent to a processor, for example a computer of a device equipping the vehicle 10, or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10. It is then used during the implementation of a method for determining the depth of one of its pixels and / or during the implementation of a method for learning a depth prediction model associated with this camera of the vision system.

[0055] Figure 2 illustrates a flowchart of the different stages of a process learning a depth prediction model, such a depth prediction model being for example associated with the vision system on board the vehicle 10 and including the first camera 11 and also being used in a method of determining the depth of a pixel of an image, according to a particular and non-limiting embodiment of the present invention.

[0056] The learning process 2 is for example implemented by a device on board the vehicle 10 implementing the depth determination process for the vision system on board a vehicle 10 or by the device 3 of [Fig.3].

[0057] Such a learning process 2 comprises two learning phases. A first learning phase is based on the processing of images acquired by the camera 11 and allows learning the depth prediction model for pixels corresponding to objects, particularly static ones, present in a three-dimensional scene observed by the camera 11, while the second learning phase is based on the processing of synthetic images representing dynamic objects, these synthetic images being generated by a synthetic image generation model and being annotated, that is to say, presenting target depths determined by the synthetic image generation model during the generation of a synthetic image.

[0058] The learning process 2 first includes steps associated with the first phase of learning.

[0059] In a first step 21, representative data of a first image and a second image acquired by the camera 11 at two distinct acquisition times are received; that is, a first image acquired by the camera 11 at a first acquisition time and a second image acquired by the camera 11 at a second acquisition time distinct from the first acquisition time are received. Image reception consists of receiving representative image data, which are, for example recorded in a device embedded in the vehicle 10 and comprising memory accessible by a computer implementing the learning process 2. The first and second acquisition times are sufficiently close in time so that the observed three-dimensional scene is the same but viewed from two different points of view while the vehicle 10 is in motion. The first and second images are acquired at time intervals on the order of one or more milliseconds.

[0060] According to a particular embodiment, the positions of the pixels in the first and second images are not expressed in Cartesian coordinates but are encoded in another reference frame associated with these images.

[0061] Position coding is performed by transforming the position into a combined function using a sinusoidal curve and a cosine curve, following the sub-operations below: • In a first operation, the abscissa x in the horizontal direction and the ordinate y in the vertical direction in pixel coordinates (0,1,2,3,4,5,6...) are normalized in a range 0 to 2;T. • In a second operation, a denominator that increases in pairs is calculated, the denominator thus describes the following series: 1, 1, 2, 2, 3, 3... with the magnitudes greater than 1 to decrease the magnitude of the coordinate value, and preferably with an exponential increase. • In a third operation, the values ​​of the abscissa x and ordinate y are divided by the denominator. • In a fourth operation, a sine function is applied to the digits in odd positions and a cosine function to the digits in even positions of the x-coordinate and y-coordinate. The result forms two matrices with the height and width of the characteristic map respectively for x and y. • In a fifth operation, the two position coding matrices are added to all channels of the two feature maps.

[0062] Such position coding subsequently makes it possible to preserve the position of the pixels in the original image and to evaluate a level of distortion associated with them, which is especially important when the images exhibit strong distortion, such as when the images are acquired by a wide-angle camera.

[0063] In a second step 22, bounding boxes associated with objects of the three-dimensional scene are determined in the first and second images and an object type is associated with each bounding box,

[0064] Bounding boxes are determined by an object detection model, which detects pixels in an image corresponding to an object and determines a bounding box in this training image around these pixels.

[0065] The detection of an object in an image and the determination of a bounding box are known to those skilled in the art, these operations being carried out for example using algorithms such as Yolo7®, centemet® or vfnet® presented respectively in the following documents: • “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors”, written by Chien-Yao Wang, Alexey Bochkovskiy and Hong-Yuan Mark Liao and published in July 2022, • “CenterNet: Keypoint Triplets for Object Detection,” authored by Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian, ​​and published in April 2019, and • “VarifocalNet: An loU-aware Dense Object Detector”, written by Haoyang Zhang, Ying Wang, Feras Dayoub and Niko Sünderhauf and published in August 2020.

[0066] The type of object associated with a bounding box is then compared to a type of target object belonging to a set of target objects comprising: - a car, - a truck, - a coach, - a motorcycle, and - a bicycle.

[0067] Such target objects correspond to objects in the three-dimensional scene that can have a high velocity in the observed three-dimensional scene; that is, they are objects capable of moving rapidly within the observed three-dimensional scene. Note that while target objects are capable of moving rapidly, they can nevertheless be stationary within the observed three-dimensional scene; for example, a truck can be parked or stopped at a stop sign. Such an object is stationary and could have been considered in the first learning phase; however, it is preferred here not to consider it in the subsequent steps of the first learning phase, and a truck, whether moving or stationary, will be addressed in the second learning phase.

[0068] In a third step 23, first depths associated with first pixels are predicted, the first pixels being pixels of the first image not contained within a bounding box corresponding to a target object type. Similarly, second depths associated with second pixels are predicted, the second pixels being pixels of the second image not contained within a bounding box corresponding to a target object type.

[0069] Such a depth prediction model is known to those skilled in the art, and is described for example in the following documents: - “Digging Into Self-Supervised Monocular Depth Estimation” by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019, and - “HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation” by Xiaoyang Lyu, Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu, Xinxin Chen and Yi Yuan published in December 2020.

[0070] Pixels corresponding to an object whose type belongs to the target object type are not taken into consideration at this stage. Indeed, such objects are considered dynamic, and a predicted depth for such an object is therefore unreliable because the object's own motion within the three-dimensional scene is unknown and misleads the depth prediction model. Thus, only the static elements of the three-dimensional scene observed by the camera 11 are taken into consideration during the first learning phase.

[0071] In a fourth step 24, third and fourth images are generated from respectively the first and second images, the depths associated with the first pixels, respectively second pixels, and a movement of the vehicle 10 between the two time instants of acquisition.

[0072] Generating a third image from the first image acquired by camera 11 consists of reprojecting a pixel from the first image acquired at a first time instant into the three-dimensional scene as a point, and then projecting this point into the image plane of camera 11 at its position at a second time instant of acquisition corresponding to the time instant of acquisition of the second image, so as to obtain a third image corresponding to a view of the three-dimensional scene from the viewpoint of camera 11 at the second time instant of acquisition. The image plane of camera 11 corresponds to a plane defined in the camera 11's frame of reference, normal to the optical axis of camera 11 and located at the focal length of camera 11.Thus, the third image generated from the first image is comparable to the second image, except for the areas of the second image contained within bounding boxes associated with objects of the target object type. Similarly, the fourth image generated from the second image is comparable to the first image, except for the areas of the first image contained within bounding boxes associated with objects of the target object type.

[0073] Since the positions of the camera 11 at the first and second acquisition times are not coincident when the vehicle 10 is moving, and since objects may obscure other objects in the scene or even move between the two acquisition times, the third and fourth generated images are not identical to the first and second images acquired by the camera 11. Furthermore, depth prediction includes the prediction of camera 11's movement. The differences between the first and second time points, as well as the models used to generate the third and fourth images, are not error-free. Therefore, comparing a generated image to an acquired image allows us to assess the accuracy of the different models used and the predicted depth values.

[0074] According to a particular embodiment, the third image, respectively the fourth image, is generated by the following function:

[0075] [Math.l] = D(pt) ) ] )

[0076] With: • Ps the coordinates of a pixel from the third image, respectively fourth image, • A function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K is a direction prediction model associated with camera 11, • Let T be a matrix comprising the extrinsic parameters, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the first image, respectively second image, and of the depth j associated with it.

[0077] According to a first variant, the direction prediction model corresponds to an intrinsic matrix of the camera 11. This first variant is particularly applicable to a pinhole camera and calibrated.

[0078] According to a second variant, the direction prediction model corresponds to a learned model implementing a different neural network. Such a direction prediction model is known to those skilled in the art; it is notably presented in the document "Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion." This second variant is suitable for wide-angle or fisheye cameras, or even for uncalibrated cameras. According to this second variant, a direction prediction step associated with the pixels of the first image and the pixels of the second image is necessary.

[0079] The matrix comprising the extrinsic parameters of the moving camera 11 can be obtained in several ways. For example, from data emitted by a driver assistance system of the vehicle 10, called the AD AS system, or from sensors on board the vehicle 10. According to another example, the extrinsic parameters corresponding to the movement of the camera 11 between the first and second time instants are determined in an additional step by a motion prediction model, for example by the one presented in the document "Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion".

[0080] In a fifth step 25, the depth prediction model is learned for the first time by minimizing a first loss error determined from: • a first intermediate error determined by comparing the third image to the second image, and • a second intermediate error determined by comparing the fourth image to the first image.

[0081] According to a particular embodiment, the first and second intermediate errors comprise a photometric error determined by the following function: [Math.2] Mp) = EJ (1-«) • l / (p)-Hp) 1 +«• ( ï-issiM(Hp).Hp) ))]

[0082] With: • L*(p) the photometric error associated with a pixel p defined by its coordinates in the first image, respectively second image, • Ip is a pixel value in the first image, respectively second image, • jp a value of pixel p in the fourth image, respectively third image, • SSIM is a function that takes into account a local structure, and • “a weighting factor depending in particular on the type of road environment.

[0083] It should be noted that a photometric error is lower the more accurate the depth prediction model, i.e., the more effective its training. Conversely, if the depth prediction model lacks accuracy, for example because its training is incomplete, then a photometric error is not negligible.

[0084] According to another particular embodiment, the first and second intermediate errors include a reconstruction error determined by the following function: [Math.3]

[0085] With: • Hmooth(P) is a reconstruction error for a pixelp of the third image, respectively fourth image, • Dmt a depth associated with the pixelp, • VF is a parameter matrix, • 0 the order of a smoothing gradient, • an L1 norm of the second-order depth gradients is calculated with VF = 1, et0 = 2, • x and y are the dimensions of the third image, and the fourth image respectively, • P is a hyperparameter dependent on the road environment in which vehicle 10 is traveling, and • IpP) a value of the pixel p in the third image, respectively fourth image.

[0086] This function is generally used to deal with the discontinuity at the edge of objects (in English "edge aware smoothness").

[0087] According to a particular embodiment, a first intermediate error, or second intermediate error, associated with a pixel of the first image, or second image, contained within a bounding box corresponding to a target object type, is zero; that is, a first or second intermediate error associated with a pixel that is neither a first pixel nor a second pixel is zero. Thus, pixels contained within bounding boxes corresponding to an object of the target object type are not taken into account in determining the first and second intermediate errors. The presence of a potentially dynamic object is therefore not detrimental to the first learning phase.

[0088] According to a particular embodiment, the first loss error is determined by the following function: [Math.4]

[0089] With: • The first mistake of loss, • L^p} the first intermediate error for a pixelp of the first image, and • L2(p} the second intermediate error for a pixel p of the second image corresponding to the pixel p of the first image.

[0090] The first learning of the depth prediction model consists of adjusting the input parameters of the convolutional neural network implementing the depth prediction model in order to minimize the first loss error previously calculated.

[0091] According to a particular embodiment, the first learning phase is repeated for more pairs of images acquired by the camera 11, for example for pairs of images comprising images distinct from the first and second images, and / or for pairs of images comprising the same first or second image, this first or second image being called the reference image and the other images forming different pairs of images, where the other image is acquired at different time points. For example, the reference image is an image acquired at time t, the images of the different pairs of images are acquired at times t+1, t+2, t+3 and t+4, each increment being determined by the image acquisition frequency of camera 11. Thus, according to this particular embodiment, the first phase of process 2 takes into account series of five images acquired consecutively by camera 11.

[0092] The learning process 2 then includes steps associated with the second phase of learning.

[0093] In a sixth step 26, a synthetic image and target depths are received. The target depths are depths associated with pixels of the synthetic image, which represents an object corresponding to the type of target object.

[0094] Such images are generated by a synthetic image generation model known to the person skilled in the art, for example using the synthetic image generation model of the aiSim® simulation tool developed by aiMotive®.

[0095] Thus, for each pixel of the synthetic image generated by the synthetic image generation model, a target depth is associated. This target depth is then an annotated data point, which is reliable and usable for training the depth prediction model.

[0096] Using this synthetic image generation model, it is possible to create a set of synthetic images forming a training set that can be used to train the depth prediction model during this second learning phase. These images represent different objects corresponding to the target object types and are shown in different positions, sizes, and locations within the generated images. These generated images do not necessarily show any scenery or three-dimensional background, but simply the objects in mid-air within the images.

[0097] In a seventh step 27, depths associated with the pixels of the synthetic image are determined by the depth prediction model.

[0098] In an eighth step 28, the depth prediction model is learned a second time by minimizing a second loss error determined by comparing the predicted depths to the target depths associated with the pixels of the synthetic image. The second error is, for example, equal to the sum of the errors determined for pixels of the synthetic image, each error determined for a pixel of the synthetic image being equal to the absolute value of the difference between the predicted depth and the target depth associated with that pixel.

[0099] As with the first learning, the second learning of the depth prediction model consists of adjusting the input parameters of the network of convolutional neurons implementing the depth prediction model in order to minimize the second loss error previously calculated.

[0100] Once the training of the depth model is complete, for example when a first and a second error are less than a threshold value defining an acceptable level of accuracy, it is then possible to use the depth prediction model to predict depths associated with pixels of images acquired by the camera 11.

[0101] Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object in the three-dimensional scene to which a pixel is associated, the determination of a depth of a pixel then corresponding to a measurement of a distance separating an object from the vehicle carrying the vision system.

[0102] If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another road user, the ADAS is then able to determine this distance precisely. For example, if the ADAS is designed to activate the braking system of the vehicle 10 in the event of a risk of collision with another road user, and the distance between the vehicle 10 and that road user decreases sharply, then the ADAS is able to detect this sudden closeness and activate the braking system of the vehicle 10 to avoid a possible accident.

[0103] Figure 3 schematically illustrates a device 3 configured to learn a depth prediction model associated with a vehicle-mounted vision system and / or to predict the depth associated with a pixel of an image acquired by a camera of the vehicle-mounted vision system, according to a particular and non-limiting embodiment of the present invention. The device 3 corresponds, for example, to a device embedded in the vehicle 10, for example, a computer associated with the monocular vision system.

[0104] Device 3 is, for example, configured to carry out the steps described opposite Figures 1 and 2. Examples of such a device 3 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of device 3, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 3 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0105] The device 3 comprises one (or more) processor(s) 30 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 3. The processor 30 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 3 further comprises at least one memory 31, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0106] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 31.

[0107] According to various particular and non-limiting embodiments, the device 3 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0108] According to a particular and non-limiting embodiment, the device 3 includes a block 32 of interface elements for communicating with external devices. The interface elements of the block 32 include one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HD MI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0109] According to another particular and non-limiting embodiment, the device 3 includes a communication interface 33 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 330. The communication interface 33 corresponds by example to a transmitter configured to transmit and receive information and / or data via communication channel 330. Communication interface 33 corresponds for example to a wired network of type CAN (from the English "Controller Area Network" or in French "Réseau de contrôlers"), CAN FD (from the English "Controller Area Network Flexible Data-Rate" or in French "Réseau de contrôlers à débit de données flexible"), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0110] According to a particular and non-limiting embodiment, the device 3 can provide output signals to one or more external devices, such as a display screen 340, touch or not, one or more speakers 350 and / or other peripherals 360 via the output interfaces 34, 35, 36 respectively. According to a variant, one or more of the external devices is integrated into the device 3.

[0111] Of course, the present invention is not limited to the embodiments described above but extends to a method for determining the depth of a pixel in an image acquired by a vision system, and / or for measuring the distance between an object and a vehicle equipped with a vision system, the depth and / or distance being predicted and / or measured via a depth prediction model learned according to the learning method described above, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0112] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 3 of [Fig.3].

Claims

1. Demands Method for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle (10), the vision system comprising a camera (11) arranged to acquire an image of a three-dimensional scene taking place near the vehicle (10), said method being implemented by at least one processor, and being characterized in that it comprises two learning phases, • a first learning phase comprising the following steps: - reception (21) of data representative of a first image and a second image acquired by the camera (11) at two distinct time moments of acquisition; - determination (22) of bounding boxes associated with objects in the three-dimensional scene in the first and second images, one type of object being associated with each bounding box, - prediction (23) of depths associated with pixels of the first image not included in a bounding box of which an associated object type corresponds to a target object type, called first pixels, and of depths associated with pixels of the second image not included in a bounding box of which an associated object type corresponds to said target object type, called second pixels, by said depth prediction model; - generation (24) of a third image from the first pixels, the depths associated with the first pixels and a movement of the vehicle (10) between the two time instants of acquisition and of a fourth image from the second pixels, the depths associated with the second pixels and said movement of the vehicle (10); - first learning (25) of the depth prediction model by minimizing a first loss error determined from a first intermediate error determined by comparing the third image to the second image and a second intermediate error determined by comparing the fourth image to the first image; and • a second learning phase comprising the following steps: - reception (26) of a synthetic image and target depths associated with pixels of the synthetic image representing an object corresponding to the type of target object, said synthetic image and said target depths being generated by a synthetic image generation model; - determination (27) of depths associated with said pixels of the synthetic image by the depth prediction model; - second learning (28) of the depth prediction model by minimizing a second loss error determined by comparing the predicted depths with the target depths associated with said pixels of the synthetic image.

2. A method according to claim 1, wherein the third image, respectively fourth image, is generated by the following function: px=P^c\ D(pt) )]) With: • Ps the coordinates of a pixel of the third image, respectively fourth image, •77 a function for going from homogeneous coordinates to pixel coordinates by removing one dimension of a vector, • K a direction prediction model associated with the camera (11), • T a matrix comprising the extrinsic parameters, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the first image, respectively second image, and the depth j associated with it.

3. A method according to claim 1 or 2, wherein said first and second intermediate errors comprise a photometric error determined by the following function: L,(p) = Er[ (1-a) • \l(p)-î(p) | +«• (l-iSSIM(j(p),î(p) ) ) ] With: • the photometric error associated with a pixelp defined by its coordinates in the first image, respectively second image, • l(p) a value of pixel p in the first image, respectively second image, * ï(p) a value of pixel p in the fourth image, respectively third image, • SSIM a function which takes into account a local structure, and • has a weighting factor depending in particular on a type of road environment.

4. A method according to any one of claims 1 to 3, wherein said first and second intermediate errors comprise a reconstruction error determined by the following function: L^ip) = With: • ^smooth^P) a reconstruction error for a pixelp of the third image, respectively fourth image, • a depth associated with pixel p, • VF a parameter matrix, • ° the order of a smoothing gradient, • an L1 norm of second-order depth gradients is calculated with VF = 1, et0 = 2, • x and dimensions of the third image, respectively fourth image, • / 1 a hyperparameter dependent on the road environment in which the vehicle travels (10), and • It{p) a value of pixelp in the third image, respectively fourth image.

5. A method according to any one of claims 1 to 4, wherein the first loss error is determined by the following function: L' = Y^iïx(L}(p),L2(p) ) With: • L' the first loss error, • L^p) the first intermediate error for a pixelp of the first image, and • L2(p) the second intermediate error for a pixelp of the second image corresponding to the pixelp of the first image.

6. A method according to any one of claims 1 to 5, wherein a first intermediate error, respectively second intermediate error, associated with a pixel of the first image, respectively second image, contained within a bounding box corresponding to a target object type is null.

7. A method according to any one of claims 1 to 6, wherein said type of target object belongs to a set of target objects comprising: - a car, - a truck, - a coach, - a motorcycle, and - a bicycle.

8. A computer program comprising instructions for carrying out the method according to any one of the preceding claims, when such instructions are executed by a processor.

9. Device (3) configured to learn a depth prediction model by a vision system embedded in a vehicle (10), said device (3) comprising a memory (31) associated with at least one processor (30) configured to implement the steps of the method according to any one of claims 1 to 7.

10. Vehicle (10) comprising the device (3) according to claim 9.

Images