Method and device for determining a calibration fault in a vision system on board a vehicle

Abstract

The invention relates to a method or a device for determining a calibration fault in a vision system on board a vehicle, the method comprising a step of receiving (31) a first error from a depth prediction model associated with the vision system, the first error being determined by a loss function during a phase of training the depth prediction model, a step of receiving (33) image data representative of a set of images acquired by the vision system, and a step of determining (34) a second error by applying the loss function to the image data. The method further comprises a step (35) of determining a conformity indicator on the basis of a result of comparing the second error with a threshold error determined on the basis of the first error, and a step (36) of updating a non-conformity rate on the basis of the conformity indicator. The method then comprises a step (37) of determining the calibration defect of the vision system on the basis of the non-conformity rate.

Description

DESCRIPTION Title: Method and device for determining a calibration fault in a vision system on board a vehicle Technical field

[0001] The present invention claims priority from French application 2400564 filed on 19.01.2024, the content of which (text, drawings and claims) is incorporated herein by reference. The present invention relates to methods and devices for determining a calibration fault in a vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for alerting a driver of a calibration fault in a vision system on board a vehicle. Technological background

[0002] Many modern vehicles are equipped with so-called ADAS (Advanced Driver Assistance System). ADAS are passive and active safety systems designed to eliminate human error in the operation of all types of vehicles. ADAS uses advanced technologies to assist the driver while driving and thus improve their performance. ADAS uses a combination of sensor technologies to perceive the environment around a vehicle, then provides information to the driver or influences certain vehicle systems.

[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control, and automatic parking systems.

[0004] ADAS systems embedded in a vehicle are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras are used to detect and locate other road users or possible obstacles around a vehicle in order to, for example: • to adapt the vehicle lighting according to the presence of other users; • automatically regulate the vehicle speed; • to act on the braking system in the event of a risk of impact with an object.

[0005] In addition, the quality of the data emitted by a vision system, i.e. the accuracy of the positioning of objects in the three-dimensional scene, has a direct impact on the proper functioning of the driver assistance systems using this data. It is therefore important to control the accuracy of a model for predicting the distance and position of an object in the scene, both following calibration of the vision system and throughout the entire lifespan of the vision system. Indeed, a fault in the vision system can occur during its lifespan, whether temporary or permanent. Summary of the present invention

[0006] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0007] Another object of the present invention is to ensure the quality of the data resulting from the processing of an image acquired by a vision system on board a vehicle.

[0008] Another object of the present invention is to improve road safety, in particular by improving the operational safety of ADAS systems powered by data obtained from a wide-angle camera.

[0009] According to a first aspect, the present invention relates to a method for determining a calibration fault of a vision system on board a vehicle, the vision system comprising a set of at least one camera, each camera of the set being arranged so as to acquire an image of a three-dimensional scene, the method being implemented by at least one processor, and being characterized in that it comprises the following steps: - receiving data representative of a non-conformity rate and a first error of a depth prediction model associated with the vision system, the first error being determined by a loss function during a learning phase of the depth prediction model; - receiving image data representative of a set of images acquired by the vision system; - determination of a second error by applying the loss function to the image data; - determination of a conformity indicator based on a result of a comparison of the second error with a threshold error, the threshold error being determined based on the first error; - updating the non-compliance rate based on the compliance indicator; - determination of a calibration fault in the vision system based on the result of a comparison of the non-conformity rate with a threshold value.

[0010] Such a method for determining a calibration defect of the vision system thus makes it possible to test the quality or accuracy of depths predicted by the depth prediction system at a current time in comparison with the quality or accuracy of depths predicted by the depth prediction system at the end of the learning phase. In the event of several recurring non-conformities, a calibration defect of the vision system is then determined or detected.

[0011] According to a variant of the method, the threshold error is proportional to the first error, a ratio between the threshold error and the first error being strictly greater than 1.

[0012] According to another variant, the method further comprises a step of receiving meteorological data representative of meteorological conditions of an environment of the vehicle, a test frequency of the vision system being determined as a function of the meteorological data.

[0013] According to another variant, the method further comprises a step of controlling the display of graphic content on a screen embedded in the vehicle, the graphic content comprising information representative of the calibration fault.

[0014] According to a further variant of the method, the vision system is a stereoscopic vision system comprising at least two cameras.

[0015] According to another variant of the method, the vision system is a monoscopic vision system comprising a single camera.

[0016] According to yet another variant of the method, a camera of the vision system is a wide-angle camera.

[0017] According to a second aspect, the present invention relates to a device for determining a calibration fault of a vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0018] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0019] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0020] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0021] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which a program is recorded computer system comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0022] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording medium or a hard disk.

[0023] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or terrestrial radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from a network such as the Internet.

[0024] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0025] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 4, in which:

[0026] [Fig. 1] schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

[0027] [Fig. 2] schematically illustrates a monoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

[0028] [Fig. 3] illustrates a flowchart of the different stages of a process for determining a calibration fault in an on-board vision system. vehicle of figure 1 or figure 2, according to a particular and non-limiting exemplary embodiment of the present invention;

[0029] [Fig. 4] schematically illustrates a device configured to determine a calibration fault of a vision system on board the vehicle of FIG. 1 or FIG. 2, according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0030] A method and a device for determining a calibration fault of a vision system on board a vehicle will now be described in the following with joint reference to figures 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.

[0031] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to identify and distinguish different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0032] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a calibration fault of a vision system on board a vehicle is for example implemented by a computer of the on-board system of the vehicle controlling this vision system.

[0033] Indeed, the method comprises receiving data representative of a first error of a depth prediction model associated with the vision system determined by a loss function during a phase of learning the depth prediction model, receiving image data representative of a set of images acquired by the vision system and determining a second error by applying the loss function to the image data.

[0034] A compliance indicator is determined based on a result of a comparison of the second error to a threshold error determined based on the first error and a non-compliance rate is updated based on the compliance indicator.

[0035] The vision system calibration defect is then determined based on the non-conformity rate.

[0036] Figure 1 schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0037] The first vehicle 10 is located in a first environment 1 corresponding, for example, to a road environment formed of a network of roads accessible to the first vehicle 10.

[0038] In this example, the first vehicle 10 corresponds to a vehicle with a thermal engine, with electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The first vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the first vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0039] The first vehicle 10 advantageously comprises at least two on-board cameras, a first camera 11 and a second camera 12, configured to acquire images of a three-dimensional scene taking place in the environment of the first vehicle 10 from distinct observation positions. The first camera 11 and the second camera 12 form a stereoscopic vision system when used together as illustrated in FIG. 1. The first camera 11 forms a monoscopic vision system when used alone, likewise the second camera 12 forms another monoscopic vision system. when used alone. The present invention, however, extends to any vision system comprising at least two cameras, for example 2, 3 or 5 cameras.

[0040] The intrinsic parameters of the first camera 11 characterize the transformation which associates, for an image point, subsequently called “point”, its three-dimensional coordinates in the frame of reference of the first camera 11 with the pixel coordinates in an image acquired by the first camera 11. These parameters do not change if the first camera 11 is moved. The intrinsic parameters of the first camera 11 include in particular a first focal length f1 associated with the first camera 11.

[0041] The intrinsic parameters of the second camera 12 characterize, for their part, the transformation which associates, for an image point, its three-dimensional coordinates in the frame of reference of the second camera 12 with the pixel coordinates in an image acquired by the second camera 12. These parameters do not change if the second camera 12 is moved. The intrinsic parameters of the second camera 12 include in particular a second focal length f2 associated with the second camera 12.

[0042] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account, that their correction is processed at the time of image acquisition or at the time of calibration.

[0043] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the first vehicle 10 or at the top of the windshield of the first vehicle 10, the second point of view is for example located on or in the right rearview mirror of the first vehicle 10 or at the top of the windshield of the first vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the first vehicle 10, the second camera 12 is located in the right rearview mirror of the first vehicle 10.

[0044] A first marker is associated with the first camera 11: - the direction of the x axis is defined horizontal and normal to the optical axis of the first camera 11. The distance B separating the optical center of the first camera 11 from the projection of the optical center of the second camera 12 on the horizontal plane passing through the optical center of the first camera 11 is called the reference base (in English “baseline”); - the direction of the y axis is defined vertical and normal to the optical axis of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.

[0045] The extrinsic parameters related to the position of the cameras 11, 12 are the following parameters: - three translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - three rotations in the x, y and z directions: 0x, 0y and 0z.

[0046] The extrinsic parameters are determined, for example, during a calibration phase of the stereoscopic vision system comprising the first camera 11 and the second camera 12.

[0047] A major constraint of the stereoscopic vision system used in automotive applications, for example, is the large distance between the two cameras. Indeed, to be able to cover a measuring range of 200 meters, the reference base must reach 60 cm for the cameras commonly used in this field.

[0048] The first and second cameras 11, 12 acquire images of a scene located in front of the first vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0049] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

[0050] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system comprising the second camera 12.

[0051] Field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system comprising the first camera 11.

[0052] Finally, field 19 is not visible to any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0053] According to a particular exemplary embodiment, the field of vision of the second camera 12 covers at least half of the field of vision of the first camera 11.

[0054] It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the first vehicle 10 by equipping it with differently placed and oriented cameras.

[0055] The images acquired by the first and second cameras 11, 12 at an acquisition time instant are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric coordinates (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

[0056] Each pixel of an acquired image is representative of an object of the three-dimensional scene present in the field of vision of the first or second camera 11, 12. Indeed, a pixel of an acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When the light strikes this object, photons are emitted or reflected, captured by a photosensitive sensor of the first or second camera 11, 12 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thereby capturing visual details. The combination of millions of pixels creates an image faithfully representing the physical object observed by the first or second camera 11, 12. An image point previously presented is thus a point on a surface of an object in the three-dimensional scene observed by the stereoscopic vision system comprising the first and second cameras 11, 12.

[0057] The images acquired by the first and second cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0058] According to a particular embodiment, the first camera 11 is of the “wide angle” type, a wide angle camera being for example equipped with a lens designed to acquire an image representative of a three-dimensional scene perceived according to a field of vision wider than that of a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens allows a larger portion of the three-dimensional scene unfolding in front of or around the camera to be captured, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle a of the field of vision of the first camera 11 is, for example, equal to 120°, 145°, 180° or 360°, whereas a standard camera offers, for example, an open field of vision at an angle of 45° or less. Such a first camera 11 corresponds, for example, to a camera equipped with mirrors or even to a “fisheye” camera (in French “fish eye”). Wide-angle lenses have a shorter focal length compared to standard lenses, which makes them suitable for acquiring images of landscapes, architecture, road intersections or any other subject requiring an extended perspective.Wide-angle cameras, for example, are used to capture immersive and dynamic images with extended depth of field.

[0059] According to a particular exemplary embodiment, the image acquired by the first camera 11 comprises a distortion equal to 0.5%, 0.8% or greater than 1%. The measurement of such a distortion corresponds to the determination of a ratio between: - the maximum spacing of a pixel of the image from a straight line of the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the edge of the image, or at the corners of the edge of the image, and - the length of this edge.

[0060] Commonly, distortion is considered, in the world of photography, as: • negligible if it is less than 0.3%, • not very sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • annoying if it is greater than or equal to 1% or more.

[0061] A barrel distortion is characterized by a positive percentage, while a crescent distortion is characterized by a negative percentage.

[0062] The images acquired by the first and second cameras 11, 12 are, for example, sent to a computer of a device equipping the first vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the first vehicle 10.

[0063] Depths associated with pixels of the images acquired by the stereoscopic vision system are for example determined by a first depth prediction model, such a depth prediction model being, for example, implemented by a neural network. The first depth prediction model, to be reliable and accurate, is in particular learned in a first learning phase by minimizing an error determined by a first loss function. Such a first learning phase is known to those skilled in the art. Different methods are known and applicable depending on the types of the first and second cameras 11, 12 and the extrinsic parameters of the stereoscopic vision system, for example the method described in the document: “UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos” by Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang and Wei Xu published in June 2019.This method implements different operations, an operation making it possible in particular to determine a photometric error by comparing images generated from images acquired by the stereoscopic vision system and predicted depths for pixels of these images to images acquired by this same stereoscopic vision system, the function making it possible to determine this photometric error being called loss function, and an operation of minimizing this photometric error by adjusting parameters of the depth prediction model. At the end of learning, a residual error deemed acceptable remains, this error then corresponds to the first error received during a method of determining a calibration fault of the vision system on board the first vehicle 10.

[0064] A method for determining a calibration fault in the stereoscopic vision system on board the first vehicle 10 is advantageously implemented by the first vehicle 10, i.e. by a processor, a computer or a combination of computers of the onboard system of the first vehicle 10, for example by the computer(s) in charge of the stereoscopic vision system of the first vehicle 10 or by the device 4 of FIG. 4.

[0065] Figure 2 schematically illustrates a monoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0066] The second vehicle 20 is located in a second environment 2 corresponding, for example, to a road environment formed of a network of roads accessible to the vehicle 20.

[0067] In this example, the second vehicle 20 corresponds to a vehicle with a thermal engine, with electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The second vehicle 20 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the second vehicle 20 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0068] The second vehicle 20 advantageously comprises at least one third on-board camera 21 configured to acquire images of a three-dimensional scene taking place in the environment of the second vehicle 20 from a determined observation position. The third camera 21 forms a monoscopic vision system.

[0069] The intrinsic parameters of the third camera 21 characterize the transformation which associates, for an image point, subsequently called “point”, its three-dimensional coordinates in the frame of reference of the third camera 21 with the pixel coordinates in an image acquired by the third camera 21. These parameters do not change if the third camera 21 is moved. The intrinsic parameters of the third camera 21 include in particular a third focal length f3 associated with the third camera 21.

[0070] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the third camera 21 is assumed to be perfect, that is to say that the distortions are not taken into account, that their correction is processed at the time of image acquisition or at the time of calibration.

[0071] This third camera 21 is arranged so as to acquire an image of a three-dimensional scene from a determined point of view, for example located on or in the left rearview mirror of the second vehicle 20 or at the top of the windshield of the second vehicle 20. In this example, the third camera 21 is located at the top of the windshield of the second vehicle 20.

[0072] The third camera 21 acquires images of a scene located in front of the second vehicle 20, the third camera 21 covering an acquisition field 22 in which an object 23 is for example positioned. The presence of the object 23 defines an occlusion field 24 for the monoscopic vision system comprising the third camera 21.

[0073] It is obvious that it is possible to use such a monoscopic vision system to take images of scenes located on the sides or behind the second vehicle 20 by equipping it with differently placed and oriented cameras.

[0074] The images acquired by the third camera 21 at an acquisition time instant are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric coordinates (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

[0075] Each pixel of the image acquired by the third camera 21 is representative of an object in the three-dimensional scene present in the field of vision of the third camera 21 . Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When the light strikes this object, photons are emitted or reflected, captured by a photosensitive sensor of the third camera 21 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image faithfully representing the physical object observed by the third camera 21 .An image point is thus a point on a surface of an object in the three-dimensional scene observed by the third camera 21.

[0076] The images acquired by the third camera 21 represent views of the same scene, for example acquired at different acquisition time instants. When the second vehicle 20 is in motion, then these images are acquired by the third camera 21 from different observation positions or viewpoints. On this three-dimensional scene are for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0077] According to a particular embodiment, the third camera 21 is of the “wide angle” type. The angle P of the field of vision of the third camera 21 is for example equal to 120°, 145°, 180° or 360°. Such a third camera 21 corresponds by example to a camera equipped with mirrors or even to a “fisheye” camera (in French “fish eye”).

[0078] According to a particular exemplary embodiment, an image acquired by the third camera 21 comprises a distortion equal to 0.5%, 0.8% or greater than 1%.

[0079] The images acquired by the third camera 21 are, for example, sent to a computer of a device equipping the second vehicle 20 or stored in a memory of a device accessible to a computer of a device equipping the second vehicle 20.

[0080] A method for determining a calibration fault in the stereoscopic vision system on board the second vehicle 20 is advantageously implemented by the second vehicle 20, i.e. by a processor, a computer or a combination of computers of the onboard system of the second vehicle 20, for example by the computer(s) in charge of the monoscopic vision system of the second vehicle 20 or by the device 4 of FIG. 4.

[0081] Depths associated with pixels of the images acquired by the monoscopic vision system are for example determined by a second depth prediction model, such a depth prediction model being, for example, implemented by a neural network. The second depth prediction model, to be reliable and accurate, is in particular learned in a second learning phase by minimizing an error determined by a second loss function. Such a second learning phase is known to those skilled in the art.Different methods are known and applicable depending on the type of the third camera 21 , for example the method described in the document: “Digging Into Self-Supervised Monocular Depth Estimation” by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019, or, for wide-angle cameras, “Neural Ray Surfaces for Self-Supervised Learning of Depth and Egomotion” by Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich and Adrien Gaidon published in August 2020.

[0082] Generally speaking, any self-supervised or self-learned vision system, i.e. one whose depth prediction model is learned from images acquired by the vision system itself and without using annotated learning data, comprises a loss function making it possible to evaluate the performance of the depth prediction model and to improve its accuracy by an iterative method implemented during a learning phase and aimed at minimizing an error determined by the loss function. At the end of the learning phase, a residual error corresponding to a minimum error obtained during the learning phase remains, this residual error then corresponding to the first error received during a method for determining a calibration fault of the vision system on board the first vehicle 10.

[0083] Optionally, the learning phases are implemented after a phase of calibration of the cameras of the vision system. The calibration of a camera of a vision system is also known to those skilled in the art, and is for example described in the following document “A Flexible New Technique for Camera Calibration” by Zhengyou Zhang published in December 1998, or in this document: “Single View Point Omnidirectional Camera Calibration from Planar Grids”, by Christopher Mei and Patrick Rives published in April 2007.

[0084] Figure 3 illustrates a flowchart of the different steps of a method for determining a calibration fault of a vision system on board a vehicle, for example in the first vehicle 10 of Figure 1 or in the second vehicle 20 of Figure 2, according to a particular and non-limiting exemplary embodiment of the present invention.

[0085] In the description associated with Figure 3, the term “vehicle 10, 20” refers indifferently to the first vehicle 10 of Figure 1 or to the second vehicle 20 of Figure 2, the method 3 for determining a calibration fault of a vision system being applicable both to a stereoscopic vision system and to a monoscopic vision system. Similarly, the term “phase learning phase” refers to the first learning phase when “the vehicle” refers to the first vehicle 10 while it refers to the second learning phase when “the vehicle” refers to the second vehicle 20.

[0086] The method 3 is for example implemented by the device on board the vehicle 10, 20 implementing the method for determining a depth by on-board vision system or by the device 4 of FIG. 4.

[0087] In a step 31, data representative of a non-conformity rate and a first error of a depth prediction model associated with the vision system are received. The first error is determined by a loss function during a training phase of the depth prediction model as described with reference to FIGS. 1 or 2.

[0088] According to a particular exemplary embodiment, the non-conformity rate is for example equal to 0 at the start of the process. When the following steps, i.e. from 33 to 37, are repeated or iterative, then the non-conformity rate is the rate previously determined, for example from the last iterations. More details on this calculation are provided below in the description of step 36.

[0089] In a step 33, image data representative of a set of images acquired by the vision system are received. The number of images depends in particular on the type of vision system.

[0090] In a step 34, a second error is determined by applying the loss function to the image data. The loss function is the same loss function as that used during the training phase of the depth prediction model. Thus, the second error corresponds to a common error, determined under the current conditions of use of the vision system.

[0091] In a step 35, a conformity indicator is determined based on a result of a comparison of the second error with a threshold error, the threshold error being determined based on the first error. Thus, the first error corresponds to the minimum error obtained at the end of the learning phase of the prediction model of depth. This is then ideal and corresponds to a perfectly calibrated vision system.

[0092] According to a particular embodiment, the threshold error is proportional to the first error, thus the threshold error is equal to 1.5x, 2x or 3x the first error. Thus, the ratio between the threshold error and the first error is strictly greater than 1. Indeed, if the ratio were 1 or less then the conformity indicator would represent a non-conformity even when the vision system implements a just learned depth prediction model.

[0093] The compliance indicator is, according to a first particular embodiment, equal to the ratio between the second error and the threshold error.

[0094] According to a second particular embodiment, the conformity indicator is equal to 1 when the second error is greater than the threshold error and equal to 0 otherwise.

[0095] In a step 36, the non-conformity rate is updated based on the conformity indicator.

[0096] For example, the non-compliance rate is equal to the average of the last N compliance indicators including the compliance indicator determined in step 35 when this method is implemented iteratively, the number N being for example defined by a user or a manufacturer. For example, the number N is equal to 10 or 100. Thus, the non-compliance rate is representative of a number of second errors last determined greater than the threshold error. It should be noted that in this example, the last N compliance indicators are for example recorded or stored in the memory of the device implementing the method.

[0097] According to another particular embodiment, the non-conformity rate is equal to the average or a weighted average of the non-conformity rate received and the conformity indicator, for example determined from the following function: T' = 0.9 x T + 0.1 x le, with: • T' the updated non-conformity rate, • T the rate of non-conformities received at step 31, and • the compliance indicator determined in step 35.

[0098] According to the second particular embodiment, the = E2 / Es, with: • the conformity indicator determined in step 35, • E2 the second error, and • Is the threshold error.

[0099] In a step 37, a calibration fault of the vision system is determined based on a result of a comparison of the non-conformity rate with a threshold value.

[0100] The threshold value is for example defined in such a way as to determine a calibration fault of the vision system when the rate of non-conformities is representative of a number of conformity indicators greater than a threshold. Indeed, the aim is not to determine a non-conformity fault from the first non-conformity determined but when several non-conformities are determined over a certain number N of tests for example.

[0101] According to a particular exemplary embodiment, the method 3 for determining a calibration fault of the vision system further comprises a step 32 of receiving meteorological data representative of meteorological conditions of an environment 1, 2 of the vehicle 10, 20, a test frequency of the vision system being determined as a function of the meteorological data. Indeed, a calibration fault occurs for example when a lens or an objective of a camera is covered with drops of transparent water, which happens more frequently in rainy weather for example. Thus, the meteorological data are for example obtained by analyzing an image acquired by the vision system, in which it is possible to perceive that the environment in which the vehicle 10, 20 is traveling is humid, or received from an on-board system of the vehicle 10, 20 when windshield wipers are for example in operation.

[0102] According to another particular exemplary embodiment, the method 3 for determining a calibration fault of the vision system further comprises a step of controlling the display of graphic content on a screen on board the vehicle 10, 20, the graphic content comprising information representative of the fault of calibration when this is determined. Such a screen is for example part of an infotainment system on board the vehicle 10, 20 and is located in the passenger compartment of the vehicle 10, 20, it is for example arranged on a dashboard or a central console. This screen is then linked to the device implementing the method 3 for determining a calibration fault of the vision system, for example via a wired connection of the communication bus type.

[0103] According to yet another particular exemplary embodiment, data are transmitted to a driver assistance system called ADAS (from the English “Advanced Driver-Assistance System” or in French “Système d'aide à la conduite supérieur”) receiving as input data the depths predicted by the vision system on board the vehicle 10, 20. This data is for example representative of the calibration fault when it is determined. The ADAS then switches, for example, to a degraded or safety mode, in order to continue to operate with non-guaranteed input data, or is even deactivated. In a manner similar to the previous example, the ADAS is linked to the device implementing the method 3 for determining a calibration fault of the vision system, for example via a wired connection of the communication bus type.

[0104] Such a method for determining a calibration fault of the vision system thus makes it possible to test the quality or accuracy of depths predicted by the depth prediction system at a current time in comparison with the quality or accuracy of depths predicted by the depth prediction system at the end of the learning phase. In the event of several recurring non-conformities, a calibration fault of the vision system is then determined or detected, making it possible, for example, to alert the driver of the vehicle carrying the vision system so that he can take into consideration a possible failure of the vision system. Similarly, an ADAS powered by depths is, for example, switched to a degraded operating mode in order to maintain an adequate level of safety. A user of the vehicle 10, 20 is then able to plan a maintenance operation, a calibration or simply a cleaning of the vision system. embedded in the vehicle 10, 20 in order to find an operational vision system as it was at the end of the learning phase.

[0105] Figure 4 schematically illustrates a device 4 configured for determining a calibration fault of a vision system on board a vehicle, for example in the first vehicle 10 of Figure 1 or in the second vehicle 20 of Figure 2, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device on board the first vehicle 10 or in the second vehicle 20, for example a computer.

[0106] The device 4 is for example configured for the implementation of the operations described with regard to figures 1 or 2 and / or steps described with regard to figure 3. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0107] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0108] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored in the memory 41.

[0109] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0110] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HDMI interface (from the English "High Definition Multimedia Interface" or "High Definition Multimedia Interface" in French); - LIN interface (from the English “Local Interconnect Network”).

[0111] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN type (from the English “Controller Area Network” or in French “Network of controllers”), CAN FD (from the English “Controller Area Network Flexible Data-Rate” or in French “Flexible data rate controller network”), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0112] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

[0113] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for detecting a calibration fault in a vision system embedded in a vehicle, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0114] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of FIG. 4.

Claims

CLAIMS 1. Method for determining a calibration fault of a vision system on board a vehicle (10, 20), the vision system comprising a set of at least one camera, each camera of said set being arranged so as to acquire an image of a three-dimensional scene, said method being implemented by at least one processor, and being characterized in that it comprises the following steps: - reception (31) of data representative of a non-conformity rate and of a first error of a depth prediction model associated with the vision system, the first error being determined by a loss function during a learning phase of the depth prediction model; - reception (33) of image data representative of a set of images acquired by the vision system; - determining (34) a second error by applying the loss function to the image data; - determination (35) of a conformity indicator as a function of a result of a comparison of the second error with a threshold error, the threshold error being determined as a function of the first error; - update (36) of the non-conformity rate based on the conformity indicator; - determination (37) of a calibration fault of the vision system based on a result of a comparison of the non-conformity rate with a threshold value.

2. Method according to claim 1, for which the threshold error is proportional to the first error, a ratio between the threshold error and the first error being strictly greater than 1.

3. Method according to claim 1 or 2, further comprising a step of receiving (32) meteorological data representative of meteorological conditions of an environment (1, 2) of the vehicle (10, 20), a test frequency of the vision system being determined as a function of the meteorological data.

4. Method according to one of claims 1 to 3, further comprising a step of controlling the display of graphic content on a screen on board the vehicle (10, 20), said graphic content comprising information representative of the calibration fault.

5. Method according to one of claims 1 to 4, for which the vision system is a stereoscopic vision system comprising at least two cameras.

6. Method according to one of claims 1 to 4, for which the vision system is a monoscopic vision system comprising a single camera.

7. Method according to one of claims 1 to 6, for which a camera of the vision system is a wide-angle camera.

8. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor.

9. Device (4) for detecting a calibration fault in a vision system on board a vehicle (10, 20), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 7.

10. Vehicle (10, 20) comprising the device (4) according to claim 9.

Images