Method and device for detecting a calibration fault in a non-parallel stereoscopic vision system.

Abstract

The invention relates to a method for detecting a calibration defect in a stereoscopic vision system installed in a vehicle (10). The stereoscopic vision system comprises an array of at least two cameras (11, 12) arranged so that each acquires an image of a scene from a different viewpoint, the optical axes representing an orientation of the field of view of each camera being oriented non-parallel. The method includes a step of determining a first representative value of a calibration defect in the stereoscopic vision system based on an average of ratios between a distance of a component of an optical flux calculated for each pixel of a first image along a determined direction and a norm of the optical flux calculated for each pixel. Figure 1

Description

Title of the invention: Method and device for detecting a calibration defect in a non-parallel stereoscopic vision system. Technical field

[0001] The present invention relates to methods and devices for detecting a calibration defect in a non-parallel stereo vision system on board a vehicle, for example in a motor vehicle. Technological background

[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0004] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the lighting of the vehicle according to the presence of other users; - to automatically regulate the speed of the vehicle; - to act on the braking system in the event of a risk of impact with an object.

[0005] The reliability of the data from these cameras and the proper functioning of the driving assistance devices using this data therefore depend on the correct calibration of the cameras.

[0006] Summary of the present invention

[0007] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0008] Another object of the present invention is to improve the reliability of the data from these cameras.

[0009] Another object of the present invention is to improve road safety.

[0010] According to a first aspect, the present invention relates to a method for detecting a calibration fault in a stereoscopic vision system on board a vehicle, the stereoscopic vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view, the optical axes representative of an orientation of the field of vision of each camera being oriented in a non-parallel manner, the method being characterized in that it comprises the following steps: - reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the set of cameras at the same first acquisition time instant tl; - detection of at least one object in the scene by analysis of the first image, a set of first pixels of the first image being associated with at least one object; - determination of a set of second pixels in the second image corresponding to the set of first pixels, by implementing an optical flow calculation method as a function of the set of first pixels, an optical flow being representative of a displacement vector between each first pixel of the set of first pixels and the second pixel corresponding to each first pixel; - determination of a first value representative of a calibration defect of the stereoscopic vision system as a function of an average of ratios between a distance of a component of the optical flow calculated for each pixel in a determined direction and a standard of the optical flow calculated for each pixel.

[0011] Determining the first value representative of a calibration fault in the stereoscopic vision system makes it possible to detect poor calibration of the stereoscopic vision system or a change in extrinsic parameters of this system.

[0012] Subsequently, the comparison of this first value representative of a calibration fault of the stereoscopic vision system with a first threshold value makes it possible to alert the user to the fact that the reliability of the system is no longer assured and that a new calibration of this system is necessary.

[0013] According to a method variant, the determination of a first value representative of a calibration defect of the stereoscopic vision system is obtained by the following formula: t where: - 'Ln' is said value representing the calibration defect, - 'N' is a number of pixels in said first image with each pixel listed by its coordinates (i,j), - 'F; / is the optical flow corresponding to a pixel with coordinates (i,j) in the first image, - 'ef is a unit vector defining the determined direction.

[0014] According to another variant, the method further comprises a step of rectifying the first and second images as a function of extrinsic parameters of the stereoscopic vision system, the extrinsic parameters comprising a parameter representative of a rotation and a parameter representative of a translation in three dimensions between the first camera and the second camera, so as to make the epipolar lines horizontal in the first and second rectified images.

[0015] Image rectification thus makes analysis easier, rectification simplifies the matching of pixels in stereo images. In fact, the pixel corresponding in the second image to a pixel in the first image (and vice versa) is positioned on the same horizontal line.

[0016] According to another variant, the method further comprises a step of determining a first visibility mask associated with the first image and representative of a set of pixels of the first image having at least one corresponding pixel in the second image, the first visibility mask being determined by the optical flow calculation method, the first value representative of a calibration defect being further determined as a function of the first visibility mask.

[0017] The use of the first visibility mask for determining a first value representative of a calibration defect of the stereoscopic vision system thus makes it possible to exclude aberrant optical flow values.

[0018] According to another variant, the method further comprises the following steps: - calculation of a first depth associated with at least one object from coordinates of the sets of first and second pixels, focal lengths of the first and second cameras and a distance between the first camera and the second camera; - reception of third data representative of a third image acquired by the first camera at a second time instant t2 prior to the first time instant t1 of acquisition of the first image; - determination of a set of third pixels in the third image corresponding to the set of first pixels determined by the optical flow calculation method; - calculation of a second depth associated with at least one object from coordinates of the sets of first and third pixels, the focal length of the first camera, of the translation and of the rotation of the first camera between the first time instant tl of acquisition and the second time instant of acquisition; - determination of a second value representative of a calibration defect of the stereoscopic vision system by comparison of the first and second depths of the first pixels calculated.

[0019] Determining the second value representative of a calibration fault in the stereoscopic vision system makes it possible to detect poor calibration of the stereoscopic vision system or a change in extrinsic parameters of this system.

[0020] Subsequently, the comparison of this second value representative of a calibration fault of the stereoscopic vision system with a second threshold value makes it possible to alert the user to the fact that the reliability of the system is no longer assured and that a new calibration of this system is necessary.

[0021] According to an additional variant, the determination of a second value representative of a calibration defect of the stereoscopic vision system is obtained by the following formula: T--LV IrV, where: ~ N ~ - 'Ld' is the second value representing a calibration fault; - 'N' is a number of pixels in the first image with each pixel listed by its coordinates (i,j); - is the first depth calculated for a pixel with coordinates (i,j) associated with the at least one object in the first image; and - is the second depth calculated for a pixel with coordinates (i,j) associated with at least one object in the first image.

[0022] According to another variant, the method further comprises a step of determining a second visibility mask associated with the first image and representative of a set of pixels of the first image having at least one corresponding pixel in the third image, the second visibility mask being determined by the optical flow calculation method, the second value representative of a calibration defect being further determined as a function of the second visibility mask.

[0023] The use of the second visibility mask for determining a second value representative of a calibration defect of the stereoscopic vision system thus makes it possible to exclude aberrant optical flow values.

[0024] According to another variant, the method further comprises a step of determining a mask of dynamic objects associated with the first image and representative of a set of pixels from the first image associated with at least one moving object in the scene, the second value representative of a calibration defect being further determined as a function of the dynamic object mask.

[0025] The use of the dynamic object mask for determining a second value representative of a calibration defect of the stereoscopic vision system thus makes it possible to exclude optical flow values ​​distorted by the inherent movement of an object in the scene.

[0026] According to a second aspect, the present invention relates to a device for detecting a calibration fault in a stereoscopic vision system on board a vehicle, the device comprising a memory associated with a processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0027] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0028] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0029] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0030] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0031] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.

[0032] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0033] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0034] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 4, in which:

[0035] [Fig-1] schematically illustrates a non-parallel stereoscopic vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention;

[0036] [Fig.2] schematically illustrates a device configured for the detection of a calibration fault of a stereoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0037] [Fig.3] illustrates a flowchart of the different operations of a process for detecting a calibration defect of a non-parallel stereoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0038] [Fig.4] illustrates a flowchart of the different steps of a method for detecting a calibration fault in a non-parallel stereoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention.

[0039] Description of the examples of embodiment

[0040] A method and a device for detecting a calibration fault in a stereoscopic vision system on board a vehicle will now be described in the following with joint reference to FIGS. 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.

[0041] According to a particular and non-limiting example of embodiment of the present invention, a method for detecting a calibration fault in a stereoscopic vision system on board a vehicle, called a stereo system, is for example implemented by a computer of the on-board system of the vehicle controlling this stereo system.

[0042] The stereo system comprises a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view, the optical axes representing an orientation of the field of vision of each camera being oriented in a non-parallel manner. The stereoscopic vision system is thus said to be non-parallel.

[0043] For this purpose, the method comprises receiving first and second data respectively representative of a first and second image acquired by a first and second camera respectively at the same acquisition time instant t1, detecting at least one object in the scene by analyzing the first image, a set of first pixels of the first image being associated with at least one object, determining a set of second pixels in the second image corresponding to the set of first pixels, by implementing an optical flow calculation method as a function of the set of first pixels,an optical flow being representative of a displacement vector between each first pixel of the set of first pixels and the second pixel corresponding to each first pixel and the determination of a first value representative of a calibration defect of the stereo system as a function of an average of ratios between a distance of a component of the optical flow calculated for each pixel in a determined direction and a standard of the optical flow calculated for each pixel.

[0044] [Fig. 1] schematically illustrates a non-parallel stereo system equipping a vehicle 10 moving in an environment 1, according to a particular and non-limiting exemplary embodiment of the present invention;

[0045] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0046] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0047] It should be noted that the vehicle 10 is either moving or stationary in the environment 1.

[0048] The vehicle 10 advantageously comprises several on-board cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms the stereoscopic vision system. Two cameras 11 and 12 are illustrated in [Fig.l]. The invention is however not limited to a stereoscopic vision system comprising two cameras but extends to any stereoscopic vision system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras.

[0049] The two cameras 11, 12 have known intrinsic parameters. These parameters consist in particular of: - the focal length fl of the first camera 11; - the focal length f2 of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the direction C2 of the optical axis of the second camera 12; and - the respective resolutions of the cameras 11, 12.

[0050] The intrinsic parameters characterize the transformation which associates, for an image point, the camera coordinates with the pixel coordinates, in each camera. These parameters do not change if the camera is moved.

[0051] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account or that their correction is processed at the time of image acquisition.

[0052] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.

[0053] A first marker is associated with the first camera 11: - the direction of the y axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the reference base (in English “baseline”) and the direction separating the two cameras 11, 12 is the y axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.

[0054] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - 3 translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.

[0055] Determining the extrinsic parameters constitutes the problem of calibrating a stereoscopic vision system.

[0056] A main constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the "baseline" must reach 60cm for the cameras commonly used in this field.

[0057] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera covering only a first acquisition field 13, the second camera covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic view of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic view of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic view of the scene by the stereo system composed of the two cameras 11, 12.

[0058] The directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.

[0059] It is obvious that it is possible to use such a stereo system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with differently placed and oriented cameras.

[0060] The images acquired by the cameras 11, 12 at a given first time instant tl are presented in the form of data representing pixels characterized by: - ​​coordinates in each image; and - data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric coordinates (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

[0061] The images acquired by the cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0062] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0063] [Fig. 2] schematically illustrates a device 4 configured for detecting a calibration fault in a stereoscopic vision system on board a vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds, for example, to a device on board the first vehicle 10, for example a computer.

[0064] The device 4 is for example configured for the implementation of the operations and / or steps described with regard to figures 1, 3 and 4. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0065] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0066] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the 4L memory.

[0067] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0068] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HD MI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0069] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0070] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 (projection system) via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

[0071] [Fig. 3] illustrates a flowchart of the different operations of a process for detecting a calibration fault in a non-parallel stereo system on board the vehicle of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention.

[0072] The process is for example implemented by one or more processors of one or more computers on board the vehicle 10, for example by a computer controlling the stereo system.

[0073] In a first operation 21, the computer receives first data representative of a first image acquired by a first camera 11 of the set of cameras at a first acquisition time instant t1.

[0074] In a second operation 22, the computer receives second data representative of a second image acquired by a second camera 12 of the set of cameras at the same first acquisition time instant t1.

[0075] The two images received correspond to two views of the same scene taking place around the vehicle 10 at the same first given time instant tl.

[0076] In order to facilitate the analysis of the two received images, these images are rectified in a third operation 26, according to a method known to those skilled in the art. Such a method is described, for example, in “Projective Rectification of Non-Calibrated Infrared Stereo Images with Global Consideration of Distortion Minimization” by Benoit Ducarouge, Thierry Sentenac, Florian Bugarin and Michel Devy of July 16, 2009.

[0077] The rectification method consists of reorienting the epipolar lines so that they are parallel with the horizontal axis of the image. This method is described by a transformation which projects the epipoles to infinity and whose corresponding points are necessarily on the same ordinate.

[0078] A rectification algorithm consists, for example, of 4 steps: - Rotate (virtually) the first camera 11 so that the epipole goes to infinity along the horizontal axis of the reference frame associated with it; - Apply the same rotation to the second camera to end up in the initial geometric configuration; - Rotate the second camera by the rotation associated with the rotation matrix 'R', corresponding to the extrinsic parameter of the starting stereo system; - Adjust the scale in both camera markers.

[0079] It should be noted that this operation 26 is optional, in fact the rectification simplifies the matching of the pixels of the stereo images. The pixel corresponding in the second image to a pixel of the first image (and vice versa) is positioned on the same line. From the knowledge of the epipolar geometry and therefore of a fundamental matrix of the stereo system, the objective is then to determine a pair of projective transformations, called homographies, which reorient the epipolar projections parallel to the lines of the images, therefore to the horizontal axis of the rectified cameras.

[0080] The following operation 23 consists of detecting at least one object in the scene by analyzing the first image, a set of first pixels of the first image being associated with at least one object. This operation amounts to extracting the characteristic elements (in English “feature extraction”) of the first image.

[0081] A method for extracting features is known to those skilled in the art, such as the SIFT method (from the English “Scale Invariant Feature Transform”) or the “Corner Detection” method.

[0082] The following operation consists of determining 24 a set of second pixels in the second image corresponding to the set of first pixels, this operation is called “stereo matching” (in English “feature matching”).

[0083] Stereo matching or disparity estimation is the process of finding pixels in stereoscopic views that correspond to the same 3D point in the scene. Rectified epipolar geometry simplifies this process of finding correspondences on the same epipolar line. There is no need to calculate the coordinates of the 3D point to find the corresponding pixel on the same line in the other image. Disparity is the distance d between a pixel and its horizontal correspondence in the other image.

[0084] The stereo matching operation 24 is performed by implementing an optical flow calculation method based on the set of first pixels. Such a method is notably described in “UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos” by Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang and Wei Xu from June 2019.

[0085] The optical flow calculation method is carried out, for example, by a convolutional neural network (CNN). This type of tool is commonly used in image processing.

[0086] The output data of this operation 24 is an optical flow representative of a displacement vector between each first pixel of the set of first pixels of the first image and the second pixel corresponding to each first pixel in the second image.

[0087] This optical flow is used in the operation 25 of determining a first value representative of a calibration defect of the stereoscopic vision system.

[0088] This first value is defined as a function of an average of ratios between a distance of a component of the optical flow calculated for each pixel along a determined direction and a norm of the optical flow calculated for each pixel. It is obtained by the following formula: [Math.l] t _ jl y

[0089] with: - 'Ln' is the first value representative of a calibration fault; - 'N' is a number of pixels in the first image with each pixel listed by its coordinates (i,j); - 'F; / is the optical flow vector corresponding to a pixel with coordinates (i,j) in the first image; and - 'ef is a unit vector defining a specific direction: that of the y axis brought back into the first image.

[0090] It should be noted that in the case where the images have been rectified, the unit vector 'e / is the horizontal unit vector and has coordinates (1,0).

[0091] It is possible that a pixel of the first image does not find a corresponding pixel in the second image. This phenomenon is explained by the fact that areas of the first image may be occluded in the second image. Indeed, the difference in point of view of the two cameras 11, 12 does not allow the two cameras 11, 12 to see all the elements of the scene. An object present in the scene may mask a second object of the scene, the second object being visible from the point of view of the first camera 11 but being masked by the first object from the point of view of the second camera 12.

[0092] An operation 27 for determining the occluded areas of the first image consists of determining a first visibility mask associated with the first image and representative of a set of pixels of the first image having at least one corresponding pixel in the second image, the first visibility mask being determined by the optical flow calculation method previously described.

[0093] The visibility mask is, for example, defined by the following formula: [Math.2]

[0094] With: - is a function that gives 1 or 0 depending on the logical operator; - the Ps index indicates the “source” pixel which is in the second image; - the Pt index indicates the “target” pixel which is in the first image; - Fs^t is the optical flow; - E is a summation operator.

[0095] Determining the visibility mask makes it possible not to take into account possible aberrant values ​​of optical flow for pixels of the first image which do not have a correspondence in the second image. By using the visibility mask in the operation 25 of determining a first value representative of a calibration defect of the stereoscopic vision system, the formula then becomes:

[0096] [Math.3]

[0097] Thus, the determination 25 of a first value representative of a calibration fault of the stereoscopic vision system is made more reliable.

[0098] Depending on the resolution of the rectified images, it is possible to take into consideration only a part of the pixels around the average optical flow value F^t for the determination of the calibration error. This discrimination makes it possible to avoid taking into account aberrant values ​​which may result from a defect in the convolutional neural network. For example, one could calculate a mean M and a standard deviation o of the norms of the optical flows obtained for all the pixels of the first image and limit oneself to the pixels whose norm of the optical flow is such that:

[0099] Ml,96*o< [Math.4] Ir mII <M+1,96* o.

[0100] The present invention therefore consists in using the optical flow F to verify the calibration of a stereo system, in particular to verify the extrinsic parameters which are Ry and Rz corresponding respectively to the rotations around the y and z axes and Tx and Tz corresponding respectively to the translations along the x and z axes.

[0101] In the case where the first and second images are rectified, the optical flow F^ is normally horizontal. The calibration error resulting from the rotation of the pixels is described by the average deviation of the calculated horizontal line.

[0102] This definition is more easily interpretable than the commonly used epipolar constraint. For a certain angular deviation, the effect of noise on the projections of objects of different distances is similar because the disparity and the projection of the following optical flow ei are both proportional to the distance separating the first camera 11 from the detected object.

[0103] It is thus easier to define a threshold value according to the performance of the convolutional neural network and the quality of the stereo system from which it is appropriate to recalibrate the stereo system.

[0104] The determination 25 of a first value representative of a calibration defect of the stereoscopic vision system makes it possible to verify four of the six extrinsic parameters of the stereo system. In order to determine a possible calibration defect of the other two extrinsic parameters of the stereo system, Ty and Rx, the process comprises the following operations.

[0105] In an operation 28, a first depth associated with at least one object is calculated from coordinates of the sets of first and second pixels, the focal length fl of the first camera 11 and the distance B between the first camera 11 and the second camera 12.

[0106] The first depth associated with at least one object is calculated, for example, according to a formula known to those skilled in the art:

[0107] [Math.5] Dts(Pt)=fx^

[0108] With: ■ Dts( Pt ) is the depth of pixel Pt of the first image; - the index s means “stereo” because the depth is calculated here for the stereo system; - the index t means “target”; - f is the focal length fl in pixel units; - B is the distance between the two cameras of the stereo system; - d is the disparity of the pixel Pt in pixel units, defined as the displacement of a pixel from the first image to the second image, this displacement is horizontal when the images are rectified.

[0109] This calculation is metrically accurate, meaning that the depth measured here is absolute and not relative. In the case where the extrinsic parameters of the system remain unchanged, this depth calculation is very accurate.

[0110] Conversely, if the distance B between the cameras 11, 12 has been modified after calibration of the stereo system, the computer remains able to calculate a depth for the pixels of the first image but this calculated depth is wrong. This depth being inversely proportional to the distance B, the calculated depths will be inversely proportional to the offset of the distance B. This results in a scale defect, the calculated depths between various objects are relatively correct, that is to say that a first object detected closer to the vehicle 10 than a second object by the stereo system is in fact closer to the vehicle 10. Nevertheless, the depth of the first object, therefore the distance separating the vehicle from the first object is not correct. This results in incorrect information being communicated to any AD AS systems and therefore incorrect action or information being output from the AD AS system receiving this depth data.

[0111] In an operation 31, the computer receives third data representative of a third image acquired by the first camera 11 at a second acquisition time instant t2.

[0112] This instant t2 is prior to the first time instant tl of acquisition.

[0113] The third data have, for example, been saved in a memory associated with the computer or in a memory of a device on board the vehicle 10 and accessible to the computer implementing the process.

[0114] If the vehicle 10 is moving, then the third image corresponds to a third view of the scene taken from a third point of view, that of the first camera 11 at its position at the second time instant t2.

[0115] This position of the first camera 11 at the second instant t2 is defined by the movement of the vehicle 10 between the first and second instants t1 and t2. This movement is therefore linked to the speed of the vehicle 10 during the time separating the first and second instants t1 and t2.

[0116] The first camera in the two positions defined at the first and second time instants t1 and t2 forms a monoscopic vision system. The intrinsic parameters of this system remain the same as those defined previously related to the first camera 11 for the stereoscopic vision system. The extrinsic parameters of this monoscopic vision system are the following parameters: - 3 translations in the x, y and z directions: Tx', Ty' and Tz' constituting the translation vector T'; and - 3 rotations around the x, y and z axes: Rx', Ry' and Rz', constituting the rotation matrix R'.

[0117] According to a first embodiment, these extrinsic parameters of the monoscopic vision system are, for example, received from a computer processing data linked to the movement of the vehicle 10. The speed of the vehicle 10 can be defined by the absolute speed of movement of the vehicle, the rotation speed of a wheel or the rotation speed of a shaft linked to the propulsion system of the vehicle 10. The pivoting of the vehicle can be determined by the angle of the steering wheel or the wheels for example.

[0118] According to a second embodiment, these parameters are obtained using an accelerometer-type sensor embedded in the vehicle 10 and / or directly linked to the first camera 11 or to the monoscopic vision system.

[0119] In order to facilitate the analysis of the first and third images received, these images are, for example, rectified according to the extrinsic parameters of the vision system. monoscopic in an additional operation, following a method known to those skilled in the art as specified previously.

[0120] The following operation 32 consists of determining a set of third pixels in the third image corresponding to the set of first pixels determined by the optical flow calculation method during operation 23.

[0121] According to another embodiment, a new set of first pixels is determined in the first image which is rectified according to the extrinsic parameters of the monoscopic system. It is however possible to associate this new set of first pixels with the set of first pixels defined during operation 23, the two rectifications of the first image being geometrically linked. Subsequently, these two possible sets of first pixels are considered as a single set, the geometric characteristics linking them being known it is easy to identify this same set in each first image.

[0122] In a step 33, a second depth associated with at least one object is calculated from coordinates of the sets of first and third pixels, the focal distance fl of the first camera 11, the translation T' and the rotation R' of the first camera 11 between the first time instant t1 of acquisition and the second time instant t2 of acquisition.

[0123] The second depth associated with at least one object is calculated, for example, according to a formula known to those skilled in the art:

[0124] [Math.6]

[0125] With: ■ ( ^ ) is the depth of the pixel Pt of the first image; - the index m means “mono” because the depth is calculated here for the monoscopic vision system; - the index t means “target”; - f is the focal length fl in pixel units; - D is the distance between the two positions of the first camera 11 between the first and second time instants tl and t2; - d is the disparity of the pixel Pt in pixel units, defined as the displacement of a pixel from the first image to the third image, this displacement is horizontal when the images are rectified.

[0126] In order to verify the calibration of the stereo system, it is appropriate to determine in an operation 29 a second value representative of a calibration defect of the stereoscopic vision system by comparing said first and second depths of the first pixels calculated during previous operations 28, 33.

[0127] Thus, it is possible, for the same pixels of the first image, to compare the depths calculated by the monoscopic and stereoscopic vision systems, following, for example, the formula:

[0128] [Math.7]

[0129] with: - 'Ld' is the second value representing a calibration fault; - 'N' is a number of pixels in said first image with each pixel listed by its coordinates (i,j); - 'DSi j is said first depth calculated during operation 28 for a pixel of coordinates (i,j) in said first image; and - is said second depth calculated during operation 33 for a pixel with coordinates (i,j) in said first image.

[0130] In order to make the calculation of the second value representative of a calibration fault of the stereoscopic vision system more reliable, it is appropriate here to add filters in order to take into account only the relevant pixels when calculating the second value.

[0131] An operation 34 for determining the occluded areas of the first image consists of determining a second visibility mask associated with the first image and representative of a set of pixels of the first image having at least one corresponding pixel in the third image, the second visibility mask being determined by the optical flow calculation method previously described.

[0132] The determination of the visibility mask Vm j .for a pixel of coordinates (i,j) makes it possible not to take into account possible aberrant values ​​of optical flow for pixels of the first image which have no correspondence in the third image. The use of the visibility mask in the operation 29 of determining a second value representative of a calibration defect of the stereoscopic vision system thus makes this operation more reliable.

[0133] An operation 35 for determining the static zones of the first image consists of determining a mask of the dynamic objects associated with the first image and representative of a set of pixels of the first image associated with an object in motion in the scene.

[0134] Indeed, monoscopic vision systems are not capable of determining the depth of a moving object in a scene. In order not to take into account irrelevant values, it is appropriate to take into account only the pixels of the first image associated with stationary objects in the scene. The use of the mask of dynamic objects P-Sj for a pixel of coordinates (i,j) in the operation 29 of determining a second value representative of a calibration fault of the stereoscopic vision system greatly makes this operation more reliable.

[0135] By combining the calculation of the second value representative of a calibration defect of the stereoscopic vision system with the visibility and dynamic object masks, the formula for calculating this second value then becomes:

[0136] [Math. 8]

[0137] With: - 'Ld' is the second value representing a calibration fault, - 'N' is a number of pixels in said first image with each pixel listed by its coordinates (i,j), - is said first depth calculated during operation 28 for a pixel with coordinates (i,j) in said first image, - this is said second depth calculated during operation 33 for a pixel with coordinates (i,j) in said first image, - It II indicates the absolute value; ■ Pj y) is the visibility mask of the stereo system defined during operation 27; - Vm(Pi.j) is the visibility mask of the monocular vision system defined during operation 34; and -Md{Pij) is the mask of dynamic objects defined during operation 35.

[0138] The determination 29 of a second value representative of a calibration defect of the stereoscopic vision system makes it possible to verify two of the six extrinsic parameters of the stereo system: Ty, including the distance B from the “baseline”, and Rx.

[0139] This definition is also more interpretable than the epipolar constraint mentioned previously.

[0140] From the calculation of the first and second values ​​representative of a calibration fault of the stereo system, it is possible to define one or more threshold values ​​from which various scenarios can be triggered. For example, it is possible to display in the vehicle 10, on a screen or via the lighting of a warning light, information intended to warn the driver that the stereo vision system and / or the AD AS using data from this stereo system are not able to perform all of their function(s).

[0141] The driver of the vehicle 10 is then able to detect the need for a new calibration of the stereo system. The driver can thus park the vehicle 10 and launch a self-calibration procedure of the stereo system for example. Such a This procedure is known to those skilled in the art and is described in the document “High-Precision Online Markerless Stereo Extrinsic Calibration” by Yonggen Ling and Shaojie Shen dated March 26, 2019.

[0142] After such a self-calibration procedure, it is possible that the second value representative of a calibration fault of the stereo system remains non-zero. The persistence of this error is in this case due to a calibration fault of the distance B of the “baseline”, following for example a deformation of the structure of the vehicle 10. An intervention by a specialist is then necessary in this case to recalibrate the stereo system.

[0143] In summary, this invention proposes two complementary ways to detect a calibration fault of the stereo system. The two operations of calculating these values ​​thus make it possible to detect a calibration fault linked to all the extrinsic parameters of the stereo system.

[0144] [Fig. 4] illustrates a flowchart of the different steps of a method for detecting a calibration fault of a non-parallel stereo system on board the vehicle of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention. The method is for example implemented by a device on board the first vehicle 10 or by the device 4 of [Fig. 2].

[0145] In a first step 21, first data representative of a first image acquired by a first camera 11 of said set of cameras at a first acquisition time instant t1 are received.

[0146] In a second step 22, second data representative of a second image acquired by a second camera 12 of said set of cameras at the same first acquisition time instant t1 are received.

[0147] In a third step 23, at least one object is detected in the scene by analyzing the first image, a set of first pixels of the first image being associated with at least one object.

[0148] In a fourth step 24, a set of second pixels corresponding to the set of first pixels is determined in the second image, by implementing an optical flow calculation method as a function of the set of first pixels, an optical flow being representative of a displacement vector between each first pixel of the set of first pixels and the second pixel corresponding to each first pixel.

[0149] In a step 25, a first value representative of a calibration defect of the stereoscopic vision system is determined as a function of an average of ratios between a distance of a component of the optical flow calculated for each pixel along a determined direction and a standard of the optical flow calculated for each pixel.

[0150] According to a variant, the variants and examples of the operations described in relation to Figures 1 and 3 apply to the steps of the method of [Fig.4].

[0151] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for detecting a calibration fault in a stereoscopic vision system on board a vehicle, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0152] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of [Fig.2].

Claims

1. Claims Method for detecting a calibration fault in a stereoscopic vision system on board a vehicle (10), the stereoscopic vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a scene from a different point of view, the optical axes representative of an orientation of the field of vision of each camera being oriented in a non-parallel manner, said method comprising the following steps: - reception (21, 22) of first and second data respectively representative of a first and second image acquired by respectively a first and second camera (11, 12) of said set of cameras at the same first acquisition time instant; - detection (23) of at least one object in the scene by analysis of said first image, a set of first pixels of the first image being associated with said at least one object; - determination (24) of a set of second pixels in said second image corresponding to said set of first pixels, by implementing an optical flow calculation method as a function of the set of first pixels, an optical flow being representative of a displacement vector between each first pixel of said set of first pixels and the second pixel corresponding to said each first pixel; - determination (25) of a first value representative of a calibration defect of said stereoscopic vision system as a function of an average of ratios between a distance of a component of said optical flow calculated for each pixel in a determined direction and a standard of said optical flow calculated for each pixel, said method being characterized in that it comprises the following steps: - calculation (28) of a first depth associated with said at least one object from coordinates of said sets of first and second pixels, focal lengths of the first and second cameras (11, 12) and a distance between the first camera (11) and the second camera (12); - reception (31) of third data representative of a third image acquired by said first camera (11) at a second time instant prior to said first time instant of acquisition of said first image; - determination (32) of a set of third pixels in said third image corresponding to said set of first pixels determined by said optical flow calculation method; - calculation (33) of a second depth associated with said at least one object from coordinates of said sets of first and third pixels, of said focal length of the first camera (11), of a displacement of said first camera (11) between said first time instant and said second time instant; - determination (29) of a second value representative of a calibration defect of said stereoscopic vision system by comparison of said first and second depths.

2. Method according to claim 1, for which said determination (25) of a first value representative of a calibration defect of said stereoscopic vision system is obtained by the following formula: j dy , where: - 'Ln' is said first value representative of a calibration defect; - 'N' is a number of pixels in said first image with each pixel enumerated by its coordinates (i,j); - 'Fij' is a vector of said optical flow corresponding to a pixel of coordinates (i,j) in said first image; and - 'ef is a unit vector defining said determined direction.

3. Method according to claim 1 or 2, further comprising a step of rectifying (26) the first and second images according to extrinsic parameters of the stereoscopic vision system, said extrinsic parameters comprising a parameter representative of a rotation and a parameter representative of a translation in three dimensions between the first camera (11) and the second camera (12), so as to make the epipolar lines horizontal in the first and second rectified images.

4. Method according to one of claims 1 to 3, further comprising a step of determining (27) a first mask of visibility associated with said first image and representative of a set of pixels of said first image having at least one corresponding pixel in said second image, said first visibility mask being determined by the optical flow calculation method, said first value representative of a calibration defect being further determined as a function of said first visibility mask.

5. Method according to one of the preceding claims, for which said determination (29) of a second value representative of a calibration defect of said stereoscopic vision system is obtained by the following formula: ld=0,1^-0^-^- - 'Ld' is said second value representative of a calibration defect; - 'N' is a number of pixels in said first image with each pixel enumerated by its coordinates (i,j); - j is said first calculated depth (28) for a pixel of coordinates (i,j) associated with said at least one object in said first image; and - 'D™ / is said second calculated depth (33) for a pixel of coordinates (i,j) associated with said at least one object in said first image.

6. Method according to one of the preceding claims, further comprising a step of determining (34) a second visibility mask associated with said first image and representative of a set of pixels of said first image having at least one corresponding pixel in said third image, said second visibility mask being determined by the optical flow calculation method, said second value representative of a calibration defect being further determined as a function of said second visibility mask.

7. Method according to one of the preceding claims, further comprising a step of determining (35) a mask of dynamic objects associated with said first image and representative of a set of pixels of said first image associated with at least one object in motion in said scene, said second value representative of a calibration defect being further determined as a function of said mask of dynamic objects.

8. 26 Device (4) for detecting a calibration fault in a stereo vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 7.

9. Vehicle (10) comprising the device (4) according to claim 8.

Images