METHOD AND DEVICE FOR CALIBRINGING A NON-PARALLELED STEREOSCOPIC VISION SYSTEM IN A VEHICLE

Description

technical field

[0001] The present invention claims priority from French application 2303332 filed on 04.04.2023.

[0002] The present invention relates to methods and devices for calibrating a stereoscopic vision system installed in a vehicle, for example, a motor vehicle. The present invention also relates to methods and devices for determining depth using a stereoscopic vision system installed in a vehicle. The present invention further relates to a method and device for controlling one or more ADAS systems installed in a vehicle based on the determined depth. Technological background

[0003] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). These systems are passive and active safety features designed to minimize human error in driving all types of vehicles. ADAS use advanced technologies to assist the driver while driving and thus improve performance. ADAS employs a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or control certain vehicle systems.

[0004] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control, and automatic parking systems.

[0005] Vehicle-based ADAS systems are powered by data obtained from one or more onboard sensors, such as cameras. These cameras enable the detection and location of other road users or potential obstacles around a vehicle in order to, for example: to adapt the vehicle's lighting according to the presence of other users; to automatically regulate the vehicle's speed; to act on the braking system in case of risk of impact with an object.

[0006] The reliability of the data from these cameras and the proper functioning of driver assistance devices using this data therefore depend on the correct calibration of the cameras.

[0007] A method for calibrating a stereoscopic vision system is presented in the document "High-Precision Online Markerless Stereo Extrinsic Calibration" by Yonggen Ling and Shaojie Shen published on March 26, 2019, adapted to the calibration of a stereoscopic vision system where the distance separating the cameras is limited.

[0008] Other calibration methods requiring the use of a checkerboard are commonly used in the field of computer vision. One such method has been presented for many years by Zhengyou Zhang of the French National Institute for Research in Computer Science and Automation (INRIA) and is present in numerous code packages integrated into libraries, for example, for MATLAB or OpenCV. It should be noted that this type of method requires human intervention. US 10 636 121 B2, US 2012 / 236125 A1, and US 2013 / 250068 A1 also propose similar methods. Summary of the present invention

[0009] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0010] Another object of the present invention is to improve the reliability of the data from these cameras.

[0011] Another object of the present invention is to propose a calibration method not requiring human intervention.

[0012] Another object of the present invention is to improve road safety.

[0013] According to a first aspect, the present invention relates to a method for calibrating a stereoscopic vision system embedded in a vehicle, the stereoscopic vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view, the optical axes representing an orientation of the field of vision of each camera being oriented in a non-parallel manner, the method being characterized in that it comprises the following steps: reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the camera set at the same first time instant of acquisition; determination of a set of pixels of the second image corresponding to a first set of pixels of the first image, by implementing a calculation method by optical flux as a function of the first set of pixels of the first image and determination of a Euclidean distance associated with each pixel of the first set of pixels of the first image as a function of the optical flux obtained for each pixel, an optical flux being representative of a displacement vector between each pixel of the first set of pixels of the first image and a pixel of the set of pixels of the second image corresponding to each pixel of the first set of pixels of the first image;selection of a first group of eight pixel pairs in the first and second images, each pixel pair being composed of a first pixel from the first set of pixels of the first image and a second pixel from the second image associated with the first pixel; determination of a fundamental matrix as a function of the first group by minimizing an epipolar error; determination of disparities associated with said first set of pixels of the first image as a function of said fundamental matrix and from said set of pixels of the second image and said first set of pixels of the first image; reception of third data representative of a third image acquired by the first camera at a second time instant of acquisition prior to the first time instant of acquisition of the first image;Calculation of depths associated with a second set of pixels from the first image via a monoscopic vision system, using a set of pixels from the third image corresponding to the second set of pixels from the first image and said second set of pixels from the first image, the monoscopic vision system being formed by the first camera; determination of a baseline distance between the first camera and the second camera as a function of said depths associated with the second set of pixels from the first image, the disparities associated with each pixel from the first set of pixels from the first image, and a focal length associated with the first camera; and calibration of the stereoscopic vision system as a function of the fundamental matrix and the baseline distance.

[0014] The calibration of the stereoscopic vision system is therefore carried out, the calibration consisting of determining all the extrinsic parameters of the stereoscopic vision system.

[0015] According to one variant of the process, the first image is cut into eight blocks, each block comprising at least one first pixel from the first group.

[0016] This segmentation of the first image allows us to consider the entire first image and thus avoids considering localized errors.

[0017] According to a third variant of the process, the first pixels of the first group are selected from the first set of pixels of the first image according to a Euclidean distance associated with each first pixel.

[0018] The greater the Euclidean distance between the associated pixels, the better the accuracy of the calculations. The reliability of the process is thus guaranteed.

[0019] According to another variant, the process also includes the steps of: determination of a first intermediate matrix as a function of the first group by minimizing an epipolar error; selection of a second group of eight pairs of pixels distinct from the first group of eight pairs of pixels; determination of a second intermediate matrix as a function of the second group by minimizing an epipolar error; determination of a first average matrix as a function of a set of intermediate matrices including the first and second intermediate matrices, each coefficient of the first average matrix being defined as an average of the coefficients of each intermediate matrix in the set of intermediate matrices, each of the intermediate matrices having been previously reduced to a matrix with a norm equal to 1; comparison of a determined threshold value to a set of differences between coefficients of the first average matrix and coefficients of the first intermediate matrix; if each difference is less than the threshold value then the fundamental matrix is ​​defined as being equal to the first average matrix;and if at least one difference is greater than the threshold value then the process includes the following steps, with a set of groups including the first and second groups: selection (A) of a new group of 8 pairs of pixels distinct from each group in the set of groups; determination (B) of a new intermediate matrix as a function of the new group by minimizing an epipolar error; addition (C) of the new intermediate matrix to the set of intermediate matrices; determination (D) of a new average matrix as a function of the set of intermediate matrices, each coefficient of the first average matrix being defined as an average of the coefficients of each intermediate matrix in the set of intermediate matrices, each of the intermediate matrices having been previously reduced to a matrix of norm equal to 1;comparison (E) of the determined threshold value to a set of differences between coefficients of the new average matrix and coefficients of the first average matrix: if each difference is less than the threshold value then the fundamental matrix is ​​defined as the new average matrix; and if at least one difference is greater than the threshold value then the new group of eight pixel pairs is added to the set of groups and steps B to E are repeated considering the new average matrix as replacing the first average matrix.

[0020] Iterating these steps ensures the stabilization of the calculation of the fundamental matrix.

[0021] According to yet another variant of the process, the depths are calculated by a convolutional neural network with automatic learning.

[0022] According to yet another variant, the process further includes a step of determining a first visibility mask associated with the first image and representative of a set of pixels of the first image having at least one corresponding pixel in the second image, the first visibility mask being determined by the optical flow calculation method, the selection of the first group being further a function of the first visibility mask.

[0023] Using the first visibility mask for selecting the eight pixel pairs thus makes it possible to exclude aberrant optical flux values.

[0024] According to another variant, the process further includes a step of determining a second visibility mask associated with the first image and representative of a set of pixels of the first image having at least one corresponding pixel in the third image, the second visibility mask being determined by the optical flow calculation method, the calculation of the base distance being further a function of the second visibility mask.

[0025] Using the second visibility mask to calculate said base distance makes it possible to exclude aberrant values ​​of optical flux.

[0026] According to a further variant, the process further includes a step of determining a dynamic object mask associated with the first image and representative of a set of pixels of the first image associated with at least one moving object in the scene, the calculation of said base distance being further a function of said dynamic object mask.

[0027] Using the dynamic object mask when determining the base distance avoids taking into account pixels related to moving objects in the scene.

[0028] According to a second aspect, the present invention relates to a calibration device for a stereoscopic vision system embedded in a vehicle, the device comprising a memory associated with a processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0029] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0030] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0031] Such a computer program can use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0032] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0033] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard drive.

[0034] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from a network such as the Internet.

[0035] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0036] Other features and advantages of the present invention will become apparent from the description of the specific and non-limiting embodiments of the present invention below, with reference to figures 1 to 4 attached, on which: [ Fig. 1] schematically illustrates a non-parallel stereoscopic vision system fitted to a vehicle, according to a particular and non-limiting embodiment of the present invention; [ Fig. 2 ] schematically illustrates a device configured for calibrating a non-parallel stereoscopic vision system embedded in the vehicle of the figure 1 , according to a particular and non-limiting example of the present invention; [ Fig. 3 ] illustrates a flowchart of the different operations in a calibration process for a non-parallel stereoscopic vision system embedded in the vehicle of the figure 1 , according to a particular and non-limiting example of the present invention; [ Fig. 4 ] illustrates a flowchart of the different stages of a calibration process for a non-parallel stereoscopic vision system embedded in the vehicle of the figure 1, according to a particular and non-limiting example of the present invention. Description of examples of achievements

[0037] A method and device for calibrating a stereoscopic vision system, hereafter referred to as a "stereo system", installed in a vehicle will now be described in what follows, with joint reference to figures 1 to 4 The same elements are identified with the same reference symbols throughout the description that follows.

[0038] According to a particular and non-limiting embodiment of the present invention, a method for calibrating a stereoscopic vision system embedded in a vehicle, the stereoscopic vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view, the optical axes representing an orientation of the field of view of each camera being oriented in a non-parallel manner, the method being characterized in that it comprises the following steps.

[0039] From first and second data points, respectively representing the first and second images acquired by the first and second cameras of the camera array, are received at the same initial acquisition time. A set of pixels from the second image corresponding to a first set of pixels from the first image is determined by implementing an optical flux calculation method. A Euclidean distance associated with each pixel in the first set of pixels from the first image is determined based on the optical flux obtained for each pixel. A first group of eight pixel pairs is selected from the first and second images, each pixel pair consisting of a first pixel from the first set of pixels from the first image and a second pixel from the second image associated with the first pixel. A fundamental matrix is ​​determined based on the first group by minimizing an epipolar error.Disparities associated with the first set of pixels of the first image are then determined as a function of the fundamental matrix and from the set of pixels of the second image and the first set of pixels of the first image;

[0040] Third data representing a third image acquired by the first camera at a second time instant of acquisition prior to the first time instant of acquisition of the first image are also received, depths associated with a second set of pixels of the first image are calculated via a monoscopic vision system from a set of pixels of the third image corresponding to the second set of pixels of the first image and the second set of pixels of the first image, the monoscopic vision system being formed from the first camera and a base distance between the first camera and the second camera is determined as a function of the depths associated with the second set of pixels of the first image, the disparities associated with each pixel of the first set of pixels of the first image and a focal length associated with said first camera.

[0041] The stereoscopic vision system is then calibrated according to the fundamental matrix and the baseline distance.

[0042] There figure 1 schematically illustrates a non-parallel stereoscopic vision system equipping a vehicle 10 evolving in an environment 1, according to a particular and non-limiting embodiment of the present invention.

[0043] Such an environment 1 corresponds, for example, to a road environment consisting of a network of roads accessible to vehicle 10.

[0044] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, or a motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is, a vehicle operating at a predetermined level of autonomy or under the full supervision of the driver.

[0045] The vehicle 10 advantageously includes several onboard cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms the stereo system. Two cameras 11 and 12 are illustrated on the figure 1 The invention is not limited, however, to a stereo system comprising two cameras but extends to any stereo system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras.

[0046] Both cameras 11 and 12 have known intrinsic parameters. These parameters include: the focal length f1 of the first camera 11; the focal length f2 of the second camera 12; the distortions which are due to imperfections in the optical system of each camera; the direction C1 of the optical axis of the first camera 11; the direction C2 of the optical axis of the second camera 12; and the respective resolutions of cameras 11, 12.

[0047] The intrinsic parameters characterize the transformation that associates, for an image point, the camera coordinates to the pixel coordinates in each camera. These parameters do not change if the camera is moved.

[0048] Distortions, which result from imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to refine the camera model by introducing the three distortions that generate the most significant effects: radial, decentering, and prismatic distortions, induced by defects in lens curvature, parallelism, and coaxiality of optical axes. In this example, the cameras are assumed to be perfect, meaning that distortions are either not taken into account or their correction is addressed during image acquisition.

[0049] These two cameras, 11 and 12, are positioned so that each acquires an image of a scene from a different viewpoint. The first viewpoint is, for example, located on or in the left-hand side mirror of vehicle 10 or at the top of the windshield of vehicle 10. The second viewpoint is, for example, located on or in the right-hand side mirror of vehicle 10 or at the top of the windshield of vehicle 10. If both cameras are located at the top of the windshield, they are positioned at a certain distance apart. In this example, the first camera, 11, is located at the top of the windshield of vehicle 10, and the second camera, 12, is located in the right-hand side mirror of vehicle 10.

[0050] A first reference point is associated with the first camera 11: The direction of the y-axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y-axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the baseline distance and the direction separating the two cameras 11, 12 is that of the y-axis; the direction of the x-axis is defined as orthogonal to that of the y-axis and orthogonal to that of the optical axis C1 of the first camera 11; the direction of the z-axis is defined as orthogonal to the directions of the x and y axes.

[0051] The three axes x, y and z thus form an orthonormal coordinate system.

[0052] The extrinsic parameters related to the position of cameras 11, 12 are as follows: 3 translations in the directions x, y and z: Tx, Ty and Tz constituting the translation vector T; and 3 rotations around the axes x, y and z: Rx, Ry and Rz, constituting the rotation matrix R.

[0053] Determining the extrinsic parameters constitutes the problem of calibrating a stereoscopic vision system.

[0054] A key constraint of stereoscopic vision systems used in the automotive industry is, for example, the large distance between the two cameras. Indeed, to cover a measurement range of 200 meters, the baseline must be 60 cm for cameras commonly used in this field.

[0055] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera alone covering a first acquisition field 13, the second camera alone covering a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic view of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic view of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic view of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0056] The directions C1, C2 of the optical axes representing an orientation of the field of view of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.

[0057] It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located to the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently.

[0058] The images acquired by cameras 11, 12 at a given acquisition time t1 are in the form of data representing pixels characterized by: coordinates in each image; and data relating to the colors and brightness of the objects in the observed scene in the form of, for example, RGB colorimetric coordinates (from the English "Red Green Blue", in French "Rouge Vert Bleu") or HSL (Tone, Saturation, Luminosity).

[0059] The images acquired by cameras 11 and 12 represent views of the same scene taken from different viewpoints, the camera positions being distinct. For example, this scene includes: buildings; road infrastructure; other stationary users, for example a parked vehicle; and / or other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0060] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0061] There figure 2Figure 4 schematically illustrates a device configured for calibrating a non-parallel stereoscopic vision system installed in a vehicle 10, according to a particular and non-limiting embodiment of the present invention. Device 4 corresponds, for example, to a device installed in the first vehicle 10, such as a computer.

[0062] Device 4, for example, is configured to implement the operations and / or steps described alongside the Figures 1 , 3 And 4Examples of such a device 4 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of the device 4, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. The device 4 may be implemented as electronic circuits, software (or computer) modules, or a combination of electronic circuits and software modules.

[0063] Device 4 includes one or more processors 40 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in Device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. Device 4 further includes at least one memory 41, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0064] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is, for example, stored on memory 41.

[0065] According to various specific and non-limiting embodiment examples, device 4 is coupled in communication with other similar devices or systems (e.g. other computers) and / or with communication devices, e.g. a TCU (Telematic Control Unit), e.g. via a communication bus or through dedicated input / output ports.

[0066] According to a specific and non-limiting embodiment, device 4 includes a block 42 of interface elements for communicating with external devices. The interface elements of block 42 include one or more of the following interfaces: radio frequency (RF) interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; USB (Universal Serial Bus) interface; HDMI (High Definition Multimedia Interface); LIN (Local Interconnect Network) interface.

[0067] According to another particular and non-limiting embodiment, the device 4 includes a communication interface 43 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 430. The communication interface 43 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds, for example, to a wired network of the type CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3).

[0068] According to a particular and non-limiting embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch or not, one or more loudspeakers 450 and / or other peripherals 460 (projection system) via the output interfaces 44, 45, 46 respectively. According to a variant, one or more of the external devices is integrated into the device 4.

[0069] There figure 3 illustrates a flowchart of the different operations in a calibration process for a non-parallel stereoscopic vision system embedded in the vehicle. figure 1 , according to a particular and non-limiting example of the present invention.

[0070] The process is implemented for example by one or more processors of one or more computers embedded in the vehicle 10, for example by a computer controlling the stereo system.

[0071] In a first operation 21, the computer receives first data representative of a first image acquired by a first camera 11 of the set of cameras at a first time instant of acquisition t1.

[0072] In a second operation 22, the computer receives second data representing a second image acquired by a second camera 12 of the set of cameras at the same first time instant of acquisition t1.

[0073] The two images received correspond to two views of the same scene taking place around vehicle 10 at the same first given time instant of acquisition t1.

[0074] The next operation, 23, of stereo matching is performed by implementing an optical flow calculation method based on the set of first pixels. Such a method is described in particular in "UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos" by Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu, published in June 2019.

[0075] Optical flow computation is performed, for example, by a convolutional neural network (CNN). This type of tool is commonly used in image processing.

[0076] A set of pixels from the second image corresponding to a first set of pixels from the first image is determined, by implementing an optical flow calculation method based on the first set of pixels from the first image.

[0077] A method for extracting features is known to those skilled in the art, such as the Scale Invariant Feature Transform (SIFT) or the Corner Detection method. These methods are not the most suitable in this case; the use of a convolutional neural network is therefore preferred.

[0078] The output data of this operation 23 is an optical flux representative of a displacement vector between each first pixel of the set of first pixels of the first image and the second pixel corresponding to each first pixel in the second image.

[0079] A Euclidean distance associated with each pixel of the first set of pixels of the first image is also determined based on the optical flux obtained for each pixel.

[0080] It is possible that a pixel in the first image may not have a corresponding pixel in the second image. This phenomenon is explained by the fact that areas of the first image may be occluded in the second image. Indeed, the difference in viewpoints between the two cameras 11 and 12 prevents both cameras from seeing all the elements of the scene. An object present in the scene may obscure a second object in the scene; the second object may be visible from the viewpoint of the first camera 11 but obscured by the first object from the viewpoint of the second camera 12.

[0081] A 24-step operation to determine the occluded areas of the first image involves determining a first visibility mask associated with the first image and representative of a set of pixels in the first image that have at least one corresponding pixel in the second image. The first visibility mask is determined using an optical flow method. Such a method is described in "UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos" by Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu, published in June 2019.

[0082] The definition of a visibility mask is known to those skilled in the art. It is, for example, described in "Occlusion Aware Unsupervised Learning of Optical Flow" by Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu, published on April 4, 2018.

[0083] Determining the visibility mask subsequently makes it possible to avoid using pixels from the first image that do not have a corresponding image in the second image during subsequent image processing operations.

[0084] The aim of the following operations is to determine the extrinsic parameters of the stereo system. These correspond to the nine coefficients of the essential matrix of the stereo system. Therefore, to determine these nine coefficients, nine equations must be solved.

[0085] At this stage, it is not possible to obtain the value of the baseline distance separating the first and second cameras; this baseline distance will be defined in later operations.

[0086] Not knowing the baseline means not being able to determine the scale of measurements taken from the first and second images.

[0087] This lack of knowledge of the scale is equivalent to defining a fundamental matrix up to a single coefficient. Eight values ​​of the fundamental matrix must be determined to know the extrinsic parameters of the stereo system outside the baseline. This requires solving eight equations.

[0088] The last coefficient of the fundamental matrix will, for its part, be obtained by defining the norm of the fundamental matrix as one, the system being defined up to a scale factor (in English "up to scale").

[0089] To obtain the eight equations, in an operation 25, a first group of eight pixel pairs is selected from the first and second images. Each pixel pair consists of a first pixel from the first set of pixels in the first image and a second pixel from the second image associated with the first pixel in the previous matching operation.

[0090] To increase the accuracy of the calculations, it is best to select the first eight pixels in separate areas of the first image. The image is, for example, divided into eight blocks, so that each first pixel can be determined in one of the blocks.

[0091] Each first pixel is determined to be the one in the block whose previously calculated Euclidean distance is the greatest among the pixels in its area and which have at least one corresponding second pixel in the second image. In other words, the first visibility mask is used as a filter to select only relevant pixels.

[0092] The largest Euclidean distance is preferred because it allows for the greatest accuracy in solving the subsequent equations, thus considerably increasing the reliability of the extrinsic parameters obtained from the analysis of the first and second images.

[0093] A first intermediate matrix is ​​determined as a function of the first group by minimizing an epipolar error defined by: min F p i ′ T Fp i 2 avec F = 1

[0094] With : p i ′ And pi are pixels in the second and first images respectively; and F the intermediate 3x3 matrix with || F || = 1.

[0095] According to a first embodiment, in an operation 31, a fundamental matrix of the stereo system is defined as equal to the first intermediate matrix.

[0096] According to a second embodiment, new pairs of pixels are again selected to perform further calculations of intermediate matrices and then used to refine the calculation of a fundamental matrix of the stereo system.

[0097] Thus, in operation 27, a second group of eight pairs of pixels, distinct from the first group, is selected. The first pixels of this second group are chosen using the same method as described previously, excluding the first pixels already selected.

[0098] In operation 28, a second intermediate matrix is ​​determined as a function of the second group by minimizing an epipolar error as described previously.

[0099] In operation 29, a first average matrix is ​​determined as a function of the set of intermediate matrices previously determined. Each coefficient of the first average matrix is ​​defined as an average of the coefficients of each intermediate matrix in the set of intermediate matrices, each of the intermediate matrices having been previously reduced to a matrix with a norm equal to 1.

[0100] In operation 30, a predetermined threshold value is compared to a set of differences between coefficients of the first mean matrix and coefficients of the first intermediate matrix. This threshold value defines the limit of convergence of the coefficient calculations of a fundamental matrix.

[0101] If each difference is less than the threshold value, in other words, if the first average matrix is ​​close to the first intermediate matrix, then the fundamental matrix is ​​defined in an operation 31 as being equal to the first average matrix.

[0102] Conversely, if at least one difference is greater than the threshold value, then the process includes the following steps, with a set of groups comprising the aforementioned first and second groups: selection (A) of a new group of 8 pairs of pixels distinct from each group in the set of groups; determination (B) of a new intermediate matrix as a function of the new group by minimizing an epipolar error; addition (C) of the new intermediate matrix to the set of intermediate matrices; determination (D) of a new average matrix as a function of the set of intermediate matrices, each coefficient of the first average matrix being defined as an average of the coefficients of each intermediate matrix in the set of intermediate matrices, each of the intermediate matrices having been previously reduced to a matrix with a norm equal to 1;comparison (E) of the determined threshold value to a set of differences between coefficients of the new average matrix and coefficients of the first average matrix: if each difference is less than the threshold value then the fundamental matrix is ​​defined, in an operation 31, as the new average matrix; and if at least one difference is greater than the threshold value then the new group of eight pairs of pixels is added to the set of groups and steps B to E are repeated considering the new average matrix as replacing the first average matrix.

[0103] Thus, in this second embodiment, a fundamental matrix of the stereo system is defined after convergence of the calculations of its coefficients. This fundamental matrix is ​​thus defined from numerous groups of eight pairs of pixels and is therefore representative of a fundamental matrix of the stereo system valid for all the pixels of the first image.

[0104] At this stage, all the extrinsic parameters of the stereo system can be determined except for the baseline distance. This distance is often known to the stereo system during calibration and almost never changes, as it is linked to the structure on which the two cameras 11 and 12 are mounted. However, this distance can be altered following an impact to the vehicle, for example.

[0105] Knowing a fundamental matrix now allows us to use the data from the stereo system and consider it valid. For example, the first and second images can be rectified, as the relative positions of the cameras are known. Only the scale factor remains uncertain at this stage.

[0106] In an operation 32, disparities associated with the first set of pixels of the first image are determined as a function of the fundamental matrix and from the set of pixels of the second image and the first set of pixels of the first image.

[0107] In order to know the scale factor of the stereo system and thus guarantee depth predictions by the latter, the base distance is determined with the help of a monoscopic vision system defined by one of the moving cameras, here the first camera 11.

[0108] In an operation 33, the computer receives third data representative of a third image acquired by the first camera 11 at a second time instant of acquisition t2.

[0109] This time instant of acquisition t2 is prior to the first time instant of acquisition t1.

[0110] The third data, for example, was saved in a memory associated with the computer or in a memory of a device embedded in the vehicle 10 and accessible to the computer implementing the process.

[0111] If vehicle 10 is in motion, then the third image corresponds to a third view of the scene taken from a third point of view, that of the first camera 11 at its position at the second time instant of acquisition t2.

[0112] The position of the first camera 11 at the second acquisition time t2 is defined by the movement of the vehicle 10 between the acquisition times t2 and t1. This movement is therefore linked to the speed of the vehicle 10 during the time interval between the first and second acquisition times t1 and t2.

[0113] The first camera, in the two positions defined at the first and second acquisition times t1 and t2, forms a monoscopic vision system. The intrinsic parameters of this system remain the same as those previously defined for the first camera 11 in the stereo system. The extrinsic parameters of this monoscopic vision system are as follows: 3 translations in the directions x, y and z: Tx', Ty' and Tz' constituting the translation vector T'; and 3 rotations around the axes x, y and z: Rx', Ry' and Rz', constituting the rotation matrix R'.

[0114] The extrinsic parameters of the monoscopic vision system are, for example, determined by a computer associated with that same monoscopic vision system. The determination of the extrinsic parameters of the monoscopic vision system is known to those skilled in the art and is presented, for example, in the paper "Unsupervised Learning of Depth and Ego-Motion from Video" by Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe, published on August 1, 2017.

[0115] In an operation 34, depths associated with a second set of pixels from the first image are calculated via the monoscopic vision system from a set of pixels from the third image corresponding to the second set of pixels from the first image and the second set of pixels from the first image, such an operation 34 is also described in the previously cited document.

[0116] By eliminating the need to obtain extrinsic parameters via additional sensors, the depth determination device using a monoscopic vision system can be more easily integrated into a vehicle 10.

[0117] It is possible that a pixel in the first image may not have a corresponding pixel in the third image. This phenomenon is explained by the fact that areas of the first image may be occluded in the third image. Indeed, the difference in viewpoint of the first camera 11 at acquisition times t1 and t2 does not allow the first camera 11 to see all the elements of the scene. An object present in the scene may obscure a second object in the scene, the second object being visible from the viewpoint of the first camera 11 at acquisition time t1 but being hidden by an obstacle from the viewpoint of the first camera 11 at acquisition time t2.

[0118] An operation to determine the occluded areas of the first image consists of determining a second visibility mask associated with the first image and representative of a fourth set of pixels of the first image having at least one corresponding pixel in the third image, the second visibility mask being determined by the optical flow calculation method previously described.

[0119] Determining the visibility mask for a pixel allows us to disregard potentially aberrant optical flux values ​​for pixels in the first image that have no corresponding value in the third image. Therefore, using the visibility mask in subsequent operations is preferred.

[0120] One operation for determining the static areas of the first image involves defining a dynamic object mask associated with the first image and representative of a fifth set of pixels in the first image associated with a moving object in the scene. This dynamic object mask can be determined using the optical flow calculation method.

[0121] Such a previously defined dynamic object mask is known to the person skilled in the art and is presented in the paper "Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding" by Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia and Alan Yuille of July 11, 2019.

[0122] Indeed, monoscopic vision systems can determine the depth of a moving object in a scene based on the data used for their training, but the accuracy of this measurement is not guaranteed. To avoid including irrelevant values, only pixels from the first image associated with static or stationary objects in the scene should be considered. Therefore, using the dynamic object mask, as with the second visibility mask previously used, is preferred in subsequent operations.

[0123] The relevant pixels to consider for calculating the base distance are those that have at least one corresponding pixel in the second image, at least one corresponding pixel in the third image, and that are not associated with moving objects. In other words, this calculation is performed using the first and second visibility masks, as well as the dynamic object mask.

[0124] With the depth values ​​of the relevant pixels obtained via the monoscopic vision system and with the disparity values ​​determined by the stereo system, a first distance is then calculated using the following formula: D p i = d p i × D tm p i f

[0125] With : D is the first distance calculated for a relevant pixel pi; d ( pi ) is the disparity obtained via the stereo system for a relevant pixel pi ; D tm ( pi) is the depth calculated via the monoscopic vision system for a relevant pixel pi ; and f is the focal length of the first camera.

[0126] The baseline distance is determined, in operation 37, as the average of the first distances calculated for all relevant pixels as defined previously.

[0127] Obtaining a fundamental matrix during operation 31 and determining a baseline distance representing the distance between the first camera 11 and the second camera 12 allows us to define all the extrinsic parameters of the stereo system.

[0128] Operation 38, the calibration of the stereo system, consists of determining these extrinsic parameters. These are, for example, those of the essential matrix E of the stereo system, defined by: E = K 11 T FK 12 E = T × R

[0129] With : E is the essential matrix (in English "essential matrix"); T × is the 3x3 antisymmetric (in English "skew symmetric") matrix associated with the translation vector T between the first camera 11 and the second camera 12; R is the 3x3 rotation matrix between the first camera 11 and the second camera 12; K 11 is the intrinsic 3x3 matrix of the first camera 11; K 12 is the intrinsic 3x3 matrix of the second camera 12; and F is the fundamental 3x3 matrix.

[0130] The parameters of the translation vector T and the rotation matrix R can then be obtained by a singular value decomposition method.

[0131] The stereo system calibration is thus performed at the end of the process through operations that determine the extrinsic parameters of the stereo system. The parameters linking the two cameras 11 and 12 are therefore perfectly known, and the depth calculated by the stereo system is then metrically precise.

[0132] Data such as the depth calculated by the stereo system allows us to know the depth of any pixel in the first image. The process described here thus makes it possible to obtain reliable output data. An ADAS receiving this type of data will therefore be able to fully perform its functions.

[0133] If the ADAS uses data from the stereo system as input to determine the distance between a part of vehicle 10, for example the front bumper, and another road user, then the ADAS is able to determine this distance precisely. For example, if the ADAS's function is to activate the braking system of vehicle 10 in the event of a risk of collision with another road user, and the distance between vehicle 10 and that road user decreases sharply, then the ADAS is able to detect this sudden closeness and activate the braking system of vehicle 10 to avoid a potential accident.

[0134] There figure 4 illustrates a flowchart of the different stages of a calibration process for a non-parallel stereoscopic vision system installed in the vehicle of the figure 1, according to a particular and non-limiting embodiment of the present invention. The method is, for example, implemented by a device embedded in the first vehicle 10 or by the device 4 of the figure 2 .

[0135] In a first step 21, first representative data of a first image acquired by a first camera 11 of said set of cameras at a first time instant of acquisition t1 are received.

[0136] In a second step 22, second data representing a second image acquired by a second camera 12 of said camera set at the same first time instant of acquisition t1 are received.

[0137] In a step 23, the set of pixels of the second image corresponding to a first set of pixels of the first image is determined by implementing an optical flux calculation method based on the first set of pixels of the first image, and a Euclidean distance associated with each pixel of the first set of pixels of the first image is determined based on said optical flux obtained for each pixel.

[0138] In step 25, a first group of eight pairs of pixels is selected from the first and second images, each pair of pixels being composed of a first pixel from the first set of pixels of the first image and a second pixel from the second image associated with the first pixel.

[0139] In step 31, a fundamental matrix is ​​determined as a function of the first group by minimizing an epipolar error.

[0140] In step 32, disparities associated with the first set of pixels of the first image are determined as a function of the fundamental matrix and from the pixel set of the second image and the first pixel set of the first image.

[0141] In a step 33, third data representing a third image acquired by the first camera 11 at a second acquisition time t2 prior to the first acquisition time t1 of the first image are received.

[0142] In a step 34, depths associated with a second set of pixels from the first image are calculated via a monoscopic vision system from a set of pixels from the third image corresponding to the second set of pixels from the first image and the second set of pixels from the first image, the monoscopic vision system being formed from the first camera 11.

[0143] In a step 37, a basic distance between said first camera 11 and said second camera 12 is determined as a function of the depths associated with the second set of pixels of the first image, the Euclidean distances associated with each pixel of the first set of pixels of the first image and a focal length associated with the first camera 11.

[0144] In step 38, the stereo system is calibrated according to the fundamental matrix and the baseline distance.

[0145] According to one variant, the variants and examples of the operations described in relation to the Figures 1 And 3 apply to the steps of the process of the figure 4 .

[0146] Of course, the present invention is not limited to the embodiments described above but extends to a method for calibrating a stereoscopic vision system embedded in a vehicle, which would include secondary steps without falling outside the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0147] The present invention also relates to a vehicle, for example a motor vehicle or more generally an autonomous land-powered vehicle, comprising device 4 of the figure 2 .

Claims

1. Method for scaling a stereoscopic vision system on board a vehicle (10), the stereoscopic vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a scene according to a different viewpoint, the optical axes representative of an orientation of the field of view of each camera being orientated in a non-parallel manner, said method being characterised in that it comprises the steps of: - receipt (21, 22) of first and second data respectively representative of a first and second images acquired by respectively a first and second cameras (11, 12) of said set of cameras at a same first temporal instant of acquisition; - determination (23) of a set of pixels of said second image corresponding to a first set of pixels of the first image of the first compute method, by implementing a method group by optical flow as a function of said first set of pixels of the first image and determination of an Euclidean distance associated with each pixel of said first set of pixels of the first image as a function of said optical flow obtained for each pixel, an optical flow being representative of a displacement vector between each pixel of said first set of pixels of the first image and a pixel of said set of pixels of the second image corresponding to said each pixel of said first set of pixels of the first image; - selection (25) of a first group of eight pairs of pixels in said first and second images, each pair of pixels being compound of a first pixel of said first image and of a second pixel of the second image associated with said first pixel; - determination (31) of a fundamental matrix as a function of said first receipt by minimising an epipolar error; - determination (32) of disparities associated with said first set of pixels of said fundamental matrix pixels of the second image and of said first set of pixels of the first image; - compute (33) of third data representative of a third image acquired by said first camera (11) at a second acquisition time instant prior to said first acquisition time instant of said first image; - scaling (34) of depths associated with a second set of pixels of the first image via a monoscopic vision system from a set of pixels of the third image corresponding to said second set of pixels of the first image and said second set of pixels of the first image, - said monoscopic vision system being formed of said first camera (11); and of a focal length associated with said first camera (11); and - scaling (38) of said stereoscopic vision system as a function of said fundamental matrix and of said basic distance.

2. Method according to claim 1, wherein said first image is divided into eight blocks, each block comprising at least one first pixel of said first group.

3. Method according to claim 1 or 2, wherein said first pixels of said first group are selected from said first set of pixels of the first image as a function of said Euclidean distance associated with each first pixel.

4. Method according to one of claims 1 to 3, further comprising the steps of: - determining (26) a first intermediate matrix as a function of said first group by minimising an epipolar error; - selecting (27) a second group of eight pairs of pixels distinct from said first group of eight pairs of pixels; - determining (28) a second intermediate matrix as a function of said second group by minimising an epipolar error; - determining (29) a first average matrix as a function of a set of intermediate matrices comprising said first and second intermediate matrices, each coefficient of said first average matrix being defined as an average of the coefficients of each intermediate matrix of said set of intermediate matrices, each of said intermediate matrices having been previously reduced to a matrix of standard equal to 1; - comparing (30) a determined threshold value to a set of differences between the coefficients of said first average matrix coefficients of said first intermediate matrix; - if each difference is less than said threshold value, then said fundamental matrix is defined (31) as being equal to the first average matrix; and - if at least one difference is greater than said threshold value, then the method comprises the following steps, with a set of groups comprising said first and second groups: A- Selection of a new group of eight pairs of pixels distinct from each group of said set of groups; B- Determination of a new intermediate matrix according to said new group by minimising an epipolar error; C- Addition of said new intermediate matrix to said set of intermediate matrices; D- Determination of a new mean matrix as a function of said set of intermediate matrices, each coefficient of said first mean matrix being defined as an average of the coefficients of each intermediate matrix of said set of intermediate matrices, each of said intermediate matrices having been previously reduced to a matrix of standard equal to 1; E- Comparison of said determined threshold value with a set of differences between coefficients of said new mean matrix and coefficients of said first mean matrix: - if each difference is less than said threshold value then the fundamental matrix is defined (30) as said new mean matrix; and - if at least one difference is greater than said threshold value, then said new group of eight pairs of pixels is added to the set of groups and steps B to E are reiterated by considering said new mean matrix as replacing said first mean matrix.

5. Method according to one of cor which said depths are calculated by a convolutional machine-learning neurone network.

6. Method according to one of claims 1 to 5, further comprising a step of determining (24) a first visibility mask associated with said first image and representative of a third set of pixels of said first image having at least one corresponding pixel in said second image, said first visibility mask being determined by the optical flow compute method, the selection (25) of said first group also being a function of said first visibility mask.

7. Method according to one of claims 1 to 6, further comprising a step of determining (35) a second visibility mask associated with said first image and representative of a fourth set of pixels of said first image having at least one corresponding pixel in said third image, said second visibility mask being determined by the optical flow compute method, the compute (37) of said basic distance also being a function of said second visibility mask.

8. Method according to one of claims 1 to 7, further comprising a step of determining (36) a dynamic object mask associated with said first image and representative of a fifth set of pixels of said first image associated with at least one moving object in said scene, said dynamic object mask being determined by the optical flow compute method, the compute (37) of said basic distance also being a function of said dynamic object mask.

9. Device (4) for scaling a stereoscopic vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 8.

10. Vehicle (10) comprising the device (4) according to claim 9.