Method and system for reconstructing an image

Abstract

The invention relates to a method for reconstructing an image (P) that is visible from a reference position and with a reference field of view, and using at least two image capture units, each image capture unit having a position and a field of view, and being associated with a transformation, each transformation being configured to associate reference co-ordinates seen from the reference position and with the reference field of view with corresponding associated co-ordinates in the captured image (IMGj), the method comprising the following steps: - each of the image capture units capturing (E2) an image; and - filling (E6) the reconstructed image (IMG0) from the captured images by using, for each captured image, the corresponding associated co-ordinates. The invention also relates to an associated system for reconstructing an image.

Description

Description Title of the invention: Method and system for reconstructing an image Technical field

[0001] The present invention relates to the technical field of image processing, and in particular the field of image reconstruction.

[0002] The invention relates more particularly to a method of reconstructing an image.

[0003] It also concerns an image reconstruction system.

[0004] The invention finds a particularly advantageous application in the reconstruction of an image inside the passenger compartment of a motor vehicle. Technological background

[0005] With the rise of wide-angle digital imaging such as in multi-function mobile phones or drones, the performance requirements for wide-angle imaging are increasing.

[0006] In panoramic mode, cameras and other imaging devices are quickly limited due to aberrations, particularly spherical aberrations, created by the optical elements.

[0007] In this context, it is known to replace a single panoramic shot with image processing allowing to reconstruct panoramas from several images having a classic field of view.

[0008] Existing methods are based on recognizing objects in captured images and blending images into their boundaries.

[0009] These methods achieve good results with distant objects whose viewing angles change little from one image to the next. However, they offer very unsatisfactory results with nearby objects. Indeed, the reconstructed images then contain numerous artifacts. Summary of the invention

[0010] In this context, the present invention proposes a novel image processing method.

[0011] In order to overcome the aforementioned drawbacks of the prior art, the present invention provides a method for reconstructing an image visible from a reference position and with a reference field of view and using at least two image capture units, each image capture unit having a position and a field of view, and being associated with a transformation, each transformation being configured to associate reference coordinates seen from the reference position and with the reference field of view with corresponding associated coordinates on the captured image, said image reconstruction method comprising the following steps: - capturing an image by each of the image capture units, and - filling the reconstructed image from the captured images using, for each captured image, the corresponding associated coordinates.

[0012] Thus, thanks to the invention, the reconstructed image is based on mathematical correspondences rather than on object recognition. The invention makes it possible to avoid many artifacts, particularly with nearby objects.

[0013] For example, it may be provided that a value of a reference coordinate pixel of the reconstructed image is calculated from at least one of the values ​​of the corresponding associated coordinate pixel of the captured images.

[0014] Furthermore, the method for reconstructing an image may comprise, prior to the image capture step, a phase for defining the transformations during which the transformations are defined using at least one point of an environment in which the image capture units are positioned, said point being modeled in a three-dimensional model of the environment.

[0015] In addition, the at least one point of the environment may be visible in one of the captured images on a pixel whose corresponding associated coordinates are determined, during the phase of defining the transformations, by projecting said at least one point of the three-dimensional model onto the captured image.

[0016] The at least one point of the environment may be visible in the reconstructed image on a pixel whose corresponding associated coordinates are determined, during the phase of defining the transformations, by projecting said at least one point of the three-dimensional model onto a reference plane representing the image visible from the reference position and with the reference field of view.

[0017] Thus, the position of at least one point of the environment is known on the captured images and on the reconstructed image thanks to the three-dimensional model. In this way, a correspondence is established between the reference coordinates and the corresponding associated coordinates. This correspondence makes it possible to define the aforementioned transformations.

[0018] Furthermore, the transformation definition phase may include a step in which each transformation is defined on pixels of the reconstructed image by a mesh transformation method from the associated coordinates and references of at least one point of the environment.

[0019] In one embodiment, an intermediate step in which image processing using the transformations is applied to the corresponding captured images to obtain transformed images.

[0020] In this same embodiment, the coordinates of an object visible on the transformed images may be identical to the coordinates of this same object on the reconstructed image.

[0021] Finally, the environment in which the imaging device is positioned may be the passenger compartment of a motor vehicle.

[0022] The invention also relates to a system for reconstructing an image visible from a reference position and with a reference field of view comprising at least two image capture units and a control unit, each image capture unit having a position and a field of view, being associated with a transformation and being configured to capture an image, each transformation being configured to associate reference coordinates seen from the reference position and with the reference field of view with corresponding associated coordinates on the captured image, the unit of command being configured to fill the reconstructed image from the captured images and respecting the coordinate transformation.

[0023] The various features, variations and embodiments of the invention may be combined with each other in various combinations to the extent that they are not incompatible or mutually exclusive. Brief description of the figures

[0024] In addition, various other characteristics of the invention emerge from the appended description given with reference to the drawings which illustrate non-limiting embodiments of the invention and where:

[0025] [Fig. 1] is a flowchart of the image reconstruction process described here,

[0026] [Fig. 2] is a flowchart of the transformation definition phase according to the image reconstruction method described here,

[0027] [Fig. 3] is a three-dimensional model of the passenger compartment of a vehicle as used in one embodiment of the image reconstruction method,

[0028] [Fig. 4] is a schematic representation of a projection of the three-dimensional model shown in Figure 1 onto images captured by image capture units of a reconstruction system,

[0029] [Fig. 5] is a schematic representation of a second embodiment of the image reconstruction method, and

[0030] [Fig. 6] is a schematic representation of the image reconstruction system described here.

[0031] It should be noted that in these figures the structural and / or functional elements common to the different variants may have the same references. Detailed description

[0032] A method of reconstructing an image according to the invention, as shown schematically in Figure 1 and designated as a whole by the reference P, is implemented by an image reconstruction system 100. It makes it possible to reconstruct an image visible from a reference position and with a field reference view, called IMGO reconstructed image, from several initial images, called captured images, which are captured, i.e. taken, (by image capture units) at different positions and with different fields of view.

[0033] The image reconstruction system 100, shown in FIG. 6, comprises a control unit 110 and K image capture units. The embodiment described here comprises two image capture units 121, 122, thus, K = 2. Each image capture unit 120j, j ranging from 1 to K, has a position and a field of view and is associated with a transformation Tj, j ranging from 1 to K.

[0034] In the example described here, the reconstruction system 100 is placed in the passenger compartment of a motor vehicle. The first image capture unit 121 is here placed on the interior ceiling of the vehicle, above the first row of automobile seats. The second image capture unit 122 is here placed on the interior ceiling of the vehicle, above the second row of automobile seats. Depending on the size of the passenger compartment, the reconstruction system 100 could comprise other image capture units.

[0035] The image reconstruction method P here allows the two captured images IMGj to be combined into a single reconstructed image IMGO. The reconstructed image IMG0 is defined as an image captured by a virtual camera C0 with a reference field of view and located at the reference position, here on the ceiling of the passenger compartment and in the center of the vehicle.

[0036] The CO virtual camera can, for example, simulate an image capture unit comprising a hypergonal lens, known in particular as a "fisheye". A hypergonal lens is a lens with a very short focal length (from a few millimeters to a few tens of millimeters) and / or a very wide field angle (generally greater than 100°).

[0037] For example, here, the virtual camera CO allows to simulate an equidistant hypergonal lens, that is to say a lens having a representation function r = fa where r corresponds to a position of an image of an object (the distance to the center of the image), a corresponds to an angular position of the object (the angle between the object and the optical axis of the lens) and f being the focal length of the lens.

[0038] For this, each transformation Tj is configured to associate reference coordinates of the reconstructed image IMGO seen from the reference position and with the reference field of view with respectively corresponding associated coordinates on the captured image IMGj. These transformations make it possible to establish a correspondence between the pixels of the captured images IMGj and the pixels of the reconstructed image IMGO.

[0039] The 120j image capture units here are cameras with hypergon lenses.

[0040] The image reconstruction process P begins at step E2 in which each image capturing unit 120j captures an image IMGj.

[0041] The image reconstruction process continues in step E4 where the reconstructed IMGO image is initialized. The reconstructed IMGO image is constructed as a pixel matrix. The pixels are set to an initial value that can correspond to black, for example. The pixels are here defined according to the (R, G, B) standard and can contain three values ​​corresponding to red R, green G and blue B.

[0042] Each R, G or B channel can act as an independent matrix. For example, during the process, the channels will be processed independently of each other.

[0043] Alternatively, the pixels may be defined according to another standard, such as the YUV standard, for example, in which Y represents the luminance signal, and U and V represent the chrominance. In another embodiment, the pixels may be defined according to a standard that takes into account light in the infrared.

[0044] The initialization of the reconstructed image IMGO can be performed by the control unit 110.

[0045] The image reconstruction process then continues to step E6 in which the reconstructed image IMGO is constructed from the captured images IMGj.

[0046] The reconstructed image IMGO is filled from the captured images IMG1JMG2 using, for each captured image IMGj, the corresponding associated coordinates.

[0047] For example, for a pixel of the reconstructed image IMGO having reference coordinates (u,v), its R, G and B values ​​are calculated from the R, G and B values ​​of the pixels of the captured images IMG1, IMG2 with corresponding associated coordinates calculated using the transformations Tl, T2.

[0048] Thus R0(u, v) = f R1(T1(U, V)), R2(T2 U, V))^ in which R0 corresponds to the R channel of the reference image IMGO, RI corresponds to the R channel of the first captured image IMG1, R2 corresponds to the R channel of the second captured image IMG2 and f is a function which allows to calculate the value of the pixel.

[0049] For example, here f is an average and (a, h) =

[0050] The G and B values ​​can be determined identically.

[0051] Alternatively, the R value of a reference coordinate pixel in the reconstructed image is a weighted average of the R values ​​of the pixels in the captured images IMGj of corresponding associated coordinates. The weighting of the average may be determined based on the pixel's distance from the edge of the corresponding captured image IMGj.

[0052] If the corresponding associated coordinates do not equal an integer, then they can be rounded to the nearest integer.

[0053] Alternatively, if the corresponding associated coordinates are not an integer, then one can consider using an average of the pixels of neighboring coordinates. For example, a bilateral filter algorithm can be used.

[0054] If the corresponding associated coordinates do not exist on one of the captured images IMGj, then the captured image IMGj is not taken into account. In this case, the average is performed on the other captured images. For example, here, if the coordinates Tl(u,v) do not exist on the first captured image IMG1, then the value of the pixel with coordinates (u,v) on the reconstructed image IMGO will correspond to the value of the pixel with coordinates T2(u,v) of the second captured image.

[0055] Finally, if the corresponding associated coordinates are not defined on any of the corresponding captured images IMGj, no value is copied into the reference coordinate pixel of the reconstructed image IMGO.

[0056] This step of filling the reconstructed image E6 is carried out by the control unit 110.

[0057] Preferably, the method of reconstructing an image P may comprise a phase of defining the transformations D prior to the steps described above.

[0058] For example, the phase of defining D transformations can advantageously be carried out during the vehicle design. The phase of defining D transformations is carried out here only once.

[0059] Thus, when using the image reconstruction system 100, steps E2 to E6 are performed in real time during an image capture using the transformations T1, T2 defined upstream during the phase of defining the transformations D. This allows a saving of computing resources and therefore of time.

[0060] An example of the D transformation definition phase is shown schematically in Figure 2.

[0061] The phase of defining the transformations D can begin at step D2 in which a set of N points Pi(x,y,z), i ranging from 1 to N, of the environment in which the image capture units are positioned is selected on a three-dimensional model (hereinafter 3D model) 10 of said environment. N is at least greater than 1. Preferably, N can be greater than 2000, or even greater than 10,000. In practice, for example, between 20,000 and 50,000 points can be used.

[0062] Here, the image capture units being positioned in a motor vehicle, the 3D model 10 represents at least part of the passenger compartment of the vehicle. This 3D model 10 is represented in figure 3.

[0063] Preferably, and in order to avoid the creation of artifacts during the reconstruction of the IMG0 image, the moving objects of the vehicle, such as the steering wheel or the seat backs, are not represented on the 3D model 10.

[0064] For the same reason, small objects such as buttons or door handles are not shown here.

[0065] Each point Pi(x,y,z) of the set of points in the environment is preferably visible in at least one of the captured images IMGj.

[0066] The phase of determining the transformations D continues in step D4 in which the associated coordinates Cj,i(u,v) at which each point Pi(x,y,z) is visible on one of the captured images IMGj are determined. The associated coordinates Cj,i(u,v) are for example calculated by projecting the 3D model onto each of the captured images IMGj.

[0067] The projection of the 3D model onto an image is carried out digitally using position and field of view settings that can be digitally controlled. These settings are adjusted in particular using known parameters of the corresponding image capture unit. These parameters here include the focal length, geometric distortion, position and orientation of the image capture unit. The projection of the 3D model of the car interior onto the captured image IMG1 by the first capture unit 121 is shown schematically in FIG. 4.

[0068] The phase of determining the transformations D continues in step D6 in which the virtual camera C0 is defined. The virtual camera C0 is defined at the reference position and with the reference field of view. The virtual camera CO can also be defined using parameters such as the focal length, the field of view, the geometric distortion, the position or the orientation of the camera.

[0069] The definition of the virtual camera C0 also allows us to define a virtual reference image IMGOref. This virtual reference image IMGOref is the image captured by the virtual camera C0. It can be identified as a reference plane representing the image visible from the reference position and with the reference field of view.

[0070] The phase of determining the transformations D continues in step D8 at the reference coordinates C0,i(u,v) at which each point Pi(x,y,z) is visible on the virtual reference image IMGOref are determined. The coordinates of reference C0,i(u,v) are for example calculated by projecting each point of the 3D model onto the virtual reference image IMGOref.

[0071] The phase of determining the transformations D continues in step D10 in which each transformation Tj is defined on the set of pixels of the virtual reference image IMGOref. For example, each transformation Tj can be defined by interpolation from the correspondences established in the previous steps between the reference coordinates C0,i(u,v) and the corresponding associated coordinates Cj,i(u,v) on the set of points Pi(x,y,z), i ranging from 1 to N.

[0072] The transformation is preferably defined by mesh (or "mesh-based transformation" in English). The set of points Pi(x,y,z) of the environment on the virtual image IMGOref defines the vertices of the mesh. These vertices define meshes. The interpolation is defined linearly for each mesh.

[0073] Other data processing methods can be used to completely define the transformation from the reference coordinates C0,i(u,v) and the corresponding associated coordinates Cj,i(u,v).

[0074] Using these coordinates defined using a 3D model allows for high-quality reconstruction. In particular, the reconstruction of objects close to the 120j image capture units (between 0 and 2 meters) can be very complicated to perform and generate many artifacts. Indeed, the angle at which a nearby object is seen by a 120j image capture unit can be very different from one 120j image capture unit to another. On the other hand, this angle varies little with distant objects.

[0075] A second embodiment of the method for reconstructing an image using an intermediate step is shown in Figure 5. The intermediate step can take place following the step E2 of capturing the images IMG1 and IMG2 by each of the image capture units 121, 122. During this intermediate step, image processing operations TU, TI2 associated with the image capture units 121, 122 and using the transformations T1, T2 are applied to the corresponding captured images IMG1, IMG2. These image processing operations use the functions inverses of the transformations Tl, T2 defined above, in order to obtain the reference coordinates from the corresponding associated coordinates.

[0076] Thus, for each pixel with coordinates (uj,vj) of a captured image IMGj, j ranging from 1 to K, representing a point Pi(x,y,z) of the environment, new coordinates (u0,v0) are defined at which the point Pi(x,y,z) is visible by the virtual camera CO. This operation is performed for all pixels of the captured images IMG1 and IMG2 in order to obtain transformed images IMG'l, IMG'2.

[0077] The coordinates of an object visible on the transformed images IMG'l, IMG'2 can therefore be identical to the coordinates of this same object on the reconstructed image IMG0.

[0078] The reconstructed image IMG0 can then be filled directly using the transformed images, without any further changes in coordinates.

[0079] So for a pixel with coordinates (u,v) of the reconstructed image IMG0: R0(u, v) = (Æ'l(it, v), R'2(u, v)) where R0 corresponds to the R channel of the reference image IMG0, R'1 corresponds to the R channel of the first transformed image IMG'l, R'2 corresponds to the R channel of the second transformed image IMG'2 and f is a function that calculates the pixel value. For example, here f is an average.

[0080] If the coordinates (u,v) do not exist on one of the transformed images IMG'j, then this one is not taken into account. For example, if the coordinates (u,v) do not exist on the first transformed image IMG'l, then the value of the pixel with coordinates (u,v) of the reconstructed image IMG0 is equal to the value of the pixel with coordinates (u,v) of the second transformed image IMG'2.

[0081] The values ​​of the G and B channels can be determined identically.

[0082] Alternatively, other standards for defining pixels can be used, such as the YUV standard. The method described here then applies in a similar manner.

Claims

Claims 1. A method for reconstructing an image (P) visible from a reference position and with a reference field of view and using at least two image capture units (121, 122), each image capture unit (120j) having a position and a field of view, and being associated with a transformation (Tj), each transformation (Tj) being configured to associate reference coordinates ((u,v)) seen from the reference position and with the reference field of view with corresponding associated coordinates (Tj(u,v)) on the captured image (IMGj), said image reconstruction method (P) comprising the following steps: - capture (E2) of an image (IMGj) by each of the image capture units (120j), and - filling (E6) of the reconstructed image (IMGO) from the captured images (IMG1, IMG2) using, for each captured image (IMGj), the corresponding associated coordinates (Tj(u,v)).

2. Method for reconstructing an image according to claim 1, wherein a value of a reference coordinate pixel ((u,v)) of the reconstructed image (IMGO) is calculated from at least one of the values ​​of the corresponding associated coordinate pixel (Tj(u,v)) of the captured images (IMGj) obtained by the corresponding transformation (Tj).

3. Method for reconstructing an image (P) according to claim 2, in which the value of a pixel of reference coordinates ((u,v)) of the reconstructed image (IMGO) is calculated using a weighted average of the values ​​of the pixel of corresponding associated coordinates (Tj(u,v)) of the captured images (IMGj), the weighting being determined as a function of a distance of said pixel of corresponding associated coordinates (Tj(u,v)) to the edge of the corresponding captured image IMGj.

4. Method for reconstructing an image (P) according to one of claims 1 to 3, comprising, prior to the image capture step (E2), a phase of defining the transformations (D) during which the transformations (T1, T2) are defined using at least one point (Pi(x,y,z)) of an environment in which the image capture units (121,122) are positioned, said point (Pi(x,y,z)) being modeled by a three-dimensional model (10) of the environment.

5. Method for reconstructing an image (P) according to claim 4, in which the at least one point of the environment (Pi(x,y,z)) is visible in one of the captured images (I MGj) on a pixel whose corresponding associated coordinates (Cj,i(u,v)) are determined, during the phase of defining the transformations (D), by projecting said at least one point (Pi(x,y,z)) of the three-dimensional model (10) onto the captured image.

6. Method for reconstructing an image (P) according to one of claims 4 to 5, in which the at least one point of the environment (Pi(x,y,z)) is visible in the reconstructed image (IMGO) on a pixel whose corresponding associated coordinates (Cj,i(u,v)) are determined, during the phase of defining the transformations (D), by projecting said at least one point (Pi(x,y,z)) of the three-dimensional model (10) onto a reference plane representing the image visible from the reference position and with the reference field of view.

7. Method for reconstructing an image (P) according to one of claims 4 to 6, in which the phase of defining the transformations (D) further comprises a step in which each transformation (Tj) is defined on pixels of the reconstructed image (IMGO) by a mesh transformation method from the reference coordinates (C0,i(u,v)) and the corresponding associated coordinates (Cj,i(u,v)) of the at least one point (Pi(x,y,z)) of the environment.

8. Method for reconstructing an image (P) according to one of claims 4 to 7, in which the environment in which the image capture units (121, 122) are positioned is the passenger compartment of a motor vehicle.

9. Method for reconstructing an image (P) according to one of claims 1 to 8, in which the reference field of view corresponds to the field of view of a hypergonal optical objective.

10. Method for reconstructing an image (P) according to one of claims 1 to 9, comprising an intermediate step in which an image processing using the transformations (T1, T2) are applied to the corresponding captured images (IMG1, IMG2) in order to obtain transformed images (IMG'1, IMG'2).

11. Method for reconstructing an image (P) according to claim 10, in which the coordinates of an object visible on the transformed images (IMG'l, IMG'2) are identical to the coordinates of this same object on the reconstructed image (IMG'l, IMG'2).

12. System for reconstructing an image (100) visible from a reference position and with a reference field of view comprising at least two image capture units (121, 122) and a control unit (110), each image capture unit (120j) having a position and a field of view, being associated with a transformation (Tj) and being configured to capture an image (IMGj), each transformation (Tj) being configured to associate reference coordinates (C0,i(u,v)) seen from the reference position and with the reference field of view with corresponding associated coordinates (Cj,i(u,v)) on the captured image (IMGj), the control unit (110) being configured to fill the reconstructed image (IMG0) from the captured images (IMG1, IMG2) using, for each captured image (IMGj), the corresponding associated coordinates (Cj,i(u,v)).

Images