Method and system for reconstructing images

Abstract

A method (P) is proposed for reconstructing an image visible from a reference position and having a reference field of view, and using at least two image capture units, each having a position and a field of view and related to a transformation, each transformation is configured to correlate reference coordinates seen from a reference position and a reference field of view with corresponding associated coordinates on the captured image (IMGj), said method comprising the steps of:-capturing (E2) an image via each image capture unit; and-for each captured image, filling (E6) the reconstructed image (IMG0) based on the captured image using the corresponding associated coordinates. A system for reconstructing a relevant image is also presented.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to the field of image reconstruction.

[0002] More particularly, the present invention relates to a method for reconstructing an image.

[0003] It also relates to a system for reconstructing an image.

[0004] The invention is particularly advantageously suitable for image reconstruction in the passenger compartment of a motor vehicle. Background Art

[0005] With the growth of wide-angle digital imaging found in multi-purpose mobile phones or drones, performance requirements in wide-angle imaging have increased.

[0006] In panoramic mode, cameras and other imaging devices quickly become limited by aberrations introduced by the optics, particularly spherical aberration.

[0007] In this case, the known single panoramic shot is replaced by image processing for reconstructing a panorama from multiple images with a regular field of view.

[0008] Existing methods are based on identifying objects in the captured image and any fading of the image within its boundaries.

[0009] These methods produce good results for distant objects whose viewpoint barely changes from one image to another. However, they provide very unsatisfactory results for nearby objects. In fact, the reconstructed images contain many artifacts. Summary of the Invention

[0010] Under this background, the present invention proposes a new image processing method.

[0011] In order to overcome the above-mentioned shortcomings of the prior art, the present invention proposes a method for reconstructing an image visible from a reference position and having a reference field of view, and using at least two image capture units, each image capture unit having a position and a field of view, and associated with a transformation, each transformation being configured to relate reference coordinates seen from the reference position and having the reference field of view to corresponding associated coordinates on the captured image, the image reconstruction method comprising the following steps:

[0012] - capturing an image via each image capture unit; and

[0013] - For each captured image, fill the reconstructed image based on the captured image using the corresponding associated coordinates.

[0014] Thus, according to the invention, the reconstructed image is based on mathematical matching rather than on object recognition.The invention allows avoiding a large number of artifacts, in particular for nearby objects.

[0015] For example, provision may be made for calculating the value of a reference coordinate pixel of the reconstructed image based on at least one value of a corresponding associated coordinate pixel of the captured image.

[0016] Furthermore, the method for reconstructing an image may comprise, before the image capturing step, a phase of defining a transformation, during which phase the transformation is defined with the aid of at least one point of the environment in which the image capturing unit is located, wherein said point is modeled in a three-dimensional model of the environment.

[0017] Furthermore, during the phase of defining the transformation, by projecting said at least one point of the three-dimensional model onto the captured images, at least one point of the environment may be visible in one of the captured images at the pixel for which the associated coordinates are determined.

[0018] During the phase of defining the transformation, at least one point of the environment is visible in the reconstructed image at the pixel for which the associated coordinates are determined, by projecting said at least one point of the three-dimensional model onto a reference plane representing a visible image from a reference position and having a reference field of view.

[0019] Thus, the position of at least one point of the environment is known in both the captured and reconstructed images by means of the three-dimensional model. In this way, a match is established between the reference coordinates and the corresponding associated coordinates. This match allows the aforementioned transformation to be defined.

[0020] Furthermore, the stage of defining transformations may comprise a step in which each transformation is defined on a pixel of the reconstructed image using a grid-based transformation method based on relative coordinates of at least one point of the environment and a reference.

[0021] In one embodiment, an intermediate step is provided, wherein image processing using a transformation is applied to the corresponding captured image in order to obtain a transformed image.

[0022] In this same embodiment, the coordinates of a visible object on the transformed image may be the same as the coordinates of the same object on the reconstructed image.

[0023] Finally, the environment in which the imaging device is located may be the passenger compartment of a motor vehicle.

[0024] The present invention also relates to a system for reconstructing an image visible from a reference position and having a reference field of view, comprising at least two image capture units, each having a position and a field of view, associated with a transformation and configured to capture an image, each transformation configured to relate reference coordinates seen from the reference position and having the reference field of view to corresponding associated coordinates on the captured image, and a control unit configured to fill in the reconstructed image based on the captured image and in accordance with the coordinate transformation.

[0025] The various features, variations and embodiments of the present invention may be combined with one another in various combinations, as long as they are not mutually incompatible or mutually exclusive. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Furthermore, various other features of the present invention will become apparent from the accompanying description provided with reference to the accompanying drawings, which illustrate non-limiting embodiments of the present invention, in which:

[0027] Figure 1 is a flow chart of the image reconstruction method described herein;

[0028] Figure 2 is a flow chart of the stages of defining a transformation according to the image reconstruction method described herein;

[0029] Figure 3 is a three-dimensional model of a vehicle passenger compartment used in one embodiment of the image reconstruction method;

[0030] Figure 4 yes Figure 1 A schematic diagram of the projection of the three-dimensional model onto an image captured by an image capture unit of the reconstruction system is shown;

[0031] Figure 5 is a schematic diagram of a second embodiment of an image reconstruction method; and

[0032] Figure 6 is a schematic diagram of the image reconstruction system described here.

[0033] It should be noted that in these figures, structural and / or functional elements common to the various variations may have the same reference numerals. DETAILED DESCRIPTION

[0034] like Figure 1 The method for reconstructing an image according to the invention, schematically shown in FIG and designated as a whole by the reference symbol P, is implemented by a system for reconstructing an image 100. It allows an image visible from a reference position and having a reference field of view (called reconstructed image IMG0) to be reconstructed based on several initial images (called captured images) captured at different positions and with different fields of view, i.e. taken (by an image capture unit).

[0035] like Figure 6 As shown, a system 100 for reconstructing an image includes a control unit 110 and K image capture units. The embodiment described herein includes two image capture units 121 and 122, so K = 2. Each image capture unit 120j (where j ranges from 1 to K) has a position and a field of view, and is associated with a transformation Tj.

[0036] In the example described herein, reconstruction system 100 is placed in the passenger compartment of a motor vehicle. In this case, first image capture unit 121 is placed on the interior ceiling of the vehicle, above the first row of vehicle seats. In this case, second image capture unit 122 is placed on the interior ceiling of the vehicle, above the second row of vehicle seats. Depending on the size of the passenger compartment, reconstruction system 100 may include additional image capture units.

[0037] In this case, the image reconstruction method P allows combining the two captured images IMGj into a single reconstructed image IMG0. The reconstructed image IMG0 is defined as the image captured by a virtual camera C0 having a reference field of view and located at a reference position, in this case on the ceiling of the passenger compartment and in the center of the vehicle.

[0038] The virtual camera CO may simulate an image capture unit comprising, for example, a fisheye objective lens. A fisheye objective lens is an objective lens with a very short focal length (from a few millimeters to tens of millimeters) and / or a very wide field of view (typically greater than 100°).

[0039] For example, in this case the virtual camera CO allows to simulate an equidistant fisheye objective, i.e. has the representation function of the objective, where r corresponds to the image position of the object (the distance to the center of the image), α corresponds to the angular position of the object (the angle between the object and the optical axis of the objective), and f is the focal length of the lens.

[0040] To this end, each transformation Tj is configured to relate the reference coordinates of the reconstructed image IMG0 seen from a reference position and with a reference field of view to the corresponding associated coordinates on the captured image IMGj. These transformations allow a match to be established between the pixels of the captured image IMGj and the pixels of the reconstructed image IMG0.

[0041] The image capture unit 120j is in this case a camera including a fisheye objective.

[0042] The image reconstruction method P starts with a step E2 during which each image capturing unit 120j captures an image IMGj.

[0043] The image reconstruction method continues with step E4, during which the reconstructed image IMG0 is initialized. The reconstructed image IMG0 is constructed as a pixel matrix. For example, the pixels are set to initial values that may correspond to black. In this case, a pixel is defined according to the (R, G, B) standard and may include three values corresponding to red (R), green (G), and blue (B).

[0044] Each R, G or B channel can be used as an independent matrix. For example, during this method, the channels will be processed independently of each other.

[0045] Alternatively, pixels may be defined according to another standard, such as the YUV standard, where Y represents a luminance signal and U and V represent chrominance. In another embodiment, pixels may be defined according to a standard that takes into account light in the infrared band.

[0046] The reconstructed image IMG0 may be initialized by the control unit 110 .

[0047] The method for reconstructing an image then continues with step E6 during which a reconstructed image IMG0 is constructed based on the captured images IMGj.

[0048] For each captured image IMGj, the reconstructed image IMG0 is filled based on the captured images IMG1, IMG2 using the corresponding associated coordinates.

[0049] For example, for a pixel of the reconstructed image IMG0 with reference coordinates (u,v), its R, G and B values are calculated based on the R, G and B values of the pixels of the captured images IMG1, IMG2, with corresponding associated coordinates calculated by transformations T1, T2.

[0050] therefore, , where R0 corresponds to the channel R of the reference image IMG0 , R1 corresponds to the channel R of the first captured image IMG1 , R2 corresponds to the channel R of the second captured image IMG2 , and f is a function allowing the calculation of pixel values.

[0051] For example, in this case, f is the mean, and .

[0052] The G and B values can be determined in the same way.

[0053] Alternatively, the R value of the reference coordinate pixel on the reconstructed image is a weighted average of the R values of the pixels of the captured image IMGj with corresponding associated coordinates. The weight of the average can be determined according to the distance of the pixel to the edge of the corresponding captured image IMGj.

[0054] If the corresponding associated coordinates do not correspond to integers, they may be rounded to the nearest integer.

[0055] As a variant, if the corresponding associated coordinate is not an integer, the average value of the pixels at the adjacent coordinates can be considered. For example, a bilateral filtering algorithm can be used.

[0056] If the corresponding associated coordinates do not exist on one of the captured images IMGj, the captured image IMGj is not considered. In this case, the other captured images are averaged. For example, in this case, if the coordinate T1(u,v) does not exist on the first captured image IMG1, the pixel value of the coordinate (u,v) on the reconstructed image IMG0 will correspond to the pixel value of the coordinate T2(u,v) of the second captured image.

[0057] Finally, if the corresponding associated coordinates are not defined on any corresponding captured image IMGj, no value is copied to the pixel of the reference coordinates of the reconstructed image IMG0.

[0058] This step of filling the reconstructed image E6 is performed by the control unit 110 .

[0059] Preferably, the method P for reconstructing an image may comprise, prior to the above steps, a stage D of defining a transformation.

[0060] For example, phase D of defining the transformation can advantageously be performed when designing the vehicle. In this case, phase D of defining the transformation is performed once.

[0061] Thus, when using the image reconstruction system 100 , steps E2 to E6 are performed in real time when the image is captured using the transformations T1 , T2 defined upstream during stage D of defining the transformations. This saves computing resources and time.

[0062] Figure 2 An example of defining stage D of a transformation is schematically shown.

[0063] The stage D of defining the transformation may begin with a step D2 during which a set of N points Pi (x, y, z) of the environment in which the image capture unit is located is selected on a three-dimensional model of said environment (hereinafter referred to as 3D model) 10, where i ranges from 1 to N. N is at least greater than 1. Preferably, N may be greater than 2000, or even greater than 10000. For example, in practice, between 20000 and 50000 points may be used.

[0064] In this case, in the case where the image capture unit is located in a motor vehicle, the 3D model 10 represents at least a portion of the passenger compartment of the vehicle. Figure 3 Shown in.

[0065] Preferably, and in order to avoid artifacts when reconstructing the image IMG0 , no moving objects of the vehicle, such as a steering wheel or seat backs, are shown on the 3D model 10 .

[0066] For the same reason, small objects such as buttons or door handles are not shown in this case.

[0067] Each point Pi(x,y,z) of the set of points of the environment is preferably visible in at least one captured image IMGj.

[0068] The transformation determination phase D continues with a step D4 in which the associated coordinates Cs,i(u,v) are determined at which each point Pi(x,y,z) is visible on one of the captured images IMGj. The associated coordinates Cs,i(u,v) are calculated, for example, by projecting the 3D model onto each captured image IMGj.

[0069] The 3D model is digitally projected onto the image with the help of digitally controllable position and field of view settings. It should be noted that these settings are adjusted using known parameters of the corresponding image capture unit. In this case, these parameters include focal length, geometric distortion, position and orientation of the image capture unit. Figure 4 Schematically shown in FIG. 1 is a projection of a 3D model of a passenger compartment of a car onto an image IMG1 captured by the first capturing unit 121 .

[0070] Phase D of determining the transformation continues with step D6, in which a virtual camera C0 is defined. The virtual camera C0 is defined at a reference position and has a reference field of view. The virtual camera C0 can also be defined by parameters such as the focal length, field of view, geometric distortion, position or even orientation of the camera.

[0071] The definition of the virtual camera C0 also allows the definition of a reference virtual image IMG0ref. This reference virtual image IMG0ref is an image captured by the virtual camera C0. It can be identified as a reference plane representing a visible image from a reference position and with a reference field of view.

[0072] Phase D of determining the transformation continues in step D8 by determining the reference coordinates C0,i(u,v) at which each point Pi(x,y,z) is visible on the reference virtual image IMG0ref. For example, the reference coordinates C0,i(u,v) are calculated by projecting each point of the 3D model onto the reference virtual image IMG0ref.

[0073] The stage D of determining the transformations continues with a step D10 in which each transformation Tj is defined on all pixels of the reference virtual image IMG0ref. For example, each transformation Tj can be defined by interpolating the matches established in the previous step between the reference coordinates C0,i(u,v) and the corresponding associated coordinates Cs,i(u,v) of the set of points Pi(x,y,z), where i ranges from 1 to N.

[0074] The transformation is preferably defined by a grid-based transformation. The set of points Pi (x, y, z) of the environment on the virtual image IMG0ref defines the peaks of the grid. These peaks define the grid. Interpolation is defined linearly for each grid.

[0075] Other data processing methods may be used in order to fully define the transformation based on the reference coordinates C0,i(u,v) and the corresponding context coordinates Cs,i(u,v).

[0076] Using these defined coordinates using a 3D model allows for high-quality reconstruction. It should be noted that performing reconstruction of close objects (between 0 and 2 meters) for image capture unit 120j can be very complex and can produce numerous artifacts. In practice, the angle at which image capture unit 120j sees nearby objects can vary significantly from one image capture unit 120j to another. However, this angle hardly changes for distant objects.

[0077] Figure 5 A second embodiment of a method for reconstructing an image using an intermediate step is shown. The intermediate step may be performed after step E2 of capturing images IMG1 and IMG2 using each image capture unit 121, 122. During this intermediate step, relevant image processing operations TI1, TI2 are applied to the corresponding captured images IMG1, IMG2 on the image capture units 121, 122 using transformations T1, T2. These image processing operations use the inverse functions of the transformations T1, T2 defined above to obtain reference coordinates based on the corresponding associated coordinates.

[0078] Thus, for each pixel of the captured image IMGj with coordinates (uj, vj), where j ranges from 1 to K and represents a point Pi(x,y,z) of the environment, new coordinates (u0, v0) are defined at which the point Pi(x,y,z) can be seen by the virtual camera C0. This operation is performed for all pixels of the captured images IMG1 and IMG2 in order to obtain the transformed images IMG'1, IMG'2.

[0079] The coordinates of the visible object in the transformed images IMG'1, IMG'2 may therefore be identical to the coordinates of the same object in the reconstructed image IMG0.

[0080] The reconstructed image IMG0 can then be filled directly with the aid of the transformed image without requiring new changes in coordinates.

[0081] Therefore, for the pixel at coordinate (u,v) of the reconstructed image IMG0:

[0082] , where R0 corresponds to the channel R of the reference image IMG0, R'1 corresponds to the channel R of the first transformed image IMG'1, R'2 corresponds to the channel R of the second transformed image IMG'2, and f is a function that allows the calculation of pixel values. For example, in this case, f is the mean value.

[0083] If the coordinate (u, v) does not exist in one of the transformed images IMG'j, it is not taken into account. For example, if the coordinate (u, v) does not exist in the first transformed image IMG'1, the pixel value of the coordinate (u, v) of the reconstructed image IMG0 is equal to the pixel value of the coordinate (u, v) of the second transformed image IMG'2.

[0084] The values of the G and B channels can be determined in the same way.

[0085] Alternatively, other standards for defining pixels may be used, such as the YUV standard.The method described here is then applied in a similar manner.

Claims

1. A method (P) for reconstructing an image visible from a reference position and having a reference field of view, using at least two image capture units (121, 122), each image capture unit (120j) having a position and a field of view, and associated with a transformation (Tj), each transformation (Tj) being configured to relate reference coordinates ((u,v)) seen from the reference position and having the reference field of view to corresponding associated coordinates (Tj(u,v)) on a captured image (IMGj), the image reconstruction method (P) comprising the following steps: - capturing (E2) an image (IMGj) via each image capture unit (120j); and - For each captured image (IMGj), reconstruct the image (IMG0) by filling (E6) based on the captured images (IMG1, IMG2) using the corresponding associated coordinates (Tj(u,v)).

2. The method for reconstructing an image according to claim 1, wherein: The reference coordinate pixel ((u,v)) value of the reconstructed image (IMG0) is calculated based on at least one of the corresponding associated coordinate pixel (Tj(u,v)) values of the captured image (IMGj) obtained by the corresponding transformation (Tj).

3. Method (P) for reconstructing an image according to claim 2, wherein The reference coordinate pixel ((u, v)) value of the reconstructed image (IMG0) is calculated with the aid of the weighted average value of the corresponding associated coordinate pixel (Tj(u, v)) value of the captured image (IMGj), and the weight is determined as a function of the distance from the corresponding associated coordinate pixel (Tj(u, v)) to the edge of the corresponding captured image IMGj.

4. The method (P) for reconstructing an image according to any one of claims 1 to 3, comprising, before the image capture step (E2), a phase (D) of defining a transformation, during which the transformation (T1, T2) is defined with the aid of at least one point (Pi(x,y,z)) of the environment in which the image capture unit (121, 122) is located, wherein said point (Pi(x,y,z)) is modeled by a three-dimensional model (10) of the environment.

5. Method (P) for reconstructing an image according to claim 4, wherein During the phase (D) of defining the transformation, by projecting said at least one point (Pi(x,y,z)) of the three-dimensional model (10) onto said captured image, at least one point (Pi(x,y,z)) of the environment is visible in one of the captured images (IMGj) at the pixel for which the corresponding associated coordinates (Cs,i(u,v)) are determined.

6. Method (P) for reconstructing an image according to any one of claims 4 to 5, wherein During a phase (D) of defining a transformation, at least one point (Pi(x,y,z)) of the environment is visible in a reconstructed image (IMG0) at a pixel for which associated coordinates (Cs,i(u,v)) are determined, by projecting said at least one point (Pi(x,y,z)) of the three-dimensional model (10) onto a reference plane representing a visible image from a reference position and having a reference field of view.

7. Method (P) for reconstructing an image according to any one of claims 4 to 6, wherein The stage (D) of defining transformations further comprises a step wherein each transformation (Tj) is defined on a pixel of the reconstructed image (IMG0) using a grid-based transformation method based on reference coordinates (C0,i(u,v)) and corresponding associated coordinates (Cs,i(u,v)) of at least one point (Pi(x,y,z)) of the environment.

8. Method (P) for reconstructing an image according to any one of claims 4 to 7, wherein The environment in which the image capture units (121, 122) are located is a passenger compartment of a motor vehicle.

9. Method (P) for reconstructing an image according to any one of claims 1 to 8, wherein The reference field of view corresponds to the field of view of a fisheye lens.

10. Method (P) for reconstructing an image according to any one of claims 1 to 9, comprising an intermediate step in which image processing using a transformation (T1, T2) is applied to the corresponding captured image (IMG1, IMG2) in order to obtain a transformed image (IMG1, IMG'2).

11. Method (P) for reconstructing an image according to claim 10, wherein The coordinates of a visible object on the transformed images (IMG'1, IMG'2) are the same as the coordinates of the same object on the reconstructed image (IMG0).

12. A system (100) for reconstructing an image visible from a reference position and having a reference field of view, comprising at least two image capture units (121, 122) and a control unit (110), each image capture unit (120j) having a position and a field of view, associated with a transformation (Tj) and configured to capture images (IMGj), each transformation (Tj) being configured to associate reference coordinates (C0,i(u,v)) as seen from the reference position and having the reference field of view with corresponding associated coordinates (Cs,i(u,v)) on the captured image (IMGj), the control unit (110) being configured to, for each captured image (IMGj), fill a reconstructed image (IMG0) based on the captured images (IMG1, IMG2) using the corresponding associated coordinates (Cs,i(u,v)).

Images