Modular image interpolation method

Abstract

To provide a user of optical capture means, for example one or more cameras on a motor vehicle, that have visibility restrictions with a comfortable visual experience by depicting areas having incomplete and / or erroneous information correctly.SOLUTION: A method includes the steps of: identifying at least one area of incomplete and / or erroneous display in a render image based on existing visibility restrictions; generating masks that enclose the at least one area of incomplete and / or erroneous display, as masked areas 30; reconstructing image data in unmasked areas 10 of the render image by means of digital inpainting; synthesizing the reconstructed image data, together with the masked areas 30, to produce a correction image; and displaying the completed and / or debugged correction image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for capturing a scene as at least one raw image by at least one optical capture means mounted on a mobile means, The invention further relates to an image processing system and to a means for its use and transportation. [Background technology]

[0002] Mobility means, for example, in particular vehicles, are increasingly being equipped with assistance systems, which also include camera devices that convey to the passengers and in particular the respective vehicle driver a visual impression of the scene outside one or more interior spaces, for example to support, facilitate or monitor a parking process.

[0003] Nowadays, most parking systems use multiple cameras to provide a good impression of the surroundings when parking the vehicle. However, even multiple cameras cannot cover the entire area around the vehicle. For example, there is no camera image of the area under the vehicle. This results in black areas that can be unpleasant for the observer. Summary of the Invention [Problem to be solved by the invention]

[0004] The object of the present invention is therefore to provide a comfortable visual experience for users of optical capture means with limited visibility, for example one or more on-board cameras, by correctly displaying areas with incomplete and / or erroneous information. [Means for solving the problem]

[0005] This problem is solved by a method having the features of claim 1. Further solutions to the problem are an image processing system according to claim 12, a method for its use according to claim 13 as well as a means for transport according to claim 14. According to the invention, in a first step, the erroneous parts of the rendered image are identified based on a visibility limit. In a next step, a mask is generated from the erroneous parts of the rendered image, and the rendered image is repaired only within this mask. Then the image data in the masked areas is repaired by digital inpainting, of which there are several possible solutions. After carrying out this repair, in a final display step, the completed and / or error-corrected modified image is displayed. Thus, in this case, an improved viewing experience is provided to the user by repairing the erroneous data based on the existing data.

[0006] In an advantageous variant, which further improves the user's visual experience, the corrected images are further optimized by post-processing into an optimized image, which is displayed instead of the respective corrected image, whereby any remaining artifacts etc. are smoothed out and the image after image restoration is adjusted to be more comfortable for the eye. For this purpose, the rendered image can be post-processed, for example to increase sharpness, reduce contrast and / or harmonize colors.

[0007] In order to allow the user to react quickly to the image scenery being displayed to him, in an advantageous variant of the method, the rendered image, the restored corrected image or the optimized image are each displayed to the observer as a displayable image in real time or with negligible delay, the refresh rate of the displayable images preferably being at least 5 fps.

[0008] In a manageable variant of the method according to the invention, visibility limits are determined based on at least a three-dimensional model of the respective vehicle and on the placement of the optical capture means in order to identify incomplete and / or erroneous image regions in the rendered image. Different scenes can also be depicted using suitable methods based on the known visibility limits. In particular, in a scene with the presence of one vehicle, a 3D model of the vehicle and the camera housing can be used to recognize parts of the rendered scene that do not have image data.

[0009] For multiple or persistent use, the data on the visibility restrictions, the geometric model (and pattern) of the surrounding area and (particularly preferred) the landscape data already generated once can be prepared in advance in at least one database in an advantageous variant of the method, so that parts of the respective rendered image that do not require restoration in relation to a mask can be easily protected or hidden from further processing in advance.

[0010] In a preferred variation, the image data to be restored can be generated using machine learning methods, by using known and / or already generated image data and possibly an iterative mask.

[0011] It is particularly preferred that the image data can be restored using an artificial neural network that has access to at least one database and is trained using the data.

[0012] In an advantageous variant of the method according to the invention, incomplete and / or erroneous image data can be restored by edge-based methods, whereby edges or object transitions are searched for in the rendered image, whereby the algorithmic processing frequently does not provide closed edge paths, which must then be joined together using additive methods to enclose the object.

[0013] In this case, the edge-based method is preferably the Level Set method, and more preferably the Fast Marching method. The former is a numerical method for approximately tracking geometric objects and their motion, which allows the calculation of curves and surfaces in a spatially constant coordinate system without the need to parameterize the object. The Fast Marching method, a special method for numerically solving boundary value problems, solves boundary value problems by the Eikonal equation, and evolves closed surfaces as functions of time and velocity.

[0014] In a further advantageous variation, edge-based methods can be used for dimensionality reduction or feature extraction, which can then be propagated in incomplete and / or erroneous image regions using diffusion methods to perform predictions of information, using suitable machine learning approaches, such as Markov Random Fields (MRFs), which can be used to segment digital images and classified surfaces, and which can be used to estimate the interactions between components of the field, or their influence on each other.

[0015] As already mentioned above, an image processing system which performs one variation of the above method, for example its use in a parking assistance system of a mobile means which performs a parking maneuver at a limited speed, as well as a mobile means equipped with such a system, particularly preferably also the vehicle itself, solves the task posed.

[0016] The above-mentioned implementations and their developments can be freely combined with each other, as long as it is meaningful. Further possible implementations and developments as well as implementations of the invention also include combinations of the above-mentioned inventive features and the inventive features described below in connection with the embodiments, which are not specifically described. Furthermore, individual aspects added by the person skilled in the art as improvements or supplements to each basic form of the invention are also included.

[0017] The invention will now be explained in more detail with reference to exemplary embodiments illustrated in schematic drawings. [Brief description of the drawings]

[0018] [Figure 1a] A schematic perspective view of the exterior area behind the vehicle captured by the optical capture means is displayed as a rendered image. [Figure 1b] A schematic perspective view of the exterior area behind the vehicle captured by the optical capture means is shown as a corrected image produced by the method according to the invention. [Figure 2a] A schematic perspective view of other exterior areas of the vehicle captured by the optical capture means is displayed as a rendered image. [Figure 2b] A schematic perspective view of another exterior area of ​​the vehicle captured by the optical capture means is displayed as a corrected image (FIG. 2b) produced by the method according to the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The accompanying drawings are intended to provide a further understanding of implementations of the present invention. They serve to illustrate implementations and, in combination with the description, to explain the principles and concepts of the present invention. Other implementations and many of the advantages described above will become apparent with reference to the drawings. Elements of the drawings are not necessarily drawn to scale.

[0020] In the representations of the figures, elements, features and components that are the same, have the same function and have the same effect are - unless otherwise stated - respectively labeled with the same reference numerals.

[0021] 1a and 1b show in schematic perspective a scene of the exterior behind a vehicle captured by an optical capture means. In the rendered image of the scene shown in 1a, a roughly rectangular area 10 is visible in which image data of the scene is incompletely displayed due to the absence of the housing of the optical capture means (not shown), configured as a camera. However, since the dimensions of said housing are known and can be prepared in a database, the missing image data can be restored by the method according to the invention, applying digital inpainting. In so doing, a self-consistent image is generated which takes the whole image as a reference and which provides the user with a better visual experience during subsequent observation.

[0022] Said area 10 where image data is missing is recognized by an edge 20, which separates said area 10 from an area 30 where image data is known. This is shown in the first identification step. In the next step of the method, the area 30 where image data is known is masked, i.e. a mask is generated. This mask surrounds the area 10 of the image that is incomplete and / or erroneous with the area 30 that is masked and not processed, so that the area 10 is repaired, but the area 30 is not processed in order to be correctly captured, rendered and outputtable image data. In the subsequent repair step, the unmasked area of ​​the rendered image is repaired by digital inpainting, and then the corrected image (FIG. 1b) is synthesized together with the masked area. The contour line of the rendered image that touches the edge 20 of the mask continues along its imaginary extension into the unmasked area 10 of the image, and the structure of the area also continues around the edge 20 of the mask. In this regard, various regions are defined in the unmasked area by contour lines, the edges of the regions are filled with respective colors, and texture is added to the regions of interest, if necessary.

[0023] It can be seen in Figures 1a and 1b that edges 40a, 40b, 40c of the corrected image are correctly displayed, while the extension of edge 40d shows negligible discontinuity since the area above is dark.

[0024] In addition, in Figures 2a and 2b, which are a rendered image (Figure 2a) and a corrected image (Figure 2b) of a view from above of the scenery to the side of a motor vehicle, it can be seen that not only is the image data of a rectangular area 10 that is missing in the rendered image displayed as a structure 50 in the corrected image, but also that a shadow area 60 on the opposite side of the light source, not shown, has been satisfactorily restored by the restoration method of the present invention.

[0025] Furthermore, the invention relates to a method for processing images in case a scene is captured as at least one raw image, in particular by means of optical capture means preferably mounted on a mobile means, and the image data of said scene is displayed incompletely and / or erroneously in at least one area in a subsequently rendered rendered image, said method comprising the following steps: - identifying at least one area of ​​the representation in the rendered image that is incomplete and / or contains errors based on the existing visibility limitations; generating a mask enclosing at least one region of the representation that is incomplete and / or erroneous as a mask region 30; - inpainting the image data of the unmasked areas 10 of the rendered image by digital inpainting and compositing them together with the masked areas 30 into a corrected image; and - Displaying the completed and / or error corrected modified image.

[0026] This advantageously improves the visual experience of a user of a system having optical capture means. This is because a complete and continuous view of the scenery is provided in the modified image.

[0027] In the above description, various features for improving the accuracy of the display are described in one or more examples. However, it is to be clearly understood that the above description is merely illustrative and is in no way intended to limit the present invention. However, they serve to cover all alternatives, modifications, and equivalents of the various features and embodiments. In fact, many other examples will be immediately and directly apparent to those skilled in the art based on the above description due to their expertise.

[0028] These embodiments have been chosen and described in order to show the principles underlying the invention and its possible application in practice, so that those skilled in the art can best adapt and use the invention and its various embodiments for the intended use. In the claims and the description, both "comprising" and "having" are used as neutral terms with the corresponding "containing". Moreover, the use of the indefinite articles "ein", "einer" and "eine" (which encompass the German meaning of "one") does not inherently exclude the possibility of a plurality of the feature or component in question. [Explanation of symbols]

[0029] 10 Unmasked Area 20 Mask Edge 30 Masked Areas 40a, 40b, 40c, 40d Edge 50 Structure 60 Shadow area

Claims

1. 1. A method for processing an image, comprising:

1. A method in which a scene is captured as at least one raw image by at least one optical capture means mounted on a vehicle, and image data of the scene is displayed incompletely and / or with errors in at least one region of a subsequently rendered image, comprising: The method comprises: - identifying said at least one region of incomplete and / or erroneous representation in the rendered image based on the existing visibility limitations, - data relating to visibility limitations determined based on the three-dimensional model of the vehicle and the placement of the optical capture means, and allowing recognition of areas in the rendered scene that are free of image data; a geometric model of the surrounding area of ​​the vehicle; - image data of the scene is stored in at least one database; - generating a mask encompassing said at least one region of the representation that is incomplete and / or contains errors as a mask region (30); - inpainting image data of the unmasked areas (10) of the rendered image by digital inpainting, the image data of the unmasked areas (10) being inpainted using an artificial neural network having access to said at least one database and trained with data therein relating to said visibility limitations, a geometric model of the area around the vehicle and image data of the scenery; - combining the restored image data together with the mask region (30) into a modified image; - displaying the completed and / or error-corrected modified image. 。

2. As a further step, 2. A method according to claim 1, further comprising the step of optimizing said modified images by post-processing into optimized images and displaying said optimized images instead of the respective modified images.

3. 3. A method according to claim 1 or 2, wherein the rendered image, the restored corrected image and / or the optimized image are each displayed as a displayable image to an observer in real time or with negligible delay.

4. 4. The method according to claim 1, wherein the visibility limit is determined based on at least a three-dimensional model of the respective vehicle and on the placement of the optical capture means in order to identify at least one area of ​​incomplete and / or erroneous representation in the rendered image.

5. 5. The method according to any one of claims 1 to 4, wherein the incomplete and / or erroneous image data is restored based on an edge-based method.

6. The method of claim 5 , wherein the edge-based method is a level set method or a fast marching method.

7. The method of claim 5 , wherein the edge-based method uses diffusion techniques to predict information in regions of an image that are incomplete and / or contain errors.

8. A method according to any one of claims 5 to 7, wherein a Markov Random Field method is used to predict information in regions of the image that are incomplete and / or contain errors.

9. 9. An image processing system comprising at least one optical capture means provided and arranged for capturing at least one raw image and for digitizing the at least one image, and display means for displaying the processed image, and for carrying out the method according to any one of claims 1 to 8.

10. Use of the image processing system according to claim 9 in a parking assistance system for a vehicle which performs parking maneuvers at limited speeds.

11. A vehicle equipped with a system according to claim 9 or employing the method of use of said system according to claim 10.

Images