Modular image interpolation method

Abstract

To provide a user of optical capture means, for example one or more cameras on a motor vehicle, that have visibility restrictions with a comfortable visual experience by depicting areas having incomplete and / or erroneous information correctly.SOLUTION: A method includes the steps of: identifying at least one area of incomplete and / or erroneous display in a render image based on existing visibility restrictions; generating masks that enclose the at least one area of incomplete and / or erroneous display, as masked areas 30; reconstructing image data in unmasked areas 10 of the render image by means of digital inpainting; synthesizing the reconstructed image data, together with the masked areas 30, to produce a correction image; and displaying the completed and / or debugged correction image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for capturing a scene as at least one raw image by at least one optical capture means mounted on a mobile means; The present invention also relates to an image processing system, a method for processing an image, and a method for using and transporting the same, wherein the image data of the scene is displayed at least incompletely and / or erroneously in at least one region of the rendered image after the capture. [Background technology]

[0002] Means of transport, for example, in particular vehicles, are increasingly being equipped with assistance systems, including camera devices that provide passengers and in particular the respective vehicle driver with a visual impression of the scene outside one or more interior spaces of the vehicle, for example to support, facilitate or monitor the parking process.

[0003] Nowadays, most parking systems use multiple cameras to provide a good impression of the surroundings when parking a vehicle. However, even with multiple cameras, it is not possible to cover the entire area around the vehicle. For example, there is no camera image of the area under the vehicle. This results in black areas that can be unpleasant for the viewer. Summary of the Invention [Problem to be solved by the invention]

[0004] The object of the present invention is therefore to provide a pleasant visual experience for users of optical capture means with limited visibility, such as one or more on-board cameras, by correctly displaying areas with incomplete and / or erroneous information. [Means for solving the problem]

[0005] This problem is solved by a method having the features of claim 1. 9 Image processing system according to claim 10 and the use thereof according to the claims. 11 The transportation means according to the present invention is a solution to the above problem. According to the invention, in a first step, erroneous parts of the rendered image are identified based on visibility constraints. In a second step, a mask is generated from the erroneous parts of the rendered image, and the rendered image is repaired only within this mask. The image data in the masked areas is then repaired by digital inpainting, of which there are several possible methods. After this repair, the final and / or error-corrected modified image is displayed in a display step. Thus, in this case, an improved viewing experience is provided to the user by repairing the erroneous data based on the existing data.

[0006] In an advantageous variation that further improves the user's visual experience, the corrected images are further post-processed to generate optimized images, which are then displayed instead of the respective corrected images. Any remaining artifacts are then smoothed out, and the restored images are adjusted to be more pleasing to the eye. To this end, the rendered images can be post-processed, for example, to increase sharpness, reduce contrast, and / or harmonize colors.

[0007] In order to allow the user to react quickly to the image scenery being displayed to him, in an advantageous variant of the method, the rendered image, the restored corrected image or the optimized image are each displayed to the observer as a displayable image in real time or with negligible delay, preferably with a refresh rate of at least 5 fps.

[0008] In a manageable variant of the method according to the invention, the visibility constraints are based on at least a three-dimensional model of the respective vehicle, in order to identify incomplete and / or erroneous image regions in the rendered image, and optical capture means Different landscapes can be depicted using appropriate methods based on known visibility limitations. The accompanying scenario So , 3D model of the vehicle and a 3D model of the camera housing, image data within the rendered landscape of You can recognize what's missing.

[0009] For multiple or persistent use, data on visibility limitations, geometric models (and patterns) of the surrounding area, and (particularly preferably) already generated scenery Image of In an advantageous variation of the method, the data is stored in at least one database. Save in In this way, portions of each rendered image that do not require restoration can be easily protected or hidden from further processing in advance using a mask.

[0010] Known and / or previously generated image data and Depending on the situation teeth Repeating Mask By preserving ,In the preferred variation, repair Should The image data can be generated using machine learning methods.

[0011] in this case, The image data is It is particularly preferred that the repair can be performed using an artificial neural network, which , access at least one database, Database data Trained using do.

[0012] In an advantageous variant of the method according to the invention, incomplete and / or erroneous image data can be repaired by edge-based methods, whereby edges or object transitions are searched for in the rendered image, whereby algorithmic processing often does not provide closed edge paths, which must be joined together using additive methods to enclose the object.

[0013] In this case, the edge-based method is preferably the level set method, or more preferably the fast marching method. The former is a numerical method for approximately tracking geometric objects and their motion, and can be used to calculate curves and surfaces in a spatially constant coordinate system without the need to parameterize the object. The fast marching method, a special method for numerically solving boundary value problems, solves boundary value problems using the eikonal equation, and evolves a closed surface as a function of time and velocity.

[0014] In a further advantageous variation, edge-based methods can be used for dimensionality reduction or feature extraction, which can then be propagated in incomplete and / or erroneous image regions using diffusion methods to perform information predictions, using suitable machine learning approaches, such as Markov Random Fields (MRFs), which can be used to segment digital images or classified surfaces and can be used to estimate the interactions between field components or their influence on each other.

[0015] As already mentioned above, an image processing system that performs one variation of the above method, for example its use in a parking assistance system for a vehicle that performs a parking maneuver at a limited speed, as well as a vehicle equipped with such a system, particularly preferably a vehicle itself, solves the problem posed.

[0016] The above-described embodiments and their developments can be freely combined with each other as long as it is meaningful. Further possible embodiments, developments, and embodiments of the present invention also include not specifically described combinations of the above-described inventive features and the inventive features described below in connection with the embodiments. Furthermore, individual aspects added by those skilled in the art as improvements or supplements to each basic form of the present invention are also included.

[0017] The invention will now be explained in more detail with reference to exemplary embodiments illustrated in the diagrammatic drawings. [Brief explanation of the drawings]

[0018] [Figure 1a] A schematic perspective view of the exterior area behind the vehicle captured by the optical capture means is displayed as a rendered image. [Figure 1b] A schematic perspective view of the exterior area behind the vehicle captured by the optical capture means is shown as a modified image produced by the method according to the invention. [Figure 2a] A schematic perspective view of other exterior areas of the vehicle captured by the optical capture means is displayed as a rendered image. [Figure 2b] A schematic perspective view of another exterior area of ​​the vehicle captured by the optical capture means is displayed as a modified image (FIG. 2b) produced by the method according to the invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] The accompanying drawings are intended to provide a further understanding of implementations of the present invention. They illustrate implementations and, in combination with the description, serve to explain the principles and concepts of the present invention. Other implementations and many of the advantages described above will become apparent with reference to the drawings. Components in the drawings are not necessarily drawn to scale.

[0020] In the representations of the figures, elements, features and components that are the same, have the same function and have the same effect are respectively labeled with the same reference numerals - unless otherwise stated.

[0021] 1a and 1b show a schematic perspective view of an external scene behind a vehicle captured by an optical capture means. In the rendered image of the scene shown in FIG. 1a, a roughly rectangular area 10 is visible, in which the image data of the scene is incomplete due to the presence of the housing of the optical capture means (not shown), which is configured as a camera. However, since the dimensions of the housing are known and available in a database, the missing image data can be restored by the method according to the present invention, applying digital inpainting. A self-consistent image is generated, which is based on the entire image itself and provides the user with a better visual experience during subsequent viewing.

[0022] The regions 10 where image data is missing are identified by borders 20, which separate the regions 10 from regions 30 for which image data is known. This is shown in the initial identification step. In the next step of the method, the regions 30 for which image data is known are masked, i.e., a mask is generated. This mask surrounds the regions 10 of the image that contain incomplete and / or erroneous data with the masked regions 30 that will not be processed, thereby repairing the regions 10 but not processing the regions 30 for which image data can be correctly captured, rendered, and output. In the subsequent repair step, the unmasked regions of the rendered image are repaired by digital inpainting, and then combined with the masked regions to form a corrected image (FIG. 1b). The contour lines of the rendered image that touch the mask borders 20 continue along their imaginary extensions into the unmasked regions 10 of the image, and the region structure also continues around the mask borders 20. In this case, various regions are defined in the unmasked area by contour lines, the edges of the regions are filled with respective colors, and texture is added to the region of interest, if necessary.

[0023] It can be seen in Figures 1a and 1b that edges 40a, 40b, 40c of the corrected image are correctly displayed, while the extension of edge 40d shows negligible discontinuity since the area above is dark.

[0024] Furthermore, in Figures 2a and 2b, which are a rendered image (Figure 2a) and a corrected image (Figure 2b) of a view from above of the scenery to the side of a motor vehicle, it can be seen that not only is the image data of a rectangular area 10 that is missing in the rendered image displayed as a structure 50 in the corrected image, but also that the shadow area 60 on the opposite side of the light source (not shown) has been satisfactorily restored by the restoration method of the present invention.

[0025] Furthermore, the present invention relates to a method for processing an image in the case where a scene is captured as at least one raw image, in particular by means of optical capture means preferably mounted on a mobile means, and the image data of said scene is displayed incompletely and / or with errors in at least one region in a subsequently rendered image, said method comprising the following steps: - Identifying at least one area of ​​the representation in the rendered image that is incomplete and / or contains errors based on the visibility limitations that exist. generating a mask enclosing as a mask region 30 at least one region of the representation that is incomplete and / or erroneous; - inpainting the image data of the unmasked areas 10 of the rendered image by digital inpainting and merging them together with the masked areas 30 into a corrected image; and - displaying said completed and / or error corrected modified image.

[0026] This effectively improves the visual experience of users of systems with optical capture means. This is because a complete and continuous view of the scenery is provided in the modified image.

[0027] In the above description, various features for improving the accuracy of display are described in one or more examples. However, it should be clearly understood that the above description is merely an illustrative example and is in no way intended to be limiting. However, it serves to cover all alternatives, modifications, and equivalents of the various features and embodiments. In fact, many other examples will be immediately and directly apparent to those skilled in the art based on the above description, due to their expertise.

[0028] These embodiments have been chosen and described to illustrate the principles underlying the invention and its potential application in practice, thereby enabling those skilled in the art to adapt and use the invention and its various embodiments to their intended uses. In the above and in the description, both "comprises" (beinhaltend) and "has" (aufweisend) are used as neutral terms for the corresponding "comprises" (umfassend). Furthermore, the use of the indefinite articles "ein," "einer," and "eine" (which encompass the German meaning of "one") does not inherently exclude the possibility of a plurality of such features or components. The present application relates to the invention described in the claims, but may also include the following configurations as other aspects. 1. 1. A method for processing an image, comprising: 1. A method in which a scene is captured as at least one raw image by at least one optical capture means mounted on a mobile means, and image data of the scene is displayed incompletely and / or with errors in at least one region of a rendered image that is subsequently rendered, comprising: The method comprises: - identifying said at least one region of incomplete and / or erroneous representation in the rendered image based on existing visibility limitations; - generating a mask encompassing said at least one region of the incomplete and / or erroneous representation as a mask region (30); - inpainting the image data of the unmasked areas (10) of said rendered image by digital inpainting; - combining the restored image data together with the mask region (30) into a modified image; - displaying the completed and / or error corrected modified image. 2. As a further step, - optimizing the modified images into optimized images by post-processing and displaying the optimized images instead of the respective modified images. 3. 3. The method according to claim 1 or 2, wherein the rendered image, the restored corrected image and / or the optimized image are each displayed as a displayable image to an observer in real time or with negligible delay. 4. A method according to any one of 1 to 3 above, wherein the visibility limit is determined based on at least a three-dimensional model of each vehicle and the installation of the optical capture means in order to identify at least one area of ​​incomplete and / or erroneous representation in the rendered image. 5. 5. The method according to any one of 1 to 4 above, wherein the data on visibility limitations, the geometric model of the surrounding area, and the scenery data that has already been generated are prepared in advance in at least one database. 6. 6. The method according to any one of 1 to 5 above, wherein the image data to be restored is generated with the aid of a machine learning approach. 7. 7. A method according to any one of claims 1 to 6, wherein the image data is restored using an artificial neural network which has access to at least one database and is trained using data from said database. 8. 8. The method according to any one of claims 1 to 7, wherein the image data containing incomplete and / or errors is restored based on an edge-based method. 9. 9. The method according to claim 8, wherein the edge-based method is a level set method, particularly preferably a fast marching method. 10. 10. The method of claim 8 or 9, wherein the edge-based method uses diffusion techniques to predict information in regions of the image that are incomplete and / or contain errors. 11. 11. The method according to any one of claims 6 to 10, wherein a Markov Random Field method is used to predict information in regions of the image that are incomplete and / or contain errors. 12. 12. An image processing system comprising at least one optical capture means provided and configured for capturing at least one raw image and for digitally processing the at least one image, and display means for displaying the processed image, and for carrying out the method according to any one of 1 to 11 above. 13. 13. Use of an image processing system according to claim 12 in a parking assistance system for a vehicle that performs parking maneuvers at limited speeds. 14. A means of transport equipped with the system described in 12 above or employing the method of using the system described in 13 above. [Explanation of symbols]

[0029] 10 Unmasked Area 20 Mask Edge 30 Masked Areas 40a, 40b, 40c, 40d Edge 50 Structure 60 Shadow area

Claims

1. 1. A method for processing an image, comprising:

1. A method in which a scene is captured as at least one raw image by at least one optical capture means mounted on a vehicle, and image data of the scene is displayed incompletely and / or with errors in at least one region of a subsequently rendered image, comprising: The method comprises: - identifying said at least one region of incomplete and / or erroneous representation in the rendered image based on the existing visibility limitations, - data relating to visibility limitations determined based on the three-dimensional model of the vehicle and the placement of the optical capture means, and allowing recognition of areas in the rendered scene that are free of image data; a geometric model of the surrounding area of ​​the vehicle; - image data of the scene is stored in at least one database; - generating a mask encompassing said at least one region of the representation that is incomplete and / or contains errors as a mask region (30); - inpainting image data of the unmasked areas (10) of the rendered image by digital inpainting, the image data of the unmasked areas (10) being inpainted using an artificial neural network having access to said at least one database and trained with data therein relating to said visibility limitations, a geometric model of the area around the vehicle and image data of the scenery; - combining the restored image data together with the mask region (30) into a modified image; - displaying the completed and / or error-corrected modified image. 。

2. As a further step, 2. A method according to claim 1, further comprising the step of optimizing said modified images by post-processing into optimized images and displaying said optimized images instead of the respective modified images.

3. 3. A method according to claim 1 or 2, wherein the rendered image, the restored corrected image and / or the optimized image are each displayed as a displayable image to an observer in real time or with negligible delay.

4. 4. The method according to claim 1, wherein the visibility limit is determined based on at least a three-dimensional model of the respective vehicle and on the placement of the optical capture means in order to identify at least one area of ​​incomplete and / or erroneous representation in the rendered image.

5. 5. The method according to any one of claims 1 to 4, wherein the incomplete and / or erroneous image data is restored based on an edge-based method.

6. The method of claim 5 , wherein the edge-based method is a level set method or a fast marching method.

7. The method of claim 5 , wherein the edge-based method uses diffusion techniques to predict information in regions of an image that are incomplete and / or contain errors.

8. A method according to any one of claims 5 to 7, wherein a Markov Random Field method is used to predict information in regions of the image that are incomplete and / or contain errors.

9. 9. An image processing system comprising at least one optical capture means provided and arranged for capturing at least one raw image and for digitizing the at least one image, and display means for displaying the processed image, and for carrying out the method according to any one of claims 1 to 8.

10. Use of the image processing system according to claim 9 in a parking assistance system for a vehicle which performs parking maneuvers at limited speeds.

11. A vehicle equipped with a system according to claim 9 or employing the method of use of said system according to claim 10.

Images