Dynamic reflections on the vehicle model based on camera images in the surround view system

Abstract

The invention relates to a process implemented in a means of transport (10), comprising at least the following steps: a) determining a visual influence (12) of an environment (14) of a means of transport (10) on the means of transport (10) itself, b) generating an image (16) of the means of transport (10) taking into account the visual influence (12) of the environment (14), and c) the process steps a) and b) are repeated at least once.

Description

Technical field The present invention relates to a process that is implemented in a means of transport and comprises at least a determination of a visual influence of an environment of a means of transport on the means of transport itself and a generation of an image of the means of transport taking into account the visual influence of the environment. State of the art Early parking assistance products for vehicles used ultrasonic parking sensors and / or a single rear-view camera to detect and maintain distances to objects around the vehicle, providing drivers with an audible alert or a rear-view video through a fisheye lens. These early products had some drawbacks: the alert only provided a proximity warning, not the position of the object(s) relative to the vehicle, and the rear-view camera had a limited field of view. Multi-camera systems overcome these problems and are becoming increasingly available. Most systems have four wide-angle cameras: one at the front of the vehicle, one at the rear, and one in each of the side-mounted rearview mirrors. The four cameras have overlapping fields of view, collectively covering the entire area around the vehicle and acting as an omnidirectional camera.Videos from the cameras are sent to the processor, which synthesizes a bird's-eye view of the vehicle by merging video feeds, correcting distortion, and transforming the perspective. US10,017,114 B2 relates to a vehicle vision system comprising a camera, a display unit, and a controller. The camera is positioned on the vehicle and has a field of view outside the vehicle. The camera is functional and capable of capturing image data. The display unit includes a display screen located in a display area on the vehicle's windshield. The display unit is electrically switchable from a non-display state to a display state. When the vehicle is driven, the controller, in response to a determination that the driver's gaze is directed toward the display area for a threshold period, switches the display unit from its non-display state to its display state. When in its display state, the display unit displays images on the display screen derived from captured image data. Disclosure of the invention According to the present invention, a process implemented in a means of transport is proposed, wherein the process comprises at least a determination of a visual influence of an environment of a means of transport on the means of transport itself and a generation of an image of the means of transport taking into account the visual influence of the environment. A means of transport is a device used for the carriage of either persons and / or goods. The means of transport is not restricted in its route; therefore, its movement can occur by land, water, or air. An environment is understood as the entirety of all elements located in such a way that their visual influence affects the appearance of the means of transport. An environment is associated with the physical location of the means of transport and a specific time. A visual influence is understood as any type of optical phenomenon involving the emission of visible light by active and passive light sources, as well as a discontinuity in light intensity, e.g., a shadow. This includes sources of natural and artificial lighting, the reflection of visible light by objects with a reflective surface, and the reflection of objects onto the surface of the means of transport. The image is a photograph captured by a camera system and processed by a computer. In the context of the process according to the invention, the computer processes the photograph in such a way that a model of the physical appearance of the means of transport is inserted into the photograph, depending on the visual influence of the environment. The image refers to a photograph captured by a camera system and processed by a computer. In the context of the process according to the invention, the computer processes the photograph in such a way that a model of the physical appearance of the means of transport is inserted into the photograph, depending on the visual influence of the environment. In an advantageous embodiment of the process according to the invention, the visual influence is a reflection on a surface of the means of transport. This includes visible light reflected by surfaces of objects located in the vicinity of the means of transport. The source of the reflected visible light includes light of natural origin, light of artificial origin, and light originating from another reflection. In another advantageous embodiment of the process according to the invention, the environment comprises at least one object with a reflective surface and / or an object that is reflected on the surface of the means of transport. These objects include other means of transport, static objects, living beings, as well as the surface characteristics of the ground and the sky. Other means of transport can be of the same type as the means of transport itself or of a different type. Static objects include buildings, vegetation, and traffic-related infrastructure. The surface characteristics of the ground include paved and unpaved roads, depending on their environmental conditions, as well as waterways. In another advantageous embodiment of the process according to the invention, either the environment comprises at least one light source or the means of transport is equipped with at least one light source. A typical light source for the means of transport itself can be a headlight, a flashing system, or interior lighting. A typical light source for the environment includes all sources of natural light, other means of transport, traffic-related infrastructure such as streetlights, signal lights such as traffic lights, hazard warning lights, aviation obstruction lighting, beacons, and other such elements, as well as non-traffic-related light sources such as interior lighting of buildings. In another advantageous embodiment of the process according to the invention, the environment of the means of transport is captured by a camera system. The camera system comprises at least one camera and is mounted in such a way that the environment of the means of transport itself is covered. In another advantageous embodiment of the process according to the invention, the means of transport is a vehicle, e.g., a car, a truck, a motorcycle, a bus, a commercial vehicle, a boat, a ship, an aircraft, or a rail vehicle. The camera system preferably covers the largest possible area, but it can also cover an area of ​​less than 360 degrees around the means of transport. In another advantageous embodiment of the process according to the invention, the image is a computer-generated 3D image. A processing unit processes the photograph captured by the camera system as follows: if the camera system comprises two or more independent cameras, the photographs are combined in such a way as to generate a continuous image of the environment. In the next step, the processing unit integrates a computer-generated model of the means of transport itself into the continuous image of the environment. The computer-generated model of the means of transport takes into account the visual influence of the environment on the means of transport itself. In another advantageous embodiment of the process according to the invention, in a first step an image of the environment, captured by the camera system, is projected onto a concave 3D vessel, and in a second step the visual influence of the environment on the means of transport is determined. The photographs from the camera system are projected onto a three-dimensional concave vessel, e.g. a bowl, to generate a continuous image of the environment, thereby determining the visual appearance of the means of transport. In another advantageous embodiment of the process according to the invention, the visual influence of the environment on a surface of the means of transport not covered by the camera system is reconstructed by extending the visual influence of the environment of an adjacent captured surface to the uncovered surface and combining it with at least one other captured surface. Determining the visual influence on the upper surface of the means of transport is made possible by interpolating adjacent surface areas. The way in which these surfaces are interpolated depends on the number of adjacent surfaces. For example, for an uncovered surface with four adjacent surfaces, extending the uppermost of the adjacent surfaces to the uncovered surface using triangles and subsequently blurring the edges yields positive results. In another advantageous embodiment of the process according to the invention, the resolution of the image of the environment captured by the camera system is adjustable. The processing unit, which determines the visual influence of the environment on the means of transport itself, has a finite set of operations that it can process. Depending on the extent of the visual influence of the environment on the means of transport itself and the frequency of changes in this visual influence, the processing power of the processing unit may be exceeded. In this case, it is preferred instead to adjust the resolution of the image of the environment captured by the camera system in such a way as to guarantee a smooth display of the moving image, rather than displaying a detailed moving image at a lower frequency. Advantages of the invention Conventional surround-view systems, especially those commonly used in the automotive industry, employ static vehicle models. This means the vehicle model always appears the same in the system. Depending on environmental conditions, the vehicle model will appear unrealistic, as it is impossible to account for all possible scenarios. The vehicle's appearance will change drastically depending on whether it is outdoors or indoors, or whether it is day or night. Therefore, by applying the process according to the invention, the 3D vehicle model will appear more realistic. In an image where authentic photographic material and computer-generated images are combined, it is crucial to provide consistent lighting and reflections on the computer-generated object so that the composition appears credible. Current technologies lack this feature. The more reflective the material on the model, the more pronounced this problem becomes. A dynamic representation of the vehicle is made possible by the continuous updating of a cube map. Every 3D environment, especially the one mentioned earlier, requires different lighting on the model. Without this invention, the model would appear out of place. Furthermore, commonly used methods have difficulties with 360-degree coverage, as implementing a camera system that covers the top and bottom of the vehicle is associated with a poor cost-benefit ratio. The process according to the invention solves this problem by its ability to approximate the appearance of surfaces not covered by the camera system. Thus, a credible appearance of the respective surface can be determined without the need for additional software. For a surface with four adjacent surfaces, this is achieved by extending the upper part of the side surfaces into a triangular shape, with the vertices of the triangle representing the two corners of the upper surface and the center of the upper surface. Subsequently, a blurring effect is applied to eliminate any sharp edges that might arise at the connection points of these triangles.The blurring is achieved in a way that makes the triangle increasingly transparent as the focus shifts away from the triangle's contour line. The same technique is applied to the lower part of the triangle, the only difference being the extension of the blur on the lower side of the faces. Brief description of the drawings Embodiments of the invention are explained in more detail with reference to the drawings and the following description. The drawings show: Fig. 1 a means of transport located in an environment and equipped with a camera system and a computer. The environment includes objects with reflective surfaces, objects reflected on the surface of the means of transport, and light sources; Fig. 2 a dynamically generated cube with captured and uncaptured surfaces; and Fig. 3 a top view of a reconstructed, uncaptured surface on the dynamic cube from Fig. 2. Embodiments of the invention In the following description of embodiments of the invention, identical or similar elements are designated by the same reference numerals, and repeated descriptions of these elements are omitted in individual cases. The figures represent the subject matter of the invention only schematically. Fig. 1 shows a means of transport 10 located in an environment 14. The environment 14 comprises objects 18 with a reflective surface, objects 19 that are reflected on the surface of the means of transport itself, and at least one light source 20, the means of transport 10 being equipped with the light source 20. The effect of the environment 14 on the visual appearance of the means of transport 10 is referred to as visual influence 12. Furthermore, the means of transport 10 is equipped with a camera system 22. The coverage of the camera system 22 can vary, i.e., captured surfaces 30 are distinguished, meaning surfaces of the means of transport 10 that are located in such a way that the visual influence 12 of the environment 14 can be determined by the camera system, as well as uncaptured surfaces 32.Surfaces that are localized in such a way that the location of the sources 18, 19, 20 for the visual influence 12 on the surface exceeds the range of the camera system 22 are both visualized. After obtaining an image of the environment 24, a computer 26 processes the image of the environment 24 in such a way that an image 16 is generated which includes a computer-generated 3D representation of the means of transport 10, taking into account its appearance within an environment that is localized within its surroundings. Fig. 2 shows a generated dynamic cube 40 with an unscanned surface 32 on its top side. The unscanned surface 32 according to Fig. 2 is a top surface 34. The top surface 34 is bounded by at least one first edge 36 of the top surface or an adjacent second edge 38 of the top surface. As can be deduced from Fig. 2, the side faces 42 form the captured surfaces 30. The upper part of the captured surfaces 30, i.e., the side faces 42, i.e., the sections 44, is projected onto the uncaptured surface 32, i.e., the upper surface 34. Thus, the appearance of the uncaptured surface 32, i.e., the upper surface 34, is estimated by means of the four adjacent captured surfaces 30, i.e., the respective side faces 42. Consequently, the upper part of each captured surface 30 is projected onto the uncaptured surface 32 and extended by means of triangles. Fig. 3 shows the pattern of the upper surface 34 according to Fig. 2. As can be deduced from Fig. 3, i.e., the top view of the unsampled surface 32, it is formed by the patterns of the four adjacent side surfaces 42, i.e., the captured surfaces 30 of the generated dynamic cube 40, as shown in Fig. 2. The top view 3 is divided by a center line 48 into a light section 50 and a more heavily shaded section 52. The reference number 54 indicates the center point of the upper surface 34 and is located on the center line 48, which divides the upper surface 34, i.e., the unsampled surface 32, into the light section 50 and the shaded section 52, respectively. As schematically shown in Fig. 3, each of the captured surfaces 30, i.e., the four faces 42 of the generated dynamic cube 40, is extended, i.e., projected, forming triangular sections 46 of the uncaptured surface 32 onto the top surface 34. Thus, the uncaptured surface 32, i.e., the top surface 34, is reconstructed by the triangular shapes 46, the projection of the top parts 44 of the faces 42, as shown in Fig. 2. The invention is not limited to the embodiments described herein and the aspects highlighted therein. Instead, a large number of modifications are possible within the scope specified by the claims, which are within the realm of what is known to a person skilled in the art. QUOTES INCLUDED IN THE DESCRIPTION This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature US 10 017 114 B2

[0003]

Claims

The process, which is implemented in a means of transport (10), comprises at least the following steps: a) determining a visual influence (12) of an environment (14) of a means of transport (10) on the means of transport (10) itself, b) generating an image (16) of the means of transport (10) taking into account the visual influence (12) of the environment (14), and c) the process steps a) and b) are repeated at least once. Process according to claim 1, wherein the visual influence (12) is a reflection on a surface (30, 32) of the transport means (10). Process according to claim 1, wherein the environment (14) comprises at least one object with a reflective surface (18) and / or an object (19) that is reflected on the surface of the means of transport. Process according to claim 1, wherein either the environment (14) comprises at least one light source (20) or the means of transport (10) is equipped with at least one light source (20). Process according to claim 1, wherein the environment (14) is captured by a camera system (22). Process according to claim 1, wherein the means of transport (10) is a vehicle. Process according to claim 1, wherein the image (16) is a computer-generated 3D image. Process according to claims 1 and 5, wherein the camera system (22) covers an environment of less than 360 degrees. Process according to claims 1 and 5, wherein in a first step an image of the environment (24) captured by the camera system (22) is projected onto a concave 3D vessel, and in a second step the visual influence (12) of the environment (14) on the means of transport (10) is determined. Process according to claims 1 and 7, wherein the computer-generated 3D image (16) of the means of transport (10) is generated by a computer (26) using a cube map. Process according to claims 1, 5 and 8, wherein the visual influence (12) of the environment (14) on a surface (32) of the means of transport (10) not detected by the camera system (22) is reconstructed by extending the visual influence (12) of the environment (14) of an adjacent detected surface (30) to the undetected surface (32) and combining it with at least one other detected surface (30). Process according to claims 1 and 5, wherein the resolution of the image of the environment (24) captured by the camera system (22) is adjustable.

Images