Method and apparatus for generating surround view and motorized vehicle

Abstract

The present invention relates to a method for generating a surround view of a motorized vehicle having a body and parts that move relative to the body, in particular steered parts. The method includes creating a first mask that shows a silhouette of the body projected onto a ground surface, and creating a second mask that shows silhouettes of the moving parts of the vehicle projected onto the ground surface in their current state. The method further includes combining the first mask and the second mask to create a mask of the motorized vehicle, and using the motorized vehicle mask to create a surround view of the vehicle's surroundings from camera images provided by an on-board camera of the motorized vehicle.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for generating a surround view, and a corresponding motorized vehicle. [Background technology]

[0002] Driver assistance systems can support drivers when driving motorized vehicles by using camera systems that generate and output to the driver a visual image of the vehicle's surroundings, which the driver can use to better understand the situation when, for example, pulling out of a parking space or parking the motorized vehicle in order to operate the motorized vehicle more quickly and safely.

[0003] For this purpose, surround-view camera systems are known that are able to depict the entire surroundings of a motorized vehicle by combining the images of several on-board cameras. The real-world images produced by these cameras can then be combined with the surroundings image.

[0004] Graphic representation of the vehicle surroundings can be performed from various viewpoints. For example, the "ball" view is known, where textures from multiple cameras are projected to form a virtual three-dimensional "ball" that depicts the entire surroundings of the motorized vehicle. Another known perspective is the "top view" representation.

[0005] Some areas of the vehicle's surroundings cannot be captured by the on-board camera because they are covered by parts of the motor vehicle or are outside the field of view of the on-board camera, such as the area under the motor vehicle.

[0006] From DE 10 2020 213 146 B3, a camera system for capturing the surroundings for a vehicle is known, whereby the vehicle body is captured in the camera image, boundary points are defined from the body boundaries of the vehicle body, and the camera coordinates are transformed into the vehicle coordinate system in order to determine texture boundaries in which the camera cannot participate.

[0007] Some of the peripheral information may be missing, especially in spatial regions close to the vehicle boundary, in order to obscure vehicle parts that would typically be projected extraneously on the ground.

[0008] Even if a conservative approach is used, where the entire vehicle periphery is covered but the ground blind area is set to a rectangle of a certain size, there will still be ground areas covered by the rectangle in the output, even though they are properly captured by the camera. In other words, there are typically ground blind areas that are ground areas close to the vehicle boundary that are covered by the rectangular blind area and are not visible in the visualization output, e.g., in a top view or ball view. That is, while the vehicle is driving, the rectangular blind area may cover surrounding objects, thereby hindering their visibility. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] DE 10 2020 213 146 B3 Summary of the Invention [Problem to be solved by the invention]

[0010] Therefore, an object of the present invention is to reduce the blind spot area. [Means for solving the problem]

[0011] This problem is solved by a method and a device for generating a surround view of the surroundings of a motorized vehicle, and by a motorized vehicle, having the features of the independent claims.

[0012] Further preferred embodiments are the subject of the dependent claims.

[0013] According to a first aspect, the present invention provides a method for generating a surround view of a motorized vehicle having a body and parts that move relative to the body, in particular steered parts. The method includes creating a first mask representing a silhouette of the body projected onto a ground surface, and creating a second mask representing silhouettes of the moving parts of the vehicle projected onto the ground surface in a current state. The method further includes combining the first mask and the second mask to create a mask of the motorized vehicle, and using the motorized vehicle mask to create a surround view of the vehicle's surroundings from camera images provided by an on-board camera of the motorized vehicle.

[0014] According to a second aspect, the present invention provides an apparatus for generating a surround view of a motorized vehicle having a body and parts that move relative to the body, the apparatus comprising an interface for receiving camera images from an on-board camera of the motorized vehicle, and computation means configured to generate a first mask indicative of a silhouette of the body projected on a ground surface, and to generate a second mask indicative of a silhouette of the moving parts of the vehicle projected on the ground surface in a current state, the computation means being further configured to generate a mask of the motorized vehicle by combining the first mask and the second mask, and to generate a surround view of the motorized vehicle from the received camera images using the motorized vehicle mask.

[0015] According to a third aspect, the invention also relates to a motorized vehicle comprising a device for generating a surround view of the surroundings of the motorized vehicle according to the invention.

[0016] The present invention allows for a consistent depiction of the entire area of ​​the vehicle's surroundings that is visible from the on-board cameras around the vehicle. This is achieved by creating a geometric mask of the vehicle that covers the vehicle parts in the multi-camera visualization.

[0017] The mask can be defined as a polygonal surface.

[0018] According to a preferred development of the method for creating a surround view of the surroundings of a motorized vehicle, the moving parts include at least the steered wheels of the motorized vehicle, and the second mask is determined as a function of the steering angle of the motorized vehicle at the current time, and the current position of the wheels is determined based on the steering angle at the current time.

[0019] According to a preferred development of the method for creating a surround view of the surroundings of a motor vehicle, a virtual tire is created for at least one wheel of the motor vehicle in an on-board camera coordinate system of an on-board camera capturing said tire of the motor vehicle depending on the steering angle at the time, said virtual tire in said on-board camera coordinate system being projected onto the ground from the viewing angle of the on-board camera in order to create a silhouette of the tire of the motor vehicle projected onto the ground. A mask for the vehicle tire can thus be created on the basis of a reprojection of the virtual tire, taking into account the current position of the tire.

[0020] According to a preferred development of the method for creating a surround view of the surroundings of a motor vehicle, the virtual tire is further created depending on the size and the position of at least one tire of the motor vehicle, the size being specifiable for the motor vehicle or for a model of the motor vehicle.

[0021] According to a preferred development of the method for creating a surround view around a motor vehicle, the virtual tire is modeled by a cylinder, the cylinder being capable of being described by a grid network.

[0022] According to a preferred development of the method for creating a surround view of the surroundings of a motor vehicle, for each on-board camera, a boundary line route is determined which indicates the boundary of the vehicle body in the camera image of the on-board camera. Each boundary line route is projected onto the ground. Then, intersections of the boundary line route projected onto the ground are determined, in which the silhouette of the vehicle body projected onto the ground is determined as a surface bounded by sections extending between the determined intersections of the boundary line route. This ensures that no image information depicting any parts of the vehicle body is used.

[0023] According to a preferred development of the method for creating a surround view of the periphery of a motor vehicle, the boundary route for each camera image is determined automatically. In this way, the method can be performed more quickly. The automated determination can be performed model-by-model, i.e. not individually for each motor vehicle. Furthermore, it is also possible to use a virtual three-dimensional model of the motor vehicle. Based on the real camera extrinsics, for each vehicle, after calibration, the exact boundary route can be extracted completely within the virtual periphery. Alternatively, it is also possible to use the three-dimensional model of the motor vehicle in order to estimate the polygonal surface of the first mask directly from the projection of the virtual vehicle.

[0024] According to a preferred development of the method for generating a surround view of the surroundings of a motor vehicle, a boundary route for each camera image is generated by means of a virtual model of the vehicle body.

[0025] According to a preferred development of the method for creating a surround view of the surroundings of a motorized vehicle, for each determined intersection point adjacent overlapping areas of the on-board cameras are determined, at least one of the overlapping areas being shifted horizontally with respect to an outer point of the silhouette projected on the ground of the moving parts of the motorized vehicle, i.e. the overlapping areas with the projection areas of the cameras can be adjusted to avoid undesired depiction of non-transparent parts.

[0026] According to a preferred development of the method for creating a surround view of the surroundings of a motor vehicle, the surround view of the surroundings of the motor vehicle is created using camera images created by a plurality of on-board cameras, but in the areas masked by the mask of the motor vehicle, no camera data of the camera images is inserted, instead an artificial image of the motor vehicle can be inserted there.

[0027] According to a preferred development of the method for generating a surround view of the surroundings of a motor vehicle, the surround view of the surroundings of the motor vehicle is a ball-shaped view of the surroundings of the motor vehicle or a top view of the surroundings of the motor vehicle from above.

[0028] According to a preferred development of the method for generating a surround view of the surroundings of a motorized vehicle, the generated surround view of the surroundings of the motorized vehicle is displayed on a display means of the motorized vehicle, which surround view can be displayed when using the parking assistant or when reversing.

[0029] According to a preferred development, the motor vehicle comprises display means arranged to be able to display the generated surround view of the surroundings of the motor vehicle.

[0030] The invention will now be explained in more detail with reference to exemplary embodiments depicted in the drawings, in which: FIG. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a schematic block diagram of a motorized vehicle equipped with an apparatus for generating a surround view of the motorized vehicle's surroundings according to an embodiment of the present invention; [Diagram 2] Figure 2 is a schematic depiction of the camera image to illustrate the border route; [Diagram 3] FIG. 3 is a schematic representation to illustrate the creation of a first mask; [Figure 4] Figure 4 is a schematic depiction of the capture area of ​​the vehicle-mounted camera; [Diagram 5]Figure 5 Schematic depiction of the unshifted overlap region; [Figure 6] Figure 6 Schematic depiction of the shifted overlap region; [Figure 7] Fig. 7 is a schematic depiction to illustrate the creation of a mask for a motor vehicle; [Figure 8] FIG. 8 is a schematic representation of a mask for a motor vehicle; and [Figure 9] FIG. 9 illustrates a flow chart of a method for generating a surround view of a motorized vehicle according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Insofar as this is meaningful, the individual features and developments described can be combined with one another in any way.Further possible features and developments as well as embodiments of the invention also include not specifically described combinations of the inventive features described above or in the following in connection with the examples.

[0033] The accompanying drawings are intended to provide a further understanding of the embodiments of the present invention. They serve to illustrate the embodiments and, in combination with the specification, to explain the principles and concepts of the present invention. Other embodiments and many of the advantages described above will become apparent with reference to the drawings. Elements of the figures are not necessarily drawn to scale, with the same reference numerals indicating components having the same or similar effect.

[0034] Fig. 1 shows a schematic block diagram of a motorized vehicle 1 with a device 2 for generating a surround view of the surroundings of the motorized vehicle 1. The motorized vehicle 1, in the definition of the present invention, includes a body including a body and non-steerable wheels. Furthermore, the motorized vehicle 1 also includes parts that are movable relative to the body, in particular swivelable or steerable parts, i.e. parts that can affect the change of the silhouette of the body 1 (in a top view). The movable parts preferably include steerable tires of the motorized vehicle 1. Furthermore, the movable parts can also include steerable components of a construction vehicle or the blades of a snowplow.

[0035] The device 2 includes an interface 3 that is coupled to the on-board cameras 5 of the motor vehicle 1, either wirelessly or by wire, and receives camera images from the on-board cameras 5. For example, four, six, eight, or even ten on-board cameras 5 may be provided. In one embodiment, the cameras are located in the front area, the rear area, and the side mirrors. The present invention is not limited by the specific number of on-board cameras 5. These on-board cameras 5 are preferably fish-eye cameras with a large capture area, preferably at least 160 degrees.

[0036] The present invention may be used in any application where "blind spots" occur, including, for example, towing applications.

[0037] The device 2 has or can derive further knowledge from the intrinsic and extrinsic camera parameters of the on-board camera 5 .

[0038] In our definition, the intrinsic camera parameters are internal and fixedly coupled to a specific on-board camera 5. They thereby enable the attribution between camera coordinates and pixel coordinates.

[0039] Extrinsic parameters may be external to the camera and vary relative to the world image, i.e. depend on the pose, position and orientation of the on-board camera 5 in the global coordinate system.

[0040] Furthermore, the device 2 includes computing means 4, which may comprise a microcontroller, microprocessor or the like, for performing computing operations.

[0041] The calculation means 4 creates a mask of the motor vehicle 1. In this case, the calculation means 4 creates a first mask that shows the silhouette of the vehicle body projected on the ground. That is, the first mask shows, for example, a body mask.

[0042] The computing means 4 also generate a second mask which shows the silhouette of the moving parts of the vehicle in their current state projected on the ground. The second mask thus includes, for example, the steerable tires of the motor vehicle 1 in their current tire positions. The virtual tire model is then used to assign the virtual tires in the global coordinate system which corresponds as accurately as possible to the real tire positions and tire sizes, taking into account the current steering angle information. The virtual tires are projected in such a way that they are positioned as if they were captured by the corresponding vehicle-mounted camera 5. For example, the left tire is projected onto the left camera and the right tire onto the right camera. The virtual tire model is then projected back onto the ground. The projected virtual tire model is then cut to correspond to the used visualization approach at the specific vehicle longitudinal position.

[0043] The computation means 4 combines the first mask and the second mask to create a motor vehicle mask, which may include all areas that are encompassed by either the first mask or the second mask.

[0044] Furthermore, the computing means 4 creates a surround view of the surroundings of the motorized vehicle 1 from the received camera images, taking into account the extrinsic and intrinsic camera parameters of the corresponding on-board camera 5. In so doing, the image information created from the camera images is projected only onto the area lying outside the mask of the motorized vehicle 1.

[0045] The surround view can be output to a display means 6, for example an in-vehicle display.

[0046] 2 shows a schematic representation of a camera image to explain a boundary line route 21 used to create a first mask (body mask), where the boundary line route 21 corresponds to a boundary between an area belonging to the vehicle body in a camera image (e.g., a camera image of a front camera) and an area outside the vehicle body, e.g., the ground.

[0047] A boundary line route 21 is created for each vehicle-mounted camera 5. This creation can be performed automatically or manually. The boundary line route 21 can be determined individually for each motorized vehicle 1. Alternatively, the boundary line route 21 can be extracted only once for a specific vehicle model and the projected geometry can be used for all motorized vehicles 1 of the same model. In doing so, it is possible to take into account small offsets to enlarge the first mask and take into account possible deviations.

[0048] FIG. 3 shows a schematic representation for explaining the creation of the first mask. Here, a top view of the motor vehicle 1 is depicted. In a further embodiment, a ball view is used. In the simplest case (FIG. 3, far left), the vehicle body is modeled by a rectangle 31. To obtain a more accurate model of the vehicle body, boundary line routes 21 to 24 are determined for each of the four vehicle-mounted cameras 1 in total and projected onto the ground (FIG. 3, center left). Then, the intersections of the boundary line routes 21 to 24 projected onto the ground are determined, and the boundary line routes 21 to 24 are limited to polygonal sections of the boundary line routes 21 to 24 extending between the determined intersections (FIG. 3, center right). The silhouette of the vehicle body projected onto the ground, which is the first mask 32, is determined as a surface surrounded by the sections of the boundary line routes 21 to 24 (FIG. 3, far right).

[0049] FIG. 4 shows a schematic representation of the capture areas 41 to 48 of the four on-board cameras 5 of a motor vehicle 1. The capture areas 41 to 48 are first defined relative to a rectangle 31 modeling the motor vehicle 1, whereby the first four regions 42, 44, 45, 47 are each captured by only one on-board camera. Furthermore, the second four regions 41, 43, 46, 48, which are overlapping regions, are each captured by two on-board cameras 5 (FIG. 4, left). Using the first mask 32, the capture areas 41 to 48 are shifted. The overlapping regions 41, 43, 46, 48 still have a rectangular shape, whereas the other first regions 42, 44, 45, 47 are limited by the first mask 32 and have good, not necessarily straight, sides.

[0050] 5 shows a schematic representation of the unshifted overlap region 41, 43. If tires 51, 52 of the motor vehicle 1 are additionally to be considered, these tires 51, 52 may extend into the unshifted overlap region 41, 43. For this purpose, it is first determined whether the tires 51, 52 are beyond the first mask 32 previously defined by the boundary routes 21 to 24. This is performed with the aid of a virtual tire, which is created based on the tire dimensions, position and current steering angle. The virtual tire may be depicted in a cylindrical net model.

[0051] The virtual tire is depicted in the on-board camera coordinate system as if it had been photographed by the on-board camera 5. The virtual tire is then projected onto the ground.

[0052] FIG. 6 shows a schematic depiction of the shifted overlapping areas 41, 43. At each intersection of the boundary routes 21 to 24, the adjacent overlapping areas 41, 43 of the capture areas of these on-board cameras are shifted horizontally to points outside the silhouettes of the corresponding tires 51, 52 projected on the ground. In this way, the overlapping areas 41, 43 are dynamically processed. In this case, rectangular overlapping areas 41, 43 are formed when on-board cameras with large capture areas up to 180 degrees (fisheye cameras) are used. By shifting the overlapping areas 41, 43, it is possible to avoid moving parts or steered tires being captured by the side on-board cameras.

[0053] 7 shows a schematic representation to illustrate the creation of a mask 72 of a motor vehicle 1. Thus, the first mask 32 and the motor vehicle steerable tire second mask 71 are combined to create the mask 72 of the motor vehicle 1.

[0054] 8 shows a schematic depiction of a mask 72 of a motorized vehicle 1. To create a surround view of the motorized vehicle 1's surroundings, the camera images of multiple on-board cameras 5 are projected into areas of the motorized vehicle 1 that are not masked by the mask 72.

[0055] FIG. 9 illustrates a flow chart of a method for generating a surround view of a motorized vehicle 1 according to an embodiment of the present invention.

[0056] In a first step S1 a first mask 32 is created which represents the silhouette of the vehicle body projected onto the ground, followed by a second step S2 which represents the silhouette of the moving parts of the vehicle in their current state projected onto the ground.

[0057] The method further includes a step S3 of creating a mask of the motor vehicle 1 by combining the first mask and the second mask. In a final step S4, a surround view of the surroundings of the motor vehicle 1, which can be output to a display means 6 of the motor vehicle 1, is created from the camera images of the on-board camera 5 of the motor vehicle 1 using the mask of the motor vehicle 1. [Explanation of symbols]

[0058] 1. Motorized Vehicles 2. Equipment for creating surround view 3. Interface 4 Calculation means 5. Car Camera 6 Display means 21-24 Border Route 31 rectangle 32 First Mask 41-48 Capture area 51, 52 Tires 71 Second Mask 72 Motor vehicle mask S1-S4 Method Steps

Claims

1. 1. A method for generating a surround view of a motorized vehicle (1) having a body and parts that move relative to the body, the method comprising the steps of: A step (S1) of creating a first mask (32) representing the silhouette of the vehicle body projected onto the ground; creating (S2) a second mask (71) showing the silhouette of the moving parts of the vehicle (1) projected on the ground in their current state; creating (S3) a mask (72) of the motor vehicle (1) by combining the first mask (32) and the second mask (71); and A step (S4) of creating a surround view of the surroundings of the motorized vehicle (1) from the camera image of the on-board camera (6) of the motorized vehicle (1) using a mask (72) of the motorized vehicle (1).

2. 2. The method according to claim 1, characterized in that the moving parts include at least the steered wheels of the motor vehicle (1), and the second mask (71) is indexed depending on the steering angle of the motor vehicle (1) at that time.

3. 3. The method according to claim 2, characterized in that for at least one wheel of the motor vehicle (1), a virtual tire is created in an on-board camera coordinate system of an on-board camera (6) capturing said tire of the motor vehicle (1) depending on the steering angle at the time, and the virtual tire in said on-board camera coordinate system is projected onto the ground in order to create a silhouette of the tire of the motor vehicle (1) projected onto the ground from the viewing angle of the on-board camera (6).

4. 4. The method of claim 3, wherein the virtual tire is further created depending on the size and location of at least one tire on the motor vehicle.

5. 5. A method according to claim 3 or 4, characterized in that the virtual tire is modelled by a cylinder.

6. A method according to one of the preceding claims, characterized in that for each vehicle-mounted camera (6) a boundary line route (21) is determined which indicates the boundary of the vehicle body in the camera image of the vehicle-mounted camera (6), each boundary line route (21) is projected onto the ground, the intersections of the boundary line route projected onto the ground are determined, and the silhouette of the vehicle body projected onto the ground is determined as a surface enclosed by sections extending between the determined intersections of the boundary line routes.

7. 7. The method according to claim 6, characterized in that the boundary route (21) for each camera image is determined automatically.

8. 8. A method according to claim 6 or 7, characterized in that the boundary route (21) for each camera image is generated using a virtual model of the vehicle body.

9. 9. The method according to claim 6, wherein for each determined intersection point adjacent overlap areas of the complementary areas of the on-board camera (6) are determined and at least one of the overlap areas is shifted horizontally to an outer point of the silhouette of the moving parts of the motor vehicle (1) projected onto the ground.

10. 10. The method according to claim 9, characterized in that a surround view of the surroundings of the motor vehicle (1) is created on the basis of camera images generated by an on-board camera (6), and camera data of the camera images is not inserted in areas of the motor vehicle (1) that are masked by a mask (72).

11. 10. The method according to claim 1, characterized in that the surround view of the surroundings of the motor vehicle (1) is a ball-shaped view of the surroundings of the motor vehicle (1) or a top view of the surroundings of the motor vehicle (1) from above.

12. 10. A method according to claim 1, characterized in that the created surround view of the surroundings of the motorized vehicle (1) is displayed on a display means (2) of the motorized vehicle (1).

13. 1. An apparatus (2) for generating a surround view of a motorized vehicle (1) having a body and parts that move relative to the body, the apparatus comprising: an interface (3) configured to receive camera images from an on-board camera (6) of the motorized vehicle (1); and A computing means (4), adapted to carry out: creating a first mask (32) representing the silhouette of the vehicle body projected onto the ground; creating a second mask (71) showing the silhouette of the moving parts of the vehicle (1) projected on the ground in their current state; creating a mask (72) of the motor vehicle (1) by combining the first mask (32) and the second mask; and From the received camera images, a mask (72) of the motorized vehicle (1) is used to create a surround view of the surroundings of the motorized vehicle (1).

14. A motor vehicle (1) equipped with a device (2) for generating a surround view of the surroundings of the motor vehicle (1) according to claim 13.

15. 15. A motor vehicle (1) according to claim 14, equipped with display means (6) arranged to be able to display a surround view of the created motor vehicle (1) surroundings.

Images