Method for the creation of a representation of surroundings representing vehicle surroundings, control device, driver assistance system, and vehicle

Abstract

A method for creating a representation (16) of surroundings which represents vehicle surroundings and is composed of multiple camera images captured from different positions of the vehicle (1), the method involving the steps of: - providing at least two camera images captured at different positions of the vehicle (1); - identifying at least one object (13) protruding from a base (14) in at least one first of the camera images (12); - identifying an image sector (15) that is concealed by the at least one object (13) in the at least one first camera image (12); - acquiring, in at least one second of the camera images, image data at least partly corresponding to the concealed image sector (15) in the at least one first camera image (12); - creating the representation (16) of the surroundings from the at least one first camera image (12) and the image data from the at least one second camera image, wherein the image data from the at least one second camera image is at least partly superposed onto the image sector (15) concealed by the object (13) in the at least one first camera image (12).

Description

[0001] Description

[0002] Method for generating an environmental representation representing a vehicle environment, control device, driver assistance system and vehicle

[0003] The invention relates to a method for generating an environmental representation representing a vehicle's surroundings. Furthermore, the invention relates to a control device, a driver assistance system, and a vehicle.

[0004] Modern vehicles are increasingly equipped with driver assistance systems that support the driver in performing driving maneuvers by displaying a view of the surroundings. Such views can, for example, be so-called surround-view displays, which depict at least part of the vehicle's surroundings. Such a view of the surroundings is typically generated based on camera images captured using one or more of the vehicle's cameras.

[0005] The surrounding area can be displayed from a virtual camera position that differs from the actual camera arrangement on the vehicle. For this purpose, the camera images can be projected onto a virtual projection surface, for example, using the projection surface to determine a view of the surrounding area from a different vantage point. However, it may happen that individual areas of the vehicle's surroundings are obscured by objects in the vehicle's surroundings.

[0006] DE 10 2015 223 176 A1 discloses a driver assistance system for a vehicle with environmental sensors that detect the vehicle's surroundings. The sensor data from the environmental sensors is evaluated to detect obstacles in the vehicle's surroundings, and masking areas in the vehicle's surroundings are determined based on the detected obstacles. The masking areas are concealed by the obstacles and limit the field of view of the driver assistance system's optical environmental sensors. The determined masking areas can be covered with textured surfaces when displaying the camera images.

[0007] DE 10 2018 203 590 A1 describes a surround view system for a vehicle. Using an evaluation unit of the surround view system, an object is detected in recorded environmental data and the object's 3D shape is determined. A projection surface used to generate a surround view is then supplemented with the specific 3D shape of the object. Image areas obscured by the object can be further filled with a predefined color, with an interpolation of surrounding areas, and / or with a reflection of environmental data from the front of the object.

[0008] In order to give the driver the most complete overview possible of the vehicle's surroundings, it is desirable that there are as few obscured areas as possible in an environmental view.

[0009] The invention is based on the object of specifying an improved method for generating an environmental representation representing a vehicle environment, which in particular avoids or at least reduces the occurrence of areas without a representation of the actual vehicle environment.

[0010] This object is achieved according to the invention by a method for generating an environmental representation representing a vehicle environment from several camera images recorded at different positions of the vehicle, the method comprising the following steps:

[0011] - Providing at least two camera images taken at different positions of the vehicle,

[0012] - Determining at least one object protruding from a base surface in at least a first of the camera images, - Determining an image area of ​​the at least one first camera image that is hidden by the at least one object,

[0013] - Determining image data which at least partially correspond to the hidden image area in the at least one first camera image in at least a second of the camera images,

[0014] - generating the environmental representation from the at least one first camera image and the image data of the at least one second camera image, wherein the image area hidden by the object in the at least one first camera image is at least partially superimposed with the image data from the at least one second camera image.

[0015] According to the invention, the surroundings representation is generated from multiple camera images, with at least two of the camera images being recorded from different positions of the vehicle. This can be the case, for example, if the vehicle has moved between the respective recordings of the camera images. The camera images recorded at different positions of the vehicle or from different locations have an at least slightly different viewpoint or angle of the vehicle's surroundings.

[0016] The camera images can be recorded by one or more cameras. The one or more cameras can, in particular, be cameras of the vehicle, which can, for example, each be designed as a front camera, a rear-view camera, and / or a side camera arranged on the side of the vehicle and capturing a lateral area. In particular, the one or more cameras that provide the camera images can be part of a surround-view system of the vehicle. The camera images can be transmitted from the cameras to a control device designed to carry out the method via an internal or external vehicle communication connection. In at least a first of the camera images, an object is detected that protrudes from a base surface. The base surface can, for example, be a road surface or the like.The object can in particular be an object which obscures an area behind the object, starting from the camera.

[0017] According to the invention, an object can be detected which is located within a predetermined distance around the vehicle and / or which has a diameter in relation to the roadway plane below a predetermined limit and / or a height above a further predetermined limit. In this way, for example, objects can be detected which are relatively close to the vehicle and have a post-like shape. Thus, in particular, objects can be detected which obscure a narrow area of ​​the vehicle's surroundings in the first camera image. The object can be, for example, a post, a pole, a tree trunk, a pillar or the like. Additionally or alternatively, other types of objects can also be detected.

[0018] Subsequently, the image area obscured by the at least one object in the first camera image is determined. In other words, the part of the first camera image in which a section of the vehicle surroundings behind the object is obscured by the object is determined. This can, in particular, be the image area of ​​the first camera image in which the object is depicted.

[0019] In at least a second of the camera images, image data is then determined which at least partially correspond to the image area obscured by the object. Because the second camera image was taken when the vehicle was at a different location, the object is also shown in the second camera image from a different observation position or from a different viewing angle. A section of the vehicle's surroundings that is obscured from the perspective of the first camera image can therefore be visible in at least one second camera image. The image data determined in the at least one second camera image therefore show at least a part of the vehicle's surroundings that is obscured by the object in the first camera image. In particular, the image data can also describe the obscured area completely.

[0020] The second camera image can, in particular, have been recorded by the same camera as the first camera image. In the case of a moving vehicle, the second camera image can be recorded at a different time than the first camera image. It is possible for the image data to be obtained from a single second camera image or for the image data to be obtained from multiple second camera images, with a portion of the image data being able to be obtained from each of the multiple second camera images.

[0021] The environmental information is then generated from the at least one first camera image and the image data of the at least one second camera image, wherein the image area obscured by the object in the at least one first camera image is at least partially overlaid with the image data from the at least one second camera image. This ensures that at least part of the image area of ​​the first camera image obscured by the object now contains image data corresponding to the actual vehicle surroundings. This has the advantage that the actual vehicle surroundings can be better recognized in the environmental representation. In the environmental representation, only the image data obtained from the second camera image can be displayed in the image area obscured by the object, or a combination or mixture of the image data with a representation of the object can be generated.

[0022] The surroundings display can, for example, be displayed on a vehicle display device so that it is visible to the driver. The use of the surroundings display according to the invention advantageously reduces blind spots or areas that cannot be seen, thereby improving the recognition of the current traffic situation. In particular, other road users and / or hazards can be recognized earlier and / or more accurately, which can advantageously improve road safety.

[0023] According to the invention, it can be provided that the at least one second camera image was recorded in a moving vehicle prior to the at least one first camera image. The second camera image can be a camera image recorded immediately before the first camera image. Alternatively, one or more second camera images can be used, and these can be selected from a plurality of camera images recorded prior to the first camera image.

[0024] In a preferred embodiment of the invention, it can be provided that the at least one first camera image and the at least one second camera image are generated by a camera arrangement comprising at least one camera, wherein the camera images are stored, in particular during a movement of the vehicle, in a memory device of the vehicle, in particular in a ring buffer. In particular, the at least one first camera image and the at least one second camera image can be generated by the same camera, since in this case there are only slight differences in the respective environmental areas imaged. By using a ring buffer, camera images recorded continuously within a predetermined, previous period of time can advantageously be stored, wherein camera images from further back in time are automatically discarded.

[0025] According to the invention, the object in the at least one first camera image can be identified using an analysis of the image content, in particular using a structure-from-motion algorithm and / or semantic segmentation, and / or using sensor data from at least one distance sensor of the vehicle. The at least one distance sensor can be, for example, an ultrasonic sensor, a radar sensor, and / or a lidar sensor.

[0026] The image content can be evaluated, for example, by a vehicle control system configured to carry out the method. A structure-from-motion algorithm can detect and differentiate between various static objects and moving objects in the vehicle's surroundings. Semantic segmentation allows different image sections to be assigned to different object types from a predefined set of object types based on their image content.

[0027] According to the invention, it can be provided that the image area obscured by the at least one object in the at least one first camera image is determined based on a contour of the at least one object determined from the first camera image. In particular, the obscured image area can be determined based on an outer contour of the object. For objects that are not closed bodies, for example, a fence or the like, the obscured image area can also comprise several sub-areas, each obscured by a portion of the object.

[0028] In a preferred embodiment of the invention, it can be provided that the image data corresponding to the obscured image area in the at least one second camera image is determined by evaluating the image content of the at least one second camera image and / or based on content information describing the image content of the at least one second camera image. The determination can be carried out, in particular, taking into account a distance traveled by the vehicle between the recording times of the first camera image and the second camera image.

[0029] According to the invention, it can be provided that, in the case of image content which corresponds only to a partial area of ​​the hidden image area, the further partial area of ​​the hidden area is filled with image data in a predetermined color and / or only with image data of the object. Depending on the extent of the object or of the image area of ​​the first camera image hidden by the object, it can happen that image data is not available for the entire hidden image area in the available second camera images. In this case, the partial area of ​​the hidden image area for which no image data is available can be displayed in a predetermined color, for example black. Additionally or alternatively, only the image data from the first camera image describing the object can be displayed in this part.

[0030] In a preferred embodiment of the invention, the object can be represented partially transparently in the environment representation. The image data for the hidden area obtained from the at least one second image can, for example, be overlaid in a partially transparent manner with the image data of the object present in this area, so that both the object and the area hidden by the object can be viewed in the environment representation. The overlay of the image data from the at least one second camera image with the first camera image can, for example, be generated using alpha blending. Additionally or alternatively, other methods for at least partially transparently overlaying images can also be used.

[0031] Furthermore, the invention relates to a control device configured to carry out a method according to the invention. The control device can provide the camera images, for example, by being configured to receive them via a communication connection and / or by retrieving them from a storage device of the vehicle.

[0032] The invention also relates to a driver assistance system comprising at least one camera and a control device according to the invention.

[0033] For a vehicle according to the invention, it is provided that it comprises a driver assistance system according to the invention.

[0034] All advantages and configurations described above with respect to the method according to the invention apply correspondingly to the control device according to the invention, the driver assistance system according to the invention, and the vehicle according to the invention, and vice versa. Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show:

[0035] Fig. 1 shows an embodiment of a vehicle according to the invention comprising an embodiment of a driver assistance system according to the invention,

[0036] Fig. 2 is a block diagram of an embodiment of a method according to the invention,

[0037] Fig. 3 an example of a first camera image and

[0038] Fig. 4 shows an embodiment of an environment representation generated using the method according to the invention.

[0039] Fig. 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a

[0040] A passenger car, a truck, or another type of commercial vehicle. It is also possible for vehicle 1 to be a non-motorized vehicle such as a trailer, or a combination consisting of a towing vehicle and a trailer. Vehicle 1 can also be a rail-bound vehicle, such as a tram or the like.

[0041] The vehicle 1 comprises an exemplary embodiment of a driver assistance system 2, which comprises one or more cameras 3 and a control device 4. In the present exemplary embodiment, the vehicle 1 comprises four cameras 3, which form a camera arrangement designed as a surround-view system. A first camera 5 is arranged as a front camera of the vehicle 1, a second camera 6 as a rear-view camera of the vehicle 1, and a third camera 7 and a fourth camera 8 are each arranged as a side camera of the vehicle 1. The control device 4 can be designed, for example, as a microcontroller, as a processor, or as another type of computing device.

[0042] The driver assistance system 2 further comprises a display device 9, via which graphic information can be presented to a driver or a user of the vehicle 1. The display device 9 can, for example, be one or more screens or displays arranged in an interior of the vehicle 1.

[0043] The vehicle 1 further comprises at least one distance sensor 10. The at least one distance sensor 10 can be used to determine the distances to objects, in particular to objects located in front of, next to, and behind the vehicle. The distance sensor 10 can be designed, for example, as a radar sensor, a lidar sensor, and / or an ultrasonic sensor. The vehicle 1 can, in particular, have a plurality of distance sensors 10, which can be designed as any combination of different sensor types and arranged at different positions on the vehicle 1.

[0044] Using the cameras 3, image data can be generated from a sub-area of ​​the surroundings of the vehicle 1. The cameras 3 and the at least one distance sensor 10 are connected to the control unit 4 via a communication connection 11. The communication connection 11 can be, for example, a bus connection such as a CAN bus or the like. Image data or camera images are transmitted, in particular continuously, from the cameras 3 to the control unit 4 via the communication connection 11.

[0045] The control device 4 is designed to carry out a method for generating an environmental representation representing the vehicle's surroundings. The control device 4 uses a plurality of camera images recorded by the cameras 3 at various positions of the vehicle 1 for this purpose. The various positions of the vehicle 1 at which the camera images were recorded can, for example, arise in the case of continuously recorded camera images due to the vehicle 1 moving between the recording times of the camera images. It is possible for location information describing the position of the vehicle 1 to be assigned to the camera images.

[0046] In order to access camera images recorded at a previous vehicle position, at least some of the recorded camera images can be stored in the vehicle 1, for example, in a storage device of the control device 4 and / or in a storage device of the respective camera. The storage device can be, for example, a ring buffer that stores the camera images from the cameras 3 and, if applicable, location information associated with the camera images, at least for a specified period of time.

[0047] Fig. 2 shows a block diagram of an embodiment of a method for generating an environmental representation representing the vehicle environment.

[0048] In a first step S1, at least two camera images recorded at different positions of the vehicle are provided. The camera images can, for example, be generated by one of the cameras 3 and stored in the storage device of the vehicle 1, which may be configured as a ring buffer, so that the control device 4 can access the stored camera images.

[0049] In the present exemplary embodiment, the control device 4 uses a first camera image and a second camera image, which were recorded by the same camera 3. The second camera image was recorded before the first camera image of a moving vehicle 1. A positional relationship between the camera images can be determined by the control device 4, for example, from the image content of the camera images and / or from the location information associated with the camera images.

[0050] Subsequently, in a step S2, the control device 4 determines at least one object protruding from a base surface in at least a first of the camera images. The base surface can be, in particular, a road surface or a traffic or surrounding area adjacent to the roadway.

[0051] Fig. 3 schematically shows a first camera image 12, in which an object 13 in the form of a pole is depicted. The object 13 protrudes at least substantially perpendicularly from a base surface 14, in this case a roadway. In the first camera image 12, the object 13 can be identified, for example, using an analysis of the image content, in particular using a structure-from-motion algorithm and / or using semantic segmentation. Additionally or alternatively, the object 13 can also be detected using the at least one distance sensor 10 of the vehicle 1.

[0052] For example, an object 13 can be detected which is located within a predetermined distance around the vehicle 1 and / or which has a diameter relative to the road surface below a predetermined limit and / or a height above a further predetermined limit. The diameter of the object 13 and / or the height of the object 13 can be determined by means of the control device 4, for example by evaluating the image content of the first camera image 12, and / or with the aid of the at least one distance sensor 10. In this context, the diameter of the object 13 relative to the road surface is understood to mean the extent of the object 13 in the longitudinal direction and the transverse direction of the vehicle 1.

[0053] In step S3, the control device 4 determines the image region 15 of the first camera image 12 that is obscured by the at least one object. The obscured image region 15 can be determined, for example, based on a contour 16 of the at least one object 13 determined from the first camera image 12. In addition to an outer contour of the object 13, which can preferably be used for objects that are not a closed body, other types of contours of the object 13 can also be used to determine the obscured image content 15. For example, in the case of an object 13 designed as a fence, the obscured image region 15 can also comprise a plurality of sub-regions, each obscured by a section of the object 13.

[0054] Subsequently, in step S4 of the method, image data are determined from the second camera image, which at least partially correspond to the obscured image area 15 in the first camera image 12. Since the second camera image was recorded from a different position of the vehicle 1, the viewing angle of the object 13 and thus also an area of ​​the image background obscured by the object 13 differ.

[0055] The second camera image is selected from a plurality of second camera images available in the storage device of the vehicle 1, in particular such that the area 15 obscured in the first camera image 12 is not obscured by the object 13 in the second camera image due to the different observation point. In this way, image data can be extracted from the second camera image that corresponds to the area 15 behind the object 13 obscured in the first camera image 12.

[0056] The image data corresponding to the hidden image area 15 in the second camera image can be determined, for example, by means of an evaluation of the image content of the second camera image by the control device 4. Additionally or alternatively, it is possible to use content information assigned to the second camera image, which describes the image content of the second camera image. The content information can, for example, be a description of the image content of the second camera image obtained using semantic segmentation or similar methods. The content information of the second camera image can, for example, also be stored in the storage device, so that the content information can also be accessed at a later time for creating the environmental representation.

[0057] Furthermore, to determine the image data corresponding to the hidden image area 15 in the second camera image, a comparison can be made between the image content of the first camera image 12 and the second camera image, so that, in particular, image areas not visible in the first camera image 12 can be determined in the second camera image. By taking into account the arrangement of the image areas in the camera images, the image data of the second camera image corresponding to the hidden image area 15 can be determined, for example.

[0058] In step S5, the control device 4 generates a representation of the surroundings from the first camera image 12 and the image data of the second camera image. In this case, the image area 15 obscured by the object 13 in the first camera image 12 is at least partially overlaid with the image data from the at least one second camera image. The representation of the surroundings can be displayed, in particular, on the display device 9 of the vehicle 1 for a driver of the vehicle 1.

[0059] Fig. 4 shows an example of an environment representation 16. The object 13 can in particular be shown partially transparent in the environment representation, so that in addition to the background 17 hidden by the object 13 (in Fig.

[0060] 4 schematically shown in dashed lines), the object 13 or its contour or the like can still be recognized in the environmental representation 16. In addition to the background 17 schematically shown in Fig. 4, an at least partially transparent representation of the object 13 in the environmental representation 16 also allows other road users and / or sources of danger for the operation of the vehicle 1 to be better recognized in the environmental representation 16.

[0061] An at least partially transparent representation of the object 13 can be achieved, for example, by a blending method, for example by alpha blending. In this case, for example, the image data of the second camera image and / or the image data of the first camera image 12 describing the object can be provided with a transparency value and overlaid accordingly. It is possible that, in the case of an image content which corresponds only to a partial area of ​​the concealed image area 15, the further partial area of ​​the concealed area 15 is filled with image data in a predetermined color. This can be necessary, for example, if the object 13 conceals an image area 15 which is not completely depicted in the one or more available second camera images.Subregions of the hidden image area for which no image data is available in the second camera images can be displayed, for example, in black or another color. Alternatively, in a subregion of the hidden area for which no image data is available, the object 13 can also be displayed without transparency in the environment representation 16, so that the object 13 in the environment representation 16 is displayed at least partially transparent only in certain regions.

[0062] It is possible for the environmental representation 16 to be generated from more than one first camera image 12. In particular, first camera images from all vehicle cameras 3 can be used, for example, to generate a surround view or a 360° view of the vehicle's surroundings. In one or more of the first camera images 12, one or more objects 13 can be detected and correspondingly replaced or at least partially overlaid by image data from one or more of the second camera images.

Claims

Patent claims 1 . Method for generating an environmental representation (16) representing a vehicle environment from several camera images recorded at different positions of the vehicle (1), comprising the steps: - Providing at least two camera images taken at different positions of the vehicle (1), - Determining at least one object (13) protruding from a base surface (14) in at least a first of the camera images (12), - determining an image area (15) of the at least one first camera image (12) hidden by the at least one object (13), - Determining image data which at least partially correspond to the hidden image area (15) in the at least one first camera image (12) in at least a second of the camera images, - generating the environmental representation (16) from the at least one first camera image (12) and the image data of the at least one second camera image, wherein the image area (15) concealed by the object (13) in the at least one first camera image (12) is at least partially superimposed with the image data from the at least one second camera image.

2. Method according to claim 1, characterized in that the at least one second camera image was recorded in a moving vehicle (1) before the at least one first camera image (12).

3. Method according to claim 1 or 2, characterized in that the at least one first camera image (12) and the at least one second camera image are generated by a camera arrangement comprising at least one camera (3), wherein the camera images, in particular during a movement of the vehicle (1), in a storage device of the vehicle (1), in particular in a ring buffer.

4. Method according to one of the preceding claims, characterized in that an object (13) is determined which is arranged within a predetermined distance around the vehicle (1) and / or which has a diameter in relation to a roadway plane below a predetermined limit value and / or a height above a further predetermined limit value.

5. Method according to one of the preceding claims, characterized in that the object (13) in the at least one first camera image (12) is determined by means of an evaluation of the image content, in particular by means of a structure-from-motion algorithm and / or by means of semantic segmentation, and / or by means of sensor data from at least one distance sensor (10) of the vehicle (1).

6. Method according to one of the preceding claims, characterized in that the image area (15) hidden by the at least one object (13) in the at least one first camera image (12) is determined on the basis of a contour of the at least one object (13) determined from the first camera image (12).

7. Method according to one of the preceding claims, characterized in that the image data corresponding to the hidden image area (15) in the at least one second camera image are determined by means of an evaluation of the image content of the at least one second camera image and / or based on a content information describing the image content of the at least one second camera image is determined.

8. Method according to one of the preceding claims, characterized in that, in the case of an image content which corresponds only to a partial area of the hidden image area (15), the further partial area of the hidden area is filled with image data in a predetermined color and / or only with image data of the object (13).

9. Method according to one of the preceding claims, characterized in that the object is shown partially transparent in the environmental representation.

10. Control device configured to carry out a method according to one of the preceding claims.

11. A driver assistance system comprising at least one camera and a control device according to claim 10.

12. Vehicle comprising a driver assistance system according to claim 11.

Images