Method for reducing image artefacts in a display of the surroundings of a vehicle, computer program, computing device and vehicle

Abstract

A method of reducing image artefacts in a display of the surroundings 290 of a vehicle 200, comprises the following steps: capturing of at least one current camera image of the vehicle’s surroundings;

Description

The present invention relates to a method for reducing image artefacts in a display of the surroundings of a vehicle, wherein an artefact area in a captured current camera image is determined as a function of the determined position of a recognised object and the captured camera image with an adapted artefact area is projected onto a virtual projection surface. The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to the invention. The invention also relates to a computing device with a computing unit configured to carry out the steps of the method according to the invention. The invention also relates to a vehicle with the computing device according to the invention. Prior art WO 2018 / 149625 A1 discloses a display method wherein images of the surroundings of a vehicle are captured and parts of these images are stored. A composite overall image is then created based on the current images and the stored images. The stretching of objects depicted in camera images and close to the camera after a coordinate transformation of the camera image and projection onto a projection surface is known and is also referred to as the Manhattan effect. Such nearby objects are often no longer recognisable to the user, at least in the projection screen display, due to the stretching. The displayed object stretch is also referred to as an artefact. To reduce such artefacts, it is known, for example, that a vertical part of the projection surface, on which an object close to the vehicle is projected, is shifted towards the vehicle based on the distance between the vehicle and the object. The document EP 3 073 446 B1 discloses a method for displaying vehicle surroundings of a vehicle. In the method, at least one actual camera image is captured by means of a camera arranged on the vehicle, which represents the vehicle surroundings. The actual camera image is then spatially spanned in a virtual space and a preselected image area of the actual camera image, in which image artefacts occur, is provided in several resolutions. A virtual camera image is then generated from the viewpoint of a virtual camera at a position in the virtual space, which represents the actual camera image, wherein the resolution used for the preselected image area of the actual camera image is the one of the resolutions provided, which reduces the visual presence of image artefacts. The document discloses that the preselected image area is preselected by examining a previously generated virtual camera image for image artefacts and determining an area associated with the image artefacts in the associated actual camera image. The purpose of the present invention is to improve the reduction of image artefacts in a displayed view of a vehicle’s surroundings, in particular for image artefacts caused by dynamic or self-moving objects. Disclosure of the invention The above problem is solved in accordance with the invention according to independent claims 1, 7, 8 and 10. The present invention relates to a method for reducing image artefacts in a display of the surroundings of a vehicle. The method comprises capturing at least one current camera image of the vehicle’s surroundings, in particular by means of at least one vehicle camera, advantageously by means of at least one wide-angle camera of the vehicle. Preferably, several vehicle cameras can be arranged on the vehicle to capture current camera images, for example four wide-angle cameras. In a further method step, an object in the surroundings of the vehicle is recognised, in particular a dynamic object. The object is preferably recognised as a function of distance data, which is captured or determined, in particular by means of at least one additional distance sensor; for example, an object is recognised when the distance falls below a threshold value. Alternatively or additionally, the object is recognised in particular as a function of the captured current camera image and / or based on stored historical image data, for example if a large difference is determined in the current camera image compared to the stored historical image data. When recognising the object, it is not crucial that the recognised object is precisely identified in terms of its type or kind, but that the presence of a dynamic object is recognised at all. The object is recognised in particular by a trained machine recognition method, especially a neural network. The position and optionally the movement of the object relative to the vehicle is then determined, in particular as a function of the current camera image captured and / or the stored historical image data. In a further method step, a movement or inherent movement of the vehicle is captured or determined. The determination of the movement of the vehicle preferably takes place as a function of an optical flow determined on the basis of the captured camera image and / or as a function of a determined position of the vehicle, which is determined, for example, by means of a satellite-based navigation system, and / or as a function of a captured steering angle, which is captured, for example, by means of a steering angle sensor and / or a yaw sensor. Alternatively or additionally, the determination of the movement of the vehicle is captured or determined as a function of a captured speed of the vehicle, which is captured, for example, by means of a speed sensor, and / or as a function of a rotational speed of the wheels captured by means of a speed sensor, and / or as a function of an acceleration of the vehicle captured by means of an acceleration sensor, and / or as a function of a control parameter for the drive motor of the vehicle. Furthermore, in another method step, at least partial area of the captured camera image is stored as historical image data. The partial area of the captured camera image depicts a predefined surrounding area around the vehicle. The respective current position relative to the vehicle is assigned to the historical image data, in particular by means of a coordinate transformation of the captured camera image, which is advantageously based on the known arrangement position and orientation of the vehicle camera and / or a calibration of the at least one vehicle camera. An object area of the currently captured camera image, which represents the determined position and / or the determined movement of the recognised object, is not stored as historical image information, in particular if the identified position and / or the future position of the recognised object determined on the basis of the identified movement has a distance relative to the vehicle less than or equal to a distance threshold value. In other words, the stored historical image data, whose associated position can be assigned to the determined position and / or the determined movement of the recognised object, is not updated by the captured current camera image. The position assigned to the stored image data relative to the vehicle is then adapted or updated as a function of the determined movement of the vehicle. In other words, the assigned position of the stored historical image data relative to the vehicle is corrected or adaptation based on the captured or determined movement of the vehicle. This means, for example, that historical image data is available from the vehicle’s journey, which depicts a surrounding area below the vehicle that is currently no longer captured by any camera image. As a result, historical image data with an assigned position is advantageously also available, which represents the surrounding area at the determined position of the recognised object, wherein the historical image data at the corresponding assigned position does not contain the dynamic object. In a further method step, an artefact area is determined in the at least one captured current camera image as a function of the determined position of the recognised object, wherein the artefact area depicts at least the recognised object as an image artefact. The determined artefact area advantageously does not correspond to the object area of the captured camera image. The artefact area is advantageously smaller than the object area. The determined artefact area is then adapted, wherein the image data of the captured camera image is replaced by the stored historical image data whose assigned position represents the artefact area. Finally, the user is shown a surrounding view of the vehicle’s surroundings, wherein the captured current camera image with the adapted artefact area is projected onto a virtual projection surface. The method has the technical advantage that image artefacts based on dynamic objects in the displayed surrounding view are reduced or prevented with the virtual projection surface, wherein the image artefacts would result from their stretching after the coordinate transformation. The advantage results in particular when these image artefacts are caused by dynamic objects with a position in the close surroundings around the vehicle and / or an actual or predicted movement of the dynamic object in the close surroundings. This gives the user a quicker or easier understanding of how the vehicle is being driven, as incorrect information or artefacts are reduced. In a preferred configuration of the invention, distance data between the vehicle and objects in the surroundings of the vehicle are captured or determined in an additional step. The capturing is carried out in particular by means of at least one additional distance sensor, for example an ultrasonic sensor, radar sensor and / or lidar sensor. Alternatively or additionally, the distance data can, for example, be determined by means of the vehicle camera using a “structure from motion” method or captured by an additional stereo camera. The object in the close surroundings of the vehicle is then recognised as a function of the distance data captured or determined. Alternatively or additionally, the determination of the position of the object relative to the vehicle is carried out as a function of the captured distance data. The advantage of this design is that a nearby object is reliably recognised even if the object is particularly close to the vehicle, for example if it obscures the field of vision of the vehicle camera, or if it can no longer be recognised as such by a trained machine recognition method due to only a partial image in the captured camera image. In one embodiment of the invention, the brightness and / or at least one colour value in the adapted artefact area of the captured current camera image is additionally changed. For example, in this embodiment, the historical image data in the artefact area can be adapted to grey values and / or the brightness of the pixels of the historical image data in the artefact area can be reduced. This makes it easy for the user to understand that this area or artefact area in the displayed surrounding view has been adapted or is not represented by a captured current camera image but by historical image data. In another embodiment, a frame is inserted around the adapted artefact area of the captured current camera image. This also indicates to the user that this area in the displayed surrounding view has been adapted or is not represented by a captured current camera image but by historical image information. In a further configuration, it can be provided that the adapted artefact area of the captured current camera image is blurred and / or blurring is generated in the adapted artefact area. This design allows the user to easily recognise which displayed areas of the displayed surrounding view are based on historical image information or do not reflect a currently captured camera image. In a further development of the present invention, a stored image information and / or a synthetic object model is provided or loaded from an electronic memory as a function of the recognised object, wherein the stored image information and / or the synthetic object model represents the recognised object or an indication of the adaptation of the artefact area. The provided image information and / or the synthetic object model is then inserted at the virtual position on the virtual projection surface, which represents the identified position of the recognised object. The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to the invention. The invention also relates to a computing device, in particular a control device, a decentralised computing device ora centralised computing device. The computing device comprises at least one first signal input for providing a first input signal which represents at least one current camera image of the vehicle’s surroundings captured by a vehicle camera. The computing device also has a signal output for outputting an output signal for a display device, wherein the output signal represents at least one view of the captured camera image with an adapted artefact area on a virtual projection surface. The computing device further comprises a computing unit, in particular a processor, which is configured to perform the steps of a method according to the invention. In an advantageous further embodiment, the computing device comprises a second signal input for providing a second input signal which represents distance data between the vehicle and at least one object in the surroundings of the vehicle captured by means of a distance sensor. The invention also relates to a vehicle comprising at least one vehicle camera, wherein the vehicle camera in particular comprises a wide-angle lens. The vehicle can have an optional distance sensor. The vehicle further comprises a computing device according to the invention. Further advantages result from the following description of embodiment examples with reference to the figures. Figure 1: Flow chart of the method as a block diagram Figure 2: Vehicle Figure 3: Surrounding view with artefact Figure 4: Surrounding view with adapted artefact area Figure 5: Stored image data of the predefined surrounding area Embodiment examples Figure 1 shows a schematic block diagram of the method. In a method step 100, at least one current camera image of the vehicle’s surroundings is captured. The capturing 100 is carried out in particular by means of at least one wide-angle camera. Optionally, a capturing 110 of distance data between the vehicle and objects in the surroundings of the vehicle can be provided, in particular by means of at least one additional distance sensor, for example an ultrasonic sensor, radar sensor and / or lidar sensor. In a further method step 120, an object in the surroundings of the vehicle is recognised, wherein the object is in particular dynamic or self-moving. For example, the recognised object is a person or a cyclist or another vehicle or third-party vehicle. It is possible that the type of recognised object is unknown and only the presence of an object is recognised. The object is advantageously recognised 120 by a trained machine recognition method as a function of the captured current camera image and / or based on captured distance data. In a further method step 130, the position of the object relative to the vehicle and / or the movement of the object relative to the vehicle are determined. The determination 130 of the position of the object relative to the vehicle advantageously takes place as a function of the captured distance data. In addition, a movement or inherent movement of the vehicle is determined in method step 140. The determination 140 of the movement of the vehicle advantageously takes place based on a captured steering angle of the vehicle and / or based on a captured speed of the vehicle. In the method step 150, at least a partial area of the captured camera image is stored as historical image data, wherein the pixel positions of the captured current camera image are changed for the storage, in particular by a coordinate transformation, for example a rectification and a perspective change of the captured current camera image is carried out. In other words, historical image data in particular is generated in step 150. The object area of the currently captured camera image, which corresponds to the determined position and / or the determined movement of the recognised object, is not stored as historical image data. The stored partial area represents a predefined surrounding area of the vehicle, wherein the predefined surrounding area represents in particular a horizontal surrounding area around the vehicle and is limited by a limit distance threshold value. The current position relative to the vehicle is assigned to the historical image data. In a further step 160, the position assigned to the stored image data relative to the vehicle is adapted as a function of the determined movement of the vehicle. Subsequently, in step 170, an artefact area is identified in the at least one captured current camera image as a function of the determined position of the recognised object, wherein the artefact area depicts at least the recognised object as an image artefact, wherein the artefact area is delimited in particular by a contour of the object. The artefact area is then adapted 180, wherein the image data of the captured camera image in the artefact area is replaced by the stored historical image data whose assigned position represents the artefact area. In the optional step 181, it can be provided to change or adapt the brightness and / or at least one colour value in the adapted artefact area of the captured current camera image. Furthermore, in the optional step 182, a frame can be inserted around the adapted artefact area of the captured current camera image, for example the inserted frame runs along the boundary of the determined artefact area. In a further optional step 183, a blurring or the generation of a blur in the adapted artefact area of the captured current camera image can be provided. In addition, in the optional step 184, stored image information and / or a synthetic object model can be provided or loaded as a function of the recognised object, wherein the stored image information and / or the loaded synthetic object model represent the recognised object or an indication of the adaptation of the artefact area. In a further optional step 185, this stored image information and / or the synthetic object model is inserted at the virtual position on the virtual projection surface, which represents the determined position of the recognised object. Finally, in step 190, a surrounding view of the surroundings of the vehicle is displayed for the user by projecting the captured camera image with the adapted artefact area onto a virtual projection surface, in particular by means of a display device. Figure 2 shows a schematic top view of a vehicle 200. In this embodiment example, the vehicle 200 comprises four wide-angle cameras as vehicle cameras 210 and twelve ultrasonic sensors as distance sensors 220. The vehicle cameras 210 and the distance sensors 220 each capture different surrounding areas of the surroundings 290 of the vehicle 200 and are each connected to the computing device 230 by means of a cable. The vehicle cameras 210 are each set up to capture a camera image of a surrounding area 211, wherein in each case two captured surrounding areas 211 overlap at the corners of the vehicle 200. The distance sensors 220 are arranged to capture distance data in the captured areas 221, which represent distances between objects 240, 250 in the surroundings 290 of the vehicle 200 and the vehicle 200. The computing device 230 is advantageously set up to generate a surrounding view based on the captured current camera images, stored image data and the captured distance data, wherein the surrounding view comprises in particular a virtual projection surface onto which the captured current camera images and the stored image data are projected. The projection surface is preferably adapted as a function of the captured distance data, in particular shifted towards the vehicle in the surrounding view or in the surrounding model if a static object is recognised in the close surroundings of the vehicle. A first dynamic object 240, such as a person, is present in the surroundings in front of the vehicle 200. A further second dynamic object 250 is located to the side of the vehicle 200, representing a bicycle, for example. The directions of movement of the dynamic objects 240, 250 are indicated schematically by arrows in Figure 2. In Figure 3, a virtual surrounding view 300 of the surroundings 290 of the vehicle 200 is shown schematically from a virtual perspective from above vertically downwards and corresponding to the situation in Figure 2, wherein this type of displayed surrounding view 300 is also referred to as a top-down view or a bird’s eye view. A semi-transparent vehicle model 310 is shown in the centre of the surrounding view 300 or the surrounding model, with the hatched underbody area 330 of the vehicle model 310 shown below it in Figure 3. Stored historical image data is projected onto the underbody area 330 of the projection surface 320. The historical image data was captured by one of the vehicle cameras 210 at an earlier point in time. In an artefact area 340 of the projection surface 320 shown hatched in Figure 3, the first dynamic object 240 close to the vehicle 200 is shown in front of the vehicle. As a result of the coordinate transformation to rectify the wide-angle image and to adapt the imaging perspective, the dynamic object 240 is depicted in the surrounding view in the artefact area 340 in a highly compressed manner, so that it is barely recognisable to the user. In a further artefact area 350 of the projection surface 320, shown hatched in Figure 3, the second dynamic object 250 close to the vehicle 200 is depicted in front of the vehicle 200. Due to its proximity to the vehicle camera, the image is also greatly stretched. This stretching of nearby objects shown in Figure 3 is also known as the “Manhattan” effect. If there are several nearby objects, the surrounding view can no longer be displayed in a meaningful way, as the artefact areas distort the view in a way that is unrealistic. Figure 4 shows a schematic view of the surroundings of the embodiment example in Figure 3 with an adapted artefact area. In the artefact areas 340 and 350 of the captured current camera image shown hatched in Figure 4, the image information has been replaced by stored historical image data, so that no object is depicted as an artefact in these artefact areas 340 and 350. In order to make the presence of an object recognisable for the user or driver, image information or a synthetic object model 440 for a person and image information or a synthetic object model 450 fora bicycle were inserted on the projection surface. In addition, the adapted artefact areas 340 and 350 are exemplarily identified for the user by a frame 410 and 420 respectively. It is therefore clear to the user that in these areas 340 and 350 historical image data was used to depict the surroundings. In this embodiment example, the dimensions of the predefined surrounding area for storing the historical image data correspond to the dimensions of the surrounding view. Figure 5 schematically shows stored image data in the predefined surrounding area 500, corresponding to the representation in Figure 4. The position of the stored historical image data is continuously adapted to the movement of the vehicle, wherein the historical image data is partly replaced by captured current camera images and partly deleted. For other historical image data, only the assigned position is adapted based on the determined movement of the vehicle. This means that historical image data stored for the underbody area 330 can also be made available. The object area 510 represents the surrounding area of the vehicle 200 in which the determined position of the person as a dynamic object 240 and a predicted movement of the object 240 as well as the resulting presumed artefact formation are located. In this object area 510, the stored image data is not temporarily overwritten or updated by current camera images captured, but only the position assigned to the historical image data is adapted based on the movement of the vehicle. The object area 520 represents the surrounding area of the vehicle 200 in which the determined position of the bicycle as a dynamic object 250 and a predicted movement of the object 250 as well as the resulting presumed artefact formation are located. In this object area 520, the stored image data is not temporarily overwritten or updated by current camera images captured, but only the position assigned to the historical image data is adapted based on the movement of the vehicle.

Claims

1. Method for reducing image artefacts in a display of the surroundings (290) of a vehicle (200) comprising the steps of• capturing (100) of at least one current camera image of the surroundings (290) of the vehicle (200),• recognising (120) an object (240, 250) in the surroundings (290) of the vehicle (200), in particular a dynamic object (240, 250),• determining (130) the position and / or the movement of the object (240, 250) relative to the vehicle (200), in particular the direction of movement and the speed of movement, and• determining (140) a movement of the vehicle (200),characterised in that the following steps are performed• storage (150) of at least a partial area of the captured camera image as historical image data, which depicts a predefined surrounding area of the vehicle, wherein the current position relative to the vehicle (200) is assigned to the historical image data, wherein an object area of the currently captured camera image, which corresponds to the determined position and / or the identified movement of the recognised object, is not stored as historical image data,• adapting (160) the position relative to the vehicle associated with the stored image data as a function of the determined movement of the vehicle (200),• determining (170) an artefact area (340, 350) in the at least one captured current camera image as a function of the determined position of the recognised object, wherein the artefact area (340, 350) at least depicts the recognised object (240, 250) as an image artefact,• adapting (180) the artefact area (340, 350), wherein the image data of the captured camera image in the artefact area (340,350) are replaced by those stored historical image data whose associated location represents the artefact area (340, 350), and• display (190) of a surrounding view (400) of the surroundings (290) of the vehicle (200) by projecting the captured camera image with the adapted artefact area (340, 350) onto a virtual projection surface (320).

2. Method according to claim 1, wherein the following step is carried out• capturing (110) of distance data between the vehicle (200) and objects (240, 250) in the surroundings (290) of the vehicle (200), in particular by means of at least oneadditional distance sensor (220), for example an ultrasonic sensor, radar sensor and / or lidar sensor,• wherein the recognising (120) of the object (240, 250) takes place as a function of the captured distance data, and / or• wherein the determination of the position of the object (240, 250) relative to the vehicle (200) takes place (130) as a function of the captured distance data.

3. Method according to one of the preceding claims, wherein the following step is carried out• changing (181) the brightness and / or at least one colour value in the adapted artefact area (340, 350) of the captured current camera image.

4. Method according to one of the preceding claims, wherein the following step is carried out• insertion (182) of a frame around the adapted artefact area (340, 350) of the captured current camera image.

5. Method according to one of the preceding claims, wherein the following step is carried out• blurring (183) the adapted artefact area (340, 350) of the captured current camera image.

6. Method according to one of the preceding claims, wherein the following steps are carried out• provision (184) of stored image information and / or a synthetic object model (440, 450) as a function of the recognised object (240, 250), wherein the stored image information and / or the synthetic object model (340, 350) represent the recognised object (240, 250) or an indication of the adaptation of the artefact area (340, 350), and• insertion (185) of the stored image information and / or the synthetic object model (340, 350) at the virtual position on the virtual projection surface (320), which represents the determined position of the recognised object (240, 250).

7. Computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the steps of the method according to any one of the preceding claims.

8. Computing device (230), in particular control unit, decentralised or centralised Computing device comprising at least the following components• a first signal input for providing a first input signal which represents at least one current camera image of the surroundings (290) of the vehicle (200) captured by means of a vehicle camera (210), wherein the vehicle camera (210) comprises in particular a wide-angle lens,• a signal output for outputting an output signal for a display device, wherein the output signal represents at least one view of the captured camera image with an adapted artefact area (340, 350) on a virtual projection surface (320), and• a computing unit, in particular a processor, configured to perform the steps of the method according to any one of claims 1 through 6.

9. Computing device according to claim 8, wherein the computing device additionally comprises the following component• a second signal input for providing a second input signal representing distance data captured by a distance sensor (220) between the vehicle (200) and at least one object (240, 250) in the surroundings (290) of the vehicle (200).

10. Vehicle (100) comprising at least one vehicle camera (210), an optional distance sensor(220) and a computing device (230) according to any one of claims 8 or 9.13

Images