COMPUTER-IMPLEMENTED METHOD FOR GENERATING ENVIRONMENTALLY SYNCHRONOUS IMAGE DATA

Abstract

The present invention relates to a computer-implemented method 130 for generating environmentally synchronous image data for a transparency-generating screen 104, comprising a first acquisition step 132 in which first image data 150 are acquired, which comprise an image of the environment that is not visible to a driver of a vehicle 148; a detection step 134 in which objects 108, 112, 116 are detected in the first image data 150; a second acquisition step 136 in which positions 110, 114, 118 of the objects 108, 112, 116 are acquired in relation to a head position of the driver; and a generation step 138, in which second image data 166 are generated from the first image data 150 at least partially based on the positions 110, 114, 118 such that from the driver's head position, the second image data 166 are perceived in an environmentally synchronous manner when they are displayed on the screen 104

Description

Technical area

[0001] The present invention relates to the field of image processing. In particular, the present invention relates to a computer-implemented method for generating environmentally synchronous image data for a vehicle screen. Furthermore, the present invention relates to a computer program product, a computer-readable storage medium, a data carrier signal, a data processing device, and a vehicle. Technical background

[0002] Improving safety measures for drivers is an important goal for all vehicle manufacturers. Unsafe traffic situations can arise from opaque structures in the vehicle body, which can obscure traffic-relevant events for the driver. Such opaque structures are usually parts of the vehicle body that ensure the mechanical rigidity required for occupant safety in the event of an accident. Examples of such opaque structures are the A, B, and C pillars, as well as the dashboard and trunk. These opaque structures disrupt the driver's field of vision, creating blind spots. Traffic-safety-relevant events or objects, such as a pedestrian crossing the road, a cyclist, or an overtaking car, can be obscured and may not be recognized by the driver in time.The conflicting requirements of high rigidity of the vehicle body on the one hand and a large field of vision for the driver on the other hand usually require a compromise.

[0003] Transparent A-pillars, hoods, or tailgates have been proposed as solutions. These displays, for example in the form of liquid crystal displays (LCDs) or organic light-emitting diodes (OLEDs), are placed inside the vehicle between the opaque structure and the driver. They display images from a camera located outside the vehicle, whose field of view captures the obscured area caused by the structure. Such an arrangement creates the illusion that the structure is transparent.

[0004] For example, EP 1 878 618 B1 discloses a support unit for capturing an image of an area that is not visible to a driver and that is created due to the presence of a pillar of a vehicle.

[0005] The solutions disclosed in the prior art have the disadvantage that the images are not displayed in synchronization with the surrounding environment. Synchronization with the surrounding environment means that the image displayed on the screen should be adjusted to represent the non-visible area as it would be perceived from the driver's perspective if no structure obstructed the view. In other words, the visible area should appear to the driver as if it were closely connected to the image of the non-visible area on the screen.

[0006] It is therefore the object of the invention to provide a computer-implemented method for generating environmentally synchronous image data. Further objects of the invention are to provide a computer program product, a computer-readable storage medium, a data carrier signal, a data processing device, and a vehicle. Disclosure of the invention

[0007] The object is achieved according to the invention by a computer-implemented method and a computer program product, a computer-readable storage medium, a data carrier signal, a data processing device, and a vehicle according to the respective main claims. Advantageous embodiments can be found in the subclaims.

[0008] According to a first aspect of the invention, a computer-implemented method for generating environmentally synchronous image data for a vehicle screen comprises a first step of acquiring first image data. The screen is arranged in the interior of the vehicle on a structure that blocks the view of a driver of the vehicle to an environment outside the vehicle. The first image data comprises an image of the environment that is not visible to the driver of the vehicle.

[0009] Examples of such a structure include A-, B-, or C-pillars, as well as trunks and dashboards. Examples of screens include liquid crystal displays (LCDs) or organic light-emitting diodes (OLEDs). The screen may well have a non-planar geometry to better adapt to the shape of the structure. The vehicle could be, for example, a motor vehicle, a motorcycle, an aircraft, an excavator, or the like.

[0010] An environmentally synchronous display means that the image shown on the display corresponds to the view that would be seen from the driver's perspective if the structure did not obstruct the view.

[0011] In a further step of the method, objects are detected in the first image data. Those skilled in the art are familiar with common methods for object recognition, such as YOLOv3 or similar.

[0012] In a further step of the process, the positions of the detected objects relative to the driver's head position are recorded. This can be done, for example, using two 3D cameras, one of which is mounted outside the vehicle and monitors the hidden area. A 3D camera can capture both image data and distance data, which can be converted into positions using object recognition and appropriate calibration.

[0013] The other of the two 3D cameras can be installed inside the vehicle and used for driver monitoring. Using a known position relationship between the 3D cameras, the position of the detected objects measured by the first 3D camera, and the position of the driver's head measured by the second 3D camera, the positions of the detected objects relative to the driver's head position can be calculated.

[0014] In a further step of the method, second image data are generated from the first image data, at least partially based on the positions. The second image data are generated in such a way that, from the driver's head position, the second image data are perceived synchronously when displayed on the screen.

[0015] If the head position and thus the field of vision of the driver as well as the field of vision of the 3-D camera that monitors the hidden area are known, the initial image data can be adjusted using simple geometric considerations (see Fig. 1).

[0016] In an advantageous embodiment, the second image data is generated by means of the following steps: - Removing the first image data in the frame areas belonging to the objects, thereby generating third image data, - Manipulating the first image data in the frame areas, thereby generating manipulated frame areas, - Inserting the manipulated frame areas into the third image data, and - Interpolating the third image data in interpolation areas near the manipulated frame areas.

[0017] The frame regions associated with the objects can, for example, correspond to the bounding boxes from an object detection. If the first image data in these frame regions are removed, the third image data will be missing the objects (and, depending on the shape of the frame regions, information surrounding the objects as well).

[0018] In the next step, the first image data in the frame areas (i.e. the image data of the objects) are manipulated to create manipulated frame areas.

[0019] Advantageously, the manipulation is realized by reducing, enlarging, shifting, cropping and / or geometric distortion.

[0020] The manipulated frame regions are then reinserted into the third image data. Preferably, a manipulated frame region is reinserted exactly where the corresponding original frame region was previously removed.

[0021] This can lead to a situation where some pixels in the third image data no longer have a color value assigned; for example, if the manipulation involves reducing the size of the frame area. In this case, the gap in the frame area is larger than the manipulated frame area to be inserted, resulting in areas that have no color value assigned.

[0022] In this case, a predetermined color value can be assigned to those areas in the third image data that have no color value assigned. This ensures error-free interpolation.

[0023] In the next step, the third image data is interpolated in areas close to the manipulated frame areas. This can be done, for example, by smearing several pixels along the edges of the frame areas. The purpose of the interpolation is to smooth out the discontinuities in the third image data caused by the removal, manipulation, and insertion, and to prevent gaps without image information.

[0024] In an advantageous embodiment, a distance map is generated from the positions and a reference distance in the method step of detecting positions. For example, a 3D camera can detect such a distance map. The reference distance serves to assign a distance to those pixels in the first image data where no object was detected and, consequently, no distance was assigned. The reference distance can, for example, be a value derived from experience or be calculated as an average across all distances detected.

[0025] Advantageously, the distance map is generated by interpolating between the positions and the reference distance. The reference distance can be ignored under certain circumstances, for example, if each pixel in the image data is assigned a position. If this is not the case, for example, for pixels in the edge areas, a reference distance can be used instead of a position.

[0026] Those skilled in the art are familiar with common methods for geometrically distorting and thus adapting image data for a specific perspective using a distance map. For example, EP 3 189 493 A1 describes a method for post-processing a digital photo to correct perspective distortions in the photo.

[0027] In an advantageous embodiment, the second image data is displayed on the vehicle’s screen.

[0028] A method as described above ensures that image data from an invisible area of ​​a vehicle can be displayed on a screen of the vehicle in an environmentally synchronous manner.

[0029] According to a second aspect of the invention, a computer program product comprises instructions which, when the program is executed by a computer, cause the computer to carry out a computer-implemented method as described above.

[0030] According to a third aspect of the invention, a computer-readable storage medium comprises the computer program product as described above.

[0031] According to a fourth aspect of the invention, a data carrier signal transmits the computer program product as described above.

[0032] According to a fifth aspect of the invention, a data processing device for generating environmentally synchronous image data for a screen of a vehicle comprises a screen, wherein the screen is arranged in the interior on a structure of the vehicle which blocks the view of a driver of the vehicle to an environment outside the vehicle.

[0033] The device also includes an evaluation unit. The screen is communicatively connected to the evaluation unit.

[0034] Advantageously, the evaluation unit comprises at least one processor and a computer-readable storage medium.

[0035] The device also has a detection unit, which is also communicatively connected to the evaluation unit.

[0036] In an advantageous embodiment, the vehicle structure is an A-pillar and / or a B-pillar and / or a C-pillar and / or a trunk and / or a dashboard. These elements often obscure the view of safety-relevant processes in the vehicle's surroundings.

[0037] In a particularly advantageous embodiment, the detection unit is formed with a first 3-D camera and a second 3-D camera.

[0038] Advantageously, the first 3D camera is arranged on the exterior of the vehicle in such a way that it at least partially captures the vehicle's surroundings, which are not visible to the driver due to the vehicle's structure. The first 3D camera can capture the first image data and the positions of the objects in spatial relation to itself.

[0039] The second 3D camera is positioned inside the vehicle in such a way that it can detect at least the driver's head position. For example, the second 3D camera is designed as a driver monitoring system that can detect driver fatigue or inattention. The driver's head position is determined in spatial relation to the second 3D camera.

[0040] If the spatial relationship between the first and second 3D cameras is known, the positions of the objects relative to the driver's head position can also be determined. Since the perspectives of the first 3D camera and the driver do not match, the first image data may need to be processed beforehand to correct this perspective distortion. Those skilled in the art are familiar with common methods for achieving this. For example, US 10 832 372 B2 discloses a method for adapting an image based on the perspective and the shape of a display device for a motor vehicle.

[0041] The acquisition unit can therefore acquire the first image data and positions of the objects necessary for the process, while the evaluation unit can detect the objects, assign the position to the objects and generate the second image data.

[0042] According to a fifth aspect of the invention, a vehicle comprises a data processing device as described above. The vehicle may, for example, be a motor vehicle, a truck, an aircraft, a ship, or the like. Summary of the characters

[0043] The invention is explained in more detail below using exemplary embodiments and figures. The figures show: Fig. 1: A schematic representation of the problem underlying the invention; Fig. 2: A flowchart of the computer-implemented method for generating environmentally synchronous image data; Fig. 3: A flowchart of a first embodiment of the computer-implemented method of Fig. 2; Fig. 4: A detailed flow diagram of the computer-implemented procedure from Fig. 3; Fig. 5: A schematic overview of a second embodiment of the computer-implemented method from Fig. 2; Fig. 6: A data processing device for generating environmentally synchronous image data; and Fig. 7: A vehicle with the data processing device from Fig. 6. Detailed description of the characters

[0044] Fig. 1 shows a schematic representation of the problem underlying the invention.

[0045] In Fig. 1, an observer 100 is shown looking at a screen 104. The screen 104, in turn, is arranged on a structure 102 of a vehicle 148. The structure 102 obscures an area outside the vehicle 148 for the observer 100 because it is transparent. This area is shown in Fig. 1 is indicated via the observer field of view 106.

[0046] In the area not visible to the observer 100, there is at least partially a first object 108 at a first position 110, a second object 112 at a second position 114, and a third object 116 at a third position 118. In Fig. 1 the positions 110,114,118 are represented as distances by the two-dimensional representation.

[0047] On the side of structure 102 facing away from observer 100, a first 3D camera 120 is arranged, which is characterized by a field of view 122. The discrepancy between the observer's field of view 106 and the camera's field of view 122 is the reason why image data of the environment recorded by the first 3D camera 120, if displayed unchanged on screen 104, does not appear environmentally synchronous. For example, a first piece of image information 126 would display too much of the second object 112 on screen 104. In contrast, a second piece of image information 128 would be missing from screen 104 to appear environmentally synchronous. Only objects located at the first position 110 would appear environmentally synchronous without a changed display, because only here does the observer's field of view 106 match the camera's field of view 122.In order to correct the image data of the first 3-D camera 120 with respect to the second object 112, for example, the first image information 126 would have to be removed from them and the resulting image data scaled to the size of the screen 104.

[0048] Fig. 2 shows a flowchart of a computer-implemented method 130 for generating environmentally synchronous image data.

[0049] In a first acquisition step 132, first image data 150 are acquired. The first image data 150 comprise an image of the surroundings, which is not visible to a driver of a vehicle 148. This is due to a structure 102 of the vehicle 148 that is opaque. Examples of such structures 102 are A-, B-, or C-pillars, trunks, or dashboards. The image data are to be displayed on a screen 104 of the vehicle 148, which is arranged on the structure 102 in the interior.

[0050] In a subsequent detection step 134, objects 108, 112, 116 are detected in the first image data 150. Methods of object recognition and classification are common means for accomplishing such tasks.

[0051] In a second detection step, 136 positions 110, 114, 118 are detected for the objects 108, 112, 116, which describe their location in relation to a head position of the driver.

[0052] In a generation step 138, second image data 166 are generated. Based on the first image data 150 and the positions 110, 100, 14, 118, the second image data 166 are generated such that, from the driver's head position, they are perceived as synchronous with the environment when displayed on the screen 104.

[0053] Fig. 3 shows a flowchart of a first embodiment of the computer-implemented method 130 of Fig. 2. In Fig. 3 is a specific embodiment of the generation step 138 of Fig. 2 is explained in more detail.

[0054] First, in a removal step 140, the first image data 150 is removed from the frame areas 152, 154, 156 corresponding to the objects 108, 112, 160. The result is stored in third image data 158.

[0055] Subsequently, in a manipulation step 142, the first image data 150 are manipulated in the frame areas 152, 154, 156. In the process, manipulated frame areas 160, 162, 164 are generated.

[0056] In a further step, insertion step 144, the manipulated frame areas 160, 162, 164 are inserted into the third image data 158.

[0057] Finally, in an interpolation step 146, the third image data 158 are interpolated in interpolation areas that are close to the manipulated frame areas 160, 162, 164.

[0058] To follow the procedure Fig. 3 to explain in more detail, shows Fig. 4 a detailed flow diagram of the computer-implemented method 130 of Fig. 3.

[0059] Fig. 4a) shows a vehicle 148 with a first 3D camera 120, whose field of view 122 captures an environment outside the vehicle 148 in first image data 150. The environment is not visible to a driver of the vehicle 148 due to an opaque structure 102. The area obscured by the structure 102, which is not visible to the driver, is represented by the observer field of view 106.

[0060] In the surroundings of the vehicle 148, a first object 108 is located at a first position 110 in the form of another vehicle. A second object 112 is located at a second position 114 in the form of a cyclist. Furthermore, a third object 116 is located at a third position 118 in the form of a pedestrian. Due to the different positions 110, 114, 118 and the associated discrepancy between the camera field of view 122 and the observer field of view 106, first image data 150 displayed unchanged would not appear to be synchronized with the environment.

[0061] Fig. Figure 4b) shows how, in the first image data 150, objects 108, 112, and 116 are first detected and assigned frame regions 152, 154, and 156 as well as positions d1, d2, and d3. Frame regions 152, 154, and 156 correspond, for example, to the bounding boxes from an object detection. As shown, frame regions 152, 154, and 156 can be configured, for example, as a rectangle, a circle, or a polygon.

[0062] In Fig. Figure 4c) shows how the first image data 150 in the frame areas 152, 100, 12, 116 associated with the objects 108, 100, 12, 116 were removed. What remains is third image data 158, in which no color value is assigned in the frame areas 152, 154, 156.

[0063] Fig. 4d) shows how the first image data 150 are manipulated in the frame areas 152, 154, 156. The manipulations 142 are based at least partially on the positions d1, d2, d3 assigned to the frame areas 152, 154, 156. For example, simple geometric considerations, as described in the description of Fig. 1 mentioned above.

[0064] The manipulation 142 results in manipulated frame areas 160,162,164, which, as in Fig. 4e), are inserted again into the third image data 158. The objects 108, 112, 116 are now represented true to their position 110, 114, 118 relative to the head position of the driver of the vehicle 148, thus generating environmentally synchronous second image data 166 from the first image data 150.

[0065] Since areas may now arise in which overlapping or missing image information is present, an interpolation 146 of the third image data 158 is carried out in interpolation areas that are located near the manipulated frame areas 160, 162, 164. This is illustrated in Fig. 4e) is illustrated by slightly greyed-out color values ​​in the first manipulated frame area 160.

[0066] Fig. 5 shows a schematic overview of a second embodiment of the computer-implemented method 130 from Fig. 2.

[0067] Also Fig. 5 shows the components necessary for the computer-implemented method 130 (cf. Fig. 1). As an alternative embodiment of the Fig. In the computer-implemented method 130 shown in Figure 2, a distance map 168, 170, 172 is created from the recorded positions 110, 114, 118 in the first recording step 132.

[0068] Fig. 5 shows how, for two detected objects 108, 112, three different distance maps 168, 170, 172 can be created. Using a reference distance and the positions recorded for the objects 108, 112, a stepped first distance map 168 is generated. A second distance map 170 is created by linear interpolation. A third distance map 172 is generated by nonlinear interpolation, for example, a seventh-degree polynomial interpolation. Subsequently, the first image data 150 can be geometrically distorted using a conventional method with the aid of one of the distance maps 168, 170, 172 so that they appear as if viewed from the driver's perspective. A corresponding method is disclosed, for example, in EP 3 189 493 A1.

[0069] Fig. Figure 6 shows a data processing device 174 for generating environmentally synchronous image data. The device 174 has an evaluation unit 186, a screen 104, and a capture unit 188. The screen 104 and the capture unit 188 are communicatively connected to the evaluation unit 186.

[0070] The data processing device 174 is intended to display image data synchronously with the environment on the screen 104 in a vehicle 148. The screen 104 is arranged in the interior of the vehicle 148 on a structure 102 that blocks the view of a driver of the vehicle 148 of an environment outside the vehicle 148.

[0071] In the example of Fig. 6, the evaluation unit 186 is configured with a processor 182 and a computer-readable storage medium 184. The processor 182 can receive first image data 150 from a first 3D camera 120. Furthermore, the processor 182 can receive data from a second 3D camera 180. The detection unit 188 is formed from the first 3D camera 120 and the second 3D camera 180.

[0072] The first 3D camera 120 is arranged on the exterior of the vehicle 148 such that it at least partially captures the surroundings of the vehicle 148, which are not visible to the driver due to the structure 102 of the vehicle 148. The second 3D camera 180 is arranged in the interior of the vehicle 148 such that it can capture at least the driver's head position.

[0073] A spatial relationship between the first 3D camera 120 and the second 3D camera 180 is stored in the computer-readable storage medium 184. Furthermore, a computer program product is stored on the computer-readable storage medium 184, which includes instructions that execute the steps of a computer-implemented method 130 as described above. Thus, from the driver's head position, which is detected relative to the second 3D camera 180, the positions of the objects 108, 112, 116, which are detected relative to the first 3D camera 120, and the spatial relationship between the first 3D camera 120 and the second 3D camera 180, the positions 110, 114, 118 of the objects 108, 112, 116 in relation to the driver's head position can be determined.

[0074] Fig. 7 shows a vehicle 148 with the data processing device 174 from Fig. 6. Furthermore, Fig.7 examples of the structures 102 are shown, which may obstruct the view of a driver. List of reference symbols 100 observers 102 Structure 104 screen 106 Observer field of view 108 First Object 110, d1 First Position 112 Second object 114, d2 Second Position 116 Third Object 118, d3 Third position 120 First 3-D camera 122 Camera field of view 124 Fourth Position 126 First image information 128 Second image information 130 procedures 132 First recording step 134 Detection step 136 Second recording step 138 Generation step 140 removal steps 142 Manipulation step 144 Insertion step 146 interpolation steps 148 vehicles 150 First image data 152 First frame area 154 Second frame area 156 Third framework area 158 Third image data 160 First manipulated frame area 162 Second manipulated frame area 164 Third manipulated frame area 166 Second image data 168 First distance map 170 Second distance map 172 Third distance map 174 Device 180 Second 3-D Camera 182 processor 184 Storage medium 186 Evaluation unit 188 registration unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 1 878 618 B1

[0004] EP 3 189 493 A1 [0026, 0068] US 10 832 372 B2

[0040]

Claims

[1] A computer-implemented method (130) for generating environmentally synchronous image data for a screen (104) of a vehicle (148), wherein the screen (104) is arranged in the interior on a structure (102) of the vehicle (148), wherein the structure (102) blocks the view of a driver of the vehicle (148) of an environment outside the vehicle (148), comprising the steps: a) capturing (132) first image data (150) which comprise an image of the environment, b) detecting (134) objects (108,112,116) in the first image data (150), c) detecting (136) positions (110,114,118) of the objects (108,112,116) in relation to a head position of the driver, and d) generating (138) second image data (166) from the first image data (150) at least partially based on the positions (110, 114, 118) such that from the driver's head position the second image data (166) are perceived in synchronism with the environment when they are displayed on the screen (104). [2] Computer-implemented method according to claim 1, characterized by that the second image data (166) are generated by means of the following steps: a) removing (140) the first image data (150) in the frame areas (152, 154, 156) associated with the objects (108, 112, 116), thereby generating third image data (158), b) manipulating (142) the first image data (150) in the frame areas (152, 154, 156), thereby generating manipulated frame areas (160, 162, 164), c) inserting (144) the manipulated frame areas (160, 162, 164) into the third image data (158), and d) interpolating (146) the third image data (158) in interpolation areas near the manipulated frame areas (160,162,164). [3] Computer-implemented method according to claim 2, characterized by that the manipulation (142) of the frame areas (160,162,164) is realized by reducing, enlarging, shifting, cropping and / or geometric distortion. [4] Computer-implemented method according to claim 1, characterized by that during the detection (132) of positions (110,114,118) a distance map (168,170,172) is generated from the positions (110,114,118) and a reference distance, wherein the distance map (168,170,172) is used in the generation (138) of the second image data (166). [5] Computer-implemented method according to claim 4 characterized by that the generation of the distance map (168,170,172) is achieved by interpolation between the positions (110,114,118) and the reference distance. [6] Computer-implemented method according to one of the preceding claims, characterized by that the second image data (166) are displayed on the screen (104) of the vehicle (148). [7] A computer program product comprising instructions which, when executed by a computer, cause the computer to execute a computer-implemented method (100) according to any one of the preceding claims. [8] A computer-readable storage medium (184) comprising the computer program product of claim 7. [9] A data carrier signal which transmits the computer program product according to claim 7. [10] A data processing device (174) for generating environmentally synchronous image data for a screen (104) of a vehicle (148), wherein the screen (104) is arranged in the interior on a structure (102) of the vehicle (148) which blocks the view of a driver of the vehicle (148) to an environment outside the vehicle (148), wherein the device (174) is configured to be able to carry out the steps of a computer-implemented method (100) according to one of claims 1 to 6, the device (174) comprising a) An evaluation unit (186), b) The screen (104) which is communicatively connected to the evaluation unit (186), and c) A detection unit (188) which is communicatively connected to the evaluation unit (186). [11] Data processing device according to claim 10, characterized bythat the structure (102) of the vehicle (148) is an A-pillar and / or a B-pillar and / or a C-pillar and / or a trunk and / or a dashboard. [12] Data processing device according to claim 10 or 11, characterized by that the detection unit (188) is formed with a first 3D camera (120) and a second 3D camera (180). [13] Data processing device according to claim 12, characterized by that the first 3D camera (120) is arranged on the outside of the vehicle (148) in such a way that it at least partially captures the surroundings of the vehicle (148), which are not visible to the driver due to the structure (102) of the vehicle (148), and that the second 3D camera (180) is arranged in the interior of the vehicle (148) in such a way that it can capture at least one head position of the driver. [14] Data processing device according to one of claims 10 to 13, characterized bythat the evaluation unit (186) has at least one processor (182) and a computer-readable storage medium (184). [15] Vehicle (148) comprising a data processing device (174) according to one of claims 10 to 14.

Images