Computer-implemented method of controlling graphical user interface on vehicle display

Abstract

The invention relates to a computer-implemented method for controlling a graphical user interface on a display of a vehicle, comprising the steps of receiving a current driving mode, if the current driving mode corresponds to a height or full automation, further comprising the steps of: receiving driver-side image data comprising imaging of a driver-side rear field of view; receiving co-pilot-side image data including imaging of a co-pilot-side rear field of view; the driver side image data and the co-driver side image data are displayed in the driver side digital rearview mirror area and the co-driver side digital rearview mirror area so that the data can not be obviously displayed in the manual driving mode; identifying an object in the driver-side image data and the co-driver-side image data; the invention relates to a method for displaying a graphical user interface, in particular a driver-side digital rear-view mirror region and a passenger-side digital rear-view mirror region, for detecting whether at least one triggering object has been identified in the objects, adjusting the graphical user interface, in particular the driver-side digital rear-view mirror region and the passenger-side digital rear-view mirror region, if at least one triggering object has been identified, and displaying the adjusted graphical user interface on a display of the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of graphical user interfaces, and in particular to a computer-implemented method for controlling a graphical user interface on a vehicle display, and a corresponding computer program product, a computer-readable storage medium, a data carrier signal, a data processing device, and a vehicle. Background Art

[0002] The Society of Automotive Engineers (SAE) has defined a progressive measure of vehicle driving mode automation in the form of the SAE J3016 standard. The relevant levels are: Level 0 (no automation); Level 1 (assistance system); Level 2 (semi-automation); Level 3 (conditional automation); Level 4 (high automation); and Level 5 (full automation).

[0003] In highly automated (Level 4) or fully automated (Level 5) vehicles, the driver no longer needs to intervene in the driving process. The driver no longer needs to observe the environment and events occurring in it. Therefore, it is not necessary to display certain information in the vehicle and this information could even be disruptive to the driver. Nevertheless, such content is currently still displayed inside the vehicle, for example in a graphical user interface on a display, regardless of the driving mode.

[0004] Therefore, in order to improve the driver's understanding and acceptance of automated driving, it is best not to display unnecessary information at all when the vehicle is driving in a semi-automated or fully automated manner, or to display this information in a way that is less intrusive to the driver. In this way, the driver can perform tasks unrelated to driving and improve their awareness of the vehicle's current driving mode. Summary of the invention

[0005] It is therefore an object of the present invention to provide a method for improving the display of a graphical user interface on a vehicle display taking into account the current driving mode.

[0006] According to the invention, this is achieved by a computer-implemented method and a computer program product, a computer-readable storage medium, a data carrier signal, a data processing device and a vehicle according to the respective independent claims. Advantageous embodiments can be derived from the dependent claims.

[0007] According to a first aspect of the invention, a computer-implemented method for controlling a graphical user interface on a vehicle display comprises a step of receiving a current driving mode. The driving mode characterizes the degree of autonomous driving of the vehicle. The driving mode may be provided, for example, by a central computing unit of the vehicle that knows the current driving mode, or by any other component in the vehicle that is aware of the current driving mode and can pass on this information.

[0008] The current driving mode is advantageously characterized by a classification according to SAE J3016. This standard describes the classification and definition of the degree of automation of motor vehicles traveling on the road in 6 levels. The first 3 levels 0-2 correspond to a low degree of automation, in which the driver monitors the driving area. The three highest levels 3-5 characterize the degree of automation in which the system monitors the driving area; specifically, level 4 describes a high degree of automation and level 5 describes full automation.

[0009] According to a first aspect of the invention, a user interface of a vehicle display includes at least one driver-side digital rearview mirror area and at least one passenger-side digital rearview mirror area. In this case, the display may be in the form of a single module, such as an LCD display. However, the display may also consist of multiple screens. The driver-side digital rearview mirror area is typically located in the vehicle near the driver-side rearview mirror. Similarly, the passenger-side digital rearview mirror area is typically located in the vehicle near the passenger-side rearview mirror.

[0010] The graphical user interface advantageously comprises at least one additional display area. In other words, the vehicle's display does not only consist of the driver's side digital rearview mirror area and the passenger's side digital rearview mirror area, so that other information (such as navigation elements, music control elements or temperature control adjustment elements) can also be displayed on the vehicle's display.

[0011] According to a first aspect of the invention, if the current driving mode corresponds to a high or full automation, the method comprises a further step. If this is the case, driver-side image data is received, which includes imaging of the driver-side rear view field. In addition, co-pilot-side image data is also received, which includes imaging of the co-pilot-side rear view field. For this purpose, for example, a camera mounted in a conventional rearview mirror position and oriented in a direction opposite to the vehicle's forward direction can be used. In addition, the driver-side image data is displayed in the driver-side digital rearview mirror area, while the co-pilot-side image data is displayed in the co-pilot-side digital rearview mirror area in a manner that makes the data less obvious than in a manual driving mode with a lower degree of automation.

[0012] In a further step of the computer-implemented method according to the first aspect of the invention, objects are identified in the driver-side image data and the passenger-side image data. This can be done by a conventional object recognition process. Known objects may include people, cyclists, other vehicles, ambulances, police cars or fire trucks, etc.

[0013] In a further step, a check is performed to determine whether at least one triggering object can be identified among the objects. In this case, a triggering object is a previously defined object of particular importance. This can be, for example, an ambulance, a police car or a fire truck. Objects that should attract the driver's attention despite the vehicle being in highly or fully automated driving mode are generally defined as triggering objects. This increases the driver's acceptance of automated driving, since they can continue to be informed of important objects and events in their surroundings.

[0014] If at least one triggering object is identified, the method includes the step of adjusting the graphical user interface. Specifically, the driver side digital rearview mirror area and the passenger side digital rearview mirror area are modified to make them more prominent, thereby attracting the driver's attention.

[0015] The trigger object advantageously corresponds to an ambulance, a fire truck, a police car, a bicycle or a person. These objects are usually objects that require the driver's high attention when they appear near the vehicle.

[0016] The graphical user interface is advantageously adapted by a size change, a color change, a contrast change and / or a blur change of the driver-side image data and / or the passenger-side image data.

[0017] If the vehicle is controlled manually by the driver, the image data on the driver's side and the front passenger side are displayed concisely and clearly in the driver's side digital rearview mirror area and the front passenger side rearview mirror area, respectively, because their content is important to the driver in manual driving mode. If the vehicle changes the driving mode to highly automated or even fully automated mode, the display in the digital rearview mirror area can be presented less concisely, for example by changing the contrast, in order to avoid unnecessarily drawing the driver's attention to this part of the display.

[0018] In a further step of the computer-implemented method according to the first aspect of the invention, a graphical user interface is displayed on a display of the vehicle.

[0019] According to a second aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to perform a method as claimed in one of the preceding claims. The computer program product can be executed entirely on board a vehicle. However, it can also be executed partly on board a vehicle and partly outside the vehicle, for example on a server. For example, all method steps except for identifying an object and checking a triggering object can be executed inside the vehicle, while the remaining steps can be executed on a server.

[0020] According to a third aspect of the invention, a computer-readable storage medium contains instructions which, when executed by a computer, cause the computer to perform one of the methods previously described.

[0021] According to a fourth aspect of the present invention, a data carrier signal transmits a computer program product as previously described.

[0022] According to a fifth aspect of the present invention, a data processing device comprises a processor configured to enable it to perform the steps of one of the previously described computer-implemented methods. In addition, the data processing device comprises the previously described computer-readable storage medium communicatively connected to the processor. In addition, the data processing device comprises a driver-side rearview mirror camera, a passenger-side rearview mirror camera, and a display communicatively connected to the processor.

[0023] According to a sixth aspect of the invention, a vehicle comprises the data processing device described previously. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The invention is described in more detail below based on exemplary embodiments with the aid of the accompanying drawings. In the drawings:

[0025] Figure 1 A flow chart showing a computer-implemented method for controlling a graphical user interface on a vehicle display;

[0026] Figure 2 Shown for Figure 1 A display of a computer-implemented method in;

[0027] Figure 3 Shown for Figure 1 A first view of a vehicle of the computer-implemented method;

[0028] Figure 4 Shown for Figure 1 a second view of the vehicle of the computer-implemented method; and

[0029] Figure 5 Shows that it can be executed Figure 1 A data processing device in which the computer implements the method.

[0030] List of reference numerals:

[0031] 100 Computer-implemented methods

[0032] 102 Graphical User Interface

[0033] 104 Display

[0034] 106 Vehicles

[0035] 108 Driver side digital rearview mirror area

[0036] 110 Passenger side digital rearview mirror area

[0037] 112 Additional display area

[0038] 114 First Receiving Step

[0039] 116 Control Module

[0040] 118 Decision-making Steps

[0041] 120 Second receiving step

[0042] 122 Driver's side rear view

[0043] 124 Third Receiving Step

[0044] 126 Passenger side rear view

[0045] 128 Driver side mirror camera

[0046] 130 Passenger side mirror camera

[0047] 132 First Display Step

[0048] 134 Second display step

[0049] 136 Identification Steps

[0050] 138 Inspection steps

[0051] 140 Trigger Object

[0052] 142 Control Steps

[0053] 144 Adjustment steps

[0054] 146 Show steps

[0055] 148 Data processing equipment

[0056] 150 processors

[0057] 152 Computer readable storage medium DETAILED DESCRIPTION

[0058] Figure 1 A flow chart of a computer-implemented method 100 for controlling a graphical user interface 102 on a display 104 of a vehicle 106 is shown.

[0059] Before explaining the various steps of the computer-implemented method 100, Figure 2 Consider the example of display 104 .

[0060] Figure 2 Shown for Figure 1 A display 104 of the computer-implemented method 100 is provided.

[0061] The display 104 shows a graphical user interface 102, for example, via a liquid crystal display (LCD). The graphical user interface 102 consists of a driver's side digital rearview mirror area 108 and a passenger's side digital rearview mirror area 110. In addition, the graphical user interface 102 includes an additional display area 112, on which information such as speed, fuel tank level, battery charge, audio control elements, or temperature control elements can be displayed. Part or all of the display 104 can also be in the form of a touch screen to enable interaction with the display 104.

[0062] refer to Figure 1 The computer-implemented method 100 includes a first receiving step 114 in which a current driving mode is received. The current driving mode characterizes the degree of automation of the vehicle 106. This classification is generally defined according to the SAE J3016 standard, where high automation is represented by level 4 and full automation is represented by level 5. For example, the current driving mode can be obtained from Figure 3 The control module 116 is shown receiving the current driving mode.

[0063] Figure 3 Shown for Figure 1 1. A first view of a vehicle 106 of the computer-implemented method 100 in FIG. 1. The control module 116 may be designed to execute the computer-implemented method 100. To this end, it may be part of another component, such as an infotainment system. However, the control module 116 may also be a separate component in the vehicle 106. The control module 116 may execute some or all steps of the computer-implemented method 100. The control module 116 may also assume other functions of the vehicle 106, such as controlling the infotainment system.

[0064] Reference again Figure 1 , the computer-implemented method 100 comprises a decision step 118, in which a check is made as to whether the current driving mode corresponds to high automation or full automation. If this is true (see Figure 1 t branch in ), then execute the second receiving step 120. In this step, the driver side image data is received, which includes the imaging of the driver side rearward field of view 122.

[0065] In a further step of the computer-implemented method 100 , in a third receiving step 124 , passenger-side image data is received, which includes an image of a passenger-side rearward field of view 126 .

[0066] Figure 3 and Figure 4 The vehicle 106 and rearward fields of view 122, 126 are shown. Figure 3 and Figure 4In the example of FIG. 1 , the driver-side image data is recorded by the driver-side rearview mirror camera 128. Similarly, the passenger-side image data is recorded by the passenger-side rearview mirror camera 130. The driver-side rearview mirror camera 128 is configured to be disposed on the driver-side portion of the vehicle 106 and can record the driver-side image data covering the driver-side rearward field of view 122. The same is true for the passenger-side rearview mirror camera 130, which is configured to be disposed on the passenger-side portion of the vehicle 106 and can record the passenger-side image data covering the passenger-side rearward field of view 126.

[0067] Reference again Figure 1 , the computer-implemented method 100 includes a first display step 132, wherein the driver-side image data are displayed in the driver-side digital rearview mirror area 108 in a manner such that they are displayed less noticeably than in the manual driving mode. In a second display step 134, the passenger-side image data are similarly displayed in the passenger-side digital rearview mirror area 110. The display of these data is also adjusted in a manner such that they are displayed less noticeably than in the manual driving mode. For example, to achieve such a less noticeable display, the size, contrast, or color range of the image data may be adjusted, or may be displayed only in grayscale, and / or the image data may be slightly smeared by blurring.

[0068] In the recognition step 136 of the computer-implemented method 100, objects are recognized in the driver-side image data and the passenger-side image data. Since the image data is an image of the rearward field of view 122, 126, typical identifiable objects are, for example, people, other vehicles, vegetation, road signs / signs, or lanes. Figure 2 In the example of FIG. 1 , for example, the image data contains a person, a sidewalk, a tree, and a road with a lane strip, so they are imaged in the digital rearview mirror areas 108 , 110 .

[0069] In a checking step 138, it is checked whether at least one triggering object 140 has been identified among the objects. Triggering objects 140 are in particular objects which require the driver's high attention when they are in the vicinity of vehicle 106. Typical triggering objects 140 are, for example, police cars, fire trucks or ambulances.

[0070] Control step 142 checks whether at least one trigger object 140 has been identified in the object. If this is the case ( Figure 1t branch in ), an adjustment step 144 is performed, in which the display of the graphical user interface 102 and in particular the driver's side digital rearview mirror area 108 and the passenger side digital rearview mirror area 110 is changed. Before the triggering object 140 is identified, the digital rearview mirror areas 108, 110 are not displayed very clearly. It is even possible to highlight the additional display area 112, for example by a colored border. If the driver looks at the display 104, their attention will be drawn to the additional display area 112.

[0071] When a trigger object 140 is identified in the image data, the digital rearview mirror areas 108, 110 are displayed in a different way, for example again in a much more concise way. This can be achieved, for example, by resizing the image, re-adjusting the contrast or blurriness. The identified trigger object 140 can also be highlighted graphically, for example using a colored highlighted display box around it. These measures draw the driver's attention to the digital rearview mirror areas 108, 110 and increase their understanding of the current traffic situation. These measures can also help inform the driver of the next steps that the autonomous driving vehicle 106 will take in this situation. This will increase the driver's acceptance of the autonomous driving mode.

[0072] In the final step of the computer-implemented method 100, namely, the display step 146, the graphical user interface 102 is displayed on the vehicle's display 104. Thus, the driver of the vehicle 106 can see the information.

[0073] pass Figure 1 The computer-implemented method 100 in FIG. 1 , on the one hand, informs the driver of the vehicle 106 that he is no longer paying attention to his surroundings. On the other hand, in these cases, the driver is aware of what is happening around him even during driving in the autonomous driving mode. This establishes situational awareness.

[0074] Figure 5 A data processing device 148 is shown, which includes a processor 150, which is configured so that it can execute Figure 1 In addition, the data processing device 148 includes a computer-readable storage medium 152 and a driver-side rearview mirror camera 128 and a passenger-side rearview mirror camera 130, all of which are communicatively connected to the processor 150. Finally, the data processing device 148 includes a display 104, which is also communicatively connected to the processor 150.

[0075] exist Figure 4 In the example of FIG. 1 , the vehicle 106 includes all elements of the data processing device 148 , wherein the processor 150 and the computer-readable storage medium 152 are part of the control module 116 .

Claims

1. A computer-implemented method (100) for controlling a graphical user interface (102) on a display (104) of a vehicle (106), wherein: The graphical user interface (102) includes at least one driver-side digital rearview mirror area (108) and at least one passenger-side digital rearview mirror area (110), wherein the method (100) includes the following steps: a) receiving (114) the current driving mode, If the current driving mode corresponds to high automation or full automation, the method (100) comprises the following further steps: b) receiving (120) driver-side image data, the driver-side image data comprising an image of a driver-side rearward field of view (122), c) receiving (124) passenger side image data, the passenger side image data including an image of a passenger side rearward field of view (124), d) displaying (132) the driver side image data in the driver side digital rearview mirror area (108) so that the driver side image data is not displayed as clearly as in the manual driving mode, e) displaying (134) the passenger side image data in the passenger side digital rearview mirror area (110) so that the passenger side image data is not as obvious as in the manual driving mode, f) identifying (136) an object in the driver-side image data and the passenger-side image data, g) checking (138) whether at least one triggering object (140) has been identified among the objects, If at least one triggering object (140) has been identified, the method (100) comprises the following further steps: h) adjusting (144) the graphical user interface (102), in particular adjusting the driver's side digital rearview mirror area (108) and the passenger side digital rearview mirror area (110), and i) displaying (146) the adjusted graphical user interface (102) on a display (104) of the vehicle (106).

2. The computer-implemented method of claim 1, wherein: The current driving mode is characterized by a classification according to SAE J3016.

3. The computer-implemented method of claim 1 or 2, wherein: The graphical user interface (102) includes at least one additional display area (112).

4. A computer-implemented method as claimed in any one of the preceding claims, characterized in that The trigger object (140) corresponds to an ambulance, a fire truck, a police car, a bicycle or a person.

5. The computer-implemented method according to any one of the preceding claims, characterized in that The adjustment (144) of the graphical user interface (102) is achieved by changing the size, color, contrast and / or blur of the driver-side image data and / or the passenger-side image data.

6. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method as claimed in any one of the preceding claims.

7. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5.

8. A data carrier signal transmitting the computer program product according to claim 6.

9. A data processing device (148), comprising: a) a processor (150) configured such that it can perform the steps of the computer-implemented method (100) as claimed in any one of claims 1 to 5, b) a computer-readable storage medium (152) as claimed in claim 7, communicatively coupled to the processor (150), c) a driver side rear view mirror camera (128) communicatively coupled to the processor (150), d) a passenger side rearview mirror camera (130) communicatively connected to the processor (150), and e) a display (104) communicatively coupled to the processor (150).

10. A vehicle (106) comprising a data processing device (148) as claimed in claim 9.

Images