Computer-implemented method for controlling a graphical user interface on a vehicle display

Abstract

A computer-implemented method (100) for controlling a graphical user interface (102) on a display (104) of a vehicle (106) comprises a step of receiving (114) a current driving mode, wherein, if the current driving mode corresponds to a high or full automation, further comprises the step of receiving (120) driver-side image data, comprising an image of a driver-side rear view field (122); further comprises the step of receiving (124) passenger-side image data, comprising an image of a passenger-side rear view field (124); further comprises the step of displaying (132) the driver-side and passenger-side image data in the driver-side digital rearview mirror area (108) such that they are displayed less obviously than in a manual driving mode;further comprises the step of recognizing objects (136) in the driver-side image data and the passenger-side image data; further comprises the step of checking (138) whether at least one trigger object (140) has been recognized in the objects, wherein, if at least one trigger object (140) has been recognized, an adaptation (144) of the graphical user interface (102) takes place, in particular of the driver-side digital rear-view mirror area (108) and the passenger-side digital rear-view mirror area (110), as well as the display (146) of the adapted graphical user interface (102) on the display (104) of the vehicle (106).;

Description

Technical area

[0001] The present invention relates to the field of graphical user interfaces. In particular, the present invention relates to a computer-implemented method for controlling a graphical user interface on a vehicle display, as well as a corresponding computer program product, a computer-readable storage medium, a data carrier signal, a data processing device, and a vehicle. Technical background

[0002] The Society of Automotive Engineers (SAE) International has defined a progressive level of automation for a vehicle's driving mode in the form of the SAE J3016 standard. The corresponding levels are: Level 0 (No Automation); Level 1 (Assistance Systems); Level 2 (Partial Automation); Level 3 (Conditional Automation); Level 4 (High Automation); and Level 5 (Full Automation).

[0003] In a highly (Level 4) or fully automated (Level 5) vehicle, the driver no longer needs to intervene in the driving process. The driver no longer needs to observe the environment and the events occurring within it. This makes the display of some information in the vehicle's interior unnecessary; it may even be perceived as distracting for the driver. Nevertheless, such content is currently still displayed in the vehicle's interior, for example, on a graphical user interface, regardless of the driving mode.

[0004] To improve driver understanding and acceptance of automated driving, it is therefore desirable to display unnecessary information either at all or in a way that is less disruptive to the driver when the vehicle is moving in partially or fully automated mode. This allows the driver to perform non-driving-related tasks and increases awareness of the vehicle's current driving mode. Disclosure of the invention

[0005] It is therefore the object of the invention to provide a method for the improved representation of a graphical user interface on a display of a vehicle, which takes into account the current driving mode.

[0006] This is achieved according to the invention by a computer-implemented method, a computer program product, a computer-readable storage medium, a data carrier signal, a data processing device, and a vehicle according to the respective main claims. Advantageous embodiments can be found in the subclaims.

[0007] According to a first aspect of the invention, a computer-implemented method for controlling a graphical user interface on a vehicle display comprises a receiving step in which a current driving mode is received. The driving mode characterizes the degree of automated driving of the vehicle. It can be provided, for example, by a central processing unit of the vehicle, in which the current driving mode is known, or by any other components in the vehicle that have knowledge of the current driving mode and can forward it.

[0008] Advantageously, the current driving mode is characterized by a classification according to SAE J3016. This standard describes the classification and definition of the degree of automation of road vehicles in six levels. The first three levels, 0-2, correspond to a low degree of automation in which the driver monitors the driving range. The three highest levels, 3-5, characterize a degree of automation in which a system monitors the driving range. Specifically, level 4 describes high automation, and level 5 describes full automation.

[0009] According to the first aspect of the invention, the user interface of the vehicle's display has at least one driver-side digital rearview mirror area and at least one passenger-side digital rearview mirror area. The display can be designed as a single module, for example, as an LCD display. However, it is also possible for the display to consist of multiple screens. Typically, the driver-side digital rearview mirror area is mounted in the interior near the driver-side rearview mirror. Likewise, the passenger-side digital rearview mirror area is typically mounted in the interior of the vehicle near the passenger-side rearview mirror.

[0010] Advantageously, the graphical user interface has at least one additional display area. In other words, the vehicle display consists not only of a driver-side digital rearview mirror area and a passenger-side digital rearview mirror area, so that other information, such as navigation elements, music control elements, or temperature control elements, can also be displayed on the vehicle display.

[0011] According to the first aspect of the invention, the method comprises further steps if the current driving mode corresponds to high or full automation. If this is the case, driver-side image data is received, which comprises an image of a driver-side, rearward field of vision. In addition, passenger-side image data is received, which comprises an image of a passenger-side, rearward field of vision. For this purpose, for example, cameras can be used which are mounted instead of a conventional rearview mirror and are aligned opposite to the forward direction of travel of the vehicle. Furthermore, the driver-side image data is displayed in the driver-side digital rearview mirror area, and the passenger-side image data is displayed in the passenger-side digital rearview mirror area in such a way that it is presented less obviously than in a manual driving mode with low automation.

[0012] In a further step of the computer-implemented method according to the first aspect of the invention, objects are detected in the driver-side image data and the passenger-side image data. This can be done using conventional object recognition. Known objects can be, for example, people, cyclists, other vehicles, ambulances, police cars, or fire engines.

[0013] In a further step, a check is carried out to determine whether at least one trigger object can be detected among the objects. A trigger object is a previously defined object of particular importance. This could be, for example, an ambulance, a police vehicle, or a fire engine. Typically, such objects are defined as trigger objects, which are intended to attract the driver's attention despite the vehicle's high or fully automated driving mode. This increases the driver's acceptance of automated driving because they remain informed about important objects and events in their environment.

[0014] If at least one trigger object is detected, the method includes a step in which the graphical user interface is adapted. In particular, the driver-side digital rearview mirror area and the passenger-side digital rearview mirror area are modified in such a way that they become more prominent and thus attract the driver's attention.

[0015] Advantageously, the trigger object corresponds to an ambulance, a fire engine, a police vehicle, a bicycle, or a person. These are typically objects whose presence in the vicinity of the vehicle requires increased driver attention.

[0016] Advantageously, the adaptation of the graphical user interface is achieved by changing the size, color, contrast and / or blur of the driver-side and / or passenger-side image data.

[0017] If the vehicle is driven manually by the driver, the driver-side and passenger-side image data are displayed concisely and clearly in the driver-side digital rearview mirror area and the passenger-side digital rearview mirror area, respectively, because their content is relevant to the driver in manual driving mode. If the vehicle changes its driving mode to one characterized by high or even full automation, the display of the digital rearview mirror areas can be made less prominent, for example, by changing the contrast, so as not to unnecessarily draw the driver's attention to this part of the display.

[0018] In a further step of the computer-implemented method according to the first aspect of the invention, the graphical user interface is displayed on the display of the vehicle.

[0019] According to a second aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to execute the method according to one of the preceding claims. The computer program product can be executed entirely on the vehicle. However, it can also be executed partially on the vehicle and partially outside the vehicle, for example, on a server. For example, all method steps, with the exception of object detection and checking for trigger objects, can be executed inside the vehicle, with the remaining steps being executed on a server.

[0020] According to a third aspect of the invention, a computer-readable storage medium comprises instructions which, when executed by a computer, cause the computer to carry out one of the methods described above.

[0021] According to a fourth aspect of the invention, a data carrier signal transmits the computer program product described above.

[0022] According to a fifth aspect of the invention, a data processing device comprises a processor configured to perform the steps of one of the previously described computer-implemented methods. Furthermore, the data processing device comprises a computer-readable storage medium as described above, which is communicatively connected to the processor. Furthermore, the data processing device comprises a driver-side rearview mirror camera, a passenger-side rearview mirror camera, and a display, which are communicatively connected to the processor.

[0023] According to a sixth aspect of the invention, a vehicle comprises a data processing device as described above. Summary of the characters

[0024] The invention is explained in more detail below using exemplary embodiments and figures. The figures show: Fig. 1: A flowchart of the computer-implemented method for controlling a graphical user interface on a display of a vehicle; Fig. 2: A display for the computer-implemented method from Fig. 1; Fig. 3: A first view of the vehicle for the computer-implemented method from Fig. 1; Fig. 4: A second view of the vehicle for the computer-implemented method from Fig. 1; and Fig. 5: A data processing device which implements the computer-implemented method of Fig. 1 can execute. Detailed description of the characters

[0025] Fig. 1 shows a flowchart of a computer-implemented method 100 for controlling a graphical user interface 102 on a display 104 of a vehicle 106.

[0026] Before the individual steps of the computer-implemented method 100 are explained, an example of the display 104 will be explained using Fig. 2 should be considered.

[0027] Fig. 2 shows a display 104 for the computer-implemented method 100 of Fig. 1.

[0028] The display 104 shows a graphical user interface 102, for example, using a liquid crystal display (LCD). The graphical user interface 102 consists of a driver-side digital rearview mirror area 108 and a passenger-side digital rearview mirror area 110. Furthermore, the graphical user interface 102 has an additional display area 112 on which information such as speed, fuel level, battery level, audio control elements, or temperature control elements can be displayed. Parts or the entire display 104 can also be configured as a touchscreen, allowing interactions with the display 104.

[0029] With reference to Fig. 1, the computer-implemented method 100 comprises a first receiving step 114, in which a current driving mode is received. The current driving mode characterizes the level of automation of the vehicle 106. This classification is usually defined according to the SAE J3016 standard, in which high automation is designated as level 4 and full automation as level 5. The current driving mode can be received, for example, by a control module 116, which is Fig. 3 is shown.

[0030] Fig. 3 shows a first view of the vehicle 106 for the computer-implemented method 100 of Fig. 1. The control module 116 may be configured to execute the computer-implemented method 100.

[0031] For this purpose, it can be part of another component, for example, an infotainment system. However, it is also possible for the control module 116 to be a separate component in the vehicle 106. The control module 116 can execute some or all steps of the computer-implemented method 100. It is possible for the control module 116 to also assume other functions of the vehicle 106, for example, controlling an infotainment system.

[0032] Referring again to Fig. 1, the computer-implemented method 100 has a decision step 118 in which it is checked whether the current driving mode corresponds to high or full automation. If this is the case (see t-branch in Fig. 1), a second receiving step 120 is executed. In this step, image data from the driver's side is received, which includes an image of a rearward field of view 122 on the driver's side.

[0033] In a further step of the computer-implemented method 100, passenger-side image data are received in a third receiving step 124, which image data comprises an image of a passenger-side rear field of view 126.

[0034] Fig. 3 and Fig. 4 show the vehicle 106 and the rear fields of view 122, 126. In the example of Fig. 3 and Fig. 4, the driver-side image data is captured by a driver-side rearview mirror camera 128. Similarly, the passenger-side image data is captured by a passenger-side rearview mirror camera 130. The driver-side rearview mirror camera 128 is configured to be located on the driver-side portion of the vehicle 106 and to capture driver-side image data covering the driver-side rearward field of view 122. The same applies to the passenger-side rearview mirror camera 130, which is configured to be located on the passenger-side portion of the vehicle 106 and to capture passenger-side image data covering the passenger-side rearward field of view 126.

[0035] Referring again to Fig. 1, the computer-implemented method 100 comprises a first display step 132 in which the driver-side image data is displayed in the driver-side digital rearview mirror area 108 such that it is less obvious than in a manual driving mode. In a second display step 134, the passenger-side image data is similarly displayed in the passenger-side digital rearview mirror area 110. This display is also adjusted such that it is less obvious than in a manual driving mode. To achieve such a less obvious display, the image data can be adjusted, for example, in its size, contrast, or color range, or displayed only in grayscale, and / or slightly blurred.

[0036] In a detection step 136 of the computer-implemented method 100, objects are detected in the driver-side image data and the passenger-side image data. Since the image data are images of a rearward field of view 122, 126, typical detectable objects are, for example, people, other vehicles, vegetation, street signs, or lanes of a road. In the example of Fig. 2, for example, the image data contains a person, a sidewalk, a tree and a roadway with lane stripes and are thus displayed on the digital rearview mirror areas 108, 110.

[0037] In a verification step 138, a check is performed to determine whether at least one trigger object 140 has been detected among the objects. Objects that qualify as trigger objects 140 are, in particular, those whose presence in the vicinity of the vehicle 106 requires increased driver attention. Typical trigger objects 140 include, for example, police vehicles, fire engines, or ambulances.

[0038] In a control step 142, it is checked whether at least one trigger object 140 has been detected in the objects. If this is the case (t-branch in Fig. 1) an adaptation step 144 is executed, in which the graphical user interface 102, and in particular the driver-side digital rearview mirror area 108 and the passenger-side digital rearview mirror area 110, are changed in their display. Until a trigger object 140 is detected, the digital rearview mirror areas 108, 110 were displayed less obviously. The additional display area 112 may even be highlighted, for example, by a colored border. If the driver looks at the display 104, their attention is thus drawn to the additional display area 112.

[0039] Upon detection of a trigger object 140 in the image data, the digital rearview mirror areas 108, 110 are displayed differently, for example, significantly more clearly. This can be achieved, for example, by further adjusting the image size, contrast, or blur. It is also possible to graphically highlight the detected trigger objects 140, for example, by placing a color-contrasting frame around them. These measures direct the driver's attention to the digital rearview mirror areas 108, 110 and increase their understanding of the current traffic situation. These measures can also help communicate to the driver what the automated vehicle 106 will do next in such a situation. This increases the driver's acceptance of the automated driving mode.

[0040] In a final step of the computer-implemented method 100, display step 146, the graphical user interface 102 is displayed on the vehicle's display 104. Thus, the information is visible to the driver of the vehicle 106.

[0041] Through the computer-implemented process 100 on Fig. 1, the driver of vehicle 106 is informed that his surroundings are no longer of interest. Furthermore, in those situations, even while driving in automated mode, the driver is informed about what is happening around him. This builds situational awareness.

[0042] Fig. 5 shows a data processing device 148 having a processor 150 configured to execute the computer-implemented method 100 of Fig. 1. Furthermore, the data processing device 148 includes a computer-readable storage medium 152, as well as a driver-side rearview mirror camera 128 and a passenger-side rearview mirror camera 130, which are communicatively connected to the processor 150. Finally, the data processing device 148 includes a display 104, which is also communicatively connected to the processor 150.

[0043] In the example of Fig. 4, the vehicle 106 includes all elements of a data processing device 148, wherein the processor 150 and the computer-readable storage medium 152 are part of the control module 116. List of reference symbols 100 Computer-implemented procedure 102 Graphical User Interface 104 Advertisement 106 vehicles 108 Driver-side digital rearview mirror area 110 Passenger-side digital rearview mirror area 112 Additional display area 114 First reception step 116 Control module 118 Decision step 120 Second reception step 122 Driver's side rear field of vision 124 Third reception step 126 Passenger-side rear field of vision 128 Driver-side rearview mirror camera 130 Passenger-side rearview mirror camera 132 First display step 134 Second display step 136 Recognition step 138 Verification step 140 trigger objects 142 Control step 144 Adaptation step 146 display step 148 Data processing device 150 processor 152 Computer-readable storage medium

Claims

[1] Computer-implemented method (100) for controlling a graphical user interface (102) on a display (104) of a vehicle (106), wherein the graphical user interface (102) has at least one driver-side digital rearview mirror area (108) and at least one passenger-side digital rearview mirror area (110), the method (100) comprising the following steps: a) receiving (114) a current driving mode, wherein, if the current driving mode corresponds to high or full automation, the method (100) comprises the following further steps: b) receiving (120) driver-side image data comprising an image of a driver-side rear field of view (122), c) receiving (124) passenger-side image data comprising an image of a passenger-side rear field of view (124), d) displaying (132) the driver-side image data in the driver-side digital rearview mirror area (108) in such a way that they are displayed less obviously than in a manual driving mode, e) displaying (134) the passenger-side image data in the passenger-side digital rearview mirror area (110) in such a way that they are displayed less obviously than in manual driving mode, f) detecting (136) objects in the driver-side image data and the passenger-side image data, g) Checking (138) whether at least one trigger object (140) has been detected in the objects, wherein, if at least one trigger object (140) has been detected, the method (100) comprises the following further steps: h) adapting (144) the graphical user interface (102), in particular the driver-side digital rearview mirror area (108) and the passenger-side digital rearview mirror area (110), and i) displaying (146) the adapted graphical user interface (102) on the display (104) of the vehicle (106). [2] Computer-implemented method according to claim 1, characterized by that the current driving mode is characterized by a classification according to SAE J3016. [3] Computer-implemented method according to claim 1 or 2, characterized by that the graphical user interface (102) has at least one further display area (112). [4] Computer-implemented method according to one of the preceding claims, characterized by that the trigger object (140) corresponds to an ambulance, a fire engine, a police vehicle, a bicycle or a person. [5] Computer-implemented method according to one of the preceding claims, characterized bythat the adaptation (144) of the graphical user interface (102) is achieved by a size change, a color change, a contrast change and / or a blur change of the driver-side image data and / or the passenger-side image data. [6] A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of the preceding claims. [7] A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 5. [8] A data carrier signal which transmits the computer program product according to claim 6. [9] A data processing device (148) comprising: a) a processor (150) configured to perform the steps of the computer-implemented method (100) according to any one of claims 1 to 5, b) a computer-readable storage medium (152) according to claim 7, which is communicatively connected to the processor (150), c) a driver-side rearview mirror camera (128) communicatively connected to the processor (150), d) a passenger-side rearview mirror camera (130) communicatively connected to the processor (150), and e) a display (104) communicatively connected to the processor (150). [10] A vehicle (106) comprising a data processing device (148) according to claim 9.

Images