Screen control arrangement for determining a user action from a user at a screen device in a vehicle

Abstract

The invention relates to a screen control arrangement 2 for determining a user action by a user at a screen device 6 in a vehicle 1, comprising the screen device 6, a camera device 5 for capturing an operating activity of the user at the screen device 6 as a basis for camera-based operating data, wherein the screen device 6 has a sensor device 8 for capturing an operating activity of the user at the screen device 6 as a basis for sensor-based operating data, and an evaluation device 9 for determining the user action, wherein the evaluation device 9 is configured to accept the camera-based operating data and the sensor-based operating data as operating data and to determine the user action on the basis of the operating data.

Description

[0001] The invention relates to a screen control arrangement for determining a user action by a user on a screen device in a vehicle having the features of claim 1. The invention also relates to a method for determining the user action and a vehicle having the screen control arrangement and / or for implementing the method.

[0002] Modern vehicles have shifted away from switches and other mechanical controls. Such mechanical controls require significant manufacturing effort in terms of provision and integration and are not very versatile. Instead, operation is now mostly implemented via touch displays. Such touch displays offer various graphical control elements that can be activated by the user touching the touch display.

[0003] Another implementation for the actuation of graphical control elements is disclosed in WO 2013 / 001084 A1. Instead of a touch display, i.e., a mechanical actuation, the document proposes contactless detection of gestures and / or operating processes using an optical detection unit, in particular a camera.

[0004] It is an object of the present invention to propose a screen operating arrangement for determining a user action by a user on a screen device in a vehicle, which is characterized by a high level of operating reliability.

[0005] This object is achieved by a screen control arrangement, by a method, and by a vehicle. Preferred or advantageous embodiments are evident from the subclaims, the following description, and the accompanying figures.

[0006] The invention relates to a screen control arrangement for determining a user action on a screen device in a vehicle. The screen control arrangement comprises the screen device.

[0007] The vehicle can be, in particular, a road vehicle, a passenger car, a truck, a bus, or other mobile vehicle. Preferably, the vehicle has a vehicle interior in which the vehicle's users can sit. The user can be a driver, a passenger, or a co-driver of the vehicle.

[0008] The screen device comprises, in particular, a screen—also called a display. The screen can be configured, for example, as a TFT, OLED, or LED screen. In particular, the screen is configured to implement a graphical user interface, a graphical user interface, and / or a user interface (GUI) with user elements and / or implements these during operation of the screen control arrangement.

[0009] A user action on the screen device is / are understood to mean in particular the selection and / or operation of a graphical control element, in particular the graphical user interface, graphical user interface and / or user interface (GUI), on the screen.

[0010] The screen control arrangement comprises a camera device which is suitable and / or configured to record the user's operating activity on the screen device, in particular on the graphical user interface (GUI) and / or on the operating element, and to use the recorded operating activity as the basis for camera-supported operating data. The camera-supported operating data thus comprise information about the user's operating activity on the screen device. In the simplest embodiment, the camera-supported operating data can be configured as the camera images of the screen with and / or without the user. It is also possible for the camera-supported operating data to already be configured as metadata, wherein the metadata is derived from the camera images.The camera-supported operating data is generated, in particular, using camera-based skeleton recognition, 3D body pose estimation, 2D lifting (generation of 3D data from 2D images), and end-to-end learning for movements and gestures. The camera-supported operating data can, in particular, be embodied as data relating to the user's movements and / or gestures relative to the display device, in particular to the display.

[0011] The camera device is designed and / or arranged, in particular, to capture the interior of the vehicle. The camera device is understood, in particular, to be a spatially resolving sensor device. In particular, the camera device is designed as an interior camera. The camera device can be designed as a grayscale camera, an RGB camera, an IR camera, an RGB-IR camera, or even as a ToF (Time-of-Flight) or stereo-IR camera. For example, the camera device is arranged and / or can be arranged on the interior mirror of the vehicle. Optionally, the camera device is designed to be motor-driven. In particular, the camera device is used alternatively or additionally for driver and passenger monitoring.

[0012] Within the scope of the invention, it is proposed that the display device comprise a sensor device, wherein the sensor device is designed to detect an operating activity on the display device, in particular on the display screen. In particular, the sensor device detects the operating activity on the display device, in particular the display screen, with spatial resolution. The detected operating activity of the user is used as the basis for sensor-supported operating data. The sensor-supported operating data thus contain information about the user's operating activity on the display device. The sensor-supported operating data can be raw data or previously evaluated data, in particular metadata.

[0013] The screen control arrangement has an evaluation device for determining the user action. The evaluation device is designed, in particular, as a digital data processing device. The evaluation device can be designed as a separate digital data processing device or, for example, can be integrated into the screen device and / or the camera device. Evaluation via a central, digital data processing device in the vehicle is also possible.

[0014] The evaluation device is configured, through programming and / or circuitry, to receive the camera-based operating data and the sensor-based operating data. For example, the aforementioned operating data are buffered and / or temporarily stored in the evaluation device. In particular, the evaluation device has access to the camera-based and sensor-based operating data. The evaluation device is also configured to determine the user action based on the operating data, namely the camera-based operating data and / or the sensor-based operating data.

[0015] It is a consideration of the invention that both detection methods for the operating activity on the screen device, in particular on the screen, have advantages and disadvantages: In principle, the established sensor technology for detecting user activity, which is integrated into the display device, is accurate and reliable. However, the sensor device is not always able to detect touches accurately. This can be due to localized contamination, dust, grease, or moisture.

[0016] The camera system, or camera-based recording of operating activity, has the advantage that it is usually already present in the vehicle and, due to the position of the camera system, is resistant to dirt and robust. However, a clear assignment to small control elements on the screen is not possible with many camera angles and resolutions.

[0017] According to the invention, when determining the user action, the camera-supported operating data and / or the sensor-supported operating data can be accessed optionally in order to be able to determine the user action depending on the situation.

[0018] For example, it can be provided that if the sensor device is functioning properly, only the sensor-supported operating data is used to determine the user action and the camera-supported operating data is available but not used.

[0019] It may also be provided that if the sensor device is not functioning properly, only the camera-based operating data is used to determine the user action and the sensor-based operating data is not used due to a lack of meaningfulness.

[0020] It may also be provided that, if the sensor device is not functioning properly, a combination of the camera-based operating data and the sensor-based operating data is used to determine the user action.

[0021] The screen control system thus ensures that even under poor initial conditions, such as partially or completely dirty or a malfunctioning sensor device, user actions can be determined as accurately as possible using camera-supported operating data. This increases the operational reliability of the screen control system.

[0022] Preferably, the user action on the screen device is converted into a command, such as operating an audio player or radio, changing the volume, operating a telephone, such as selecting a telephone partner, starting a call, ending a call, querying or setting vehicle parameters, such as operating the air conditioning system, in particular changing the temperature, etc.

[0023] In a preferred embodiment of the invention, the evaluation device is configured to evaluate the camera-based and sensor-based operating data simultaneously and / or in parallel. In principle, it is possible, for example, to view both operating data as time series and determine the user action through a joint evaluation. Alternatively, it is also possible to use artificial intelligence, e.g., a neural network, in the evaluation and thus determine the user action.

[0024] In a particularly preferred embodiment of the invention, the evaluation device is designed to determine the user action based on the sensor-supported operating data. If the user action cannot be determined, or cannot be determined reliably enough, exclusively based on the sensor-supported operating data, the evaluation device is designed to determine the user action based on the camera-supported operating data. Thus, the determination of the user action based on the camera-supported operating data forms a secondary system, which, in the event of a failure of the determination of the user action based on the sensor-supported operating data, forms a fallback level as a primary system in order to maintain the overall system for determining the user action.

[0025] In a preferred development of the invention, the evaluation device is configured to accept operating options offered on the screen device. For example, the screen device forwards information about the offered operating options to the evaluation device, or the evaluation device actively retrieves the information from the screen device. The evaluation device is configured to determine the user action taking into account the operating options offered by the screen device.

[0026] This refinement allows the evaluation device to determine the user action much more precisely: Without information about the available operating options, the evaluation device must spatially assign the operating activity to the display device, in particular to the display, and then assign the spatial assignment to the operating elements to determine the user action. According to this refinement, the evaluation device can utilize a priori knowledge about the possible user action. For example, the evaluation device only needs to check whether operating data indicates a user action that matches the available operating options.

[0027] If, for example, only a single operating element is offered on the display device, in particular on the display screen, as an operating option, it is sufficient to check whether the display screen has been operated based on the operating data, in particular the sensor-supported and / or camera-supported operating data. If the evaluation device detects that the display screen has been operated, it can be assumed that the user has acted on the display and / or selected the single operating element, and thus the user action for operating and / or selecting the single operating element can be determined. In particular, the spatial resolution takes a back seat, so that the camera-supported operating data, in particular, can be used exclusively to determine the user action.

[0028] If, for example, only two operating elements are offered on the screen device, in particular on the screen, as operating options, with each of the two operating elements being assigned to one half of the screen, it is sufficient to check, regardless of the actual size of the operating elements (which can also be considerably smaller), whether the respective half of the screen has been operated in the case of the operating data, in particular in the case of sensor-supported and / or camera-supported operating data. If operation of the half of the screen is detected by the evaluation device, it can be assumed that there has been a user action to operate and / or select the operated element in the half of the screen, and the user action to operate and / or select the operated element in the half of the screen can therefore be determined. In particular, the detection of the spatial resolution largely takes a back seat or is not necessary.is reduced to the determination of the half of the screen, so that in particular the camera-supported operating data can also be used exclusively to determine the user action.

[0029] In general, it is preferred that the evaluation device be configured to take into account a spatial distribution of the operating options on the display device, in particular on the display screen. The spatial distribution of the operating options refers, in particular, to the spatial distribution of the operating elements on the display screen.

[0030] Alternatively or additionally, the evaluation device is designed to consider a temporal progression of the operating options on the screen device. For example, operating elements displayed within a dialog flow can be given priority when determining the user action. Other operating elements located outside the dialog flow can be disadvantaged when determining the user action. In this way, the camera-based operating data, in particular, can be used exclusively to determine the user action, despite poor spatial resolution.

[0031] It is particularly preferred that the evaluation device be configured to consider a content context of the operating options on the screen device. In particular, the content context is defined by a combination of the spatial distribution, the temporal progression, and preferably the dialogue flow.

[0032] In a structural implementation, it is preferred that the sensor device be designed as a touch display sensor device, which detects a user's operating activity on the screen in a spatially resolved manner. The touch display sensor device is designed, for example, as a resistive sensor device, so that mechanical contact is necessary. Alternatively, the touch display sensor device is designed as a capacitive sensor device, which detects the user's approach, in particular a finger of the user, to the screen as an operating activity. Mixed forms of resistive, capacitive, and / or other sensor devices as a touch display sensor device are also possible.

[0033] A further subject matter of the invention relates to a method for determining a user action by a user on a screen device in a vehicle, preferably using the screen operating arrangement as described above, wherein camera-supported operating data from a camera device or the camera device and sensor-supported operating data from a sensor device or the sensor device are determined as operating data, and wherein the user action is determined on the basis of the operating data, as described above.

[0034] Another object is the vehicle with the screen control arrangement.

[0035] Further features, advantages, and effects of the invention will become apparent from the following description of preferred embodiments of the invention. These show: Fig. 1 is a schematic block diagram of a vehicle with a screen control arrangement as an embodiment of the invention, Fig. 2a, b, c different representations of a screen of the screen control arrangement.

[0036] Corresponding or identical parts are provided with the same reference numerals in the figures.

[0037] The Fig. Figure 1 shows a schematic side view of a vehicle 1 with a screen control arrangement 2 as an exemplary embodiment of the invention. The vehicle 1 is designed, for example, as a passenger car. For orientation in the vehicle 1, a seat 3, e.g., a driver's seat or a passenger seat, and an interior mirror 4, which is arranged centrally in the vehicle 1, are shown.

[0038] The vehicle 1, in particular the screen control arrangement 2, has a camera device 5, wherein the camera device 5 is designed as an interior camera. In this exemplary embodiment, the camera device 5 is arranged centrally in the region of the interior mirror 4 and monitors the interior of the vehicle 1. The field of view of the camera device 5 is schematically represented by dashed lines.

[0039] The vehicle 1, in particular the screen operating arrangement 2, has a screen device 6, wherein the screen device 6 has a screen 7 and a sensor device 8 for detecting an operating activity of a user on the screen device 6, in particular on the screen 7.

[0040] The screen device 6, in particular the screen 7, can, for example, be arranged centrally on a vehicle console. It is also possible for it to protrude from the vehicle console. The screen 7 is designed to display screen contents such as images, menus, navigation aids, maps, etc. For example, it is designed as a color screen, in particular an LED or OLED screen. The sensor device 8 serves for the spatially resolved detection of an operating activity of the user on the screen device 6, in particular on the screen 7. For example, the sensor device 8 is designed as a capacitive sensor device 8. The screen device 6, in particular the screen 7, thus forms a touch display. Sensor-supported operating data can be derived based on the detected operating activity of the user on the screen device 6, in particular on the screen 7.The sensor-supported operating data includes information about the user's operating activity on the display device 6, in particular on the display 7.

[0041] In addition to recording the operating activity on the screen device 6, the operating activity can alternatively or additionally be optically recorded by the camera device 5, since this lies in the field of view of the camera device 5, as shown in the Fig. 1 is indicated schematically. Camera-supported operating data can be derived from the recorded operating activity, wherein the camera-supported operating data includes information about the user's operating activity on the display device 6, in particular on the display 7. When deriving the camera-supported operating data, algorithms such as camera-based skeleton recognition, 3D body pose estimation, 2D lifting (generation of 3D data from 2D images), and end-to-end learning for movements and gestures can be used.

[0042] The derivation of the operating data can take place in the camera device 5 or in the screen device 6. Alternatively, the derivation of the operating data can take place in any digital data processing device that is connected to the camera device 5 and / or the screen device 6.

[0043] The screen control arrangement 2 has an evaluation device 9 for determining a user action based on the user's operating activity on the screen device 6, in particular on the screen 7. The evaluation device 9 is designed, for example, as a digital data processing device.

[0044] The user action is the selection and / or operation of a graphical control element 10 ( Fig. 2), which is displayed as part of a graphical user interface (GUI) on the screen device 6, in particular on the screen 7. The user's operating activity is approaching or touching the screen device 6, in particular the screen 7. By evaluating the user's operating activity, the (desired) user action can be determined by the evaluation device 9.

[0045] For the determination, the sensor-based operating data and the camera-based operating data are transferred to the evaluation device 9 as operating data or received by it, so that both the sensor-based operating data and the camera-based operating data are present in the evaluation device 9. The evaluation device 9 is designed to determine the user action based on the operating data. Depending on the situation, the user action is determined exclusively on the sensor-based operating data, exclusively on the camera-based operating data while evaluating the sensor-based operating data, or in parallel by both the sensor-based operating data and the camera-based operating data: In principle, the sensor-based operating data is highly meaningful, so that these are generally used as the basis and the camera-based operating data is discarded.

[0046] Alternatively, despite their high informative value, the sensor-based operating data and the camera-based operating data can be evaluated in parallel in order to be able to carry out a more robust determination of the user action in the sense of sensor fusion.

[0047] A particularly relevant case occurs when the sensor-based operating data leads to a non-determination of the user action. This can occur, for example, if the sensor device 8 does not correctly detect the touch or approach due to localized contamination, dust, grease, or moisture on the display device 6, in particular on the sensor device 8.

[0048] In this case, in the sense of a fallback level, the camera device 5 can be used as a secondary system to the sensor device 8 as the primary system and the user action can be determined on the basis of the camera-supported operating data.

[0049] Under certain circumstances, a user action on the screen device 6 in the form of a display operation can also be recognized camera-based. A clear assignment to small-scale control elements 10 on the screen device 6 is not possible with many camera perspectives and resolutions.

[0050] Even without further measures, the user action can be determined from the camera-based operating data. However, this excludes control elements 10 that are too detailed depending on the camera perspective and resolution.

[0051] In general or specifically for this case, the context of the operation of the screen device 6 can be used to improve the determination of the user action. For example, a specific context can be used, e.g., if only one or two control elements 10 that can be clearly distinguished using the camera device 5 are displayed, and the user is, for example, within a dialogue flow, then a camera-based determination with a simultaneous non-determination via the sensor device 8 can be interpreted as input, and the erroneous non-determination can be corrected.

[0052] The evaluation device 9 can, for example, receive or acquire data about the operating options offered on the screen device 6. Based on the offered operating options, which form a context, the determination of the user action can be simplified.

[0053] The Fig. 2 a, b, c each show the screen 6 with a graphical user interface, which has one or more of the control elements 10.

[0054] The Fig. Figure 2a shows an embodiment in which the operating activity, indicated by the stylized finger 11, is performed on a single operating element 10. When determining the user action based on the camera-based operating data and knowing the available operating options, the evaluation device 9 can already conclude from an actuation in the direction of the screen 7 that the single operating element 10 should be selected and / or operated. This makes it possible to perform a robust determination of the user action based on the camera-based operating data without spatial resolution or with a spatial resolution that only indicates whether the finger 11 has touched or approached the screen 7 somewhere.

[0055] The Fig. Figure 2b shows an embodiment in which the operating activity is performed on two operating elements 10, one of the operating elements 10 being arranged on the left half and the other operating element 10 on the right half of the screen 7. In this embodiment, it is sufficient for the evaluation device 9 to be able to deduce from the camera-based operating data which half of the screen 7 was operated.

[0056] In both embodiments, a spatial distribution of the operating options, designed as operating elements 10, on the screen device 6, in particular on the screen 7, is thus taken into account.

[0057] The Fig.Figure 2 c illustrates the consideration of a temporal progression of the operating options by the evaluation device 9. Initially, eight operating elements 10 are shown on the screen 7. These are difficult to distinguish by the evaluation device 9 based on the camera-based operating data. However, over time, a larger operating element 10 is projected over the other operating elements 10, whereby the evaluation device 9 can consider the temporal progression of the operating options in such a way that the selection and / or operation of the larger operating element 10 is determined as the user action, since the larger operating element 10 was displayed last.

[0058] For example, a 2D RGB IR camera mounted on the interior mirror 4, serving as a camera device 5 for driver and passenger monitoring, looks obliquely from above at the displays installed in the vehicle 1, including the screen device 6, and can thus detect an operation and distinguish between the operation of different segments. With the help of skeleton-based hand gesture recognition, an operation with an outstretched index finger is recognized, as is a touch of the display via the estimated 3D coordinates. Furthermore, a specific movement sequence when touching a surface with an index finger can be learned end-to-end using CNNs. This can be used, particularly in a certain context, to correct a non-determination of the user action of a touch operation. List of reference symbols 1 vehicle 2 Screen control layout 3 seats 4 interior mirrors 5 Camera setup 6 Screen device 7 screen 8 Sensor device 9 Evaluation device 10 Control element 11 fingers QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 2013 / 001084 A1

[0003]

Claims

[1] Screen control arrangement (2) for determining a user action by a user on a screen device (6) in a vehicle (1), with the screen device (6), with a camera device (5) for recording an operating activity of the user on the screen device (6) as a basis for camera-supported operating data, characterized by , that the screen device (6) has a sensor device (8) for detecting an operating activity of the user on the screen device (6) as a basis for sensor-supported operating data, and by an evaluation device (9) for determining the user action, wherein the evaluation device (9) is designed to accept the camera-supported operating data and the sensor-supported operating data as operating data and to determine the user action on the basis of the operating data. [2] Screen control arrangement (2) according to claim 1, characterized bythat the evaluation device (9) is designed to use the camera-supported and the sensor-supported operating data simultaneously and / or in parallel to determine the user action. [3] Screen control arrangement (2) according to claim 1 or 2, characterized by that the evaluation device (9) is designed to determine the user action on the basis of the sensor-supported operating data and, in the event of a non-determination, to determine the user action on the basis of the camera-supported operating data. [4] Screen control arrangement (2) according to claim 3, characterized by that the evaluation device (9) is designed to accept operating options offered on the screen device (6) and to determine the user action taking into account the operating options offered on the screen device (6). [5] Screen control arrangement (2) according to claim 4, characterized bythat the evaluation device (9) is designed to take into account a spatial distribution of the operating options on the screen device (6). [6] Screen control arrangement (2) according to claim 4 or 5, characterized by that the evaluation device (9) is designed to take into account a temporal progression of the operating options on the screen device (6). [7] Screen operating arrangement (2) according to one of the preceding claims 4 to 6, characterized by that the evaluation device (9) is designed to take into account a content context of the operating options on the screen device (6). [8] Screen control arrangement (2) according to one of the preceding claims, characterized by that the sensor device (8) is designed as a touch display sensor device and / or spatially resolving capacitive sensor device. [9] Method for determining a user action by a user on a screen device (6), wherein camera-based operating data, which are formed on the basis of a detection of an operating activity of the user on the screen device (6) by a camera device (5), and sensor-based operating data, which are formed on the basis of a detection of an operating activity of the user on the screen device (6) by a sensor device (8), are detected as operating data and wherein the user action is determined on the basis of the operating data. [10] Vehicle (1) with the screen operating arrangement (2) according to one of claims 1 to 8 and / or for carrying out the method according to claim 9.

Images