Method for identifying a mobile terminal device in a vehicle

Abstract

A method for identifying a mobile terminal device (2) in a vehicle is disclosed. In the method, first image data is captured by means of at least one image capturing device (4) in an interior space (1) of the vehicle, wherein the first image data contains at least a first view of the terminal device (2) of a user (10) at a first point in time. Furthermore, second image data is captured by means of the image capturing device (4), wherein the second image data contains at least a second view of the terminal device (2) of the user (10) at a second point in time. The first and second image data are joined together to obtain a combined view of the terminal device (2) of the user (10). By evaluating the joined image data, content (22) displayed on the terminal device (2) of the user (10) is identified.

Description

Technical Field

[0001] This invention relates to a method for identifying mobile terminal devices in a vehicle. Specifically, it involves using an image acquisition device to capture the content displayed on the mobile terminal device in order to control the vehicle's functions. Background Technology

[0002] Modern vehicles have a variety of functions that can be controlled by the user, such as activating, starting, stopping, and configuring them. This particularly includes the functions of the infotainment system, such as entering a navigation destination, playing music clips, and calling contacts. These functions of the infotainment system are typically controlled by the user through direct interaction with the vehicle, such as by manipulating buttons or touching a touchscreen, or through voice control or gesture control.

[0003] However, operating within the vehicle itself can sometimes be cumbersome, such as when data, like navigation destinations or music titles, must be entered via a keyboard on a touchscreen. Typically, such input is easier on the user's terminal device, such as a smartphone. In some cases, the data is even already presented on the smartphone, for example, when the user has already listened to a particular song on their smartphone and now wants to hear it in the vehicle as well. Therefore, methods are known in which data is transferred from the user's terminal device to the vehicle, for example, via Bluetooth or via a cloud service.

[0004] It is also known to capture content displayed on a mobile device's screen via cameras within the vehicle. Since modern vehicles are typically equipped with cameras, for example, for monitoring the interior space, this infrastructure can be utilized to read the displayed content. The content can be displayed text recognized via OCR. Therefore, for example, a navigation destination can be easily transmitted from a smartphone to the vehicle. Machine-readable codes, such as QR codes, can also be used, captured by the vehicle's cameras.

[0005] To reliably identify displayed content, such as text or codes, sufficient capture of the displayed content is necessary. Specifically, reliable identification requires capturing the entire content or at least the portion relevant to identification. However, mirroring or reflection often occurs, particularly in external areas under ambient light, especially due to sunlight, and this mirroring or reflection can depend on the user's holding of their smartphone relative to the camera inside the vehicle. Even when it is darker inside the vehicle than in the external environment, such reflections can be particularly strong. Therefore, even if people attempt to move their smartphones back and forth, it is often difficult to hold the smartphone in front of the camera without reflections. To avoid or reduce mirroring, special films or coatings are known to be applied to smartphone displays. However, such films may not always be available and may be undesirable to some users. Summary of the Invention

[0006] The object of this invention is to provide an improved method for identifying mobile terminal devices in vehicles. In particular, the identification of content displayed on the mobile terminal device should be improved.

[0007] The solution to this objective is achieved according to the teachings of the independent claims. Different embodiments and further improvements of the invention are the technical solutions of the dependent claims.

[0008] A first aspect of the invention relates to a method, particularly a computer-implemented method, for identifying a mobile terminal device in a vehicle. In this method, first image data is acquired using at least one image acquisition device within the vehicle's interior space, wherein the first image data at least includes a first view of the user's terminal device at a first point in time. Furthermore, second image data is acquired using the image acquisition device and / or other image acquisition devices within the vehicle's interior space, wherein the second image data at least includes a second view of the user's terminal device at a second point in time. The first and second image data are combined to obtain a combined view of the user's terminal device. Content displayed on the user's terminal device is identified by evaluating the combined image data.

[0009] Therefore, the aforementioned method according to the first aspect is particularly based on evaluating image data acquired by means of an image acquisition device, such as a camera. Here, multiple views of the user's terminal device are captured by means of the image acquisition device (or, if necessary, multiple image acquisition devices), making it possible to identify the content on the terminal device, particularly the content displayed on the terminal device's screen. By acquiring multiple views and then combining these views into a single overall view, defects in the individual views that may hinder the identification of content based on each view can be eliminated. This may particularly involve reflections on the screen of a mobile terminal device, such as a smartphone. The views should be distinct to obtain a defect-free overall view. In practice, this can be easily achieved when the user rarely holds their terminal device completely still. The views can be captured by the same image acquisition device, but may also be captured by different image acquisition devices located at different positions within the vehicle's interior space. Thus, reflections can be replaced and eliminated in the overall view by corresponding portions in other views. This allows for better and more reliable identification of the displayed content.

[0010] The term "vehicle" as used herein specifically refers to passenger cars, including all types of motor vehicles, hybrid and battery-powered electric vehicles, and vehicles such as vans, buses, trucks, delivery vans, etc.

[0011] The term "image acquisition device" used here specifically refers to a camera, particularly a digital camera. A camera can capture still images (photographs) or moving images (videos), especially within the visible light range. An image acquisition device can acquire or capture such images and output the corresponding image data. An image acquisition device can also be used as a monitoring device; that is, the image acquisition device and the monitoring device can be the same device. It is understood that multiple image acquisition devices can exist within the interior space of a vehicle, located in different positions. The acquired image data can then be processed by a central control unit.

[0012] As used herein, the term "mobile terminal device" specifically refers to a portable electronic device capable of wireless communication with other devices or networks. This includes handheld devices that can typically be carried in a pocket or held in the hand. Mobile terminal devices include smartphones, tablets, laptops, smartwatches, and other similar devices. These devices typically use mobile radio networks, WLAN, Bluetooth, or UWB to establish wireless connections and enable data transmission. Mobile terminal devices are particularly configured for presenting content, especially displaying content on a monitor. The user of the terminal device can be, in particular, the driver of a vehicle. However, it is understood that the front passenger and, in principle, any passenger in the vehicle can also be a user of the terminal device in the sense of this invention.

[0013] As used herein, the term "content" for a terminal device can specifically refer to information presented on the terminal device's display device, such as a monitor. Content does not necessarily have to be displayed on the terminal device within an app whose type corresponds to a vehicle function. Content may consist solely, for example, unformatted text or machine-readable code. Nevertheless, the presentation of the content, such as font formatting, color, shape, logos, etc., can be examined, which can be used to identify vehicle functions. Code may also contain more than just text, such as additional descriptions of the content type or even corresponding control codes for vehicle functions.

[0014] The term "monitoring device" as used here specifically refers to a device suitable for monitoring the interior space of a vehicle, particularly the occupants, and especially their position and movement within that space. Image acquisition devices may be involved here. Infrared (IR) cameras (e.g., near-infrared (NIR) cameras) can be used. IR images are well-suited for monitoring because they are robust to varying lighting conditions, such as even under strong sunlight. Similarly, they can be captured in darkness. It is understood that a corresponding IR light source can be present. It is also conceivable, for example, to create a three-dimensional model of the driver's or the vehicle's interior space using a time-of-flight (TOF) camera. With the aid of this three-dimensional model, the distance to the camera can be calculated, and thus the distances between objects and their movements can be calculated.

[0015] As used herein, the term "vehicle function" (synonymous with the term "vehicle feature"), as used herein, can be understood, in particular, as the function of the vehicle's infotainment system. This can include proprietary applications of the vehicle manufacturer and / or applications implemented by external providers. Control of vehicle functions, also described herein, can in particular include invoking or activating vehicle functions corresponding to identified content based on identified content. However, control of vehicle functions may also consist solely of displaying data relating to identified content or corresponding vehicle functions (where appropriate, together with the identified content), for example, on the vehicle's screens, particularly on the infotainment system's display.

[0016] The terms “comprising,” “including,” “containing,” “having,” “having,” “with,” or any other variation thereof, used herein where necessary, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to the method or apparatus.

[0017] Furthermore, unless explicitly stated otherwise, "or" refers to an inclusive "or," not an exclusive "or." For example, condition A or B is satisfied by one of the following conditions: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0018] The term “a” or “one” as used herein is defined as meaning “one or more”. The terms “another” and “another” and any other variations thereof should be understood as meaning “at least one other”.

[0019] The term “multiple” as used here should be understood to mean “two or more”.

[0020] The terms "configuration" or "setting" for fulfilling a specific function (and its corresponding modifications) should be understood in the context of this invention as meaning that the corresponding device is already designed or set in a way that allows it to perform the function, or is at least configurable—that is, set so that it can perform the function after a corresponding setting. Here, configuration can be performed, for example, by setting parameters of the program process accordingly, or by using switches or the like to activate or deactivate the function or setting. In particular, the device can have multiple predetermined configuration or operating modes, allowing configuration to be performed by selecting one of these configuration or operating modes.

[0021] Preferred embodiments of the method are described below. These embodiments may be combined with each other and with other aspects of the invention as described herein, unless explicitly excluded or technically impossible.

[0022] In some embodiments, the method further includes determining the position and / or orientation of the mobile terminal device within the vehicle's interior space using at least one monitoring device or an image acquisition device, wherein image data are stitched together only if the position and / or orientation of the mobile terminal device at a first time point differs from its position and / or orientation at a second time point. If the position and / or orientation of the mobile terminal device changes in space, the corresponding view acquired by the same image acquisition device also changes. As briefly mentioned above, it is advantageous that the views, i.e., the stitched image data, are different. In this way, defects or errors in the individual views can be compensated for by corresponding portions in the one or more other views. The efficiency of this method can be improved if stitching is only performed when the views are not actually simultaneous, since stitching together (substantially) identical views does not offer any advantage.

[0023] In some implementations, the first and second image data are part of an image sequence acquired by means of an image acquisition device, wherein the first and second image data are selected from the image sequence. However, the second image data is selected only when the position and / or orientation of the mobile terminal device at a first time point differs from its position and / or orientation at a second time point. As just described, different views should be stitched together to obtain a holistic view from which content can be reliably identified. For this purpose, an image sequence, such as a video stream, can be acquired, wherein individual images are selected from the image sequence when they differ from previous views and then stitched together. This allows for advantageous tracking of the terminal device's position and / or orientation in space, for example, using a monitoring device, such as a time-of-flight camera. Of course, this can also be determined based on simple images from a conventional camera.

[0024] In some embodiments, the position and / or orientation of the mobile terminal device includes the angle between the mobile terminal device and the image acquisition device. Specifically, when the difference in angle exceeds a predetermined threshold, it can be confirmed that the position and / or orientation of the mobile terminal device differs at a first time point and a second time point. Particularly for mirrors or reflections, the angle of the terminal device, more precisely its display, relative to the camera (image acquisition device) is more important than, for example, the distance to the camera. Therefore, movements that change the angle are particularly suitable for altering mirror images, thereby allowing a unified view to be stitched together without defects, from which the displayed content can be reliably identified.

[0025] In some embodiments, the method further includes: confirming whether a vehicle function matches the identified content, wherein, when no function matches the content, additional image data is selected from the image sequence at at least one additional time point after the first and second time points, and the additional image data is combined with the other image data. The identified content can be used, in particular, to control vehicle functions, as will be described in more detail below. Therefore, if a corresponding vehicle function matches the content, the content can be identified as "valid." Image sequences with different views can now be captured for an extended period until enough different image data has been collected to create an overall view from which valid content can then be identified. During this "iteration," the user can move their terminal device accordingly. If natural movement or movement made by the user is insufficient, appropriate prompts can also be provided to the user.

[0026] In some embodiments, the method further includes, before acquiring image data by means of the image acquisition device(s), acquiring monitoring data for at least a portion of the interior space of the vehicle by means of at least one monitoring device; and confirming (i.e. checking) whether the user's terminal device is at least partially located in a predetermined area of ​​the monitoring data; wherein image data is acquired by means of the image acquisition device(s) only when it is confirmed that the user's terminal device is at least partially located in the predetermined area.

[0027] By implementing this measure before actually recognizing the content on the terminal device, this method can be designed with improved reliability. Image analysis is not performed permanently, which on the one hand would require additional computing power, and on the other hand could potentially lead to undesirable results, for example, when the user may not even intend to control the vehicle functions. If the user actually holds the terminal device within the acquisition area (the predetermined area), only image data is acquired for potential recognition of the content on the terminal device. For this purpose, a monitoring device that may already be present in the vehicle can be used, for example, to monitor the driver for other purposes. However, the monitoring data itself may not be suitable for recognizing the content presented on the terminal device, which is why image data is acquired as described above when it is definitively confirmed that the terminal device is within the predetermined area. However, the monitoring device can also be an image acquisition device, as explained in more detail below.

[0028] In some embodiments, the method further includes confirming whether the mobile terminal device is at least partially located in a predetermined area of ​​the interior space during use, the predetermined area being within the acquisition area of ​​the image acquisition device (see above description). Therefore, it can be stipulated that each use of the mobile terminal device is not recognized, for example, when the user is using their smartphone solely for themselves, such as typing a message or making a call. The predetermined area of ​​the interior space may be an area in front of an interior space camera, which may be located within the rearview mirror. Therefore, the user must actively hold their terminal device in front of the camera for use to be recognized.

[0029] In some embodiments, this method, before identifying content on a user's terminal device by evaluating image data, further includes: determining a portion of the image data representing the terminal device from monitoring data (particularly the aforementioned predetermined area), and confirming whether that portion contains image data suitable for identifying content on the user's terminal device, wherein content on the terminal device is identified only if it is confirmed that the portion contains image data suitable for identifying content on the terminal device. In other words, monitoring data can be used to determine the portion of the image data relevant to identification. This can be determined based on coordinates: for example, where the terminal device's display is located. For example, if the terminal device's display is not even turned on, the image data may not be suitable for identifying content, which can be determined, for example, by brightness differences in the image data. These lighting conditions may also be unsuitable for identifying content on the terminal device, for example, because the display appears too dark. This can be done, for example, by determining pixels and their saturation, for example, when the number of pixels with a defined (minimum saturation) is less than a predetermined value.

[0030] In some implementations, when stitching image data together, only image portions from the first and second image data whose saturation exceeds a threshold are considered. Saturation can be used, in particular, as an indicator of a mirror or reflection on a display, since these areas appear locally and strongly overexposed or white in the image data. These portions can then be removed by replacing them with corresponding image regions from other views that do not contain the mirror (or contain the reflection elsewhere) when stitching the image data together. Saturation can be determined using a corresponding histogram, which can be performed pixel-by-pixel.

[0031] In some implementations, image data is acquired in the visible light range using an image acquisition device. This can be done using an RGB camera. This method is best suited for recognizing content displayed on a terminal device's screen.

[0032] In some implementations, monitoring data is acquired using a monitoring device configured to collect data on at least a portion of the vehicle's interior space. This monitoring data preferably includes image data and / or three-dimensional model data in the infrared (IR) range. Driver monitoring systems typically utilize such IR cameras. This method is suitable for monitoring vehicle interior spaces because image data acquired in the IR range is robust against varying lighting conditions. In other words, the method provides reliable results not only under strong sunlight but also in darkness (particularly compared to image data in the visible light range). Data on a three-dimensional model, for example, captured by a time-of-flight camera, allows for a detailed three-dimensional view of the vehicle's interior space.

[0033] In some implementations, the identified content includes identifying text and / or identifying machine-readable codes. This identification can utilize known methods, such as using OCR to identify text from image data. The identified text (or code) can then be classified or parsed to match it with vehicle functions. For example, from the identified text, it can be determined whether it is the name of a music clip or an address. Depending on this, the identified music clip can then be played, for example, by invoking a streaming provider's application, or a navigation system can be initiated using the corresponding destination guidance to the identified address.

[0034] In some implementations, this method includes: matching the identified content with the functions of the vehicle and controlling the functions of the vehicle based on the identified content. Matching the identified content with the functions of the vehicle, in other words, identifying the type of content, allows the content to be matched with vehicle functions, so that the function can then be controlled, particularly invoked or configured, based on the identified content, especially using the identified content. This mode of operation significantly simplifies the control of vehicle functions because it is not necessary to couple, connect, or otherwise electronically link the terminal device to the vehicle. Therefore, vehicle functions can also be controlled, for example, by a user who does not need to connect their terminal device to the vehicle beforehand. Control can be performed, for example, simply by holding a smartphone to a camera in the interior space of the vehicle. From an image captured by the smartphone, or more precisely from a display with content shown thereon, the corresponding vehicle function with the appropriate content can be invoked quickly and easily. By improving image recognition by stitching together individual images as described above, vehicle functions can be controlled particularly reliably.

[0035] A second aspect of the invention relates to a system for data processing, the system having at least one processor configured to implement the method according to the first aspect of the invention. The system particularly includes at least one lighting device in the interior space of a vehicle.

[0036] In some embodiments, the system further includes at least one image acquisition device configured to acquire image data in the visible light range (RGB). As described above, the content of a terminal device, particularly the content displayed on the terminal device's screen, can be identified in this manner.

[0037] In some embodiments, the system further includes at least one monitoring device configured to acquire monitoring data for at least a portion of the vehicle's interior space, particularly in the form of image data and / or three-dimensional model data in the infrared (IR) range. As described above, it is advantageous to monitor the vehicle's interior space using an infrared camera or a time-of-flight camera.

[0038] A third aspect of the invention relates to a computer program comprising instructions which, when implemented on a system according to the second aspect, cause the system to perform the method according to the first aspect.

[0039] In particular, the computer program can be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This may be advantageous if the computer program itself is to be processed independently of the processor platform on which the one or more programs are to be implemented. In another implementation, the computer program can exist as a file on a data processing unit, particularly on a server, and can be downloaded via a data connection, such as the Internet or a dedicated data connection, such as a private network or local network. Furthermore, the computer program can have multiple cooperating individual program modules.

[0040] Accordingly, the system according to the second aspect may have a program memory in which a computer program is stored. Alternatively, the system may also be configured to access a computer program available externally, such as on one or more servers or other data processing units, via a communication connection, and in particular to exchange with that computer program data applied during the execution of the method or the computer program, or data representing the output of the computer program.

[0041] The features and advantages set forth with reference to the first aspect of the invention are also applicable to the other aspects of the invention. This also applies to the features and advantages set forth with reference to the second aspect of the invention. Attached Figure Description

[0042] Other advantages, features, and application possibilities of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0043] In the picture:

[0044] Figure 1 A flowchart illustrating a method according to one embodiment is shown;

[0045] Figure 2 The diagram shows a view of the vehicle interior space, including the terminal device held by the user within the recognition area; and

[0046] Figure 3 This shows a view of the terminal device with the displayed content. Detailed Implementation

[0047] In all the figures, the same reference numerals are used throughout for the same or corresponding elements of the invention.

[0048] exist Figure 1 The diagram illustrates a method 100 for identifying a mobile terminal device 2 in a vehicle, which, in the illustrated example, results in the control of vehicle functions. Method 100 can be implemented, in particular, in the vehicle's data processing system (not shown). See also the following... Figure 2 and Figure 3 Method 100 will be described in detail. Figure 2 A portion of the vehicle interior space 1 is shown, in which the driver is a smartphone user 10, the smartphone serving as an exemplary mobile terminal device 2. Figure 3 The image shows a smartphone 2 in the user's hand, with content 22 displayed on a display 21. Reflections 23 can obstruct the view of the displayed content 22. However, as explained below, these reflections can be eliminated by means of method 100, specifically by capturing a sequence of images and combining the image data.

[0049] In step S1, monitoring data for at least a portion of the vehicle interior space 1 can be collected. The vehicle may be equipped with a Driver Monitoring System (DMS) that monitors the driver and may also monitor other parts of the vehicle interior space 1. This monitoring can operate permanently during driving (i.e., specifically from the start of ignition to the end of ignition deactivation). For this purpose, the monitoring device 3 can be arranged in the vehicle interior space 2, for example, in the base 5 of the rearview mirror 6. From this position, the driver 10 and most of the vehicle interior space 1 can be clearly seen.

[0050] Infrared (IR) cameras (e.g., near-infrared (NIR) cameras) can be used. IR images are well-suited for surveillance because they are robust to varying lighting conditions, such as even under strong sunlight. Similarly, they can be taken in darkness. Understandably, a corresponding IR light source can be present. It is also conceivable, for example, to create a 3D model of the driver 10 or the interior space 1 of a vehicle using a time-of-flight (ToF) camera. With the aid of the 3D model, the distance to the camera can be calculated, and thus the distances between objects and their motion can be calculated.

[0051] The collected monitoring data can determine whether a mobile terminal device 2 in use has been identified. In principle, this can include any arbitrary use of the terminal device 2 by the user 10 within the vehicle, without necessarily holding the terminal device 2 in a specific area (see description below). It may be sufficient for the user 10 to simply hold the terminal device 2 in their hand. For safety or comfort reasons, to identify use, it may also be necessary to determine, for example, who is using the terminal device 2, in order to distinguish specifically between the driver and passengers; and / or whether the vehicle is stationary or in motion. For example, it could be specified that use by the driver is only identified when the vehicle is stationary, or during travel, use is always identified only when the terminal device 2 is held in area 7 as described below, which is... Figure 2 Zhongyou Figure 2 The dashed line marks it.

[0052] Area 7 is a local area derived from the monitoring data; in the case of (IR) image data, this can simply be defined as a two-dimensional area suitable for recognizing content displayed on the terminal device. Within the vehicle interior space, this may correspond to the area near camera 3 (monitoring device) (or camera 4 (image acquisition device), see below). Furthermore, the arrangement of the rearview mirror 6 at its base 5 or center within the vehicle interior space 1 allows for easy access by the user to position the terminal device 2 appropriately within area 7 for recognition. It can be understood that in Figure 2 The highlighted area 7 is merely an example. The position, size, and shape of area 7 can be adjusted according to desired requirements.

[0053] The following steps of method 100 are advantageously performed only when the terminal device 2 enters the area 7. On the one hand, the method can be implemented more reliably because no evaluation of all monitoring data is performed even if the terminal device 2 is not, for example, in the field of view of camera 3 (or camera 4, see below).

[0054] Before acquiring image data for evaluation in step S2, it can be checked beforehand whether the terminal device 2 is properly positioned in front of the camera 4. For example, in order to reliably identify the displayed content, the terminal device 2 should be held in an orientation that is neither too tilted nor too twisted. The terminal device 2 should also not be obstructed, for example, by the user 10's hand, which could be an indication that the display 21 of the terminal device 2 is not in the field of view or at least that most of the display 21 is partially obscured by the user 10's fingers or hand. In addition, excessively rapid movement of the terminal device 2 may also cause the content 22 to be unrecognizable. On the one hand, if the terminal device 2 does not move too much, the recognition of the content 22 works more reliably. On the other hand, rapid movement of the terminal device 2 in area 7 may be a sign that it has unintentionally entered area 7. Possible reflections 23 on the display 21 can then be eliminated, as explained below.

[0055] Image acquisition device 4 can specifically be a camera 4 that acquires image data in the visible light range and can therefore be called an RGB camera. While an IR camera or ToF camera, as a monitoring device 3, provides suitable images under very different lighting conditions, the monitoring data 3 does not contain (RGB) colors. However, these colors are necessary for identification on terminal devices 2, such as on the display 21 of a smartphone (see...). Figure 3 The content 22 presented on the display (within the visible light range) is necessary. On an IR image, the terminal device 2, such as a smartphone, can only be recognized as a (typically) rectangular object, on which the display 21 can at best be distinguished as a rectangular monochromatic surface.

[0056] exist Figure 2 The monitoring device 3 and the image acquisition device 4 are shown as separate units, but they can also be combined into one unit, which in particular can be an RGB-IR camera, i.e. a camera that provides image data not only in the RGB range but also in the IR range.

[0057] The image acquisition device 4 can also be arranged in the base 5 of the rearview mirror 6. Therefore, the field of view areas of the monitoring device 3 and the image acquisition device 4 can be considered largely equal, at least in area 7 (or even substantially equal if the monitoring device 3 and the image acquisition device 4 form a single unit). Therefore, image data located in area 7 is acquired in particular. To expand coverage of the interior space, it can also be specified that image data from multiple cameras are combined.

[0058] In step S2, image data is acquired, specifically multiple individual images as an image sequence. For example, a corresponding video stream can be captured using camera 4, where each frame constitutes an individual image and provides image data for method 100. When user 10 holds their terminal device in area 7, the image sequence can begin to be captured, which—in other words—means that terminal device 2 is visible in the camera image.

[0059] Then you can continue shooting until the displayed content can be identified (especially by stitching together the image data of the individual images). This is achieved by implementing hand movements (see...). Figure 2 (The arrow in the image) Orients the terminal device 2 relative to the camera 4 in the interior space 1. The user 10 moves and rotates the potentially reflective surface of the display 21 to such an extent that the reflections are located at different positions in the respective views at different points in time during the video sequence. For this purpose, information about how the terminal device is oriented in space can be additionally used to selectively choose single images in a sequence (step S3) in a simple manner and form, in which reflections occur at different locations. By stitching the selected captured images together (step S4), reflections 23 can now be removed from the image content since the content 21 displayed on the display 21 remains unchanged. Reflections 23 can preferably be completely removed or at least reduced to such an extent that valid content 21 that can subsequently be used to control vehicle functions can be identified.

[0060] Image data is now evaluated using image analysis. First, the content 22 presented on terminal device 2 is identified (step S5). Specifically, if the content is text, known methods for optical character recognition (OCR) can be used, or if the content is a code, such as a QR code, known methods for reading machine-readable codes can be used. Then, in step S6, the identified content can be matched with vehicle functions, for example, using known parsing or classification methods. For example, the identified address can be matched with the vehicle function "navigation system," and the identified music clip can be matched with the vehicle function "music player," etc. For matching, other visual features, such as formatting, color, etc., may be used if necessary, for example, to invoke a specific streaming provider or data from the code.

[0061] Then, in step S7, the corresponding vehicle function is operated according to the identified content. Controlling the vehicle function may specifically include calling or activating the vehicle function that matches the identified content, particularly configuring the identified content to the vehicle function. For example, if content 22 is an address, the address can be directly used as the set target address to call vehicle function 9 "Navigation System", thereby eliminating the need to manually enter the address in the vehicle's navigation system. If it involves an identified music segment, a streaming media provider with the identified music segment can be directly invoked, and so on.

[0062] However, vehicle control functions may also simply include displaying data corresponding to the identified content, such as on the vehicle's screen 8, particularly on the infotainment system's display. Therefore, it may be necessary to provide queries for user confirmation, such as whether the identified content is correct and / or whether the function corresponding to the identified content should be implemented. For example: "The following address has been identified. Should the target boot be initiated?", "The following music clip has been identified. Should it be played?", etc.

[0063] For example, this method can be implemented in practice as follows: The user opens an app on their smartphone, for example, to display a QR code. The user rotates the smartphone and moves it toward the interior space camera. When the smartphone becomes visible in the camera image of the interior space camera, a shot is taken and the image is stored. Simultaneously, the camera software estimates the corner points of the smartphone display and calculates the smartphone's position in 3D space. Optionally, the coordinates of the hand holding the smartphone with a reflective display surface can be calculated. If the position of the smartphone display, especially the plane angle between the interior space camera and the smartphone display, changes significantly (i.e., greater than a predetermined threshold), the relevant image is selected (from the image sequence). Finally, a complete image is calculated from the selected images, specifically by blending and stitching the individual images together (this can also be referred to as "stitching"). The stitched image is used to identify the complete QR code so that it can ultimately control the associated vehicle functions.

[0064] While at least one exemplary embodiment has been described above, it should be noted that numerous variations exist. It should also be noted that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the above description is intended to provide guidance to those skilled in the art for implementing at least one exemplary embodiment, and it is understood that different changes may be made in the manner of function and arrangement of the elements described in the exemplary embodiments without departing from the technical solutions respectively identified in the appended claims and their legal equivalents.

[0065] List of reference numerals

[0066] 100 Method for identifying mobile terminal devices in vehicles

[0067] 1. Vehicle interior space

[0068] 2. Terminal devices (smartphones)

[0069] 3. Monitoring devices (IR cameras)

[0070] 4. Image acquisition device (RGB camera)

[0071] 5. Base of rearview mirror

[0072] 6 interior rearview mirrors

[0073] 7 regions

[0074] 8. Displays of the infotainment system

[0075] 9 Vehicle Functions

[0076] 10 users (drivers)

[0077] 21 Displays of terminal devices (smartphones)

[0078] 22 Displayed Content

[0079] 23 Reflections

Claims

1. A method (100) for identifying a mobile terminal device (2) in a vehicle, wherein, The method includes the following: First image data is acquired by means of at least one image acquisition device (4) in the interior space (1) of the vehicle, wherein the first image data contains at least a first view of the user's (10) terminal device (2) at a first time point; Second image data is acquired by means of the image acquisition device (4) and / or other image acquisition devices in the interior space (1) of the vehicle, wherein the second image data at least includes a second view of the user's (10) terminal device (2) at a second time point; The first and second image data are combined to obtain a combined view of the user's (10) terminal device (2); and The content (22) displayed on the user's (10) terminal device (2) is identified by evaluating the combined image data.

2. The method according to claim 1, further comprising: The location and / or orientation of the mobile terminal device (2) within the vehicle interior space (1) is determined by means of at least one monitoring device (3) or by means of an image acquisition device (4). The image data is combined only when the position and / or orientation of the mobile terminal device (2) at the first time point is different from that at the second time point.

3. The method according to claim 2, wherein, The first and second image data are part of an image sequence acquired by means of an image acquisition device (4), wherein the method further includes the following: Select the first and second image data from the image sequence. The second image data is selected only when the position and / or orientation of the mobile terminal device (2) at the first time point is different from the position and / or orientation at the second time point.

4. The method according to claim 2 or 3, wherein, The position and / or orientation of the mobile terminal device includes the angle between the mobile terminal device (2) and the image acquisition device (4).

5. The method according to claim 4, wherein, When the difference in angles exceeds a predetermined threshold, it is confirmed that the position and / or orientation of the mobile terminal device (2) are different at the first time point and the second time point.

6. The method according to any one of claims 2 to 5, further comprising: Confirm whether the vehicle's functions (9) match the identified content (22). When no function (9) can be matched with the content (22), additional image data is selected from the image sequence at at least one additional time point after the first and second time points and the additional image data is combined with the other image data.

7. The method according to any one of the preceding claims, wherein, The method further includes the following before acquiring image data: Monitoring data for at least a portion of the interior space (1) of the vehicle is collected by means of at least one monitoring device (3) or by means of an image acquisition device (4); Confirm whether the user's (10) terminal device (2) is at least partially located in the predetermined area (7) of the monitoring data; Image data is collected only when it is confirmed that the user's (10) terminal device (2) is at least partially located in the predetermined area (7).

8. The method according to any one of the preceding claims, further comprising: Match the identified content (22) with the vehicle's functions (9); and Control the vehicle’s functions (9) based on the identified content (22).

9. The method according to any one of the preceding claims, wherein, When combining image data, the portion of the image data from the first and second image data whose saturation exceeds a threshold is considered.

10. The method according to any one of the preceding claims, wherein, The identified content (22) includes identified text and / or identified machine-readable code.

11. A system for data processing, having at least one processor configured to implement the method according to any one of the preceding claims.

12. A computer program, including instructions that, when executed on a system according to claim 11, cause the system to perform the method according to any one of claims 1 to 10.

Images