Method for identifying a mobile terminal device in a vehicle

Abstract

A method for identifying a mobile terminal device (2) in a vehicle is disclosed. In the method, image data is captured in an interior space (1) of the vehicle by means of at least one image capturing device (4), wherein the image data contains at least one view of the terminal device (2). A content displayed on the terminal device (2) is identified. To identify the content, a presentation form (24) of the displayed content is detected by analyzing the image data, and a function (9) of the vehicle associated with the displayed content (22) is determined by means of a trained machine learning model in accordance with the detected presentation form (24). Furthermore, a function parameter in the displayed content (22) is determined. A method for training the machine learning model is also disclosed.

Description

Technical Field

[0001] This invention relates to a method for identifying a moving terminal device in a vehicle. Specifically, the content displayed on the moving terminal device should be captured using an image acquisition device to control the vehicle's functions. Background Technology

[0002] Modern vehicles have a multitude of functions that can be controlled by the user, such as activating, starting, stopping, and configuring them. This is particularly true of infotainment system functions, such as inputting navigation destinations, playing music, calling contacts, or similar features. These infotainment system functions are typically controlled through direct user interaction with the vehicle, such as by manipulating buttons or touching a touchscreen, or through voice or gesture control.

[0003] However, in-vehicle operation can sometimes be cumbersome, such as when data, like navigation destinations or music titles, must be entered via a keyboard on a touchscreen. Typically, such input is simpler on the user's terminal device, such as a smartphone. In some cases, the data even already exists on the smartphone, for example, when a user has already listened to a song on their smartphone and now wants to listen to it in the vehicle. Therefore, methods are known in which data is transmitted from the user's terminal device to the vehicle, for example, via Bluetooth or via a cloud service.

[0004] It is also known that content displayed on the screen of a mobile terminal device is captured via cameras in the vehicle. Since modern vehicles are typically equipped with cameras, for example for interior space monitoring, this infrastructure can be used to read the displayed content. The content may involve displayed text, which is recognized via Optical Character Recognition (OCR). Thus, for example, a navigation destination can be easily transmitted from a smartphone to the vehicle. Machine-readable codes, such as Quick Response Matrix Codes (QR codes), can also be used, captured by cameras in the vehicle.

[0005] Even if text recognition correctly identifies the displayed text, correctly identifying or associating it with the actual relevant text can still be difficult. For example, the displayed content may contain different texts or words, of which only a portion is relevant to the correct control of the intended function. For instance, a track title must be separated from other text, such as descriptions or labels in an app, in order to actually control the music function. It may also happen, for example, that multiple points of interest (POIs) are identified via text recognition on a map, but only one of these POIs is of interest to the user. While it would be possible to process the entire identified text and provide a list to the user, this is particularly undesirable from the user's perspective. In other cases, confusing or ambiguous results may occur during text recognition, such as when a song title is identified as both a name and a location. Thus, the text recognition result cannot be clearly associated with a function, such as "media" or "navigation," meaning that clear control of vehicle functions cannot be achieved, or at least not without user intervention. Summary of the Invention

[0006] The objective of this invention is to provide an improved method for identifying a moving terminal device in a vehicle. In particular, the identification of content displayed on the moving terminal device should be improved.

[0007] The solution to the task is achieved according to the teachings of the independent claims. Various embodiments and further extensions of the invention are the technical solutions of the dependent claims.

[0008] A first aspect of the invention relates to a method, particularly computer-implemented, for identifying movement within a terminal device in a vehicle. In this method, image data is acquired within the interior space of the vehicle using at least one image acquisition device, wherein the image data contains at least one view of the terminal device. Content displayed on the terminal device is identified. To identify the content, the presentation format of the displayed content is detected by analyzing the image data, and based on the detected presentation format, the vehicle's functions associated with the displayed content are determined using a trained machine learning model. Furthermore, functional parameters within the displayed content are determined.

[0009] The method described above, according to the first aspect, is therefore particularly based on the analysis of image data acquired by an image acquisition device, such as a camera. To identify the displayed content, a trained machine learning model is used. Specifically, the model can associate the presentation, i.e., layout, of the displayed content with vehicle functions. The model is trained using relevant training data before application, as will be explained in more detail later. Therefore, the method can identify which vehicle function the displayed content should be associated with based on its presentation. For example, it can be identified whether the displayed content relates to a view of a music application on a terminal device, whereby the song title can be identified as a functional parameter, and the corresponding media or entertainment application in the vehicle can be determined. If, based on the presentation, the machine learning model identifies that the displayed content relates to a map representation or navigation application on a terminal device, it can be associated with the navigation function in the vehicle accordingly, and a location can be identified as the navigation destination as a functional parameter. In this way, the reliability of content identification can be improved because the system can associate or classify the identified content with vehicle functions (also called "domains") according to the layout of the identified content using a trained machine learning model.

[0010] The term “vehicle” as used herein specifically refers to passenger cars, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vans, buses, trucks, delivery vans and similar vehicles.

[0011] The term "image acquisition device" used herein specifically refers to a camera, particularly a digital camera. A camera can capture still images (photographs) or moving images (videos), especially in the visible light range. An image acquisition device can acquire or capture such images and output the corresponding image data. The image acquisition device can also be used as a monitoring device; that is, the image acquisition device and the monitoring device can be the same device.

[0012] The term "mobile terminal device" as used herein specifically refers to a portable electronic device capable of wirelessly communicating with other devices or networks. This electronic device relates to a small device, typically portable in a pocket or held in the hand. Mobile terminal devices include smartphones, tablets, laptops, smartwatches, and similar devices. The mobile terminal device is specifically configured for displaying content, particularly on a screen. The user of the terminal device may specifically be the driver of a vehicle. However, it should be understood that the front passenger and, in principle, any vehicle occupant may also be a user of the terminal device in the sense of this invention.

[0013] The term "content" of a terminal device (as used herein) should be understood in particular as information displayed on the display device of the terminal device, such as a screen. This content does not necessarily have to be displayed on the terminal device within an application that corresponds to a vehicle function in its type. The content may, for example, consist only of untypeset text or machine-readable code. Nevertheless, the content is considered particularly in terms of its presentation (i.e., layout, arrangement, representation), which may include, for example, font formatting, colors, shapes, logos, or the like, to identify the associated vehicle function. In particular, the displayed content also includes functional parameters for the vehicle's associated function, i.e., parameters used to invoke or execute that function. This could be, for example, the title of a song played in a media application or a location as a navigation destination.

[0014] The term "monitoring device" as used herein specifically refers to a device suitable for monitoring the interior space of a vehicle, particularly for monitoring vehicle occupants, and especially for monitoring the position and movement of said occupants within that space. The monitoring device may involve an image acquisition device. Infrared (IR) cameras (e.g., near-infrared (NIR) cameras) can be used. IR images are well-suited for monitoring because they are robust to varying lighting conditions, such as even in strong sunlight. Imaging in darkness is also possible. It should be understood that a corresponding IR light source may be present. It is also conceivable to create a three-dimensional model of the driver's or vehicle's interior space, for example, using a time-of-flight (TOF) camera. Using this three-dimensional model, distances to the camera can be calculated, and thus the spacing between objects and the movement of said objects can be calculated.

[0015] The term "vehicle function" (synonymous with the term "vehicle feature") as used herein (when applied in this document) should be understood in particular as the function of the vehicle's infotainment system, such as "navigation" or "entertainment." This can include proprietary applications of the vehicle manufacturer and / or applications implemented by external suppliers. Control of vehicle functions as described herein can in particular include invoking or initiating vehicle functions associated with identified content based on identified content, i.e., using determined function parameters, such as initiating navigation to a determined destination, starting playback of a determined song using an entertainment application, etc. However, control of vehicle functions can also include displaying only data about the identified content or data about associated vehicle functions (where applicable, together with the identified content), for example, on the vehicle's screen, particularly on the infotainment system's display screen.

[0016] The terms “comprising,” “including,” “incorporating,” “having,” “having,” “with,” or any other variations thereof, when used herein, should cover non-exclusive inclusion relationships. Thus, a method or apparatus that includes or has a list of elements, for example, is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a method or apparatus.

[0017] Furthermore, unless the opposite is explicitly stated, "or" refers to an inclusive "or" rather than an exclusive "or". For example, condition A or B is satisfied by one of the following conditions: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and not only A but also B is true (or exists).

[0018] The term “an” or “a kind” (as it is used herein) is defined in the sense of “one / a kind or a plurality / multiple kinds”. The terms “another” and “another”, and any other variations thereof, shall be understood in the sense of “at least one other”.

[0019] The term “multiple” (as it is used in this article) is understood in the sense of “two or more”.

[0020] The terms "configuration" or "setting" are used to satisfy a specific function (and its variations). In the context of this invention, it should be understood that the corresponding device already exists in a design or setting in which the device can perform the function, or that the device can at least be set, i.e., configured, such that the device can perform the function after a corresponding setting. Here, the configuration can be performed, for example, by setting corresponding parameters of the process flow or corresponding settings of switches or the like, to activate or deactivate functionality or settings. In particular, the device can have multiple predetermined configurations or operating modes, such that the configuration can be performed by selecting one of these configurations or operating modes.

[0021] Preferred embodiments of the method are described below. Unless explicitly excluded or technically impossible, these preferred embodiments can be combined with each other and with other aspects further described in this invention.

[0022] In some implementations, the identification of the functional parameter is performed within a portion of the displayed content, determined according to the detected presentation format and the identified function. In this way, functional parameters, such as text or code, can be identified selectively because the location of the functional parameter within the layout is known based on the presentation format and its associated function. Thus, for example, a song title can be easily distinguished from other equally visible descriptions. Similarly, with navigation, the destination can be easily filtered out from other locations because the defined area is known, and the destination, rather than other locations, is indeed located within that defined area in the determined view. By limiting identification to the defined area, relevant text can be easily distinguished from irrelevant text (when text is identified as a functional parameter). Furthermore, the identification can be performed more efficiently and quickly because it is not necessary to traverse the entire displayed content to search for and identify the functional parameter as needed.

[0023] In some implementations, additionally, anchor points are determined within the detected presentation format of the displayed content. These anchor points define a portion of the displayed content where the functional parameters can be identified. Such anchor points can be easily determined using a trained machine learning model. From these anchor points, a bounding box can be defined for the portion of the content where the functional parameters can be identified. The system can learn, using the machine learning model, which presentation format, location, and function to which anchor points can be found.

[0024] In some implementations, the anchor point is therefore the position within the presentation of the displayed content, determined using the machine learning model. The model, for example, may have already learned where a song title can be found in a music application, such as below the album art, where other text, such as annotations, may be disregarded.

[0025] In some implementations, determining the anchor point involves identifying markers within the detected presentation format, where the identified markers are determined as anchor points. For example, in a particular presentation format of the displayed content, specific symbols or icons may indicate locations where the functional parameter can be found. The machine learning model is then particularly able to find the markers within the presentation format, even when the location of the markers is flexible, such as POIs on a map marked with specific symbols. In this way, the correct functional parameter can be identified with improved reliability, even when the presentation format is flexible.

[0026] In some implementations, the detection of the presentation format includes identifying patterns in the displayed content and comparing those patterns with known patterns. The aforementioned markers can also be identified through pattern comparison. Pattern recognition can simplify the association of the detected presentation format with specific vehicle functions, particularly with the aid of the machine learning model. The presentation format may, for example, have specific characteristics (i.e., patterns) of content arrangement, which can then be easily associated with functions, i.e., with domains.

[0027] In some implementations, the method includes controlling vehicle functions based on identified functional parameters. As described, the identified content is associated with vehicle functions by recognizing the presentation format using the machine learning model, so that the function can be invoked or configured specifically based on the identified content (more precisely, based on the identified functional parameters). This approach significantly simplifies the control of vehicle functions because no coupling, connection, or other electronic connection between a terminal device and the vehicle is necessary. Therefore, the vehicle functions can also be controlled by a user, for example, without the user needing to pre-connect their terminal device to the vehicle. Control can be performed, for example, simply by holding a smartphone in front of a camera inside the vehicle. Images captured by the smartphone (or more precisely, from the display screen and the content shown therein) can quickly and easily invoke the corresponding vehicle function with matching content.

[0028] In some implementations, the functional parameters include text, which is recognized using text recognition. Alternatively, the functional parameters may also include machine-readable code. Such recognition can be performed using known methods, such as text recognition from the image data using OCR.

[0029] In some embodiments, the method further includes the following before acquiring the image data using the image acquisition device: acquiring monitoring data for at least a portion of the interior space of the vehicle using at least one monitoring device, and determining (i.e. checking) whether the user's terminal device is at least partially located in a pre-defined area of ​​the monitoring data, wherein the acquisition of the image data using the image acquisition device is performed only when it is confirmed that the user's terminal device is at least partially located in the pre-defined area.

[0030] The method can be designed with improved reliability by allowing for prior identification of the content on the terminal device. Inconsistent image analysis would require additional computing power and could potentially lead to undesirable results, for example, when the user may not intend to control vehicle functions. Image data is only acquired to potentially identify the content on the terminal device if the user actually keeps the terminal device within the acquisition area (pre-defined area). For this purpose, a monitoring device may be used, which may already be present in the vehicle, for example, to monitor the driver for other purposes. However, the monitoring data itself may not be suitable for identifying the content represented on the terminal device; therefore, image data is only acquired as described above after it is definitively confirmed that the terminal device is located within the pre-defined area. However, the monitoring device could also involve an image acquisition device, as explained in more detail below.

[0031] In some embodiments, the method further includes: confirming whether a mobile terminal device in use is at least partially located within a pre-defined area of ​​the interior space, the pre-defined area being within the acquisition area of ​​an image acquisition device (see above description). Therefore, it can be stipulated that each use of the mobile terminal device is not recognized, for example, when the user is using their smartphone solely for themselves, such as typing a message or making a call. The pre-defined area of ​​the interior space may be an area in front of an interior space camera, which may be located within a rearview mirror. Therefore, the user must then actively keep their terminal device in front of the camera to facilitate recognition of use.

[0032] In some implementations, the method further includes the following before identifying content on a user's terminal device by analyzing the image data: determining segments of image data from the monitoring data (particularly the pre-defined area described above), the segments representing the terminal device, and confirming whether the segments contain image data suitable for identifying content on the user's terminal device, wherein identification of the content on the terminal device is performed only if it is confirmed that the segments contain image data suitable for identifying content on the terminal device. In other words, the monitoring data can be used to determine segments of the image data that are relevant to identification. Here, the location of, for example, the display screen of the terminal device can be determined according to coordinates. When the display screen of the terminal device is not turned on at all (this can be determined, for example, by brightness differences in the image data), the image data may not be suitable for identifying content. Lighting conditions may also be unsuitable for identifying content on the terminal device, for example, because the display screen appears too dark. This can be done, for example, by determining pixels and their saturation, for example, when the number of pixels with a specific saturation (minimum saturation) is less than a pre-defined value.

[0033] In some implementations, the image data is acquired within the visible light range using the image acquisition device. This can be done using an RGB camera. This method is best suited for recognizing content displayed on the terminal device's screen.

[0034] In some implementations, the monitoring data is acquired using a monitoring device configured to acquire monitoring data for at least a portion of the vehicle's interior space, wherein the monitoring data is preferably included as image data in the infrared (IR) range and / or as three-dimensional model data. Driver monitoring systems typically operate using such IR cameras. This method is suitable for monitoring vehicle interior spaces because image data acquired in the IR range is robust to varying lighting conditions. In other words, the method provides reliable results in both strong sunlight and darkness (especially compared to image data in the visible light range). Data regarding a three-dimensional model, for example, captured using a time-of-flight camera, enables a detailed three-dimensional view of the vehicle's interior space.

[0035] A second aspect of the invention relates to a method for training a machine learning model for a terminal device used to identify movement in a vehicle, particularly in the method according to the first aspect. The method according to the second aspect provides training data, wherein the training data includes image data containing at least one view of the terminal device and associations between the image data and vehicle functions. Content displayed on the terminal device is identified (from the training data). To this end, the presentation of the displayed content is detected by analyzing the image data, and the functions of the vehicle associated with the displayed content are determined by the machine learning model according to the detected presentation, and functional parameters in the displayed content are identified. Since the training data contains associations between the image data and corresponding vehicle functions, effective training can be provided. In other words, the image data is labeled accordingly with its corresponding vehicle function (domain), thereby establishing a training process. The machine learning model may, for example, be a neural network trained using the training data.

[0036] In some embodiments of the training method, the training data defines a partial region of the displayed content, in which the functional parameter can be identified. In other words, for the image data of the training data, not only are the matching domains known, but also the parts of the image data (the partial regions) where the functional parameter, i.e., related text, can be found. As described above, the partial regions can be defined, for example, as bounding boxes at anchor points.

[0037] In some implementations, the training data includes image data of different views of the terminal device, which are associated with the same function having the same functional parameters. In this way, the model can be trained better, and the reliability of recognition can be improved. In particular, the dependence on the actual view can be reduced, that is, content can be recognized even if the detected presentation is biased, such as larger or smaller, or rotated. For example, it can be specified that content on the terminal device is represented in different views, such as scaled or scrolled, where each view thus has another segment visible. Figure One Generally, it can include arbitrary rotation, zoom, segmentation, and the like. Furthermore, the training can be performed using training data representing views of a specific function, but with variable function parameters, such as views from a music application that, while presenting themselves identically, have different song titles (as function parameters), or map views from a navigation application with different addresses as navigation destinations.

[0038] A third aspect of the invention relates to a system for data processing, the system having at least one processor configured such that the processor performs the method according to the first aspect.

[0039] In some embodiments, the system includes at least one image acquisition device configured to acquire image data within the visible light spectrum (RGB). As explained above, this allows for the identification of the content of the terminal device, particularly the content displayed on the terminal device's screen.

[0040] In some embodiments, the system further includes at least one monitoring device configured to acquire monitoring data, particularly in the form of image data in the infrared (IR) range and / or three-dimensional model data, for at least a portion of the vehicle's interior space. As explained above, it is advantageous to monitor the vehicle's interior space using an infrared camera or a time-of-flight camera.

[0041] The fourth aspect of the invention relates to a computer program having instructions that, when executed on a system according to the third aspect, cause the system to perform the method according to the first aspect.

[0042] The computer program can be stored, in particular, on a non-volatile data carrier. Preferably, the data carrier is in the form of an optical data carrier or a flash memory module. Advantageously, the computer program itself should be processed independently of the processor platform on which the one or more programs can be executed. In another implementation, the computer program can exist as a file on a data processing unit, particularly on a server, and can be downloaded via a data connection, such as the Internet or a dedicated data connection, such as a private network or local area network. Furthermore, the computer program can have multiple cooperating individual program modules.

[0043] The system according to the third aspect may accordingly have a program memory in which the computer program is stored. Alternatively, the system may also be configured to access an external computer program, for example, available on one or more servers or other data processing units, via a communication connection, particularly to exchange data with the computer program that is applied during the operation of the method or computer program, or represents the output of the computer program.

[0044] The features and advantages set forth with reference to the first aspect of the invention also apply to the other aspects of the invention. This is equally true of the features and advantages set forth with reference to the second aspect of the invention.

[0045] Further advantages, features, and applications of the invention will become apparent from the following detailed description in conjunction with the accompanying drawings. Attached Figure Description

[0046] This is shown here:

[0047] Figure 1 A flowchart illustrating a method according to one implementation is shown;

[0048] Figure 2 A view showing the interior space of a vehicle with a terminal device held by a user in the recognition area; and

[0049] Figure 3 This displays a view of the terminal device with the content shown. Detailed Implementation

[0050] In the accompanying drawings, the same reference numerals are consistently used for the same or corresponding elements.

[0051] exist Figure 1 The diagram illustrates a method 100 for identifying a moving terminal device 2 within a vehicle, which, in the illustrated example, ultimately leads to the control of vehicle functions. The method 100 can, in particular, be executed within the vehicle's data processing system (not shown). Reference is also made below. Figure 2and Figure 3 The method described herein is explained in section 100. Figure 2 A section of the vehicle interior space 1 is shown, the vehicle interior space including a driver as user 10, the user being a user of a smartphone as an example mobile terminal device 2. Figure 3 The image shows a smartphone 2 in the user's hand, which has content 22 displayed on a display screen 21.

[0052] In step S1, monitoring data can be collected for at least a portion of the vehicle interior space 1. The vehicle may be equipped with a Driver Monitoring System (DMS) that monitors the driver and potentially other parts of the vehicle interior space 1. The monitoring can be continuously active during driving (i.e., particularly with the ignition being turned on or off). For this purpose, the monitoring device 3 can be positioned within the vehicle interior space 2, for example, in the base 5 of the rearview mirror 6. From this position, the driver 10 and a wide portion of the vehicle interior space 1 can be clearly seen.

[0053] Infrared (IR) cameras (e.g., near-infrared (NIR) cameras) can be used. IR images are well-suited for surveillance because they are robust to varying lighting conditions, such as strong sunlight. Filming in darkness is also possible. It should be understood that a corresponding IR light source can exist. It is also conceivable to create a three-dimensional model of the driver 10 or the interior space 1 of the vehicle, for example, using a time-of-flight (ToF) camera. With the aid of this three-dimensional model, the distance to the camera can be calculated, and thus the spacing between objects and the motion of those objects can be calculated.

[0054] The collected monitoring data can determine whether a mobile terminal device 2 in use is identified. In principle, this can include any arbitrary use of the terminal device 2, which can be performed by the user 10 in the vehicle, without necessarily keeping the terminal device 2 in a specific area (see description below). It may be sufficient when the user 10 has the terminal device 2 in their hand or similar circumstances. For safety or comfort reasons, the identification of this use may also also confirm, for example, who is using the terminal device 2, in order to distinguish specifically between the driver and the front passenger, and / or whether the vehicle is stationary or moving. For example, it can be stipulated that use by the driver is identified only when the vehicle is stationary, or during movement, use is always identified only when the terminal device 2 is kept in area 7 as described below. Figure 2 It is shown in the middle by a dashed line.

[0055] The area 7 is a segment of the monitoring data. In (IR) image data, this can simply be a two-dimensional area, which has been determined to be suitable for identifying content represented on the terminal device. In the vehicle interior space, this can correspond to the area near camera 3 (monitoring device) (or the area near camera 4 (image acquisition device), see below). Furthermore, the arrangement structure in the middle above the base 5 of the interior rearview mirror 6 in the vehicle interior space 1 is easily accessible to the user so that the terminal device 2 can be placed in area 7 in an appropriate manner for identification. It should be understood that, in Figure 2 The area 7 shown is merely an example. The position, size, and shape of area 7 can be set according to desired requirements.

[0056] The steps of method 100 are advantageously performed only when terminal device 2 enters area 7. On the one hand, the method can be performed more reliably because analysis of all monitoring data is not performed, nor is analysis performed when terminal device 2 is not, for example, in the field of view of camera 3 (or camera 4, see below).

[0057] Before acquiring image data for analysis in step S2, it can be pre-checked that the terminal device 2 is properly positioned in front of the camera 4. For example, the terminal device 2 should be kept in one direction, neither excessively tilted nor excessively twisted, in order to reliably recognize the represented content. The terminal device 2 should also not be obstructed, for example, by the user 10's hand. This could be indicated by signs that the display screen 21 of the terminal device 2 is not within the field of view, or at least a wide portion of the display screen 21 is partially obstructed by the user 10's fingers or hand. Furthermore, excessively rapid movement of the terminal device 2 may also lead to the failure to recognize the content 22. On the one hand, the recognition of the content 22 works more reliably when the terminal device 2 does not move excessively. On the other hand, rapid movement of the terminal device 2 in area 7 can be a clue that the terminal device unintentionally enters area 7. Potential reflections 23 on the display screen 21 can then be removed as described below.

[0058] Image acquisition device 4 can in particular be a camera 4, which acquires image data within the visible range of light and can therefore be called an RGB camera. While IR cameras or ToF cameras, as monitoring devices 3, provide suitable images under very different lighting conditions, the monitoring data does not contain (RGB) colors. However, these colors are necessary for identification on terminal devices 2, such as the display screen 21 of a smartphone (see...). Figure 3The content 22 (within the visible range of light) is necessary. On an IR image, the terminal device 2, such as a smartphone, can only be identified as a (generally) rectangular object, on which the display screen 21 can at most be distinguished as a rectangular monochrome plane.

[0059] Despite Figure 2 In the diagram, the monitoring device 3 and the image acquisition device 4 are shown as separate units, but the monitoring device and the image acquisition device can also coexist in a single unit, which in particular can be an RGB-IR camera, i.e., a camera that provides image data in both the RGB and IR ranges.

[0060] The image acquisition device 4 can also be arranged in the base 5 of the rearview mirror 6. Therefore, the field of view of the monitoring device 3 and the field of view of the image acquisition device 4 can be considered to be substantially overlapping, at least in area 7 (or even substantially overlapping when the monitoring device 3 and the image acquisition device 4 are configured as a single unit). Therefore, image data located in area 7 is acquired in particular.

[0061] In step S2, image data is acquired, which includes a view of the terminal device 2 with the displayed content 22. The image data is then analyzed using image analysis, wherein the layout of the presentation format 24, i.e., the displayed content 22, is detected (step S3). Based on the presentation format 24, the content 22 can now be associated with vehicle functions 9 by applying a machine learning model (step S4). In other words, the presentation format 24 is analyzed, wherein the method 100 can classify the displayed content 22 according to the model, i.e., associate it with the corresponding vehicle function 9 (also called a domain). For example, the displayed presentation formats of a music application and a navigation application on the terminal device 2 can be distinguished from each other, thereby improving the reliability in text recognition because the method 100 already "knows" whether the recognized text belongs to, for example, an entertainment application or a navigation application. In particular, the text is identified as a functional parameter for the determined function 9 in a portion 23 of the presentation format 24 of the displayed content 22, which is pre-defined by the presentation format 24.

[0062] Before the actual application of the machine learning model in method 100, the model is trained using training data. Here, the model is trained using training data comprising a series of image data labeled with domains and associated text markers, such as bounding boxes. In other words, in the training data, it is known which vehicle function (domain) each image data belongs to, i.e., music or navigation, and where the corresponding function parameters, i.e., text describing song titles or navigation destinations, can be found in the corresponding view. Thus, method 100 can then easily identify the relevant partial region 23 in which the text can be recognized, ignoring other text, words, labels, and the like. To further improve the reliability of recognition, the model is trained, in particular, using different views of the same content 22. For example, in the image data, the content 22 may be scrolled, zoomed, or twisted, where the correct association is trained in each case. Similarly, different views of the presentation format belonging to vehicle function 9 are trained using different function parameters, such as different views of a music application with different song titles.

[0063] To identify the functional parameters, when the content is code, such as a QR code, known methods for Optical Character Recognition (OCR) or methods for reading machine-readable code can be used. The recognition can be performed quickly and reliably, especially because it is not necessary to traverse and search the entire content 22, for example, text, but only a portion of the region 23.

[0064] Subsequently, in step S6, the corresponding associated vehicle functions are operated according to the identified functional parameters. Controlling the vehicle functions may specifically include invoking or activating vehicle functions associated with the identified content, particularly based on the identified content, more precisely, based on the functional parameters; that is, the vehicle functions are configured using the identified functional parameters. For example, when the domain relates to "navigation" and the text (functional parameter) relates to an address, the address can be directly used as the pre-set destination address to invoke vehicle function 9, "Navigation System," thus eliminating the need to manually input the address into the vehicle's navigation system. When the identified functional parameters relate to a song title, the identified music can be directly used to invoke entertainment applications (such as streaming media providers).

[0065] However, vehicle function control may also consist solely of displaying data associated with the identified content, such as on the vehicle's screen 8, particularly on the infotainment system's display. Subsequently, if necessary, it may be specified to provide queries for user confirmation, such as whether the identified content is correct and / or whether the function associated with the identified content should be executed. Examples include: "The following address has been identified. Should destination guidance begin?", "The following musical piece has been identified. Should it be played?", etc.

[0066] Although at least one exemplary embodiment has been described for the foregoing, it should be noted that numerous variations exist. It should also be noted that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the apparatus and methods described herein. Rather, the foregoing description provides guidance to those skilled in the art for implementing at least one exemplary embodiment, wherein it should be understood that various changes can be made to the function and arrangement of the elements described in the exemplary embodiments without departing from the technical solutions and their legal equivalents as set forth in the appended claims.

[0067] Figure Labels

[0068] 100 Method for identifying moving terminal devices in a vehicle

[0069] 1. Vehicle interior space

[0070] 2. Terminal devices (smartphones)

[0071] 3. Monitoring device (IR camera)

[0072] 4. Image acquisition device (RGB camera)

[0073] 5. Interior rearview mirror base

[0074] 6. Interior rearview mirror

[0075] 7 areas

[0076] 8. Infotainment system display screen

[0077] 9. Vehicle Functions

[0078] 10 users (drivers)

[0079] 21. Display screen of terminal device (smartphone)

[0080] The content displayed in 22

[0081] 23. Partial areas with functional parameters

[0082] 24. The presentation format of the displayed content

Claims

1. A method (100) for identifying a mobile terminal device (2) in a vehicle, wherein, The method includes the following: - Image data is acquired in the interior space (1) of the vehicle using at least one image acquisition device (4), wherein the image data includes at least one view of a terminal device (2); - Identify the content (22) displayed on the terminal device (2), wherein the identification of the content (22) includes the following: - The presentation form (24) of the displayed content (22) is detected by analyzing the image data; - Based on the detected presentation format (24), determine the vehicle's functions (9) associated with the displayed content (22) using a trained machine learning model; and - Identify the functional parameters in the displayed content (22).

2. The method according to claim 1, wherein, The identification of the functional parameters is performed in a portion of the displayed content (22), which is determined according to the detected presentation format (24) and the determined function (9).

3. The method of claim 2, further comprising: - Determine an anchor point in the detected presentation form (24) of the displayed content, wherein the anchor point defines a portion (23) of the displayed content (22) in which the functional parameters can be identified.

4. The method according to claim 3, wherein, The anchor point is the position in the presentation (24) of the displayed content (22), which is determined by the machine learning model.

5. The method according to claim 3 or 4, wherein, The determination of the anchor point includes identifying a marker in the detected presentation form (24), wherein the identified marker is determined as the anchor point.

6. The method according to any one of the preceding claims, wherein, The detection of the presentation format (24) includes identifying the pattern of the displayed content (22) and comparing the pattern with known patterns, particularly with the aid of the machine learning model.

7. The method according to any one of the preceding claims, wherein the method further comprises: - Control the vehicle’s determined functions based on the identified functional parameters (9).

8. The method according to any one of the preceding claims, wherein, The functional parameters include text, which is identified using text recognition.

9. A method for training a machine learning model for a terminal device (2) used to identify movement in a vehicle, wherein, The method includes the following: - Provide training data, wherein the training data includes image data containing at least one view of the terminal device (2), and associate the image data with the functions (9) of the vehicle; - Identify the content (22) displayed on the terminal device (2), wherein the identification of the content (22) includes the following: - The presentation form (24) of the displayed content (22) is detected by analyzing the image data; - Based on the detected presentation format (24), the machine learning model is used to determine the vehicle's functions (9) associated with the displayed content (22); and - Identify the functional parameters in the displayed content (22).

10. The method according to claim 9, wherein, The training data defines a portion (23) of the displayed content (22), in which the functional parameters can be identified.

11. The method according to claim 9 or 10, wherein, The training data includes image data of different views of the terminal device (2), which are associated with the same function (9) having the same functional parameters.

12. A system for data processing, the system having at least one processor configured to perform the method according to any one of claims 1 to 8.

13. A computer program having instructions that, when executed on a system according to claim 12, cause the system to perform the method according to any one of claims 1 to 8.

Images