Method for operating an input interface of a vehicle, computer-readable medium, system, and vehicle

Abstract

The invention relates to a method for operating an input interface of a vehicle, the method comprising the steps of: detecting an object by means of an interior camera system of the vehicle; extracting text information relating to the detected object by means of the interior camera system of the vehicle; transmitting the extracted text information relating to the detected object by means of the interior camera system of the vehicle to a pre-trained machine learning method for extracting one or more entities from the extracted text information; extracting the one or more entities from the extracted text information by means of the pre-trained machine learning method; and controlling a vehicle function depending on the one or more extracted entities by means of the vehicle.

Description

[0001] 24-2243 1

[0002] Method for operating a vehicle input interface, computer-readable medium, system, and vehicle

[0003] The invention relates to a method for operating a vehicle input interface. The invention further relates to a computer-readable medium for operating a vehicle input interface, a system for operating a vehicle input interface, and a vehicle comprising the system for operating a vehicle input interface.

[0004] Modern vehicles often feature various input interfaces that allow users to enter information. For example, a user can manually enter data via a touchscreen or transmit information to the vehicle using a voice input interface. Manually entering data via a touchscreen can often be very time-consuming and / or prone to errors. The voice input interface can also be susceptible to errors depending on the user, country, or language. Therefore, transmitting information via the vehicle's voice input interface may require time-consuming, multiple attempts to input the information correctly before the vehicle has correctly recorded it.

[0005] It is therefore an object of the invention to operate a vehicle's input interface more efficiently. In particular, an object of the invention is to efficiently improve the acquisition of data for a vehicle function via a vehicle's input interface.

[0006] This problem is solved by the features of the independent claims. Advantageous embodiments and further developments of the invention result from the dependent claims.

[0007] According to a first aspect, the invention is characterized by a method for operating a vehicle's input interface. The method can be a computer-implemented method and / or a control unit-implemented method. The method comprises detecting an object using an interior camera system of the vehicle, extracting text information from the detected object using the 24-2243 2

[0008] The vehicle's interior camera system. The method further comprises transmitting the extracted text information of the captured object by the vehicle's interior camera system to a pre-trained machine learning algorithm to extract one or more entities from the extracted text information, extracting the one or more entities from the extracted text information by the pre-trained machine learning algorithm, and controlling a vehicle function depending on the one or more extracted entities by the vehicle.

[0009] Advantageously, the input interface can more easily capture text information as input data for vehicle functions. This allows vehicle functions to be controlled more efficiently by the vehicle user.

[0010] According to an advantageous embodiment, the object detection by the vehicle's interior camera system can comprise the object detection by an IR camera of the vehicle's interior camera system, wherein preferably the object's coordinates are captured in an image from a frame stream of the IR camera of the vehicle's interior camera system. This allows the input interface to detect the object more efficiently.

[0011] According to a further advantageous embodiment, the object in the hand of a vehicle occupant can be detected by the vehicle's interior camera system, and / or the object in the occupant's hand can be optically oriented towards the vehicle's interior camera system. This allows the input interface to be triggered efficiently.

[0012] According to a further advantageous embodiment, extracting text information from the detected object by the vehicle's interior camera system can comprise recognizing the object detected by the interior camera system's IR camera in an image from a frame stream of an RGB camera of the vehicle's interior camera system, and extracting all text information from the detected object using an optical character recognition (OCR) method based on an image from the RGB camera's frame stream. This allows the input interface to extract the text information more efficiently from the image data. 24-2243 3

[0013] According to a further advantageous embodiment, the transmission of the extracted text information of the detected object by the vehicle's interior camera system to a pre-trained machine learning algorithm for extracting one or more entities from the extracted text information can include transmitting a prompt comprising a predefined entity type and the extracted text information of the detected object to the pre-trained machine learning algorithm for extracting the one or more entities from the extracted text information. This allows one or more entities to be extracted efficiently in a context-dependent manner.

[0014] According to a further advantageous embodiment, the extraction of one or more entities from the extracted text information by the pre-trained machine learning method can comprise extracting one or more entities of the specified entity type from the extracted text information. This allows one or more entities to be extracted efficiently in a context-dependent manner.

[0015] According to a further advantageous embodiment, the pre-trained machine learning method can be a pre-trained machine learning method of the vehicle. If the vehicle's pre-trained machine learning method is unable to extract an entity from the extracted text information, it can transmit the extracted text information of the detected object via the vehicle's interior camera system to a pre-trained machine learning method on an external server for the purpose of extracting one or more entities from the extracted text information. The pre-trained machine learning method on the external server then extracts these one or more entities from the extracted text information. This efficiently improves the extraction of one or more entities.

[0016] According to a further advantageous embodiment, controlling a vehicle function based on one or more extracted entities can include determining the vehicle function based on the specified entity type and controlling the determined vehicle function based on the one or more extracted entities. This allows for more efficient control of a vehicle function. 24-2243 4

[0017] According to a further advantageous embodiment, the specified entity type can be an address, the vehicle function can be a navigation function of the vehicle, the one or more entities can be one or more addresses extracted from the extracted text information, and the one or more addresses can be used as input data for the navigation function. This allows for more efficient control of the vehicle's navigation function.

[0018] According to a further advantageous embodiment, the specified entity type can be a music track, and the vehicle function can be an audio output function of the vehicle. Furthermore, the one or more entities can be one or more music tracks extracted from the extracted text information, and these one or more music tracks can be used as input data for the audio output function. This allows for more efficient control of the vehicle's audio output function.

[0019] According to another aspect, the invention is characterized by a computer-readable medium for operating an input interface of a vehicle, wherein the input interface comprises instructions which, when executed on a computer or a control unit of the vehicle, execute the method described above.

[0020] According to another aspect, the invention is characterized by a system for operating an input interface of a vehicle, wherein the system is configured to perform the method described above.

[0021] According to another aspect, the invention is characterized by a vehicle comprising the system described above for operating an input interface of a vehicle.

[0022] Further features of the invention will become apparent from the claims, the figures, and the description of the figures. All features and combinations of features mentioned above in the description, as well as those subsequently mentioned in the description of the figures and / or shown in the figures alone, are usable not only in the combinations specified, but also in other combinations or individually. 24-2243 5

[0023] The following describes a preferred embodiment of the invention with reference to the accompanying drawings. Further details, preferred configurations, and further developments of the invention will be derived from these drawings. Specifically, the drawings schematically illustrate...

[0024] Fig. 1 shows an exemplary method for operating a vehicle input interface, and

[0025] Fig. 2 shows an exemplary system for operating a vehicle input interface.

[0026] Figure 1 shows in detail an exemplary method 100 for operating a vehicle input interface. The input interface can be a camera-based interface that uses an interior camera system of the vehicle as an input interface for one or more vehicle functions. For example, the interior camera system can optically capture and / or recognize the content of a display on a mobile device belonging to a user of the vehicle. The mobile device can be a smartphone, a smartwatch, or another smart device that has a display. The content of the display can include one or more pieces of text information.

[0027] Method 100 can detect an object using an interior camera system of the vehicle 102. The object can be a mobile device, a mobile device display, or a physical medium, such as a paper business card. Furthermore, the interior camera system can extract text information from the detected object using the interior camera system of the vehicle 104. The interior camera system can, for example, be integrated into a rearview mirror in the interior of the vehicle. The text information can be, for example, a chat history from an app, text from a website, or text from a document that can be extracted by the interior camera system.

[0028] Method 100 can transmit the extracted text information of the detected object by the vehicle's interior camera system to a pre-trained machine learning algorithm for extracting one or more entities from the extracted text information. 106 The pre-trained machine learning algorithm can be a large language model capable of processing text information in a variety of languages. Preferably, the pre-trained machine learning algorithm is executed on a control unit of the vehicle. Additionally or alternatively, the pre-trained machine learning algorithm can be executed on an external server. The transmission process can include sending a prompt to the pre-trained machine learning algorithm. The prompt can, for example, include a question asking whether an entity of a given entity type is present in the text information.A predefined entity type could be, for example, an address or a song title. The predefined entity type can define a context for extracting one or more entities. For each predefined entity type, a vehicle function is defined that can process one or more entities of that entity type as input data. An entity can be a uniquely identifiable, single piece of information. For example, the entity could be an address. For example, the entity could be a song title.

[0029] Method 100 can extract one or more entities from the extracted text information using the pre-trained machine learning algorithm. For example, Method 100 can extract one or more entities of a specified entity type from the extracted text information using the pre-trained machine learning algorithm. The entity type can be specified by the prompt to the pre-trained machine learning algorithm. The method can submit multiple prompts for a variety of specified entity types to the specified machine learning algorithm, with each prompt encompassing a specified entity type. Furthermore, for each prompt with a specified entity type, the method can extract one or more entities from the text information, provided that one or more entities of the specified entity type are present in the text information.

[0030] Finally, the method 100 can control a vehicle function depending on one or more extracted entities. Method 110 passes one or more extracted entities from the pre-trained machine learning method to a vehicle function as input data. The vehicle function can then be executed using these extracted entities. For example, if an extracted entity is an address and the vehicle function is a navigation function, the navigation function can use the address as the destination address for navigation. Additionally, the navigation function can generate one or more navigation routes to the destination address and / or automatically start navigation to the destination address.Additionally or alternatively, the address of the extracted entity can be adapted by the vehicle or an external server so that the vehicle function can use the address as the destination address for navigation. Additionally or alternatively, procedure 100 can modify the vehicle function in 24-2243 7.

[0031] Determine the dependency of the given entity type and control the determined vehicle function through the vehicle depending on the one or more extracted entities.

[0032] Figure 2 shows in detail an exemplary system 200 for operating an input interface of a vehicle 202. The vehicle 202 can include an interior camera system 204. The interior camera system 204 can capture an object 206 in the hand 208 of a user 210 of the vehicle 202. For example, the object 206 can be the display of a mobile device. The interior camera system 204 can extract text information 212 from the object 206. Furthermore, the interior camera system 204 can transmit the text information 212 to a pre-trained machine learning algorithm of the vehicle 202 for extracting one or more entities from the extracted text information. The pre-trained machine learning algorithm of the vehicle 202 can extract the one or more entities from the extracted text information. The vehicle 202 can control a vehicle function depending on the one or more extracted entities.For example, the vehicle function can display the extracted one or more entities on a display 214. If the vehicle function is a navigation function of the vehicle 202, it can display one or more addresses of the one or more entities and / or one or more navigation routes 216 to the respective addresses on the display 214. The vehicle function can be started automatically or depending on an operator interaction 210 of the vehicle 202.

[0033] Advantageously, the method 100 and / or the system 200 can perform inputs to one or more vehicle functions via the input interface by holding text information in front of an interior camera system 204 of a vehicle 202. Time-consuming manual inputs and / or recognition problems with voice assistants can be efficiently avoided. The convenience of entering complex text information, such as song titles or addresses, in the vehicle can be significantly increased. 24-2243 8

[0034] Reference symbol list

[0035] 100 procedures

[0036] 102 Capturing an object

[0037] 104 Extracting Text Information

[0038] 106 Transmitting the extracted text information

[0039] 108 Extracting one or more entities

[0040] 110 Controlling a vehicle function

[0041] 200 System

[0042] 202 vehicles

[0043] 204 Interior camera system

[0044] 206 objects

[0045] 208 Hand

[0046] 210 users

[0047] 212 Text information

[0048] 214 Display

[0049] 216 Navigation route

Claims

24-2243 9 Patent claims 1. Method for operating an input interface of a vehicle, the method comprising: Detection of an object by an interior camera system of the vehicle; Extracting text information from the captured object using the vehicle's interior camera system; Transmitting the extracted text information of the captured object by the vehicle's interior camera system to a pre-trained machine learning algorithm to extract one or more entities from the extracted text information; Extracting one or more entities from the extracted text information using the pre-trained machine learning algorithm; and Control of a vehicle function by the vehicle depending on one or more extracted entities.

2. The method of claim 1, wherein the detection of the object by the vehicle's interior camera system comprises: Detection of the object by an IR camera of the vehicle's interior camera system, wherein preferably coordinates of the object are detected in an image of an image stream of the IR camera of the vehicle's interior camera system.

3. Method according to one of the preceding claims, wherein the object in the hand of an occupant of the vehicle is detected by the vehicle's interior camera system; and / or wherein the object in the hand of the occupant is optically oriented towards the vehicle's interior camera system.

4. Method according to any of the preceding claims, comprising extracting text information from the captured object by the vehicle's interior camera system: Detection of the object captured by the IR camera of the interior camera system in an image from a stream of images from an RGB camera of the vehicle's interior camera system; and 24-2243 10 Extracting all text information of the captured object using an optical character recognition method using an image of the RGB camera image stream from the vehicle's interior camera system.

5. Method according to any of the preceding claims, wherein the transmission of the extracted text information of the detected object by the vehicle's interior camera system to a pre-trained machine learning method comprises extracting one or more entities from the extracted text information: Submitting a prompt comprising a specified entity type and the extracted text information of the captured object to the pre-trained machine learning algorithm to extract one or more entities from the extracted text information.

6. Method according to any of the preceding claims, comprising extracting the one or more entities from the extracted text information by the pre-trained machine learning method: Extracting one or more entities from the extracted text information for the specified entity type using the pre-trained machine learning algorithm.

7. A method according to any of the preceding claims, wherein the pre-trained machine learning method is a pre-trained machine learning method of the vehicle; and If the vehicle's pre-trained machine learning algorithm was unable to extract an entity from the extracted text information: Transmitting the extracted text information of the captured object by the vehicle's interior camera system to a pre-trained machine learning algorithm on an external server to extract one or more entities from the extracted text information; and Extracting one or more entities from the extracted text information using the pre-trained machine learning algorithm of the vehicle-external server.

8. A method according to any of the preceding claims, comprising controlling a vehicle function by the vehicle depending on one or more extracted entities: 24-2243 11 Determining the vehicle function depending on the specified entity type; and controlling the determined vehicle function depending on the one or more extracted entities by the vehicle.

9. A method according to any of the preceding claims, wherein the specified entity type is an address; and wherein the vehicle function is a navigation function of the vehicle; wherein the one or more entities are one or more addresses extracted from the extracted text information; and wherein the one or more addresses are used as input data for the navigation function.

10. A method according to any of the preceding claims, wherein the specified entity type is a music track; and wherein the vehicle function is an audio output function of the vehicle; wherein the one or more entities are one or more music tracks extracted from the extracted text information; and wherein the one or more music tracks are used as input data for the audio output function.

11. Computer-readable medium for operating an input interface of a vehicle, wherein the input interface comprises instructions which, when executed on a computer or a control unit of the vehicle, execute the method according to any one of claims 1 to 10.

12. System for operating an input interface of a vehicle, wherein the system is configured to execute the method according to any one of claims 1 to 10.

13. Vehicle comprising the system for operating an input interface of a vehicle according to claim 12.

Images