Computer-implemented method for data processing of a vehicle system

Abstract

The invention relates to a computer-implemented method for data processing of a vehicle system (22) of a vehicle (10), comprising the steps: - Capture of a voice input (S10) by a microphone (26) coupled to or integrated into the vehicle system (22); - Conversion of voice input into a text message (S12) by the vehicle system (22); - Transferring the text message (S14) to a computing unit (12); - Generating a response using a machine learning algorithm executed on the computing unit (12), comprising: - Extracting a task list (S16) from the text message comprising at least one task, and - Obtaining information and / or generating commands (S18) for the vehicle system (22) according to the tasks in the task list; - Transmitting the response (S20) to the vehicle system (22); and - Execution of the commands contained in the response by the vehicle system (22) and / or generation of a speech output (S22, 40) from the response to read out the information contained in the response.

Description

[0001] The invention relates to the field of data processing by a vehicle system and in particular to vehicle communication with network and cloud services.

[0002] Voice assistants are a familiar feature in modern vehicles. The driver can press a button or say a key phrase to control a vehicle's onboard computer. The onboard computer can be prompted to make a call or operate the navigation system through voice input.

[0003] However, speech recognition is tied to specific keywords and is not able to process longer input or recognize more complex tasks.

[0004] Modern machine learning algorithms, especially those trained with large language models, are capable of processing large texts and, through clever algorithms, assigning more or less weight to individual words in an input. This allows terms to be placed in context, enabling context-based text processing.

[0005] However, the input for such algorithms usually requires a complex interface with which the user must interact. Such systems are therefore unsuitable for operation while driving, especially by a driver. The driver, in particular, should be paying attention to the traffic and steering the vehicle. Furthermore, large language models require a significant amount of computing power, which a vehicle's onboard computer cannot usually provide.

[0006] Against this background, the object of the invention is to propose a method for a vehicle system that can react to complex inputs and provide an appropriate response to the driver.

[0007] The problem is solved by the subject matter of the independent claims. Disclosure of the invention

[0008] According to a first aspect of the invention, this object is achieved by a computer-implemented method for data processing of a vehicle system of a vehicle, comprising the steps: - Capturing a voice input through a microphone coupled to or integrated into the vehicle system; - Conversion of the voice input into a text message by the vehicle system; - transmitting the text message to a computing unit, wherein the computing unit is communicatively connected to the vehicle system via a network; - Generating a response using a machine learning algorithm executed on the computing unit, comprising: - Determining a task list from the text message comprising at least one task, and - Obtaining information and / or generating commands for the vehicle system according to the tasks in the task list; - Transmitting the response to the vehicle system; and - Execution of the commands contained in the response by the vehicle system and / or generation of a voice output from the response to read out the information contained in the response.

[0009] The vehicle system can, in particular, be the vehicle's on-board computer. Alternatively or additionally, the vehicle system can also include a device connected to the vehicle, in particular a smartphone or a retrofitted separate computing device.

[0010] Modern operating systems can be equipped with all kinds of software, so that a laptop, a tablet or a smartwatch, for example, can also be considered as vehicle systems or parts thereof.

[0011] The coupling of such a peripheral device with the on-board computer can, for example, be a wired coupling, in particular via a USB cable, or a wireless connection, in particular a radio or Bluetooth connection.

[0012] The vehicle system includes at least one microphone with which it can record speech spoken by the driver or other vehicle occupants as an acoustic signal.

[0013] In principle, the invention can be operated by the driver of a vehicle. However, other occupants may also be present in the vehicle who can also use the method. Mechanisms are possible that allow only the driver to execute the method, for example, voice recognition or a button accessible only to the driver. For simplicity, the driver is always referred to as the user in this invention. Unless explicitly excluded, the driver always includes other possible vehicle occupants.

[0014] The acoustic signal captured by the microphone is analyzed by a speech recognition algorithm and converted into a text message. The speech recognition algorithm is preferably executed locally by the vehicle system. Transmitting a text message requires less data than transmitting a voice message.

[0015] The text message thus generated is transmitted to a processing unit that is communicatively connected to the vehicle system via a network. The communicative connection is designed to allow the network's communication participants to exchange data with each other.

[0016] Advantageously, the computing unit connected to the vehicle via the network can be equipped with more computing power than, for example, the on-board computer or associated peripherals. The computing unit can, for example, be a server structure that offers a cloud service and can be contacted by the vehicle system.

[0017] A machine learning algorithm is executed on the processing unit. Currently, running machine learning algorithms for text recognition requires a significant amount of computing power, which is usually not available to a vehicle's onboard computer. So that the driver can still use the machine learning algorithm to process their text message, the text message is transmitted to the processing unit.

[0018] The machine learning algorithm is configured to analyze and evaluate the text message upon receipt. The evaluation involves analyzing the text message for one or more possible tasks. A task can, for example, include providing specific information or generating a specific command for the vehicle system and / or the vehicle. A text message can thus lead to one or more tasks, which are recorded in a task list.

[0019] The processing unit can then process the tasks with or without assistance and thus generate a response. The machine learning algorithm is particularly needed for analyzing the text message. The algorithm is not necessarily required for solving the tasks, although it can be used if it allows a task to be solved more efficiently.

[0020] The response is transmitted from the processing unit to the vehicle system, where it is processed. Depending on what the response contains, the vehicle system can process it. A pure text message as a response can be read aloud to the driver by a module that generates a voice output. If the response contains one or more commands to the vehicle system, these can be implemented by the vehicle system.

[0021] By transmitting the text message to the computing system and processing it using a machine learning algorithm, the driver of a vehicle can use more complex voice inputs to generate commands for the vehicle and / or the vehicle system. This achieves the object of the invention.

[0022] In one embodiment, the text message and the response are transmitted over a wireless network between the vehicle system and the computing unit.

[0023] The radio network can, in particular, be a mobile radio network. Mobile radio devices are widely used devices that can generally be easily coupled to a vehicle or its onboard computer. For the purposes of this invention, the system comprising the onboard computer and the mobile radio-capable device is referred to as the vehicle system.

[0024] The use of radio networks to transmit the text message from the vehicle system to the processing unit and the response from the processing unit to the vehicle system has the advantageous effect that even vehicles without their own radio equipment can easily be retrofitted or temporarily equipped with appropriate devices in order to process complex voice inputs into a response.

[0025] In one embodiment, the machine learning algorithm is a neural network trained with a large language model.

[0026] A machine learning algorithm is an algorithm designed to automatically detect patterns and relationships in data and make predictions or decisions. It is created by training on existing data and can then be applied to new, unknown data to generate predictions or classifications.

[0027] A machine learning algorithm can take various forms, such as linear models, decision trees, support vector machines, neural networks, and many others. It is optimized by learning from the training data, identifying patterns and rules to make the best possible predictions or classifications for new data.

[0028] The effectiveness of a machine learning algorithm depends on several factors, including the quality and quantity of training data, the choice of algorithm, the model configuration, and the evaluation of the model using evaluation metrics. The model is continuously improved and optimized to maximize accuracy and performance.

[0029] A large language model is a term that refers to language models that use massive amounts of text data to learn the ability to generate natural language. These models are commonly used in machine text processing and can perform various tasks such as machine translation, text generation, text classification, question-answering systems, and more.

[0030] Large language models are specialized models of neural networks. The architecture of a neural network comprises multiple nodes, neurons, or nodes arranged in layers. Each node receives an input and generates an output value from it. Input values ​​can be the input or parts of it, but also output values ​​from nodes in previous layers.

[0031] Due to their structure, neural networks, especially when trained with a large language model, are particularly well suited for text processing.

[0032] In another embodiment, the Large Language Model includes a transformer architecture. Transformer architectures, also referred to simply as transformers, are specific architectures for neural networks based on the mechanism of attention and used to process sequences, such as text. They have proven extremely powerful in machine text processing.

[0033] The main idea behind the Transformer is that it relies entirely on the attention mechanism to model the relationships between words in a text. Traditional sequential models such as LSTM or RNN struggle to efficiently capture long dependencies. The Transformer, on the other hand, can directly access the entire input text and model relationships between words in parallel, significantly improving training speed and model performance.

[0034] The Transformer architecture consists of a stack of identical layers. Each layer is divided into two main components: the multi-head self-attention mechanism and the position-wise feedforward network.

[0035] The multi-head self-attention mechanism is the heart of the Transformer. Here, the input text is converted into a query, a key, and a value. For each word, an attention weight is calculated, which indicates how relevant the other words in the text are to that word. Each word can then receive its representation through a weighted combination of the values ​​of the other words. This process generates word embedding vectors.

[0036] The attention weights are calculated multiple times in so-called "heads" to generate different representations of the attention context. The results are then concatenated and recombined using a linear transformation.

[0037] The position-wise feedforward network comprises two linear transformations applied independently to each word. It is a non-sequential part of the architecture because each vector is processed separately.

[0038] Since a transformer architecture has no built-in information about the word order, positional information is added through position encodings. These position encodings are added to the word embedding vectors to account for the influence of positions.

[0039] The transformer architecture can be divided into an encoder and a decoder. The encoder processes the input sequence, and the decoder generates the output sequence. For some tasks, such as machine text processing, multiple transformer layers can be used for the encoder and decoder to increase model performance.

[0040] Overall, the Transformer architecture advantageously enables highly parallel and efficient text processing, leading to increased performance in text processing tasks. By using attention mechanisms, the model can specifically access relevant information and recognize complex dependencies within text messages.

[0041] In one embodiment, at least one vehicle parameter, in particular the vehicle speed, the current fuel level or the current battery charge level, the current position, the current destination of a navigation system belonging to the vehicle system, the route selected in the navigation system, the vehicle type, the currently tuned radio station, metadata of the currently playing audio file, or another parameter of the vehicle and / or the vehicle system, is sent with the text message. The response is then generated using the at least one vehicle parameter.

[0042] The at least one vehicle parameter can provide context for the text message transmitted to the processing unit. If the task list includes tasks that require the at least one vehicle parameter to complete, it is available to the machine learning algorithm upon input. This increases the relevance of the response.

[0043] For example, a text message could be: “I am hungry and would like to eat something in the next hour.” The machine learning algorithm could determine the tasks from the text message: 1. Find a restaurant near the navigation system route. 2. Check if the restaurant can be reached within an hour. 3. Check if the restaurant is open in the next hour. 4. Create a route with a stop at the restaurant.

[0044] For these tasks, the vehicle's navigation data, in particular the currently followed route, is required.

[0045] Furthermore, the computing unit can be configured to take the vehicle battery charge level or fuel level into account when processing text messages regarding the route. In such an embodiment, the vehicle battery charge level or fuel level could be used as a parameter to find a restaurant with a charging or refueling station, allowing the vehicle to charge while the driver dines in the restaurant.

[0046] Advantageously, the inclusion of vehicle parameters increases the relevance of the response without requiring a separate request to the vehicle and / or the driver to generate the response.

[0047] In one embodiment, the at least one vehicle parameter can include an identification number used for billing the computing unit's service. This allows the operator of the computing unit to determine the number and / or volume of text messages transmitted or the responses generated thereto. This, in turn, can be used for accounting purposes, in particular for invoicing, or for assessing the correct scaling of the computing unit's resources.

[0048] In one embodiment, the at least one vehicle parameter is encoded according to a protocol for communication with the machine learning algorithm.

[0049] Coding at least one vehicle parameter has the advantageous effect of increasing the relevance of the response. There is always a risk, albeit small, that text recognition may not work properly. This applies to both text processing by the machine learning algorithm and speech recognition.

[0050] For example, the protocol may provide for at least one vehicle parameter to be identified as the expected vehicle parameter for the machine learning algorithm. If the algorithm expects a specific vehicle parameter within a data packet received as input data, it can use the vehicle parameter directly without having to extract corresponding values ​​from the text message. This minimizes potential sources of error for an erroneous response.

[0051] In one embodiment, the vehicle system comprises a smart device connected to the vehicle, wherein the smart device is configured to access a computing unit of the vehicle and read the at least one vehicle parameter. In advantageous embodiments, the connection of the smart device to the vehicle must be verified to prevent misuse of the vehicle parameters.

[0052] Smart devices can typically be connected to other devices, especially cloud services, via a wireless network. Using a smart device can simplify communication between the vehicle system and the computing unit. For example, the configuration can be set up independently of the journey and the vehicle, so the driver only needs to connect their smart device to the vehicle to operate it while driving. This improves operating convenience.

[0053] In one embodiment, the response includes a request for the vehicle system to execute the commands, and the commands are executed only after the request is confirmed.

[0054] Preferably, the request includes a description of the commands to be executed.

[0055] The query prevents the vehicle system from implementing commands that result in an incorrect or irrelevant response. An incorrect response can arise, for example, from a misinterpretation during speech recognition or the evaluation of the text message. Furthermore, the response may be based on incorrect information.

[0056] The driver can prevent the execution of incorrect or irrelevant commands by objecting to the request.

[0057] In one embodiment, the request may include a request to the vehicle system to verify authentication. In particular, the vehicle system may have exchanged certificates with the computing unit to mutually authenticate each other. This can increase the security of the text message transmission and the response.

[0058] In one embodiment, the commands of the response comprise at least one of: changing the destination and / or route of a navigation system belonging to the vehicle system, playing a defined audio file or audio stream with a media system belonging to the vehicle system, establishing a communicative connection to a contact defined in the response via a cellular network.

[0059] The commands contained in the response can adjust various settings in the vehicle system. In particular, settings related to navigation and the media system can be adjusted.

[0060] For example, the computing unit can issue a command to modify or create a new route that takes into account the driver's specific requests, tune into a media stream or radio station the driver has requested, or provide contact information for a person or institution with which the vehicle system then connects. A request for the latter case could be, in particular, "Connect me to a hotel or restaurant near my current route."

[0061] In a further embodiment, the detection of the voice input is initiated by a defined voice sequence or the pressing of a key.

[0062] Manually initiating the voice input capture advantageously prevents the voice recognition system from inadvertently listening in on potentially intimate driver data and transmitting it to the processing unit. Manually initiating the voice input capture protects the driver's data and potentially saves data volume, as no unwanted text messages are transmitted to the processing unit.

[0063] In one aspect, the invention relates to a computer program with program code for carrying out a method for data processing of a vehicle system as described above when the computer program is executed on a computer.

[0064] In a further aspect, the invention relates to a computer-readable data carrier with program code of a computer program for carrying out a method for data processing of a vehicle system as described above when the computer program is executed on a computer.

[0065] In a further aspect, the invention relates to a vehicle system for carrying out a method for data processing, the vehicle system comprising: - means for capturing speech input; - means for converting a voice input into a text message; - means for transmitting the text message and at least one vehicle parameter via a network; - means for receiving a response; - means for generating a speech output from the response; and - Means for executing the commands contained in the response.

[0066] The means for detecting the speech input can in particular be a microphone integrated into the vehicle system or coupled to the vehicle system.

[0067] The means for converting a voice input into a text message can be a text or speech recognition module.

[0068] The means for transmitting the text message and receiving the response may comprise a cellular interface.

[0069] The means for generating a speech output may comprise a module for generating an audio signal and a loudspeaker.

[0070] Preferably, the computer-implemented steps are carried out by an on-board computer of the vehicle.

[0071] The described designs and further training courses can be combined as desired.

[0072] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned. Short description of the drawings

[0073] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0074] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements illustrated in the drawings are not necessarily drawn to scale.

[0075] They show: Fig. 1 schematically shows the communication between a vehicle and a computing unit; Fig. 2 schematically shows the components of a vehicle system and a computing unit; and Fig. 3 schematically shows the sequence of a method according to one embodiment.

[0076] In the figures of the drawings, the same reference symbols designate the same or functionally equivalent elements, parts or components, unless otherwise stated.

[0077] Fig. 1 shows the interaction of a vehicle system from a vehicle 10 with a computing unit 12.

[0078] The vehicle system is preferably integrated into the vehicle 10. For example, the vehicle system can be an on-board computer of the vehicle 10. The vehicle system is configured to communicate via a network 14. The network 14 can preferably be a radio network comprising a plurality of radio masts 16. The communication participants in the network 14, including the vehicle 10 and the computing unit 12, are communicatively connected to one another via the radio masts 16.

[0079] Computing unit 12 can preferably offer a cloud service that processes the text messages originating from vehicle 10. Furthermore, computing unit 12 can be coupled to a second network 18. Computing unit 12 can access knowledge and / or information via second network 18. The knowledge can be provided, for example, by other computing units and databases (neither shown). The second network 18 can, in particular, be the Internet.

[0080] In an alternative embodiment, the vehicle system of the vehicle 10 can communicate with the computing unit 12 via the Internet.

[0081] Fig. 2 schematically shows the communication and interaction of a driver 20 with the vehicle system 22 and the computing unit 12 according to one embodiment.

[0082] The driver 20 formulates a voice input 24, which is received by a microphone 26 and converted into an audio signal. The microphone 26 transmits the audio signal to a module 28, which converts the audio signal into a text message.

[0083] The vehicle system 22 comprises its own computing unit, in particular an on-board computer, which can access a subsystem 30. From the subsystem 30, the vehicle system 22 reads at least one vehicle parameter, which is transmitted with the text message to the computing unit 12. In this embodiment, the vehicle system 22 does not perform any analysis of the text message itself, which is why possible vehicle parameters are transmitted in a lump sum with the text message. The transmission is in particular a transmission via a wireless network 14, which is preferably a radio network.

[0084] In embodiments, the vehicle system 22 may include a module that examines the text message and checks it for keywords. The vehicle parameters transmitted with the text message can then be selected based on recognized keywords, so that the computing unit receives fewer vehicle parameters. This reduces the amount of transmitted data and improves the security of the vehicle data if the computing unit 12 only receives vehicle data according to the "need-to-know" principle.

[0085] The computing unit 12 processes the incoming text message and the at least one vehicle parameter with a machine learning algorithm, which is designed in particular as a neural network 32 with a transformer architecture.

[0086] The neural network 32 can interpret the incoming text message and break it down into tasks, which are compiled into a task list. A task can include obtaining information or providing or generating commands for the vehicle system 22.

[0087] In a module 34, a response is prepared as a message, which is then transmitted to the vehicle system 22 via the network 14. The module 34 can be operated with its own machine learning algorithm. Furthermore, the module 34 can also be operated without a machine learning algorithm. In alternative embodiments, the function of the module 34 is assumed by the neural network 32, so that no independent module 34 is necessary. For the invention, it is only important that a response is transmitted from the computing unit 12 to the vehicle system 22, which response includes the most relevant information possible for the driver 20 and / or commands for the vehicle system 22.

[0088] The vehicle system 22 receives the response from the computing unit 12 and, if necessary, breaks it down into a part for the driver 20 and a part for the vehicle system 22. The part for the driver 20 is translated into an audio signal by a module 36. The audio signal is played by a loudspeaker 38. The audio signal can, for example, be a read-aloud text message 40 containing requested information or an answer to a specific question.

[0089] The loudspeaker 38 can, for example, be a loudspeaker of the media system of the vehicle or a loudspeaker system coupled to the vehicle system 22, in particular a headset or a hands-free system.

[0090] The part of the response intended for vehicle system 22 is transmitted to subsystem 30 and processed there. Subsystem 30 may, for example, be a navigation system that calculates a new route or modifies an existing route in response to the commands. Furthermore, subsystem 30 may be a media system that processes, for example, commands to tune to a specific radio station or to play an audio stream or audio file.

[0091] Fig. 3 shows schematically the sequence of the method according to an embodiment.

[0092] In step S10, driver 10 utters a statement that is captured by vehicle system 22 as voice input. The voice input is converted into a text message in step S12. In the subsequent step S14, the text message is transmitted from vehicle system 22 to computing unit 12. In embodiments, at least one vehicle parameter can be added to the text message, which is also transmitted to computing unit 12.

[0093] The computing unit 12 receives the text message and, if applicable, the at least one vehicle parameter, and uses it to determine a task list with at least one task in step S16. For this purpose, the text message is interpreted, which is preferably performed by a neural network with a transformer architecture operating on the computing unit 12.

[0094] In step S18, the computing unit 12 compiles information and / or commands into a response. To this end, the tasks in the task list are processed. Furthermore, the response may also include metadata, such as the date, the cost, or a security token. The response is transmitted from the computing unit 12 to the vehicle system 22 in step S20.

[0095] The transmissions in steps S14 and S20 preferably take place via a mobile network and / or the Internet.

[0096] The vehicle system 22 processes the response in step S22 by executing the commands contained therein and converting and playing a text message addressed to the driver 20 into a voice output.

Claims

[1] Computer-implemented method for data processing of a vehicle system (22) of a vehicle (10), comprising the steps: - detecting a voice input (S10) by a microphone (26) coupled to the vehicle system (22) or integrated into the vehicle system (22); - converting the voice input into a text message (S12) by the vehicle system (22); - transmitting the text message (S14) to a computing unit (12), wherein the computing unit (12) is communicatively connected to the vehicle system (22) via a network (14); - generating a response using a machine learning algorithm executed on the computing unit (12) comprising: - determining a task list (S16) from the text message comprising at least one task, and - Determining information and / or generating commands (S18) for the vehicle system (22) according to the tasks in the task list; - transmitting the response (S20) to the vehicle system (22); and - Execution of the commands contained in the response by the vehicle system (22) and / or generation of a voice output (S22, 40) from the response for reading out the information contained in the response. [2] The computer-implemented method of claim 1, wherein the text message and the response are transmitted over a radio network between the vehicle system (22) and the computing unit (12). [3] A computer-implemented method according to any one of the preceding claims, wherein the machine learning algorithm is a neural network (32) trained with a large language model. [4] The computer-implemented method of claim 3, wherein the large language model comprises a transformer architecture. [5] Computer-implemented method according to one of the preceding claims, wherein at least one vehicle parameter, in particular the vehicle speed, the current fuel level or the current battery charge level, the current position, the current destination of a navigation system belonging to the vehicle system (22), the route selected in the navigation system, the vehicle type, the currently tuned radio station, metadata of the currently played audio file, or another parameter of the vehicle and / or the vehicle system is sent along with the text message, and wherein the response is generated using the at least one vehicle parameter. [6] The computer-implemented method of claim 5, wherein the at least one vehicle parameter is encoded according to a protocol for communication with the machine learning algorithm. [7] Computer-implemented method according to one of claims 5 or 6, wherein the vehicle system (22) comprises a smart device connected to the vehicle (10), wherein the smart device is configured to access a computing unit of the vehicle (10) and to read out the at least one vehicle parameter. [8] A computer-implemented method according to any one of the preceding claims, wherein the response comprises a request for the vehicle system (22) to execute the commands, and the commands are executed only after confirmation of the request. [9] A computer-implemented method according to any one of the preceding claims, wherein the commands of the response comprise at least one of: Changing the destination and / or route of a navigation system belonging to the vehicle system (22), playing a defined audio file or an audio stream with a media system belonging to the vehicle system (22), establishing a communicative connection to a contact defined in the response via a mobile network. [10] Computer-implemented method according to one of the preceding claims, wherein the detection of the speech input (S10) is initiated by a defined speech sequence or the actuation of a key. [11] Computer program with program code for carrying out a method for data processing of a vehicle system (22) according to one of the preceding claims when the computer program is executed on a computer. [12] Computer-readable data carrier with program code of a computer program for carrying out a method for data processing of a vehicle system (22) according to one of claims 1 to 10 when the computer program is executed on a computer. [13] Vehicle system for carrying out a method for data processing, the vehicle system (22) comprising: - means for detecting a speech input (S10); - means for converting a voice input into a text message (S12); - means for transmitting the text message (S14) and at least one vehicle parameter via a network (14); - means for receiving a response; - means for generating a speech output (S22) from the response; and - Means for executing the commands contained in the response.

Images