Computer-implemented method for training an artificial neural network and computer-implemented method for operating a motor vehicle using the trained artificial neural network.

Abstract

The invention relates to a computer-implemented method for training an artificial neural network, which has an energy function and is trained to predict a road surface friction coefficient from image data of a road surface, comprising the steps: a) Providing a training environment that includes a database with - a first group of training datasets comprising a variety of image data labeled with measured road surface friction values, providing visual information about road surface characteristics of different road surfaces and road types, - a second group of training datasets comprising a variety of acoustic information obtained through speech input about road surface characteristics of different road surfaces and road types, as well as - a third group of training datasets, which includes a variety of tactile information about road surface characteristics of different road surfaces and road types, b) Training the artificial neural network with the training data sets of the first group as input variables and with the training data sets of the second and third groups as latent variables until the energy function, which represents a deviation between the measured road surface friction values ​​and the predicted road surface friction values, is minimized.

Description

[0001] The present invention relates to a computer-implemented method for training an artificial neural network, which has an energy function and is trained to predict a road surface friction coefficient from image data of a road surface. Furthermore, the invention relates to a computer-implemented method for operating a motor vehicle using the trained artificial neural network.

[0002] Computer-implemented methods for operating a motor vehicle that utilize artificial intelligence algorithms to calculate wheel slip are already known in various embodiments. Examples include US 2023 / 0106644 A1 and DE 11 2021 000 146 T5. Calculating the wheel slip of a motor vehicle depends, in particular, on determining the coefficient of friction of the road surface on which the vehicle is traveling. Determining the coefficient of friction is often very imprecise.

[0003] German patent DE 10 2021 131 606 A1 discloses a method for training an AI system to determine road surface conditions, whereby, for example, optical sensor data in a vehicle is acquired and labeled by a vehicle trailer equipped with a friction coefficient sensor. Furthermore, it is provided that trigger events for road friction coefficients are manually determined by a driver and incorporated into the labeling.

[0004] DE 10 2012 112 725 A1 discloses a method for estimating the coefficient of friction, wherein a camera coefficient of friction is made plausible by means of a wheel coefficient of friction.

[0005] From DE 10 2018 008 788 A1 a method for determining road surface roughness is known, wherein the texture and roughness of the road surface are determined from a camera image using a neural network, also taking into account the weather conditions, and the coefficient of friction is subsequently determined from this.

[0006] DE 10 2019 211 052 A1 discloses a method for estimating the coefficient of friction using a neural network, wherein input data includes both a friction-specific sensor type and a vehicle dynamics sensor type.

[0007] DE 10 2020 214 620 A1 describes a method for determining the coefficient of friction using a trained neural network, wherein the input data sets are acquired from a first and a second sensor arrangement, the weighting error of which is minimized by means of a corresponding training.

[0008] The invention is based on the objective of providing an improved computer-implemented method for training an artificial neural network, which has an energy function and is trained to predict a road surface friction coefficient from image data of a road surface. The present invention is further based on the objective of providing a computer-implemented method for operating a motor vehicle using the trained artificial neural network, which enables a very accurate estimation of the road surface friction coefficient during the vehicle's operation and further improves the associated control interventions of a vehicle stability control logic.

[0009] These problems are solved by a computer-implemented method for training an artificial neural network with the features of claim 1 and by a computer-implemented method for operating a motor vehicle using a trained artificial neural network with the features of claim 7. The dependent claims relate to advantageous embodiments of the invention.

[0010] A computer-implemented method according to the invention for training an artificial neural network, which has an energy function and is trained to predict a road surface friction coefficient from image data of a road surface, comprises the steps: a) Providing a training environment that includes a database with - a first group of training datasets comprising a variety of image data labeled with measured road surface friction values, providing visual information about road surface characteristics of different road surfaces and road types, - a second group of training datasets comprising a variety of acoustic information obtained through speech input about road surface characteristics of different road surfaces and road types, as well as - a third group of training datasets, which includes a variety of tactile information about road surface characteristics of different road surfaces and road types, b) Training the artificial neural network with the training data sets of the first group as input variables and with the training data sets of the second and third groups as latent variables until the energy function, which represents a deviation between the measured road surface friction values ​​and the predicted road surface friction values, is minimized.

[0011] The method according to the invention advantageously enables efficient and precise training of the artificial neural network, by means of which the corresponding road surface friction values ​​can be predicted in production operation from image data acquired that contains information about the road surface condition.

[0012] In one embodiment, it is proposed that the road surface friction coefficients used for labeling the image data of the first group of training datasets are determined by wheel slip and rolling resistance measurements in test vehicles.

[0013] In one embodiment, a user's voice input, along with acoustic information about road surface conditions, can be captured and recorded by a microphone while the vehicle is driving on different road surfaces and road types. Preferably, artificial intelligence can be used to determine the similarity between the user's voice input and the actual road surface conditions. For this purpose, a Large Language Model (LLM), in particular GPT 3 or 4, can be used. Other algorithms could include, for example, Bert or Gemini. The similarities can each be represented, for example, by a numerical value of the metrics for lexical and contextual similarity recognition. Examples of these are the dot product, cosine similarity, and Jaccard similarity.

[0014] In one embodiment, it is provided that the tactile information about road surface conditions of different road surfaces and road types is captured by means of a biomimetic sensor.

[0015] In order to enable further improved training, one embodiment proposes that additional latent variables used during the training of the artificial neural network include a tire diameter and / or a tire pressure and / or a vehicle mass and / or an ambient temperature and / or a road surface temperature.

[0016] A computer-implemented method according to the invention for operating a motor vehicle comprises the following steps: - Receiving image data recorded by at least one camera device, in particular a front camera device, of the motor vehicle, wherein the image data contains information about a road surface on which the motor vehicle is moving, - Predictions of a road surface friction coefficient using an artificial neural network trained according to one of claims 1 to 6, which receives and processes the image data as input variables, - Calculating wheel slip for each of the vehicle's wheels using the predicted road surface friction coefficient by means of a vehicle stability control logic and, if necessary, generating a kinematic control intervention for at least one of the wheels based on the wheel slip calculated for each of the vehicle's wheels.

[0017] The computer-implemented method according to the invention for operating a motor vehicle using the artificial neural network trained in the manner described above enables a very accurate estimation of the road surface friction coefficient during the journey of the motor vehicle and can thereby further improve the associated control interventions of a vehicle stability control logic of the motor vehicle.

[0018] In one embodiment, sensor data from multiple sensors, which detect kinetic and / or kinematic measurements, are used to calculate wheel slip. The sensor data used for calculating wheel slip includes, in particular, the rotational speed of the wheels and the vehicle speed. From this information, the algorithm can calculate a theoretically possible wheel speed at which the vehicle can move without uncontrolled wheel spin or slippage. The wheel slip can be predicted from the difference between the theoretically possible wheel speed and the actual wheel speed, allowing the vehicle stability control logic to make appropriate kinematic control interventions at the wheels if necessary.

[0019] In one embodiment, it is possible to use latent variables in the production operation of the artificial neural network such as tire diameter and / or tire pressure and / or vehicle mass and / or ambient temperature and / or road surface temperature and / or user voice input that subjectively describes the road surface properties.

[0020] Further features and advantages of the present invention will become clear from the following description of preferred embodiments with reference to the accompanying figures. These figures show Fig. 1 a schematic view illustrating details of a system for carrying out a procedure to determine wheel slip of a motor vehicle, Fig. 2 A schematic view showing details of a training environment for training an artificial neural network, which is implemented in a system according to Fig. 1 is used.

[0021] With reference to Fig. A system 1 for carrying out a computer-implemented method for operating a motor vehicle 100 comprises at least one vehicle-side camera device 2, in particular a front camera device, which is designed to continuously record the condition of a road surface on which the motor vehicle 100 is traveling. The image data 3 obtained in this process are fed into a trained artificial neural network 4 in which an energy function 40 is implemented. This artificial neural network 4 is trained to predict a road surface friction coefficient 5 from the measured image data 3. The road surface friction coefficient 5 depends on the road surface condition. For example, dry asphalt has a higher road surface friction coefficient 5 compared to wet asphalt or an icy road surface.

[0022] Furthermore, one or more latent variables z can be provided to the artificial neural network 4 and used by it in predicting the road surface friction coefficient 5. Latent variables can include, in particular, a tire diameter and / or a tire pressure and / or a vehicle mass 100 and / or an ambient temperature and / or a road surface temperature and / or voice input from a user that subjectively describes the road surface properties.

[0023] Details of the corresponding training of the artificial neural network 4 are explained in more detail below.

[0024] The road surface friction coefficient 5 predicted by the trained artificial neural network 4 is subsequently fed as an input to a vehicle stability control logic 6. The vehicle stability control logic 6 is configured to estimate the wheel slip for each of the wheels of the motor vehicle 100 from the predicted road surface friction coefficient 5 and sensor data from a plurality of vehicle sensors. The sensor data used for calculating the wheel slip include, in particular, the rotational speed of the wheels and the vehicle speed. Preferably, the algorithm 60 implemented in the vehicle stability control logic 6 for calculating the wheel slip also incorporates information about the wheel diameter and tire properties, such as the tread pattern, tread depth, and rubber compound.

[0025] From this information, algorithm 60 can calculate a theoretically possible wheel speed at which the vehicle 100 can move without uncontrolled wheel spin or slippage. The wheel slip can be predicted from the difference between the theoretically possible wheel speed and the actual wheel speed, allowing the vehicle stability control logic 6 to make appropriate kinematic control interventions at the wheels if necessary.

[0026] Before being deployed in production, the artificial neural network 4 is tested in a Fig. 2 training environment 7 shown, trained on training data sets which are stored in a database 8 and include visual, acoustic and tactile inputs which represent road surface friction values ​​5'.

[0027] Database 8 contains an initial group of training datasets with a large number of images 3' of different road surfaces and road types under varying ambient light conditions, labeled with their corresponding road surface friction coefficients 5'. These road surface friction coefficients 5' can be determined very precisely, for example, by wheel slip and rolling resistance measurements in test vehicles. Each image 3' is assigned its corresponding road surface friction coefficient 5' in text form and thus labeled.

[0028] Database 8 contains a second group of training datasets comprising speech input data from at least one user of vehicle 100. This data includes subjective descriptions of the properties of different road surfaces and road types in human language. These speech inputs are recorded using a microphone, either worn by the user on a wearable device such as a smartwatch or integrated into the vehicle, and processed using a natural language model. Speech recognition algorithms such as Bert, Gemini, GPT 3, GPT 4, or similar algorithms can be used.

[0029] The speech input processed by the language model can be trained on the similarity of an artificial intelligence (AI) to a target specification. For this purpose, an AI based on reinforcement learning can be used, for example. The AI ​​can then be trained in a self-learning process by using similarity as a reward. During training, a target specification describing the road condition (for example: "The road is in good condition.") and subjective statements from the user regarding the road condition are used. The higher the similarity, the greater the reward. Similarity can, for example, be a numerical value from the metrics for lexical and contextual similarity recognition. Examples of these are the dot product, cosine similarity, and Jaccard similarity.

[0030] Furthermore, database 8 contains a third group of training datasets, comprising a collection of tactile signals representing the sliding friction between a finger and the road surface, used to identify a tire's grip characteristics. Engineers can typically gain a good understanding of the material of a particular road surface by sliding their finger across it (the so-called "stick-slip strategy"). To obtain the training datasets for this third group, the user slides their finger, which may be equipped with a biomimetic sensor, across various road surfaces. The resulting tactile signals are captured, recorded using a corresponding algorithm, and classified (i.e., assigned a road surface friction value) before being stored as the third group of training datasets in database 8.

[0031] For the training of the artificial neural network 4, which is generative and has the energy function 40, a training architecture is used that can, for example, be a so-called Latent Variable Energy-Based Model (LVEBM). This model is trained using the labeled image data 3 from the first group of training datasets, which show the condition of the road surface, and latent variables z. The latent variables z can include, in particular, the subjective descriptions of the road surfaces in human language, which form the second group of training data, as well as the tactile signals from the biomimetic sensor, which form the third group of training data. Other latent variables z can be, for example, the ambient temperature, the road surface temperature, the tire diameter, the tire pressure, or the vehicle mass.

[0032] The training objective of the artificial neural network 4 is to minimize the deviation 9 between the measured road surface friction coefficients 5', with which the image data 3' are labelled, and the road surface friction coefficients 5 predicted by the artificial neural network 4. In terms of the energy function 40 of the artificial neural network 4, this means that the energy is low when there is a high degree of similarity and high when there is a low degree of similarity. The training of the artificial neural network 4 is carried out until the energy function 40, which represents the deviation 9 between the measured road surface friction coefficients 5' and the predicted road surface friction coefficients 5, is minimized.

Claims

[1] Computer-implemented method for training an artificial neural network (4) which has an energy function (40) and is trained to predict a road surface friction coefficient (5) from image data (3) of a road surface, comprising the steps: a) Providing a training environment (7) that includes a database (8) with - a first group of training datasets comprising a variety of image data (3') labelled with measured road surface friction values ​​(5') with visual information about road surface properties of different road surfaces and road types, - a second group of training datasets comprising a variety of acoustic information obtained through speech input about road surface characteristics of different road surfaces and road types, as well as - a third group of training datasets, which includes a variety of tactile information about road surface characteristics of different road surfaces and road types, b) Training the artificial neural network (4) with the training data sets of the first group as input variables and with the training data sets of the second and third groups as latent variables until the energy function (40), which represents a deviation (9) between the measured road surface friction values ​​(5') and the predicted road surface friction values ​​(5), is minimized. [2] Computer-implemented method according to claim 1, characterized by , that the road surface friction coefficients (5') used for labeling the image data (3') of the first group of training datasets are determined by wheel slip and rolling resistance measurements in test vehicles. [3] Computer-implemented method according to one of claims 1 or 2, characterized by , that the voice inputs of a user with acoustic information about road conditions during journeys of the motor vehicle (100) on different road surfaces and road types are captured and recorded by means of a microphone. [4] Computer-implemented method according to claim 3, characterized by , that artificial intelligence is used to determine similarities between the user's voice input and the actual road surface conditions. [5] Computer-implemented method according to any one of claims 1 to 4, characterized by , that tactile information about road surface conditions of different road surfaces and road types is recorded using a biomimetic sensor device. [6] Computer-implemented method according to any one of claims 1 to 5, characterized by, that further latent variables used during the training of the artificial neural network (4) include a tire diameter and / or a tire pressure and / or a vehicle mass of the motor vehicle (100) and / or an ambient temperature and / or a road surface temperature. [7] Computer-implemented method for operating a motor vehicle (100), comprising the steps: - Receiving image data (3) recorded by at least one camera device (2), in particular a front camera device, of the motor vehicle (100), wherein the image data (3) contain information about a road surface on which the motor vehicle (100) is moving, - Predictions of a road surface friction coefficient (5) using an artificial neural network (4) which has been trained according to one of claims 1 to 6 and which receives and processes the image data (3) as input variables, - Calculating a wheel slip for each of the wheels of the motor vehicle (100) using the predicted road surface friction coefficient (5) by means of a vehicle stability control logic (6) and, if necessary, generating a kinematic control intervention for at least one of the wheels based on the wheel slip calculated for each of the wheels of the motor vehicle (100). [8] Computer-implemented method according to claim 7, characterized by , that sensor data from a plurality of sensor devices that detect kinetic and / or kinematic measurements are used to calculate wheel slip. [9] Computer-implemented method according to one of claims 7 or 8, characterized by, that as latent variables in the production operation of the artificial neural network (4) a tire diameter and / or a tire pressure and / or a vehicle mass of the motor vehicle (100) and / or an ambient temperature and / or a road surface temperature and / or voice inputs of a user which subjectively describe the road surface properties are used.

Images