Distributed machine-learning model for compressing and anonymizing transmitted vehicle data
A distributed machine learning model with an encoder and decoder compresses and anonymizes vehicle sensor data, addressing inefficiencies in existing methods by primarily removing personal information, thus reducing data transmission and enhancing privacy and latency.
Patent Information
- Application Number
- PCT/EP2024/059591
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-04-09
- Publication Date
- 2024-11-21
AI Technical Summary
Existing methods for transmitting vehicle sensor data are inefficient in reducing bandwidth and latency while ensuring privacy, as deterministic compression methods transmit irrelevant data and fail to guarantee retention of relevant information, and require additional complex anonymization steps.
A distributed machine learning model with a vehicle-side encoder and receiver-side decoder is used to irreversibly compress and anonymize sensor data, primarily removing personal information to reduce data transmission and ensure privacy, utilizing a pre-trained model with artificial neural networks for efficient dimension reduction and data protection.
This approach reduces data transmission to essential information, enhances privacy protection, and lowers latency, allowing higher frame rates for image data transmission while ensuring that personal information cannot be reconstructed without knowledge of the encoder architecture.
Smart Images

Figure EP2024059591_21112024_PF_FP_ABST
Abstract
Description
[0001] DISTRIBUTED MACHINE LEARNING MODEL FOR COMPRESSION AND ANONYMISATION OF TRANSMITTED VEHICLE DATA
[0002] The invention relates to a system for transmitting data from a vehicle to a receiver, as well as a method for training a distributed machine learning model for transmitting data from a vehicle to a receiver by means of such a system.
[0003] The following information does not refer to any specific prior art document, but is based on practical considerations:
[0004] The development of automated vehicles requires large amounts of data, for example, to train and validate artificial intelligence models. To provide adequate training data, real data can be recorded, particularly by storing vehicle sensor data from real-life operation in public transport. Furthermore, systems such as those for controlling vehicles by remote drivers (also called "teleoperation" or "remote driving") use data streams from the vehicle being controlled to the remote driver to transmit sensor data, such as from camera images, radar devices, or ultrasonic sensors. To minimize the required bandwidth and reduce latency during data transmission for such purposes, special, deterministic, but also lossy, compression methods are typically used.
[0005] While the use of deterministic compression methods reduces the data stream, this also necessitates a largely direct reconstruction of sensor data at the receiving location. This results in sensor data irrelevant to the specific driving situation, for example, for performing a teleoperation. Furthermore, it cannot be guaranteed that all relevant data will be retained during compression. A deterministic compression method is typically so static that the entire data stream is transmitted to reconstruct sensor data, even if relevant information is completely missing. In this case, significantly more resources than necessary are required.
[0006] Another problem with the transmission of sensor data using deterministic compression methods is that additional, complex anonymization methods must be used to enable privacy-protecting data storage.
[0007] The object of the invention is to transmit large amounts of sensor data about the vehicle's surroundings and / or data about vehicle-internal conditions, determined in connection with the operation of a vehicle, to a receiver more efficiently while maintaining requirements for anonymization of the data.
[0008] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.
[0009] A first aspect of the invention relates to a system for transmitting data from a vehicle to a receiver, comprising an interface for providing original data of the vehicle, a vehicle-side computing unit, a vehicle-side communication module, and the receiver located remotely from the vehicle, wherein an encoder is implemented in the computing unit as the coding-side part of a pre-trained distributed machine learning model, and the computing unit is designed to irreversibly reduce the original data of the interface by means of the encoder by personal information from the environment of the vehicle in such a way that, as a result of the encoder, a compressed representation of the data is obtained that is primarily lossy with regard to the personal information.and wirelessly transmit the compressed representation via the communication module to the receiver having an implemented decoder as the decoding-side part of the pre-trained distributed machine learning model, wherein the receiver is configured to decode the compressed representation by means of the decoder.
[0010] Original vehicle data is data that has not yet been cleansed of personal data. This does not exclude the possibility that the original data may already be the result of a processing step such as a Kalman filter or edited video images; however, it has not yet been processed by the machine learning model. This data particularly includes data about the vehicle's surroundings that was determined by a corresponding sensor unit. For example, camera images, information from an ultrasonic distance sensor, a lidar unit, a radar unit, etc. can sensorically detect the vehicle's surroundings and provide descriptive information. In a further embodiment, the data can also include vehicle-internal data and states, such as a pedal position or a steering angle.
[0011] By applying the system according to the invention, a solution is provided that reduces the data to be transmitted between the vehicle and the remote receiver for processing there to essential information, while preserving the privacy of information in the originally recorded data about persons, so that no persons can be identified even when reconstructing the transmitted data at the decoder.
[0012] The core of the system is a lossy data compression algorithm, which is trained using machine learning to specifically lose private data that needs to be protected, while simultaneously anonymizing it during encoding. For this purpose, a vehicle-side encoder and a receiver-side decoder are trained with the goal of dimensionality reduction and, after training, are deployed in the system.
[0013] The machine learning model preferably comprises at least a first artificial neural network as an encoder and a second artificial neural network as a decoder. Both the encoder and the decoder can each comprise multiple artificial neural networks to perform specific tasks, such as processing various types of sensor data on the encoder side and generating various types of decoded information on the decoder side, in particular for different purposes, for example, one for teleoperation of the vehicle and another for generating information from a scenario library.
[0014] Due to the stochastic structure of neural networks, the decoded data does not contain any actual images of people. However, the model has been trained in a previous training phase to specifically transform the submitted data into an internal state with significantly lower degrees of freedom than the input data – this leads to data compression. Personal information is specifically removed, which, on the one hand, promotes data protection and, on the other hand, advantageously reduces the amount of data to be transmitted, precisely at points that the recipient would expect to be irrelevant anyway, such as for teleoperation, i.e., remote control of the vehicle.
[0015] The fact that a lossy compressed representation of the data is obtained as a result of the encoder, wherein the loss and the concomitant compression of the data occur predominantly with respect to the personal information in the data, means that such a pre-trained distributed machine learning model is used, in particular such an encoder is used that has been deliberately trained to remove personal information from the original data. This does not necessarily mean in every case that no other information at all can be removed from the data; in contrast, in a preferred embodiment, only the personal information is removed from the data, and all other data remains in the representation.
[0016] Decoding is required to use the data. This decoding is performed by the decoding part of the model. The encoder and decoder together form a distributed machine learning model (in spatial distribution).
[0017] The computing unit, including its implemented encoder, as well as the communication module and the interface, are permanently located on the vehicle to ensure maximum protection of the privacy of persons and objects belonging to persons. This means that before each data transmission, the personal information is removed from the data by the communication module, and thus cannot be affected by eavesdropping attacks on data transmission paths between the vehicle and the receiver.
[0018] Further advantages include the smaller data volumes that must be transmitted and an expected lower latency, which allows for a higher frame rate at the receiver, especially when transmitting image data as environmental data. The privacy protection of individuals is also significantly improved, particularly through the algorithmic representation of the data, which cannot be reconstructed without knowledge of the encoder architecture. By splitting the encoder / decoder architecture, original data such as sensor data containing sensitive information remains in the vehicle, and only the encoded data is forwarded. By executing the encoder processing model already in the vehicle, anonymization no longer needs to be performed at the receiver.
[0019] According to an advantageous embodiment, the machine learning model comprises an autoencoder structure with at least one artificial neural network.
[0020] An autoencoder is a type of artificial neural network typically used to efficiently encode unlabeled data. The autoencoder has two functions: an encoder function that transforms the input data and a decoder function that reconstructs the input data in its original form from the encoded representation. The goal of the autoencoder is to generate an efficient representation of the input data, typically for dimensionality reduction.
[0021] According to a further advantageous embodiment, the encoder comprises an artificial neural network whose output layer is smaller than the input layer.
[0022] The size of the output layer can be determined empirically during the training process. A smaller output layer can increase the privacy-preserving effect and data compression. At the same time, however, it is to be expected that beyond a certain reduction in the output layer, essential information can no longer be reconstructed.
[0023] According to a further advantageous embodiment, the data comprises sensor data from a plurality of sensor units of the vehicle, and wherein the encoder comprises a plurality of artificial neural networks, wherein the computing unit is designed to supply a data stream of a respective sensor unit to a respective one of the artificial neural networks.
[0024] The encoder can therefore handle sensor data of different formats, since different sensor units and bus data (camera, ultrasound, CAN bus vehicle communication) typically have different formats. This is made possible by using a structure of the respective artificial neural network that matches the respective data, e.g., n-dimensional convolutional neurons for the camera, an attention network for vehicle communication data, and a fully connected input for continuous signals.
[0025] According to a further advantageous embodiment, the data comprise sensor data from a plurality of sensor units of the vehicle, wherein the computing unit is designed to sequentially supply the sensor data of a plurality of sensor units to the artificial neural network.
[0026] As an alternative to the previous embodiment, this embodiment can also be used to transmit sequences of data (e.g., the course of the radar signal over a time interval). According to a further advantageous embodiment, the computing unit is designed to further compress the compressed representation generated by the encoder with a second, lossless compression stage before transmission by the communication module.
[0027] Accordingly, the decoder is advantageously also designed to decompress the data compressed by the second lossless compression stage in a lossless manner in order to be able to generate processed data formats.
[0028] According to a further advantageous embodiment, the receiver is a stationary control center, and in particular one of the following: data center, teleoperation center, dispatcher.
[0029] According to a further advantageous embodiment, the control center has a plurality of decoders, each of which is designed to decode into a respective domain-specific format. To decode the representation, multiple decoders can be used, each of which translates into a domain-specific format, e.g., according to the well-known "OpenScenario" format, for a simulation environment, into sensor data, or other formats.
[0030] Another aspect of the invention relates to a method for training a distributed machine learning model for transmitting data of a vehicle to a receiver, wherein the distributed machine learning model comprises an encoder and a decoder, comprising the steps:
[0031] - Providing original vehicle data as predefined input data for the machine learning model;
[0032] - Initial training of the machine learning model so that the output data of the decoder is as similar as possible to the input data of the encoder of the machine learning model; and
[0033] - Performing reinforcement learning of at least parameters of the encoder, wherein the personal information in the output data of the decoder is recognized using a helper model for recognizing personal information, and the value of a reward function is increased with increasing absence of personal information in the output data of the decoder.
[0034] According to a further advantageous embodiment, a decoder with an artificial neural network is used, wherein further parameters of the decoder are trained by fixing the parameters of the encoder trained by means of reinforcement learning and adapting the parameters of the decoder.
[0035] Advantages and preferred developments of the proposed method result from an analogous and analogous transfer of the statements made above in connection with the proposed system.
[0036] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.
[0037] They show:
[0038] Fig. 1: A method for creating a system for transmitting data from a vehicle to a receiver according to an embodiment of the invention.
[0039] Fig. 2: A system for transmitting data from a vehicle to a receiver with multiple decoders according to an embodiment of the invention.
[0040] Fig. 3: A system for transmitting various sensor data of a vehicle to a receiver according to an embodiment of the invention.
[0041] Fig. 1 shows a method for training a distributed machine learning model for transmitting data from a vehicle to a receiver 11. The goal is to maintain a distributed machine learning model in which an encoder 7 is executed in a vehicle and prepares vehicle data for transmission to a decoder 9 so that the decoder 9 can reconstruct the original data minus losses entered in the encoder 7. In a first step of the method, original vehicle data S1 is provided as predetermined input data for the machine learning model. This original data includes, in particular, sensor data, but can also include vehicle-internal data from a CAN bus, such as an accelerator pedal position or a steering angle. The sensor data is real data that was determined during a real test drive.First, an initial training S2 of the machine learning model is carried out analogously to an autoencoder based on sensor data collected in the vehicle. This means that the model is trained so that the specified input data is output as accurately as possible in the output layer. During this phase, certain sensor inputs can be excluded in the input layer after the sensor data has been received, e.g., all related sensor data values can be set to zero, whereby the measured data deactivated in the input layer is still used in the output layer to calculate the loss function and to adjust the weights. Finally, reinforcement learning is carried out S3 to increase confidence in the privacy-preserving properties of the system. Depending on the situation, one or more reinforcement learning approaches are used for further training.One or more helper models are used for facial recognition, license plate recognition, and other personal data that requires protection. The helper models are used for identification in such a way that identification features are searched for in the data of the output layer. The sensor-specific helper models used, each of which is designed as a neural network as part of an encoder network, can also consist of pre-trained neural networks, e.g. a feature layer of a convolutional neural network used for object recognition, which processes camera data. If no identification features are found, the system is rewarded. If features are found, the system is penalized. To further train the decoder, the encoder weights are fixed. This means they do not change during further training. Only the decoder, which is another neural network, is further trained.For this purpose, domain-specific data must be used together with sample sensor data from the input layer. This can also include manual labeling of sensor data, whereby only the encoded data is reconstructed and labeled to protect privacy. The model trained in this way is capable of reconstructing sensor data. This data could therefore be displayed directly on the output devices of a teleoperator (for so-called "remote driving"). For privacy-preserving data processing (no storage of anonymized data), a first instance of the system with the complete communication path: vehicle with sensor unit and encoder, receiver with decoder, can already be implemented for training purposes. This allows the system to process the vehicle's sensor data in quasi-real time in a silent testing mode without having to permanently store it.
[0042] Fig. 2 shows a system for transmitting data from a vehicle to a receiver 11. An interface 1 in the vehicle provides original sensor data from sensor units of the vehicle to a computing unit 3 of the vehicle. The computing unit 3 has an implemented encoder 7 as the coding-side part of a pre-trained distributed machine learning model, which was trained, for example, according to the method from Fig. 1. The computing unit 3 uses the original data of the interface 1 by means of the encoder 7, which comprises an artificial neural network, to irreversibly reduce personal information from the vehicle's surroundings in such a way that the result of the encoder 7 is a representation R of the data that is compressed with loss, primarily with reference to personal information.This compressed representation R is then wirelessly transmitted via the communication module 5 to the receiver 11, which implements a decoder set with the individual encoders 9a and 9b as the decoding part of the pre-trained distributed machine learning model. The decoders 9a and 9b decode the encoder's data representation R into their respective domain-specific language, e.g., for OpenScenario.
[0043] Fig. 3 shows another exemplary system analogous to that of Fig. 2, but with only one decoder output of a decoder 9. From the interface 1, in subunits 1a and 1b, respective sensor data from a plurality of sensor units of the vehicle are transferred to the computing unit 3. The encoder 7 implemented there has a plurality of artificial neural networks 7a and 7b, e.g. one consisting of: n-dimensional convolutional neurons for the camera, an attention network for vehicle communication data, a fully connected input for continuous signals.
[0044] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms. With knowledge of the disclosed inventive concept, the person skilled in the art can make various changes, for example, with regard to the function or arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. List of reference symbols.
[0045] 1 interface
[0046] 3 Computing unit
[0047] 5 Communication module
[0048] 7 encoders
[0049] 9 decoders
[0050] 11 recipients
[0051] 51 Provision
[0052] 52 Training
[0053] 53 Implementation
Claims
Patent claims 1. A system for transmitting data from a vehicle to a receiver (11), comprising an interface (1) for providing original vehicle data, a vehicle-side computing unit (3), a vehicle-side communication module (5), and the receiver (11) located remotely from the vehicle, wherein an encoder (7) is implemented in the computing unit (3) as the coding-side part of a pre-trained distributed machine learning model, and the computing unit (3) is designed to irreversibly reduce the original data of the interface (1) by means of the encoder (7) to remove personal information from the environment of the vehicle in such a way that, as a result of the encoder (7), a representation of the data is obtained that is compressed with loss, primarily with reference to personal information.and wirelessly transmitting the compressed representation via the communication module (5) to the receiver (11) with an implemented decoder (9) as the decoding-side part of the pre-trained distributed machine learning model, wherein the receiver (11) is designed to decode the compressed representation by means of the decoder (9) or to store it in encoded form.
2. The system of claim 1, wherein the machine learning model comprises an autoencoder structure with at least one artificial neural network.
3. System according to claim 2, wherein the encoder (7) comprises an artificial neural network whose output layer is smaller than the input layer.
4. System according to claim 3, wherein the data comprises sensor data from a plurality of sensor units of the vehicle, and wherein the encoder (7) comprises a plurality of artificial neural networks, wherein the computing unit (3) is designed to supply a data stream of a respective sensor unit to a respective one of the artificial neural networks.
5. System according to claim 3, wherein the data comprises sensor data from a plurality of sensor units of the vehicle, wherein the computing unit (3) is designed to transmit to the artificial neural network to sequentially feed the sensor data from several sensor units.
6. System according to one of the preceding claims, wherein the computing unit (3) is designed to further compress the compressed representation generated by the encoder (7) with a second, lossless compression stage before transmission by the communication module (5).
7. System according to one of the preceding claims, wherein the receiver (11) is a stationary control center, and in particular one of the following: data center, teleoperation center, dispatcher.
8. System according to one of the preceding claims, wherein the receiver (11) comprises a plurality of decoders (9), each of which is designed to decode into a respective domain-specific format.
9. A method for training a distributed machine learning model for transmitting data of a vehicle to a receiver (11), wherein the distributed machine learning model comprises an encoder (7) and a decoder (9), comprising the steps: - Providing (S1) original vehicle data as predefined input data for the machine learning model; - Initial training (S2) of the machine learning model such that the output data of the decoder (9) are as similar as possible to the input data of the encoder (7) of the machine learning model; and - Carrying out (S3) reinforcement learning of at least parameters of the encoder (7), wherein the personal information in the output data of the decoder (9) is recognized using a helper model for recognizing personal information, and the value of a reward function is increased with increasing absence of personal information in the output data of the decoder (9).
10. The method according to claim 9, wherein a decoder (9) with an artificial neural network is used, wherein parameters of the decoder (9) are further trained by fixing the parameters of the encoder (7) trained by means of reinforcement learning and adapting the parameters of the decoder (9).
Citation Information
Patent Citations
Computer-implemented method and computer program for processing raw data from environmental sensors of a driving system or a traffic infrastructure element.
DE102021202072A1
