COMPUTER-IMPLEMENTED METHOD, COMPUTER PROGRAM, COMPUTER-READABLE DATA CARRIER, DATA CARRIER SIGNAL AND SYSTEM FOR PREVENTING A MODEL THEFT ATTACK ON A SOFTWARE SYSTEM AND CONTROL SYSTEM FOR A DRIVE SYSTEM

Description

[0001] The invention relates to a computer-implemented method, computer program, computer-readable data carrier, data carrier signal and system for preventing a model theft attack on a software system and a control system of a driving system.

[0002] US 9,268,938 B1 discloses a system for integrity assessment and intrusion detection in computerized systems. This system uses a physical side channel to evaluate the correct functioning of the computerized system. The side channel information is information that does not change the operating state of the system and includes, among other things, sensor signals such as temperature, vibration, pressure, time, power consumption, and GPS coordinates.

[0003] Model theft attacks on machine learning models are revealed, for example, in F. Tramèr et al., Stealing Machine Learning Models via Prediction APIs, https: / / ar-xiv.org / pdf / 1609.02943.pdf.

[0004] CN 110 795 703 A discloses a method for preventing model theft attacks.

[0005] US 2019 / 0347410 A1 discloses a method for the fault tolerance of machine learning models against model theft attacks in the context of machine learning asa service.

[0006] US 2019 / 0303567 A1 concerns the detection of anomalies in bit data streams, where the bit data streams are exchanged via a data bus between two or more devices in a network and the anomaly detection is carried out using an artificial neural network.

[0007] US 2020 / 0134391 A1 concerns a method and a data processing system for detecting manipulation of a machine learning model. During a training operation, the machine learning model is provided with several input values. In response to a predetermined invalid input value, the machine learning model is trained to expect a predetermined output value. The model is verified to be unmanipulated by inputting the predetermined invalid input value during an inference operation period. If the expected output value is provided by the machine learning model in response to the predetermined input value, then the machine learning model has not been manipulated. If the expected output value is not provided, then the machine learning model has been manipulated.

[0008] EP 3 370 389 A1 discloses anomaly detection in a cloud network of vehicles. Further prior art is disclosed, for example, in DE 10 2019 115585 A1.

[0009] The goal of spying on and copying other companies' products is widespread and has become a business model in its own right for some companies. An additional risk arises if information about the internal workings of a software system falls into the hands of a malicious attacker who doesn't intend to copy the system but rather to cause it to malfunction. If this succeeds, the software manufacturer may face extortion demands, or the products may be deliberately made to behave erratically, potentially destroying the underlying hardware and damaging the manufacturer's reputation. In the automotive sector, there is also the possibility of causing accidents with personal injury, which can lead to particularly negative publicity and reputational damage.

[0010] The object of the invention was to provide an alternative system for preventing model theft attacks, abbreviated SVMA.

[0011] The subject matter of claims 1, 10 and 13 solves this problem.

[0012] In one aspect, the invention provides a computer-implemented method for preventing a model theft attack on a software system. The software system receives input data and provides output data with which it regulates and / or controls hardware components, including hardware components of a driving system. The method comprises the following steps: Receiving the input data, performing anomaly detection on the input data, including a point-by-point evaluation of individual data points, a contextual evaluation of several data points, a collective evaluation of groups of data points, and / or a time-series evaluation of the data points, whereby anomalies are detected in the case of outliers from the data points, including outliers in the data size of individual data points, outliers in the number of data points, and / or outliers in a sequence of data points, and in the case of a detected anomaly, ensuring information security of the software system, including documentation of the attack, reporting of the attack, changing the functionality of the software system, extracting a watermark, running the software system in emergency mode, and / or deactivating the software system.

[0013] According to another aspect, the invention provides a computer program for preventing a model theft attack on the aforementioned software system. The computer program comprises commands that cause a computer system running the software system to execute the steps of the method according to the invention when the computer program is running on the computer system.

[0014] According to another aspect, the invention provides a control system for a driving system. A software system runs on the control system, which receives input data from sensors of the driving system and provides output data for control units of the control system, including control units for drive control, driver assistance systems, automated driving systems, and active and / or passive safety, thereby preventing model theft attacks. the control system executes a computer program according to the invention, a computer-readable data carrier on which the computer program according to the invention is stored is connected to the control system for data transmission and the computer program according to the invention is executed, or the control system receives a data carrier signal that transmits the computer program according to the invention and executes the computer program according to the invention.

[0015] According to another aspect, the invention provides a system for preventing model theft attacks, the system comprising Input interfaces through which the system receives input data for a control and / or regulation software system, a computing unit configured to execute the inventive method and / or a computer program according to the invention to detect anomalies, wherein the computer program is stored in a memory of the computing unit and / or the system comprises a data interface through which a computer-readable data carrier and / or a data carrier signal according to the invention is received, a memory in which the detected anomalies are stored, and an output interface to provide the detected anomalies.

[0016] Advantageous embodiments of the invention will become apparent from the definitions, dependent claims, drawings and descriptions of preferred embodiments.

[0017] A model theft attack, also known as a model stealing attack, means that an attacker can derive parameters and / or functionality of a software system, such as a machine learning model, by comprehensively analyzing input data, query content (also called a query), output data, and / or other information, such as the software / network type and / or structure of the software system. By cleverly combining different input data and testing how the software system reacts, such attacks gather information about how the software system works and / or how it can be deceived.

[0018] According to one aspect of the invention, altered input data, which an attacker attempts to introduce into the software system in order to subsequently observe the output data, is detected by the anomaly detection system. Altered input data includes disturbances that are introduced into the software system via interfaces, including application programming interfaces (APIs), for example, via APIs of control electronics on an electronic test bench, and advanced driver assistance system interface specifications (ADASIS).

[0019] According to another aspect of the invention, altered input data is obtained through environmental disturbances, particularly when the attacker does not have access to the software system's interfaces, for example, when the input data is encrypted or verified before being processed by the software system. In this case, the attacker can apply interfering signals, for example, in the form of stickers, audio data, and / or laser light, to objects detected by the vehicle system's sensors, such as traffic signs, body parts, the ground, or a rolling road test bench. These disturbances are also detected by the anomaly detection system.

[0020] The system according to the invention for preventing model theft attacks has the advantage that, through anomaly detection in the input data, it makes it more difficult for an attacker to gain information about the software system. This also makes it more difficult to understand and thus copy the functionalities of the software system. Furthermore, the invention makes it more difficult for an attacker to generate damaging disruptions to prevent the correct functionality of the software system during operation. This also thwarts extortion, sabotage, and reputational damage attempts by the attacker.

[0021] A software system is a technical system or subsystem whose functionality is implemented by means of software. Software systems include user interfaces, input and / or output interfaces, and programming interfaces. According to one aspect of the invention, the software system is embedded in the vehicle system for controlling and / or regulating a drive system, including transmission control; automated operation of the vehicle system, including assisted and autonomous operation; and / or active and / or passive safety systems of the vehicle system, including brakes, restraint belts, and airbags. The fundamental concept of the invention is applicable to any software system. The exact implementation of the software system used for control is irrelevant.

[0022] According to one aspect of the invention, the software system is a traditional, controlled system with a controlled, process-oriented decision-making task. According to another aspect of the invention, the software system is a data-driven system that is trained, for example, using deep neural networks and reinforcement learning. Any combination of such systems is also conceivable and is protected by the invention.

[0023] Data-driven systems, especially those based on machine learning, such as deep neural networks, are highly vulnerable to attacks and must be protected. This includes, on the one hand, increasing robustness against deliberately calculated interference signals to ensure continued correct functionality even when the input data is corrupted. On the other hand, the fundamental step is to prevent the systems from being spied on and an attacker / competitor from gaining information that would enable such targeted attacks. The primary objective of the invention is to address the latter point and prevent spying. Should this also be achieved, the invention can also prevent the first point and increase the system's robustness against interference attacks.

[0024] According to the invention, black-box attacks are prevented. A software system or a machine learning model is a black box if the user or attacker only has access to the model or system by querying the input and output data. The black box can also be protected, for example, by encryption and other cybersecurity measures, to such an extent that an attacker cannot gain access to data within the system. Various processing and interpretation steps can be carried out within the protected system, to which the attacker has no access. The attacker's goal is to obtain information about the internal processes, decisions, and / or functions within the software system. This is prevented by the invention through various measures.

[0025] The driving system comprises land vehicles, watercraft, and aircraft. According to one aspect of the invention, the driving system is an automated road vehicle, for example, a car, bus, people mover, or truck.

[0026] Hardware components of the driving system include actuators, for example actuators for drive control, transmission control, a driver assistance system, an automated driving system and / or an active and / or passive safety system.

[0027] The point-by-point analysis of individual data points encompasses univariate anomaly detection, meaning anomaly detection in a single data dimension, and multivariate anomaly detection, meaning anomaly detection in multiple dimensions. Multivariate anomaly detection has the advantage of recognizing even more complex anomalies. For example, when detecting anomalies in children, neither a height of 1.70 m nor a weight of 20 kg would be particularly unusual. Univariate anomaly detection for height and univariate anomaly detection for weight would each fail to identify an anomaly, since a 10-year-old child can weigh 20 kg and a 16-year-old child can be 1.70 m tall. The combination of both data points for a single child is recognized as an anomaly in multivariate anomaly detection.

[0028] Contextual anomaly detection means that data points only become noticeable when viewed within a broader context. For example, the operation of a people mover fluctuates considerably between day and night. A high volume of data, which is typical and therefore normal during daytime operation, could be an indicator of a model theft attack at night. In this example, a security-relevant contextual anomaly exists, which is detected by the invention.

[0029] In a collective anomaly, individual data points are not unusual. Instead, an anomaly only becomes apparent when considering a group of data.

[0030] In addition to univariate anomalies, time series often also contain contextual or collective anomalies. According to one aspect of the invention, future anomalies are detected using time series, as a time series allows for projection into the future within the framework of predictive analytics. For this purpose, statistical methods or machine learning algorithms such as neural networks, in particular long-short-term memory networks, are used, according to another aspect of the invention. According to another aspect of the invention, the predicted values ​​and their confidence intervals are used as a definition of normality to detect anomalies when they occur.

[0031] According to one aspect of the invention, the time series correspond to a sequence of past input data. This improves anomaly detection, as the structure of normal input data can be strongly influenced by current conditions. By using temporal information, anomaly detection can better distinguish whether the current input data has actually been manipulated or whether normal usage has fundamentally changed, for example, due to different driving behavior, terrain, or weather. Consequently, the true positive rate can be improved, and there is less overhead associated with manually reviewing suspicious cases and potentially initiating further legal action.

[0032] Another aspect involves first embedding the raw input data into a condensed information vector. This vector contains as much relevant information from the sequence as possible and is sufficient for analysis. The condensed representation facilitates analysis because it reduces dimensionality and omits unimportant information. Anomaly detection is then performed on the entire embedded input data to assign the observed sequence to the described operating modes. Various techniques are available for generating such an embedded information vector. These range from simple rules, such as formal mapping, to complex techniques, such as the use of long-short-term memory networks. The most suitable method for a given case depends entirely on the structure and type of the data.Consequently, the method used will be selected depending on the specific input data.

[0033] The aforementioned variants are examples of how to detect different operating modes based on the system's input data. The invention is not dependent on these and also works with other methods that can be used for anomaly detection in the input data.

[0034] In the event of a detected anomaly, measures are initiated according to the invention to prevent the attacker from successfully obtaining information about the functionality of the software system used.

[0035] Documenting the attack allows the invention to retrospectively trace which hardware components were targeted and under what circumstances the attack was carried out. This facilitates better substantiation of potential claims for damages and helps identify those responsible for an accident if an attacker deliberately engineers a malfunction that leads to system failure. According to one aspect of the invention, the documentation is stored in a secure and / or encrypted storage area of ​​the software or vehicle system and / or sent to and stored in a cloud storage location.

[0036] By altering the functionality of the software system, the attacker gains no information about the current software system. The attacker is provided with plausible initial data, making it difficult for them to determine the software system's ongoing defenses. Furthermore, any calculated interference signals are rendered harmless to the actual, current software system.

[0037] Another measure is the insertion of a software watermark into the software system as a detection feature. For example, the watermark is inserted into the execution path of the software system and extracted when the software is run. According to one aspect of the invention, for instance, very small sub-functions are performed in a significantly different way than is typically done. This does not arouse suspicion in an attacker, but it can be clearly proven that espionage has occurred if the attacker suddenly begins to exhibit similar behavior in their products.

[0038] Running the software system in emergency mode, such as false safe, and / or disabling the software system is the most radical step to defend against an attacker. A less drastic measure can be taken by preventing the system from restarting. All of these measures could be reversed by a law-abiding user upon request or through a specific process. This process would verify that no attack or user misconduct actually occurred.

[0039] The aforementioned measures for defending against an attack can be implemented individually or in any combination according to the invention. According to one aspect of the invention, at least one point in time at which at least one of the measures becomes active is freely configurable and based on the requirements. The measures mentioned are examples of ways to prevent an attacker from obtaining relevant information about the current state of operation of a software system or from deceiving a software system. The invention does not depend on a specific measure but describes the concept, which is independent of the specific technology used.

[0040] According to another aspect of the invention, anomaly detection is performed either before the input data is fed into the software system or in parallel with the evaluation of the input data by the software system. In the upstream architecture of anomaly detection prior to the input data being fed into the software system, the input data is analyzed before it is processed by the software system. This allows an attack to be detected before the input data is fed into the software system, and countermeasures to be initiated. Performing anomaly detection in parallel with the evaluation of the input data has the advantage that no unacceptable latency occurs.

[0041] Another aspect is used to detect outliers. a probability of generating one or more of the data points from a statistical model adapted to the input data is assessed, each of the data points is evaluated in a context of its environment and / or the similarity to other data points by executing a distance and density algorithm, a given number of groups are formed from a set of similar data points by executing a clustering algorithm, the membership of one of the data points in a class is determined by executing a machine learning algorithm trained on the input data, the data points are mapped to a lower dimensionality, a comparison with stored outliers is performed and / or decision rules are applied.

[0042] The statistical models are appropriately chosen and, if necessary, parameterized.

[0043] If a sufficiently large amount of similar data exists for a data point, the distance and density method evaluates the data point as normal.

[0044] According to one aspect of the invention, the clustering method uses a machine learning model to divide the data into groups. Data points that are far removed from all groups are identified as outliers. Real clustering algorithms are optimized using training data, according to another aspect of the invention. This involves recording input data from normal operation and simulating potential attack or problem scenarios in other input data. This allows the clustering algorithm to be trained to recognize abnormal input data and assign it to different clusters, which are then allocated to the attack scenario or various malfunction scenarios. If, during actual operation, input data is more likely to be assigned to a different cluster, appropriate measures can be taken to protect the software system.Furthermore, these algorithms can be retrained and updated during operation to adapt them to the current normal state. All input data that is highly likely to be assigned to the normal cluster during operation can be used to further improve the clustering algorithm and fine-tune it to the current use case. This reduces the false positive rate and the effort required for manual follow-up checks.

[0045] In class-based methods, a machine learning classifier is trained on the training data to predict the class membership of a data point. Class-based methods typically achieve higher recognition rates because the machine learning classifier can differentiate the input data in greater detail and is not forced to form formally defined clusters.

[0046] Data points that are poorly mapped to a lower dimensionality are considered anomalies.

[0047] When comparing with stored outliers, for example, an internal database of the system for preventing model theft attacks is accessed. This database already contains typical attack patterns, and a comparison with these is sufficient to recognize that the current pattern is such a pattern or a similar one. According to one aspect of the invention, this database is kept constantly up-to-date via a wireless connection.

[0048] Decision rules are used to detect an operational scenario. For example, very precise speeds with little natural variation indicate that the driving system is on a chassis dynamometer and not in normal driving mode. A rapid succession of very different driving conditions can also suggest a systematic attack.

[0049] To prevent the invention from falsely detecting an attack during a necessary test bench trial at a regular service facility, one aspect of the invention involves temporarily disabling the system for preventing model theft attacks via a secure method, in coordination with the system owner / manufacturer. This allows regular maintenance to be performed and the security measures to be reactivated afterward.

[0050] In a further embodiment of the invention, the input data are transferred to a Normal case, in which the input data is identical to the input data that occurs during regular use of the software system; attack case, in which the input data is altered for the model theft attack; problem case, in which the input data and / or output data indicate a malfunction and / or damage to at least one of the hardware components; and fault case, in which the output data is disrupted by the model theft attack. subdivided. In the event of an attack and in the event of a malfunction, the information security of the software system is ensured according to the invention.

[0051] This allows anomaly detection to differentiate which of the aforementioned operating modes reflect the current input data. The division into the aforementioned operating modes is achieved, in particular, by combining a given number of groups from a set of similar data points using a clustering algorithm. For example, the data is clustered into four groups: normal operation, attack operation, problem operation, and fault operation.

[0052] In the event of a problem, the software system is not spied on, so no measures need to be taken to prevent the information from being obtained. Depending on the specific problem, alternative measures are taken based on an aspect of the invention to resolve the issue, such as recommending a workshop visit or recalibration.

[0053] In the event of a malfunction, a disruptive signal intended to deceive or manipulate the software system is simply a change in the input data. Therefore, this deviation can be detected as an anomaly just as in the case of an attack and countered with an appropriate combination of countermeasures.

[0054] According to another aspect of the invention, a nearest-neighbor classification algorithm, a k-means algorithm, a support vector machine algorithm, a long-short-term memory artificial neural network, an autoencoder, or a replicator neural network is executed to detect outliers.

[0055] The nearest neighbor classification algorithm, also called the k-nearest-neighbor algorithm, abbreviated kNN, is an example of the distance-density method. kNN is used for classification and regression.

[0056] The k-means algorithm is a clustering algorithm.

[0057] In an autoencoder, for example, a deep neural network learns clustering in a latent space, while the deep neural network solves a regression problem. In the autoencoder, the input data is typically fed into a hidden layer, which has fewer neurons than the input layer. This allows the network to learn a compression pattern for the input data.

[0058] Replicator neural networks feed input data through a hidden layer with a step-like activation function. This step-like activation function quantizes the output vector of the hidden layer into grid points, thereby ordering the input data into a number of clusters. Replicator neural networks are disclosed in S. Hawkins et al., Outlier Detection Using Replicator Neural Networks, https: / / togaware.com / papers / dawak02.pdf.

[0059] According to another aspect of the invention, changing the functionality of the software system includes switching to an earlier version of the software system, changing the configuration of the software system, and switching between different versions of the software system. The earlier version of the software system differs sufficiently from the current version that the attacker does not gain any valuable information, yet plausible results are still provided.

[0060] In particular, switching to an older version of the software system is combined with inserting and extracting the software watermark. Because this older software version, into which the watermark is inserted, is deliberately chosen to be visible to an attacker, the attack becomes particularly easy to detect. For example, very small sub-functions can be executed in a significantly different way than is typically done.

[0061] When switching between several different versions, it becomes significantly more difficult for the attacker to interpret the acquired data and derive valuable information about the specific functionality of the software. A self-learning attacker system, such as a replacement system trained with similar functionality using input-output pairs, may therefore fail to converge. Ideally, the different software variants reflect several cases that guarantee near-optimal behavior but differ significantly in their output values. In this case, an attacker might become suspicious because they see widely varying output values ​​for very similar input data and suspect that something is currently wrong with the system and that it is not functioning normally.However, since the output values ​​differ significantly and originate from many different software versions, it is difficult or impossible for the attacker to interpret the received data and gain relevant information about the specific functionality of the software system.

[0062] According to another aspect of the invention, abnormal input data, along with metadata including position data, time data, and / or hardware component identification numbers, are stored for documentation purposes. This improves the documentation and traceability of the attack. Position data includes, for example, GPS coordinates.

[0063] According to another aspect of the invention, anomaly detection is performed locally within the driving system or externally, encompassing a remote system or cloud. The detected anomaly is made available to the driving system, a driver or operator of the driving system, and / or the remote system. This provides two variants for the system to prevent model theft attacks. For example, anomaly detection is provided via a human-machine interface.

[0064] According to the invention, the software system comprises a machine learning model, and the theft of this model is prevented. Machine learning models, for example those based on deep neural networks, are particularly vulnerable to attacks. For instance, just a few distorted pixels in an image, such as pixels of different colors, are sufficient to disrupt the inference of an artificial neural network that receives image data as input and is trained on object classification. Similarly, even acoustic signals inaudible to humans are enough to disrupt the inference of an acoustic classifier network. The invention prevents the machine learning model from being spied on and an attacker or competitor from gaining information that would enable targeted attacks.In particular, the invention prevents an attacker from stealing model parameters and / or hyperparameters of an artificial neural network through model theft attacks. Model parameters include weights for neuron connections and activation functions. For example, in an objective function L = I + λR of an artificial neural network, I is the loss function, R is a regularization term, and λ is a hyperparameter. The regularization term prevents overfitting. The hyperparameter λ determines how well a model can generalize. Furthermore, the invention increases the robustness of the machine learning model against deliberately calculated interference signals.

[0065] According to the invention, the method according to the invention can also be used in a software system of a drive control of the driving system comprising a transmission control, a driver assistance system, an automated driving system and / or an active and / or passive safety system.

[0066] For example, the latest transmission has received a software update to improve functionality, such as reduced fuel consumption. A competitor learned of this through advertising campaigns in which the manufacturer used the suddenly lower fuel consumption as a selling point with the new transmission software and now has to adapt its own product to avoid losing market relevance. Since developing its own software would take too long, the new transmission software is to be spied on in order to be able to offer a similar update more quickly.

[0067] In the case of a gearbox whose software is not protected by the inventive system for preventing model theft attacks, a competitor can obtain all relevant information by testing various prototypes on the test bench. The functionality can be replicated from the data obtained, and a corresponding update of their own software can be quickly made available. This technological advantage is short-lived, and it cannot be proven that the competitor copied the software rather than developing it independently based on their own ideas.

[0068] If the control software is instead equipped with the inventive system for preventing model theft attacks, it is impossible for the competitor to copy the functionality. The software system would recognize that test samples are currently being run on the test bench to gather information about the system. Consequently, it would use an outdated version of the software instead of the current one and / or switch between different variants. The competitor cannot gain any useful information and even wastes time trying to spy on the functionality. This further increases the time it takes to provide a similar update. Additionally, development is more expensive because everything has to be researched and implemented independently. Therefore, the financial gain for the original developer of the software update is greater, as their exclusivity lasts longer.

[0069] If the competitor were to decide to simply copy the software, they would be copying an outdated or inferior version, as well as the embedded watermarks. This could later prove that espionage had occurred, potentially leading to legal claims and demands.

[0070] As another example, an attacker aims to disrupt an ADAS system. A stop sign is to be manipulated by affixing a sticker that appears innocuous to the human eye, causing the ADAS system to misinterpret it as a 100 km / h speed limit sign instead of a stop sign. The attacker intends to cause a serious accident through this manipulation. Once this has occurred, a ransom note is sent to the software manufacturer, threatening further attacks unless a large ransom is paid. The ADAS software manufacturer suffers significant reputational damage and substantial financial losses.

[0071] To carry out such an attack, the attacker must first calculate a suitable jamming signal, in this case, the sticker. For this, they need information about the ADAS system to calculate a specific jamming signal. Therefore, they first conduct an information-gathering attack by systematically testing various jamming methods, thereby gaining insights into how the software system reacts to disrupted input signals. For example, the attacker could place a vehicle on a rolling road dynamometer and show the vehicle's ADAS camera many different stop signs with minor modifications until the desired response is observed in the system.

[0072] With an ADAS system protected by the inventive system for preventing model theft attacks, such an attack is not possible. The attacker would not have succeeded in obtaining information about the system's functionality, and therefore such a sticker could not have been generated. No extortion attempt would have occurred, meaning the ADAS software manufacturer would have suffered no damage, neither to its reputation nor financially. Furthermore, the data relating to the attempted attack would be stored in the cloud to obtain information about the ADAS system's functionality. This can be used to file a report with the police and to reject any potential claims for damages. In addition, the detection of similar attacks can be further improved by retraining the attack detection system on the new data.

[0073] The instructions of the computer program according to the invention comprise machine instructions, source code or object code written in assembly language, an object-oriented programming language, for example C++, or in a procedural programming language, for example C. The computer program is executed wholly or partially on the vehicle system or in the remote system comprising a cloud.

[0074] Another aspect of the invention is a computer-readable data carrier for preventing model theft attacks, on which the computer program according to the invention is stored. The computer-readable data carrier is, for example, a random access memory, a read-only memory, a flash memory, a USB stick, an SD card, a CD, a DCD, or a Blu-ray disc. If the driving system is not equipped with the inventive system for preventing model theft attacks upon delivery, it can be retrofitted using the computer-readable data carrier.

[0075] Another aspect of the invention is a data carrier signal for preventing model theft attacks. The data carrier signal transmits the computer program according to the invention. The computer program can be downloaded via the data carrier signal, for example, via a network, such as the internet, or using wireless technology, such as WLAN.

[0076] The control units of the control system according to the invention comprise electronic control units, also called electronic control units.

[0077] The sensors of the driving system include sensors from a transmission control unit, such as pressure sensors; environmental perception sensors, such as cameras, lidar, radar, and microphones; and other driving system sensors, such as accelerometers. The sensor data is then further processed and transformed within the software system, undergoing various steps. Typically, the raw sensor data is first pre-processed to improve the quality of the input data and then analyzed, and the output is generated based on this analysis. According to one aspect of the invention, these last two steps are performed by a controlled system or by a trained system. The software system generates output values ​​that are used to control hardware, such as the transmission control unit, or are output via hardware to be displayed to a user, such as a smart reversing camera.

[0078] Potential attack vectors include the input of sensor data into the software system before or after preprocessing and after the generation of output data. The attack can be carried out by altering the physical environment or by manipulating sensors or sensor data that are not part of the vehicle's operating system.

[0079] The system according to the invention is partially or completely cloud-based.

[0080] The invention is illustrated in the following exemplary embodiments. They show: Fig. 1 a schematic representation of a model theft attack, Fig. 2 an embodiment of a software system which is protected against model theft attacks by the invention, Fig. 3 an example of input data for different operating modes of one of the software systems Fig. 2 , Fig. 4 an embodiment of a system according to the invention for preventing model theft attacks, Fig. 5 a schematic representation of a clustering of input data, Fig. 6 a schematic representation of a change between different software versions and Fig. 7 a schematic representation of the method according to the invention.

[0081] In the figures, identical reference symbols denote identical or functionally similar reference parts. For clarity, only the relevant reference parts are highlighted in each figure.

[0082] The goal of spying on and copying other companies' products is widespread and has become a business model in its own right for some companies. An attacker A with knowledge of the specific software system S is capable of such acts by intentionally and selectively altering the input data E of the software system S. No physical contact with the cables or interfaces is necessary. It is sufficient for the attacker A to apply interfering signals, such as stickers, audio, or laser light, to objects that are used or read by sensors.

[0083] To gain an understanding of owner O's software system S, particularly regarding owner O's machine learning model f, attacker A uses the ability to observe which changes in the output data are caused by a targeted modification of the input data E. By repeatedly showing deliberately manipulated data x1, ..., xn and analyzing the output data f(x1), ..., f(xn), conclusions can be drawn about the functionality of the software system S, and a model theft f*=f can be carried out (see...). Fig. 1 .

[0084] The procedure for an attack that aims to gain information about the software system and deliberately alters the input signals in order to observe the effect on the output data can be differentiated in the following points: Point of attack when applying the interference: o Physical medium -> shield, ground, body, roller test bench, etc. o Synthetic via interfaces / API of the control electronics -> electronic test bench o External lines / buses -> introduction of hardware components that send interference signals ∘ Internal lines / buses -> excluded by assumption, the system is self-contained, protected / encrypted. Goal of information gathering: o Understanding the basic functionality o Exact copying of the functionality o Generation of damaging interference (adversarial examples) to prevent correct functionality during normal operation ▪ Extortion ▪ Sabotage ▪ Reputational damage. Execution of the attack: o Test bench (2 types of test benches, electronic and mechanical) -> quick, try out different (more or less realistic) test patterns o Real tests in manipulated physical terrain -> time-consuming,Real test patterns; type of attack; or train a replacement system with similar functionality (using input-output pairs), which can then be analyzed / used in detail because its structure is known; structure of the underlying software system; controlled decision output -> traditional control software; learned decision output -> modern machine learning methods; combined systems; or further components, for example, preprocessing with fixed mathematical procedures, rules, for example, Fourier transform, filters, normalization.

[0085] These types of attacks are prevented by the SVMA system according to the invention to prevent model theft attacks.

[0086] Fig. 2 Figure 1 shows an exemplary software system S that is protected against model theft attacks by the method according to the invention. The software system S comprises an input module through which input data E from sensors is read. The sensors can be part of the input module. Furthermore, the software system S comprises a preprocessing module in which the raw input data E is preprocessed, for example, filtered or abstracted. The software system S also comprises an evaluation module in which the input data E is evaluated, for example, with regard to object recognition. Finally, the software system S comprises a generation module in which the output data is generated, for example, control and / or regulation signals for actuators of a drive control system, a transmission control system, a driver assistance system, an automated driving system, and / or an active and / or passive safety system.The exact implementation of the software system S is unimportant for the application of the method according to the invention.

[0087] Attacker A has no access to data / parameters within the closed software system S. They can only observe input data E and output data. The software system receives signals as input, representing the current status of relevant parts of the environment of a driving system. These signals may have already been recorded by one or more sensors, and attacker A may, for example, insert falsified map information via fixed navigation systems outside the driving system or HD live maps. Alternatively, the sensors may be part of the driving / software system and monitor the environment internally, meaning encryption and / or verification of the transmitted data is ensured, e.g., ADAS with its own camera sensors. In this case, attacker A can only introduce interference within the environment and not at any interfaces to the system. Attack points on the software system S are in Fig. 2 hinted at.

[0088] Fig. 3 Visualizes the input data E for the described operating modes: normal case, problem case, attack case, and fault case. The input data E for an attack differs from the input data E in normal case, problem case, and fault case.

[0089] Fig. 4 Figure 1 shows an embodiment of the SVMA system according to the invention for preventing model theft attacks. The SVMA system monitors the individual modules of the [system / device]. Fig. 2 The software system S shown. Detected anomalies are stored in a secure memory within the vehicle system and reported to a cloud service. The SVMA operates independently of any other sensors or components in the entire system. The only assumption is that the underlying software system S is sufficiently protected to prevent an attacker from gaining access to data within the system.

[0090] Fig. 5 shows a clustering of the input data for anomaly detection according to the in Fig. 3 described operating modes.

[0091] In Fig. 6 This illustrates the possibility of switching between different software versions. The software's optimality depends on various system parameters, which also lead to different output data. By switching between different optima, good functionality of the software system S is ensured, while an attacker A is nevertheless confused because the output data varies considerably.

[0092] The inventive method is in Fig. 7 As shown, in process step V1, the SVMA system receives the input data E. In process step V2, the anomaly detection according to the invention is performed on the input data E. In the event of a detected anomaly, the information security of the software system S is ensured according to the invention in process step V3. Bezugszeichen

[0093] S Software system V1-V3 Process steps O Owner A Attacker f Machine learning model x1,..xn Query input data f* Model theft E Input data SVMA system for preventing model theft attacks

Claims

1. Computer-implemented method for preventing a model theft attack on a software system (S), wherein the software system (S) comprises a machine learning model and theft of the machine learning model is prevented, wherein the machine learning model is a black box to which an attacker has access only via query contents of input data (E) and output data and cannot obtain access to data within the system, wherein the software system (S) receives the input data (E) and provides the output data, using which data the software system (S) performs closed-loop and / or open-loop control of hardware parts including hardware parts of a driving system, comprising the steps of • obtaining the input data (E) (V1), wherein, in the event of an attack, an attacker loads modified input data into the software system (S) in order to subsequently observe output data; • performing anomaly identification on the input data (E) comprising a selective evaluation of individual data points, a contextual evaluation of several of the data points, a collective evaluation of groups of the data points and / or a time series evaluation of the data points, wherein the modified input data are identified as anomalies in the form of outliers from the data points including outliers in the data size of individual data points, outliers in the number of data points and / or outliers in a sequence of the data points (V2), • wherein a probability of generating one or more of the data points from a statistical model adapted to the input data is evaluated in order to identify outliers; each of the data points is evaluated in a context of its environment and / or similarity to other data points by executing a distance and density algorithm; a given number of groups is formed from a set of similar data points by executing a cluster algorithm; an assignment of one of the data points to a class is determined by executing a machine learning algorithm that has been trained on the input data; the data points are mapped into a lower dimensionality; a comparison with stored outliers is carried out, and, • in the event of an identified anomaly, ensuring information security of the software system (S) including documenting the attack, reporting the attack, modifying the operation of the software system (S), extracting a watermark, running the software system (S) in emergency operation and / or deactivating the software system (S) (V3) .

2. Method according to Claim 1, wherein the anomaly identification is carried out by the software system (S) before the input data (E) are fed into the software system (S) or in parallel with the evaluation of the input data (E) . • .

3. Method according to any one of Claims 1 to 2, wherein the input data (E) are divided into • a normal case in which the input data (E) are identical to the input data (E) that arise during regular use of the software system (S), • an attack case in which the input data (E) are modified for the model theft attack, • a problem case in which the input data (E) and / or output data indicate a malfunction in and / or damage to at least one of the hardware parts and • an interference case in which the output data are interfered with by the model theft attack, and the information security of the software system (S) is ensured in the attack case and the interference case, respectively.

4. Method according to any one of Claims 1 to 3, wherein a nearest-neighbour classification algorithm, a k-means algorithm, a support vector machine algorithm, a long short-term memory artificial neural network, an autoencoder or a replicator neuronal network is executed in order to identify outliers.

5. Method according to any one of Claims 1 to 4, wherein the modification of the operation of the software system (S) includes a change to an older version of the software system (S), a change in the configuration of the software system (S) and a change between different versions of the software system (S).

6. Method according to any one of Claims 1 to 5, wherein abnormal input data (E) are stored together with meta information including position information, time information and / or hardware part identification numbers for documentation purposes.

7. Method according to any one of Claims 1 to 6, wherein the anomaly identification is carried out locally in the driving system or externally including a remote system including a cloud and the identified anomaly is provided to the driving system, to a driver or operator of the driving system and / or to the remote system.

8. Use of the method according to any one of Claims 1 to 7 in a software system (S) of an open-loop drive controller of the driving system comprising an open-loop transmission controller, a driver assistance system, an automated driving system and / or an active and / or passive safety system.

9. Computer program for preventing a model theft attack on a software system (S) that receives input data (E) and provides output data, using which data the software system (S) performs closed-loop and / or open-loop control of hardware parts including hardware parts of a driving system, comprising commands that cause a computer system running the software system (S) to execute the steps of the method according to any one of Claims 1 to 7 when the computer program is running on the computer system.

10. Computer-readable data carrier for preventing a model theft attack, on which the computer program according to Claim 9 is stored.

11. Data carrier signal for preventing a model theft attack, which signal transmits the computer program according to Claim 9.

12. Open-loop control system of a driving system, wherein a software system (S) runs on the open-loop control system, which software system receives input data (E) from sensors of the driving system and provides output data for open-loop control devices of the open-loop control system including open-loop control devices for open-loop drive controllers, driver assistance systems, automated driving systems and active and / or passive security, wherein, in order to prevent a model theft attack, • the open-loop control system executes a computer program according to Claim 9, • the computer-readable data carrier according to Claim 10 is connected to the open-loop control system for data transmitting purposes and the computer program according to Claim 9 is executed or • the open-loop control system receives the data carrier signal according to Claim 11 and executes the computer program according to Claim 9.

13. System for preventing model theft attacks, comprising • input interfaces via which the system receives input data (E) for a closed-loop and / or open-loop control software system, • a computing unit designed to execute the method according to any one of Claims 1 to 7 and / or a computer program according to Claim 9 in order to detect anomalies, wherein the computer program is stored in a memory of the computing unit and / or the system comprises a data interface via which a computer-readable data carrier according to Claim 10 and / or a data carrier signal according to Claim 11 is received, • a memory in which the identified anomalies are stored, and • an output interface to provide the identified anomalies.