Methods for detecting events in the environment of a vehicle

Abstract

The invention relates to a method for detecting events in the environment of a vehicle (1), comprising the following steps: - providing (101) measurement data, wherein the measurement data result from a measurement of an acceleration and / or a rotation rate with at least one inertial sensor (2) arranged in the vehicle (1), - analyzing (102) the measurement data with a machine learning model, wherein the analysis with the machine learning model is carried out using a hardware accelerator (3), - detecting (103) at least one event in the environment of the vehicle (1) based on the analysis of the measurement data. The invention further relates to a computer program, a device, and a storage medium for this purpose.

Description

The invention relates to a method for detecting events in a surroundings of a vehicle. The invention further relates to a computer program, an apparatus and a storage medium for this purpose.Prior ArtThe present invention relates in particular to the technical field and the application of inertial sensors and to monitoring a vehicle environment. Inertial sensors are used in vehicles, for example, and can measure the acceleration and the rotation rate of the vehicle. Inertial sensors, which can also be referred to as inertial sensors, play an important role in the automotive industry in particular. Inertial sensors are used, for example, in vehicles to measure and monitor various aspects of the vehicle dynamics. An inertial sensor may include an accelerometer and a gyroscope, which preferably cooperate to sense both linear and rotational movements.Existing inertial sensors are used in the vehicle, for example, for triggering an airbag or for interventions in steering and drive, such as an ESP sensor for detecting skidding. However, these inertial sensors and the downstream data processing may cause a high power consumption and are therefore not continuously operating, in particular, because they can discharge the battery of the vehicle in a short time.In the prior art, there are also systems, for example, which are based on cameras which permanently record the environment of the vehicle and analyze it with regard to suspect behavior or various events. Such a system, however, is not data protection compliant in some cases, for example, and may require a lot of power for camera recording as well as for downstream data processing.Disclosure of the InventionThe subject matter of the invention is a method with the features of claim 1, a computer program with the features of claim 8, an apparatus with the features of claim 9 and a computer-readable storage medium with the features of claim 10. Features and details which are described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure reference is or can always be made to the individual aspects of the invention in a mutually alternating manner.The invention relates in particular to a method for detecting events in a surroundings of a vehicle, comprising the following steps:providing measurement data, wherein the measurement data results from a measurement of an acceleration and / or a rotation rate using at least one inertial sensor arranged in the vehicle,analyzing the measurement data with a machine learning model, wherein the analyzing with the machine learning model is performed using a hardware accelerator,detecting at least one event in the environment of the vehicle on the basis of the analysis of the measurement data.The inertial sensor is preferably a MEMS inertial sensor, wherein MEMS can stand for "micro-electro-mechanical systems". It is conceivable that at least two inertial sensors are provided, since a higher number of inertial sensors can enable a more differentiated provision of the measurement data. The machine learning model may detect the event in the environment of the vehicle by pattern recognition, wherein the machine learning model has preferably been trained with corresponding patterns in training data for events to be detected. A result of analyzing the measurement data using the machine learning model may accordingly represent at least a presence or a non-presence of the at least one event. The machine learning model may perform a fast Fourier transform for the detection. The machine learning model can be designed as a neural network, in particular as a convolutional neural network. By using the hardware accelerator, a computing effort for the detection using the machine learning model can advantageously be reduced, as a result of which a power requirement for the execution of the method also correspondingly decreases. As a result, the method can be advantageously carried out without leading to rapid discharge of a battery of the vehicle.It is optionally possible that the measurement data is provided when the vehicle is in a switched-off state in order to provide the monitoring of the environment of the vehicle in the switched-off state of the vehicle. A vehicle off state may be when both an engine and an ignition of the vehicle is off. Accordingly, it may be provided that switching off the vehicle triggers activation of the at least one inertial sensor. In an alternative possibility, the inertial sensor is permanently active, i.e. both in a switched-off and in a switched-on state of the vehicle.Advantageously, the invention can provide that the method further comprises at least one of the following steps:initiating an output of a warning message depending on the detected event, wherein the warning message is preferably transmitted to a user,initiating activation of at least one further system as a function of the detected event, in particular a camera, if the at least one event has been detected, in order to additionally monitor the environment with the at least one further system.It is conceivable that the warning message is transmitted to a smartphone of the user, or owner of the vehicle. It is also possible that the outputting of the warning message is combined with the activation of the at least one further system, for example that a horn or a switching on of a lighting of the vehicle is triggered, for example in the event that the detected event is a theft or damage to the vehicle. In this case, a camera and / or a microphone can also be activated as a further system in order to advantageously be able to collect relevant data with respect to the theft or the damage of the vehicle. In the event that the detected event is an accident, a camera and a microphone, for example, can be activated as a further system in order to advantageously be able to collect relevant data relating to the accident. It is conceivable that the at least one further system is activated for a defined time period depending on the detected event.It is possible that the method further comprises the following step:initiating a transmission of the detected event to a control device of the vehicle in order to analyze the detected event by the control device and to initiate the outputting of the warning message and / or the activation of the at least one further system on the basis of the analysis.The control device can be a data processing device or comprise a data processing device, for example. It can be provided that the inertial sensor is connected to the control device via an interface, in particular a CAN interface, in order to transmit the detected event to the control device via the interface. Furthermore, the at least one further system can be connected to the control device via an interface, so that the at least one further system can activate and deactivate via said interface.Furthermore, it can be provided within the scope of the invention that the method further comprises the following step:classifying the detected event, wherein the classifying is carried out on the basis of a pattern specific to a respective class in the measurement data.A respective class can represent, for example, an environmental impact and / or damage by a person and / or theft and / or an accident. The environmental effect can be, for example, rain, hagel, roof avalanche, storm or strawberry. The damage by the person can be, for example, a scratching of the vehicle or a turning of a pane of the vehicle. Classes are also conceivable which represent opening and / or closing of a door, a bonnet and / or a fuel cap of the vehicle, in order to infer, for example, a break in the vehicle on the basis thereof. The accident can be, for example, a collision with another vehicle or bicycle or with a shopping cart. It may be advantageous that each of the preceding examples is represented by a respective class. The respective classes can be extended according to requirements. The machine learning model may be trained with training data including the specific patterns of the respective classes, the specific patterns preferably being labeled according to the respective class.Optionally, it is provided that the hardware accelerator is designed as a digital circuit and is integrated into the at least one inertial sensor. The digital circuit can be integrated into an application-specific integrated circuit (ASIC) of the inertial sensor. An application specific integrated circuit may be an electronic circuit realized as an integrated circuit. A function of the application-specific integrated circuit can therefore no longer be changed in particular. The configuration of the hardware accelerator as a digital circuit may advantageously reduce a power, memory, and computation time when using the machine learning model.According to a further possibility, it can be provided that the provision of the measurement data and the analysis of the measurement data are carried out synchronously in time with the machine learning model. Furthermore, the detection of the at least one event in the environment of the vehicle can also be carried out synchronously in time with at least one of the steps of providing the measurement data and of analyzing the measurement data using the machine learning model. For example, a sample size or time period for ascertaining the acceleration and / or the rotation rate using the at least one inertial sensor may be established, after which the analysis of the environmental data using the machine learning model is carried out synchronously in time. Further, the hardware accelerator may include a first-in-first-out buffer memory to provide the synchronous time performance.It is possible for the method according to the invention to be used in a vehicle. The vehicle may be designed, for example, as a motor vehicle and / or passenger motor vehicle and / or autonomous vehicle. The vehicle may have a vehicle device, for example for providing an autonomous driving function and / or a driver assistance system. The vehicle device can be designed to at least partially automatically control and / or accelerate and / or brake and / or steer the vehicle.The invention likewise relates to a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer program to execute the method according to the invention. Thus, the computer program according to the invention brings with it the same advantages as have been described in detail with reference to a method according to the invention.The invention likewise relates to a device for data processing which is set up to carry out the method according to the invention. The device can be, for example, a computer which executes the computer program according to the invention. The computer may include at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored and from which the computer program can be read out by the processor for execution.The invention can likewise be a computer-readable storage medium which has the computer program according to the invention and / or comprises instructions which, when executed by a computer, cause the computer to execute the method according to the invention. The storage medium is embodied, for example, as a data memory such as a hard disk and / or a nonvolatile memory and / or a memory card. The storage medium may be integrated into the computer, for example.In addition, the method according to the invention can also be embodied as a computer-implemented method.Further advantages, features and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can be essential to the invention individually or in any combination. The following are shown: FIG. 1 shows a schematic visualization of a method, a device, a storage medium and a computer program according to exemplary embodiments of the invention, FIG. 2 shows a schematic illustration of a vehicle having an inertial sensor, a control unit and a further system according to exemplary embodiments of the invention, FIG. 3 shows a schematic illustration of an inertial sensor with an integrated hardware accelerator according to exemplary embodiments of the invention.FIG. 1 schematically illustrates a method 100, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.FIG. 1 schematically illustrates a method for detecting events in a surrounding environment of a vehicle 1 according to exemplary embodiments of the invention. In a first step 101, measurement data are preferably provided, wherein the measurement data result from a measurement of an acceleration and / or a rotation rate with at least one inertial sensor 2 arranged in the vehicle 1. In a second step 102, the measurement data are preferably analyzed with a machine learning model, wherein the analysis with the machine learning model is carried out using a hardware accelerator 3. In a third step 103, at least one event in the environment of the vehicle 1 can be detected on the basis of the analysis of the measurement data.FIG. 2 shows a schematic illustration of a vehicle 1, which comprises an inertial sensor 3, a control unit 7 and a further system 4. The further system 4 is, for example, a camera which can be activated by the control device 7 if a corresponding event has been detected. Control unit 7 is connected to inertial sensor 3 and to further system 4, respectively.FIG. 3 shows a possible embodiment of the inertial sensor 2. In this case, this comprises a chip 6 and an application-specific integrated circuit (ASIC) 5. In this embodiment, a hardware accelerator 3 is integrated as a digital circuit into the application-specific integrated circuit 5.The invention enables, in particular, a detection of events, such as, for example, vandalism or an unweetter in the surroundings of a vehicle 1 with power-saving inertial sensors 2.The present invention preferably does not record images and can thus advantageously be data protection compliant. The inertial sensor 2 described in the present invention can furthermore be very power-saving and can therefore be operated continuously without too much discharging a vehicle battery.In the present invention, both the inertial sensor 2 and the evaluation logic can be permanently active in an advantageous embodiment. This is possible in particular if the digital evaluation logic contains a hardware accelerator 3 and can thus consume significantly less power. As a result, further systems 4 are preferably awakened only when an event is actually detected.One aspect of the invention is, in particular, a power-saving MEMS inertial sensor 2 in combination with a hardware accelerator 3 for neural networks. The hardware accelerator 3 is preferably implemented as a digital circuit and can advantageously provide a very power-saving calculation of the neural network.The sensor system is preferably installed in the vehicle 1 and can also be active when the vehicle 1 is switched off. After application of a supply voltage, the sensor system preferably starts with a measurement of an acceleration and a rotation rate in three spatial axes. The acceleration and rotation rate are measured in particular with a current-saving MEMS inertial sensor 2. Based on the acceleration and rotation rate, events in an environment of the vehicle 1 can be detected, such as, for example, vandalism, discomfort or theft.The detection of the event is preferably carried out by downstream data processing. The data processing can be carried out using a machine learning model, in particular a neural network, preferably using a convolutional neural network ("convolutional neural network"). The inputs to the machine learning model are, for example, the acceleration and yaw rate signals. The output of the machine learning model may indicate whether an event has been detected and, if so, which. The calculation of the machine learning model is carried out in particular with the aid of a special hardware accelerator 3 for machine learning models, in particular neural networks, and time series data. The hardware accelerator 3 can be designed as a digital circuit and integrated into the chip of the sensor. Furthermore, the hardware accelerator 3 can be adapted by at least one parameter. The digital circuit may advantageously provide a very power-saving calculation of the machine learning model. Without the digital circuit, the calculation of the machine learning model would require a very high outlay, for example in terms of current, memory and computing time.The calculation of the machine learning model is performed by a hardware accelerator 3, in particular. the hardware accelerator 3 includes storage elements, calculation elements, and parameter elements in this embodiment. The hardware accelerator 3 can advantageously have a power loss of only about 1 μW in a nominal operation.All registers in an activation memory of the hardware accelerator 3 are preferably clocked and are activated only for one cycle when they are triggered by the previous layers. A connection of FIFO buffers to a processing element of the hardware accelerator 3 comprises in particular a two-stage multiplexer for selection between channels and layers. The processing element of the hardware accelerator 3 preferably does not comprise input buffers, i.e. only one level of the buffer hierarchy is used, in contrast to comparable devices in the prior art. This is made possible in particular by a low system frequency of the hardware accelerator 3 and, for example, avoids additional leaks and circuits in additional clock gate registers. SRAM (static random access memory) based buffers are preferably used to implement a weight and bias memory of the machine learning model, in particular neural network. As a result, the hardware accelerator 3 can configure and / or reconfigure parameters of the machine learning model, in particular neural network, before the application.By means of FFT calculations and a special machine learning algorithm, an evaluation of measurement data of the inertial sensor 2 and, if appropriate, an assignment to a previously defined event learned by the algorithm can take place.If an event is detected, then this event can be communicated to a superordinate control device 7, preferably together with an associated "confidence level", via an interface, for example a CAN interface. By means of this control device 7, further actions can then be triggered, such as the activation of a further system 4, such as a camera, or the transmission of a warning message to a smartphone of a user, or owner of the vehicle.The following describes the configuration of the hardware accelerator 3 in more detail: a data flow in a sequential component of the hardware accelerator 3 is implemented, for example, as follows. First, a new sample of the measurement of inertial sensor 2 may be inserted into an activation buffer of hardware accelerator 3, all samples preferably being advanced by one tap, while the last sample is discarded. Since all samples in the buffer of the hardware accelerator 3 are aligned with new weights of the machine learning model, a subroutine is preferably triggered which calculates and accumulates partial sums within a convolution kernel, which corresponds in particular to a stationary data stream. This can then be repeated for all output channels of the hardware accelerator 3, wherein a final sum of each channel can be inserted into a separate pooling unit. This unit preferably compares the stored value with the new value and maintains the larger value. Once k=3 new values are inserted, a new value can be transferred to the next layer activation buffer, triggering another computation cycle.The foregoing explanation of the embodiments describes the present invention solely by way of examples. Of course, individual features of the embodiments can be freely combined with one another, insofar as technically expedient, without departing from the scope of the present invention.

Claims

Method for detecting events in an environment of a vehicle (1), comprising the following steps: - providing (101) measurement data, wherein the measurement data results from a measurement of an acceleration and / or a rotation rate with at least one inertial sensor (2) arranged in the vehicle (1), - analyzing (102) the measurement data with a machine learning model, wherein the analysis is carried out with the machine learning model using a hardware accelerator (3), - detecting (103) at least one event in the environment of the vehicle (1) on the basis of the analysis of the measurement data.Method according to claim 1, characterized in that the measurement data is provided when the vehicle (1) is in a switched-off state, in order to provide the monitoring of the environment of the vehicle (1) in the switched-off state of the vehicle (1).Method according to one of the preceding claims, characterized in that the method further comprises at least one of the following steps: - initiating an output of a warning message as a function of the detected event, - initiating an activation of at least one further system (4) as a function of the detected event, in particular of a camera (4), if the at least one event has been detected, in order to additionally monitor the environment with the at least one further system (4).Method according to Claim 3, characterized in that the method further comprises the following step: - initiating transmission of the detected event to a control device (7) of the vehicle (1) in order to analyze the detected event by the control device (7) and, on the basis of the analysis, to initiate the outputting of the warning message and / or the activation of the at least one further system (4).Method according to one of the preceding claims, characterized in that the method further comprises the following step: - classifying the detected event, wherein the classification is carried out on the basis of a pattern specific for a respective class in the measurement data.Method according to one of the preceding claims, characterized in that the hardware accelerator (3) is designed as a digital circuit and is integrated into the at least one inertial sensor (2).Method according to one of the preceding claims, characterized in that the provision (101) of the measurement data and the analysis (102) of the measurement data are carried out synchronously in time with the machine learning model.A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer program to execute the method (100) according to any one of the preceding claims.Data processing device (10) configured to carry out the method (100) according to any one of claims 1 to 7.A computer readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to carry out the steps of the method (100) of any one of claims 1 to 7.

Images