Sensor system, method for analyzing an interaction with a surface by machine-learning, method for generating a training data set and computer program product

Abstract

A sensor system (1) comprises one or more sensor elements (2) for providing data in response to an interaction (13) with a surface (4) of a surface element (3), wherein the system (1) is configured for analyzing the data by a machine-learning classification model and / or a machine-learning localization algorithm.

Description

[0001] Description

[0002] Sensor system, method for analyzing an interaction with a surface by machine-learning, method for generating a training data set and computer program product

[0003] The present disclosure relates to a sensor system designed to detect and classify mechanical or other interactions (e.g. radiation, heat, etc.) between a surface, in particular of a surface element, and its environment. Furthermore, the disclosure relates to a method for analyzing such an interaction by machine-learning.

[0004] Surface elements, such as the exteriors of vehicles, robots, buildings, or other valuable objects, often encounter various environmental interactions that can lead to damages, or unintended contact with other objects. The need for a sensor system capable of detecting, classifying, and responding to such interactions is crucial for maintaining the integrity and functionality of these surface elements.

[0005] Embodiments of the disclosure relate to an improved sensor system and a method for analyzing an interaction with a surface by a machine-learning classification model and / or a machine-learning localization algorithm.

[0006] According to a first aspect, a sensor system comprises one or more sensor elements for providing data in response to an interaction with a surface of a surface element, wherein the system is configured for analyzing the data by a machinelearning classification model and / or a machine-learning localization algorithm. The sensor elements can be coupled with the surface element or integrated into the surface element , for example . The sensor system may comprise a plurality of sensor elements . It is also possible that the sensor system comprises only a single sensor element . "Coupling" comprises a direct contact with the surface element or an indirect contact with the surface element via one or more further components which are in contact with the surface element .

[0007] The sensor elements may be attached to a back-side of the surface element , for example . The surface element may be part of the sensor system . It is also possible that the surface element is in a first step not a part of the sensor system, wherein the sensor system is provided separately from the surface element and the sensor elements are then coupled to or integrated in the surface element .

[0008] The at least one sensor element may be a piezoelectric sensor element . The data provided by the sensor element may be an electric signal generated by an interaction with the surface element and a resulting deformation, for example .

[0009] The sensor element may be configured to detect elastic and / or inelastic mechanical impacts , such as those caused by water drops , stones , or other particles hitting the surface . Additionally or alternatively, the sensor element may be configured to detect directly or indirectly other interactions , in particular potential harmful events such as radiation, overheating, structural micro defects , resonant vibrations etc .

[0010] Interactions may also comprise touch events , e . g . a human touching the surface . As an example , the surface may also be a screen, control console, dashboard or other kind of operating terminal or machine interface configured to be touched by a human. In this case, the interaction can be a targeted touch for input of a command or receiving feedback.

[0011] Generally, an interaction comprises any kind of impact on the surface. The impact may be a mechanical impact or another impact such as by heat or radiation, for example. The interaction or impact may be also a touch event.

[0012] The data provided by the sensor elements may be digitized by an electronic system and transmitted to a data processing device. The electronic system may comprise amplifiers and filters, for example. The electronic system may be configured to ensure reliable and accurate data transmission from the sensor element.

[0013] The data processing device may comprise a software package utilizing machine-learning for the analysis of the data provided by the sensors. The machine-learning algorithm may include artificial intelligence methods. The classification may be based on a convolutional neural network, for example.

[0014] Detected events may be classified based on, e.g., damage type, impact type, intensity, frequency, temperature, time dependency of events, amplitude, etc. The software package may initiate further actions based on the type of event detected. Furthermore, a point of contact can be localized by machine-learning or by an analytical approach.

[0015] For example, the sensor system may serve one or more of the following purposes: 1) Impact detection and classification: Detects and classifies elastic or inelastic mechanical impacts on the surface element caused by environmental factors (e.g., water drops, stones) .

[0016] 2) Damage localization and categorization: Localizes and categorizes damages to the surface element, such as scratches or other surface imperfections.

[0017] 3) Object touch detection and localization: Detects and localizes instances where the surface element comes into contact with other objects or is touched by a human being.

[0018] As an example, the machine-learning classification model may be configured to classify a type of an interaction. A type may include at least one of an elastic interaction, an inelastic interaction and a scratching interaction. Furthermore, the severity and / or type of a damage can be determined. As an example, the interaction may be classified as either an inelastic or elastic interaction. The interaction may be an impact by an object hitting or otherwise interacting with the surface, for example. An elastic interaction may comprise interactions without any damage of the surface element or only with a small damage of the surface element. For an inelastic interaction, also the severity of the interaction can be categorized. Thereby, it can be decided if a human inspection and / or maintenance actions have to be started, for example.

[0019] The classification model may be additionally or alternatively configured to classify the object generating the interaction. In particular, the material of an object generating the interaction may be classified. As an example, the classification may be either a hard or a soft material. It is also possible to differentiate between specific hard or soft materials, e.g. wood, steel or plastic in case of hard materials. Furthermore, the size of the object may be classified. As a further example, the type of interaction may be classified, e.g. differentiating between an object dropped on a surface, a punching object or a scratching object. Thereby, the source of the impact may be determined.

[0020] The classification model may be based on training data comprising interactions generated by different materials of objects and / or different sizes of objects and / or different weights of objects and / or different impact energies and / or different types of interactions. As an example, the different materials may be soft and hard materials. The different types of impact may comprise elastic or inelastic impacts. The type of interaction may additionally or alternatively comprise the type of interacting object, e.g. an object dropped on the surface, a punching object or a scratching object.

[0021] While the training data may comprise a large variety of different interactions, the classification may be made in view of a smaller subset, e.g. elastic or inelastic interactions. A large and diverse training data set helps to improve the quality of classification.

[0022] Additionally or alternatively, the sensor system may be configured for determining a location of the interaction. The sensor system may be configured to determine the location of the interaction by a machine-learning localization algorithm. A localization algorithm can be also understood as a specific classification algorithm. It is also possible that the sensor system is configured for determining the location of interaction by an analytical algorithm. An analytical algorithm may provide good results for simple shapes of a surface. However, for complex shapes or joined parts of the surface element, a machine-learning algorithm may provide better results.

[0023] In a human touch event, classifications can be made concerning "who", "where", "what" and "how", for example. Regarding "who", the person generating the touch can be determined. As an example, the type of pressure, intensity area or location can be used to discern between a man, woman or child. Regarding "where", the location of the touch on the surface can be determined. Regarding "what", the purpose of the touch can be determined, e.g. a targeted selection of an option, pressing a "button", double-clicking, etc. Regarding "how", the type of touch can be determined. As an example, a "normal" touch can be discerned from a "hasty" touch, a "slow" touch or "confused" touch. Depending on the type of touch, an emergency stop, a query or an alert can be triggered, for example. As an example, a "slow" touch may indicate fatigue or a "confused" touch may indicate alcohol or drug influence. The classification can thus be used to enhance useability and / or safety of the device, e.g. a car or a machine .

[0024] The sensor system may comprise a processor configured for carrying out the analysis. The sensor system may comprise a memory configured for storing the classification model and / or localization algorithm. The algorithm can be provided as a software package installed on the processor. It is also possible that the classification model and / or localization algorithm is cloud-based. The sensor system may comprise an alert system for alerting a user when the interaction is classified as an alertable interaction. A user may be a direct user, such as a driver of a vehicle comprising the surface. A user may be also an operator, e.g. an operator of a fleet of vehicles or an insurance company. As an example, the alert system may comprise a screen where an alert is displayed or may comprise sending an electronic message to an operator or user.

[0025] The alert system may be configured for alerting a user depending on the classification. As an example, a user may be alerted when an interaction is classified as inelastic or exceeding an allowed level of damage.

[0026] The sensor system may be utilized in various industries, including automotive (e.g. car bumpers) , robotics (e.g., movable robot chassis) , architecture (e.g. building exteriors) , and protection of valuable objects (e.g. property, military areas, industrial plants) . More specifically, e.g. in automotive, it could be possible for rental car providers to detect "invisible" impacts to the car and differentiate between the types of impact to detect severe ones early on. Thereby, advice on proper maintenance can be given to enhance security and limit down-times of a single car or the overall car fleet. The same approach can be applied for robots etc. For buildings, damages or stress, e.g. stress by catastrophes (earthquake, fire, flooding) but also vandalism etc., might be detected and proper countermeasures might be set. For valuable objects like industrial plants, detectors might be used for surveillance of tanks, pipes, concrete structures, pressurized compartments, etc. In fact, all parts critical to damage via "external" factors (see above) and "internal" factors (like malfunctioning, material aging, etc.) could be detected, classified and proper countermeasures could be started.

[0027] Furthermore, the sensor system may be used to classify and / or localize touch events on a surface intended to be touched by a user such as a machine interface or any other type of touch interface. By the classification and localization, the processing of the touch event can be improved and the security can be enhanced.

[0028] Overall, the sensor system allows an early detection of impacts, damages and generally all kinds of interactions. Interactions can be monitored and analyzed in real-time, allowing an early response. The sensor system can be utilized in versatile applications across different surface elements. Thereby, the longevity and functionality of surface elements in diverse environments can be ensured.

[0029] According to a further aspect, a method for analyzing an interaction with a surface comprises the steps of receiving data in response to an interaction with a surface of a surface element and analyzing the data by a machine-learning classification model and / or a machine-learning localization algorithm. The method may result in a classification of the interaction and / or localization of the interaction. The method may be computer-implemented. A software-package may be provided for carrying out the method.

[0030] The method may comprise any functional and / or structural features as described in connection with the sensor system. The method can be carried out by using the sensor system disclosed in the foregoing. As an example, when an interaction occurs, an electric signal may be provided by the sensor elements. The electric signal may be filtered and amplified and a wavefront detection may be carried out. Then, the electric signal is evaluated by an evaluation algorithm, which may classify the interaction and / or localize the point of interaction. As an example, a mechanical impact generated by an object or other types of interactions such as touch events can be detected.

[0031] According to a further aspect, a method for generating a training data set for the machine-learning classification model and / or machine-learning localization algorithm comprises generating interactions of at least one of different materials of objects and / or different sizes of objects and / or different weights of objects and / or different impact energies and / or different types of interaction.

[0032] The types of interactions, in particular mechanical impacts, may include objects falling on the surface, objects punching the surface and objects scratching on the surface, for example. The types of interactions may include touch events. As an example, training data sets with different types of persons generating the touch ("who") , different locations of the touch ("where") , different target of the touch ("what") and different types of touch ("how") may be generated.

[0033] According to a further aspect, a computer program product comprises instructions which, when executed on a computing device, implements one or more of the methods disclosed in the foregoing. The computer program product may be a downloadable computer program or a stored computer program, for example. The computer program product may comprise instructions comprising algorithms in the form of computer code. When this computer code is read by a computing device, the computing device implements the above-disclosed method. According to a further aspect , a computer-readable storage medium comprises the computer program product disclosed in the foregoing . The computer-readable storage medium may any data carrier, for example , such as a tangible storage medium and / or a transmission medium that participates in providing instructions to a processor for execution . The computer- readable storage medium may be , for example , including but not limited to , non-volatile media, volatile media, and transmission media . A digital file attachment to an e-mail or other sel f-contained information archive or set of archives is considered a distribution medium equivalent to a tangible storage medium .

[0034] The present disclosure comprises several aspects and embodiments . Every feature described with respect to one of the aspects and embodiments is also disclosed herein with respect to the other aspects and embodiments , even i f the respective feature is not explicitly mentioned in this context .

[0035] Further features , refinements and expediencies become apparent from the following description of the exemplary embodiments in connection with the figures . In the figures , elements of the same structure and / or functionality may be referenced by the same reference signs . It is to be understood that the embodiments shown in the figures are illustrative representations and are not necessarily drawn to scale .

[0036] Figure 1 shows an embodiment of a sensor system in a schematic view, Figure 2 shows an embodiment of a surface element with several sensor elements in a perspective view from the bottom,

[0037] Figure 3 shows a process diagram of a method for analyzing an impact on a surface ,

[0038] Figure 4 shows a more detailed process diagram of a method for analyzing an impact on a surface with a speci fic classi fication,

[0039] Figure 5 shows a test setup for locali zation of an impact on a surface ,

[0040] Figure 6 shows a wave propagation from an impact point ,

[0041] Figure 7 shows a wave propagation pattern recorded by a sensor element ,

[0042] Figure 8 shows a heat map achieved from an analytical approach,

[0043] Figure 9 shows a comparison between the analytical and the machine learning approach,

[0044] Figure 10 shows a setup for generating a training data set in a schematic view,

[0045] Figure 11 shows a schematic diagram for obj ect classi fication with a machine-learning approach,

[0046] Figure 12 shows a further setup for generating a training data set in a schematic view, Figure 13 shows a schematic diagram for locali zing an impact point with a machine-learning approach,

[0047] Figures 14 and 15 shows an example for an analytical approach for locali zing an impact point ,

[0048] Figure 16 shows an embodiment of a sensor system in a schematic view,

[0049] Figure 17 shows a further process diagram for a method for analyzing an interaction with a surface .

[0050] Figure 1 shows a sensor system 1 comprising at least one sensor element 2 . The sensor element 2 is integrated in a surface element 3 or is positioned on the surface element 3 . For example , the sensor element 2 is positioned on a side of the surface element 3 which is opposite to the side of a surface 4 on which an interaction occurs . The sensor element 2 can also be otherwise coupled to the surface element 3 .

[0051] The sensor element 2 is configured to provide an electronic signal in response to an impact on the surface 4 of the surface element 3 . The surface 4 is a surface on which an interaction, e . g . a mechanical impact or other impacts such as heat , occurs . The surface 4 may be an outer surface of an obj ect . As an example , the obj ect may be a car or another kind of vehicle . It is also possible that the surface 4 is an interface for user input , e . g . a touchscreen .

[0052] The data provided by the electronic signal is analyzed by a data processing device 5 . The data processing device 5 is configured to analyze the data based on a machine-learning classi fication model and / or a machine-learning locali zation algorithm . The analysis and processing can be also cloudbased .

[0053] The sensor element 2 may be a transducer, such as a piezoelectric transducer . As an example , the sensor element 2 may be a piezoelectric polymer sensor element . The sensor element 2 may be also another kind of piezoelectric element , e . g . a ceramic sensor element .

[0054] The sensor element 2 may be embedded in the surface element 4 or another element coupled to the surface element 4 . It is also possible that the sensor element 2 is deposited on the surface element 4 or attached to the surface element 4 by a fastening material . Also in this case , the sensor element 2 may be attached or fastened to another element which may be coupled to the surface element 4 .

[0055] The sensor system 1 may comprise only a single sensor element 2 or several sensor elements . The sensor elements 2 are coupled to the obj ect to be surveyed .

[0056] The sensor system 1 is configured to detect an interaction of an obj ect , in particular of a surface element 3 with its environment . The sensor system 1 may be an impact detector . The processing device 5 acts as a monitoring system, capable of detecting the designated surveillance parameters set for the obj ect .

[0057] Figure 2 shows a surface element 3 with several sensor elements 2 positioned on a bottom surface 6 of the surface element 3 . The bottom surface 6 is opposite to the surface 4 . The sensor elements 2 are shown to be positioned on corners of the surface element 3. However, other positions are possible .

[0058] Figure 3 shows a process diagram of a method for analyzing an interaction with a surface 4. The method can be carried out with the sensor system 1 of Fig. 1, for example. The method can be computer-implemented.

[0059] By the method, the interaction of the surface 4 with its environment can be detected. Generally, the method allows detection, localization and classification of interactions with the surface 4 and, for example, of damages of the surface element 3 caused by such interactions, e.g. mechanical or other impacts.

[0060] This includes, for example: the localization and categorization of damages of the surface element 3 like scratches, or other inelastic or elastic damages, the detection and classification of elastic or inelastic mechanical impacts coming from Objekten hitting the surface element 3, e.g. water drops or stones, the detection, localization and classification of touch or contact of the surface element 3 with other objects or with humans.

[0061] As can be seen in the process diagram, when an interaction occurs (Step A) with the surface 4, the interaction is detected by one or more sensor elements 2, in particular transducers, and an electric signal is provided and processed (Step B) . The signal is then evaluated (Step C) by a machinelearning algorithm.

[0062] Accordingly, the method comprises:

[0063] A) a mechanical interaction,

[0064] B) data acquisition by a sensor and data processing,

[0065] C) evaluation of the data by an evaluation algorithm.

[0066] The evaluation algorithm may comprise analytical methods in addition to a machine-learning algorithm. The algorithm is computer-implemented .

[0067] For localization (L) of the interaction point on the surface 4, a localization algorithm can be used, which can be based on an analytical approach or on a machine-learning approach. In particular, a statistical and / or classification model can be used.

[0068] Machine-learning and, in particular, artificial intelligencebased systems can be used when analytical models cannot be used, e.g. due to the surface shape and the structural complexity of the parts involved, in particular the surface element 3 (stiffening rips, anchor, and connection / j oining points with other parts) . The impact may then be localized by a machine-learning classification model, in addition or as an alternative to an analytical model.

[0069] For further classification (CL) , e.g. of the type of damage or the type of interaction, a machine-learning approach is used . In particular, a machine-learning classification model classifies (CL) the signal pattern. The machine learning classification model may be an Al-based classification model.

[0070] Figure 4 shows a more detailed process diagram of a method for analyzing an interaction with a surface 4 with a specific example of classification of the signal pattern. In particular, the interaction may be a mechanical impact by an object hitting the surface.

[0071] The signal provided by the transducers 2 is processed by filtering and / or amplification (step Bl) and the wavefront is detected (B2 ) .

[0072] In the shown embodiment, classification (CL) is made regarding the different types of impacts, such as elastic (Y) , i.e. non-destructive, or inelastic (X) , i.e. destructive. Also a scratching impact can be classified. A scratching impact can be seen as a specific form of an inelastic impact or as a separate class.

[0073] Other classifications, e.g. the material or size of an interacting object can be made.

[0074] Figure 5 shows a test setup 7 for localization of an interaction with a surface 4. The test setup 7 may be used for analytical determination of the point of interaction and / or for a machine-learning approach. As an example, the test setup can serve also for generating training data and / or training the machine-learning algorithm.

[0075] Different masses are dropped from different heights h onto a surface 4 at specific points C (Xc, Yc) of the surface 4 (see also black crosses in Fig. 8) . The surface 4 is equipped with one to four sensor elements 2 in corners that are used to measure the incoming wavefront, in particular a shock wave. Interaction localization can be done on different substrate materials, such as ABS, PVC, PC, PS and other plastic materials. The surface element 3 used as a sample has a geometry of a plate. The sample is fastened to a frame.

[0076] Plates with different thicknesses, in particular 2 mm, 5 mm and 10 mm can be used.

[0077] As sensor elements 2, piezo elements can be used. The piezo elements may be transducers, in particular PVDF transducers, e.g. with a geometry of 10*20mm and 3pm thickness. Beside the different substrate thicknesses, the test setup can contain different impact masses, e.g. of m=0.1g to 1.4g dropped from different heights, e.g. between h=10 and 30cm. The sensor elements 2 can be positioned in the corners of the surface element 3, e.g. as shown in Fig. 2.

[0078] The velocity v of the falling object when interacting with the surface can be calculated as v ~ ^2gh, with g being the acceleration due to gravity.

[0079] Figure 6 shows a wave propagation from an interaction or impact point C towards individual sensors D1-D4, resulting in different run times used for the localization. The wave propagation may be shown for the test setup of Fig. 5 and may be used for analytically calculating the coordinates Xc, Ycof the interaction point C.

[0080] The individual sensors 2 are located at the corners of a positions DI to D4 may be located in the corners of a square with a side length L. The wavefronts at radii rj_ to r4 arrive at di f ferent times at the sensor positions D1-D4 such that the recorded wavefront pattern can be used for calculating the interaction point C .

[0081] Figure 7 shows a wave propagation pattern recorded by a sensor element 2 .

[0082] The signal S in Volt V recorded by the sensor element is shown over the time t in seconds . A real signal is shown in a solid line and an ideal signal in a dotted line . The signal shi ft Aj_ of the incoming wavefront may be 1 mV at time tj_ .

[0083] The incoming wavefront hits the sensor at time tj_ , with an error +A for the real signal . The error in wave fronts arrival time can lead to an error in locali zing the interaction point .

[0084] For simple test structures , like plates , the locali zation of the interaction can be calculated by an analytical method based on the wave propagation ( see Fig . 6 ) and the pattern recorded by the individual sensors ( see Fig . 7 ) . For more complex structures , a machine-learning algorithm, in particular a deep learning model , is used .

[0085] The exact determination of the time for the incoming vibration wavefront can be a challenge due to a low amplitude of the wavefront , a non-periodical nature of the wave , an unpredictable shape of the wave and EMI noise within the same amplitude range of the wavefront . For stable wavefront detection, di f ferent signal-processing approaches can be used, including filtering and using derivatives .

[0086] Figure 8 shows a so-called heat map achieved from an analytical approach of the simple rectangular surface of the test setup of Fig. 5. The curves have been obtained with the test-setup of Fig. 5 with masses dropped from different heights, on different substrate materials and thicknesses and different weights.

[0087] The crosses indicate the nine different true interaction points C. the circles indicate the localized points, i.e. the points of interaction calculated from incoming wavefronts as shown in Figs. 6 and 7. The predicted points of interaction correlate with the real points of interaction with a slight asymmetric character (elongated and center-shifted) . The asymmetric character results from signal damping over distance .

[0088] Figure 9 shows a comparison between the analytical and the machine-learning, in particular deep-learning, approach for localizing an interaction point C with the test setup of Fig. 5.

[0089] The curve A shows a correlation for the analytical model and the curve M shows a correlation for a machine-learning model, in particular for a deep-learning model. Each curve shows the percentage L of localized points L being within a circle with a radius R around a true interaction point C, over the radius R of the circle. In other words, the percentage LR is the percentage of localized points L having a deviation A < R from the true interaction point C (see insert in Figure 9) .

[0090] For this simple test-setup of a plate, the analytical model determines the point of interaction slightly better than the machine-learning approach. This is mainly based on the fact of the limited number of training cycles conducted in the present machine-learning approach. However, for complex-shaped structures like robot chassis car bumpers or surfaces which contain multiple joined parts (e.g. glued or screwed) an analytical solution cannot be derived. The same holds true if one single sensor element 2 is used and / or for "indirect" parameters which shall be detected (like e.g. structural micro fractures which shall be detected out of an impact; or piping systems filled with the wrong fluid shall be detected by change in resonance frequencies; etc . ) .

[0091] Therefore, good results cannot be achieved by an analytical approach for all surfaces and parameters to be detected. However, localization is possible by machine-learning systems, in particular Al-based systems, also in complex systems and / or for deriving indirect parameters out of "just" an interaction. This is even more important in the context of interaction classification where a machine-learning approach, in particular a deep learning system, is trained for pattern recognition (e.g. an interaction-type dependent waveform) to predict the nature of the interaction (e. g. elastic (no deformation) ; inelastic (permanent deformation of the surface) ) or any kind of "indirect" effect as stated above.

[0092] Figure 10 shows a setup for generating a training data set for interaction classification, in particular impact classification for objects hitting a surface 4.

[0093] Training data sets are generated by dropping different types of masses, e.g. different materials of objects, different sizes of objects (e.g. different diameters) , different shapes of objects and / or objects with different energy impact (e.g. by varying the dropping height) on the surface 4. As an example, an object dropped on the surface 4 may be a ball 8 of steel.

[0094] The training data set can be generated for elastic deformation scenarios, ranging from small dents such as small craters Xg to mid-big craters Xg to cracks Xg through the whole plate, and up to complete penetrations X4 of the plate.

[0095] Furthermore, the training data set can additionally or alternatively be generated for elastic deformation Y where the surface 4 is not damaged.

[0096] Furthermore, training data sets can also be generated from punch tools 9. Here, an object is not dropped on the surface 4 but punches the surface 4. Also in this case, the training data set can reach from elastic impact to different sorts of inelastic impact.

[0097] Furthermore, training data sets can also be generated from scratching tools 10 (see Fig. 12) , scratching over the surface 4.

[0098] For elastic impact (no damage) , masses may be used in a range between 1.4g to 110 g, for example. For inelastic impact, masses around 4000 g may be used, for example. The energy impact for elastic impact may range from 0.6 mJ to 1.5 J. For small craters Xg, the energy impact may be about 0.4 J and overlap with the energy impact of elastic impact. For larger craters Xg up to complete penetrations X4, the energy impact may be up to 40 J, for example.

[0099] The corresponding wave patterns are recorded and used for training the machine-learning model. Table 1 shows a confusion matrix for a high energy impact dataset. The confusion matrix shows the number of impacts correctly assigned (upper left: true positive (TP) and lower right: true negative (TN) ) and the number of impacts incorrectly assigned (lower left: false negative (FN) and upper right: false positive (FP) ) in the test as well as the number of training data sets used for each case. "EXP" is the real outcome in the experiment, which can be an inelastic impact (INEL) or an elastic impact (EL) and "PRED" is the impact predicted by the machine-learning algorithm.

[0100] Table 1: Confusion matrix for high energy impact

[0101] Table 2 shows performance parameters obtained with the setup of Fig. 10.

[0102]

[0103] Table 2 : Performance parameters for high energy impacts

[0104] Focusing first on high impact energy cases, the system was trained to differentiate between elastic and inelastic cases. A 1-dimensional convolutional neural network (1DCNN) has shown the best result by correctly labeling 98% of the hits.

[0105] Out of a dataset of in total 2842 events a training set of 2273 events was used with a ratio of 80% elastic and 20% inelastic, leaving a data set of 569 events for testing. As shown in tables 1 and 2, for those high energy impacts prediction accuracy is 96,8% ( (TP+TN) / Total) .

[0106] By expanding the energy range to the low energy region of events (energy range for small dents: 0.4-1.5J) , the original model predictions rate drops to 64%, based on the high impact energy model. The system was then retrained by adding around 50% small energy events into the trainings dataset.

[0107] Tables 3 and 4 show a confusion matrix for a broader energy range and respective performance parameters.

[0108] Table 3: Confusion matrix for low energy impact Table 4 : Performance parameters for low energy impacts

[0109] The system was trained and tested with in total 3103 data sets with a ratio of 75% / 25% between elastic / inelastic events. The trainings set uses 2404 data sets and the test was performed with the remaining 699 sets. This leads to a prediction accuracy for the test set of 96,4 % which is almost the one for the pure high impact model of 96,8%.

[0110] Figure 11 shows a schematic diagram for object classification with a machine-learning approach, in particular a deep learning network. The system is trained with elastic impact events, i.e. without damage, using different object materials (NBR rubber, POM plastic, stainless steel and wood) . Rubber, plastic, steel and wood were chosen due to their wide use and differences in the set of mechanical properties. As in one of the setups in Fig. 9, a ball 8 was dropped on a surface 4, wherein balls 8 formed from the different materials were used .

[0111] The training data was randomized by choosing different heights, e.g. between 1-150 cm, and different masses, e.g. between 0,1-110 g, to provide better data variability and higher generalization ability of the resulting model.

[0112] The overall dataset contains in total 15937 points, where 12750 (~80%) points were used for training while the remaining 3187 events were used for tests. All impact events were performed in the elastic regime.

[0113] The recorded signal pattern (C) was evaluated by an algorithm (D) , in particular classified by an Al based classification. The classification was made for the material of the object, i.e. rubber (Mj_) , plastic (Mg) , steel (Mg) , wood (M4) .

[0114] Table 5 shows a confusion matrix of the complete impact energy and material range when classifying the different materials according to Fig. 11.

[0115] The 1DCNN model (increased size) correctly predicted 70 % in the case of all four classes in the four-class-model, but the prediction quality could be significantly increased when using a two-class-model (soft: rubber / hard: wood, plastic, steel to 98.5 % (submatrix in bold-faced) . Accordingly, the object was classified only as a soft or a hard material. Generally, such Al algorithms can be also used to detect various other, also indirect, effects as well.

[0116] As explained further above, the machine-learning approach can also be used for localizing a point of interaction. For simpler surface shapes, an analytical approach can be used.

[0117] For more complex structural shapes (curved, bent, tubes) the analytical approach is less applicable, as e.g. is the case for a chassis of a vehicle. Here, an AT approach based on deep learning can be used. In particular, a statistical model can be used. It is also possible that a combination of an Al approach and an analytical approach is used.

[0118] Figure 12 shows a further setup for generating a training data set in a schematic view, wherein a scratching tool 10 is used for generating an interaction.

[0119] Different shapes of scratching tooltips 11 can be used, such as a pyramid, a blade or a round tooltip. Also the tool materials, tool velocity v, pressure P on the tool 10, trajectory and time of the event can be varied.

[0120] This also allows a further classification between elastic events, scratching events and other inelastic events. Also the depth of a scratching interaction can be classified, similar to the depth of a collision as shown in Fig. 10.

[0121] Figure 13 shows a schematic diagram for localizing an interaction point with a machine-learning approach.

[0122] An interaction (Step A) is detected by a sensor element 2 (transducer) (Step B) , which then provides a signal to electronics. Key variables of the impact may be mass and speed. For the transducer, a key variable may be the conversion efficiency for converting the interaction signal to an electronic signal.

[0123] The signal is then filtered and amplified (Bl) , wherein an analog-digital-converter may be used. The fact of an interaction and the wavefront is detected by a signal processing method (B2) . A machine-learning localization algorithm (C) is carried out to localize (L) the point of interaction, which is the output of the algorithm.

[0124] The localization algorithm is structure-dependent and might significantly vary and increase its complexity when going from simple planar structures to complex shapes.

[0125] Figures 14 and 15 show an example for an analytical approach for localizing (L) an interaction point C in a sphere or half sphere. The analytical approach is based on a wavepropagation model.

[0126] A number i of sensors 2 are located at positions [r, n / 2, cp ] on a circle 12 in the coordinates of Figure 13. The interaction point C is located at a position [r, 0i, c i ] . The system of equations for interaction localization in a halfsphere and its solution is based on the systems of equations of the geodesic lines on a spherical surface, wherein v is the speed of sound, t± is the wave arrival time and To is the wave travel time between an impact point and the closest sensor. In particular the difference in the angle A0i can be determined as follows with Equation 1:

[0127] Equation 1: Set of equations for determining the difference in angle

[0128] For complex shapes, a machine-learning approach demonstrates comparable localization accuracy and could be used as an alternative or additional localization technique. Figure 16 shows an embodiment of a sensor system 1 comprising one or more sensor element 2 for sensing an interaction on a surface .

[0129] The signal from the sensor elements 2 resulting from an interaction is sent to a data processing device 5 of the sensor system 1 . The data processing device 5 comprises a memory 15 on which training data is stored and a processor 14 for executing the machine-learning algorithm . The machinelearning algorithm may be a provided as a computer-program . The processor 14 may also execute an analytic algorithm for determining a point of interaction . By the machine-learning algorithm, the interaction is classi fied and / or locali zed .

[0130] The result of the classi fication and / or locali zation is provided to an output 16 of the system 1 . In particular, the output 16 may be a user-interface . As an example , the output 16 may be a screen on which the result is displayed . It is also possible that a message is sent to a user or a provider of the system 1 . Depending on the classi fication, an alert 17 may be generated . The alert 17 may be generated on a display and / or other signals , e . g . acoustic signals may be provided . The alert 17 may be a message sent to a user or a provider of the system 1 , for example .

[0131] The system 1 may comprise any features as disclosed for the other embodiments in the foregoing .

[0132] Generally, the interaction may be any type of interaction with an impact on the surface , e . g . a mechanical impact by an obj ect hitting the surface or a user touching the surface . Figure 17 shows a further process diagram for a method for analyzing an interaction with a surface. The method may be a part of the method disclosed in Figs. 3 and 4. The method may be computer-implemented. The method may be provided in the form of a software, which when carried out by a processing device conducts the method steps. The method may be carried out by a processing device 5 and processor 14 as disclosed in Fig. 16, for example.

[0133] The method comprises the steps of receiving (R) data in response to an interaction with a surface of a surface element. The data may be provided by sensor elements coupled to the surface.

[0134] The method further comprises the step of analyzing (C) the data by a machine-learning classification model and / or a machine-learning localization algorithm. The output of the method is a classification CL and / or a localization L of the interaction. As example, the interaction may be classified as an elastic interaction (Y) or an inelastic interaction (X) .

[0135] Reference Signs

[0136] 1 sensor system

[0137] 2 sensor element

[0138] 3 surface element

[0139] 4 surface

[0140] 5 data processing device

[0141] 6 bottom surface

[0142] 7 test setup

[0143] 8 dropped obj ect

[0144] 9 punching obj ect

[0145] 10 scratching obj ect

[0146] 11 scratching tooltip

[0147] 12 circle

[0148] 13 interaction

[0149] 14 processor

[0150] 15 memory

[0151] 16 output

[0152] 17 alert

[0153] A interaction occurring

[0154] B providing electric signal

[0155] Bl filtering and ampli fication

[0156] B2 wavefront detection

[0157] C evaluation

[0158] CL classi fication

[0159] L locali zation

[0160] Y elastic

[0161] X inelastic

[0162] XI inelastic interaction, small crater

[0163] X2 inelastic interaction, mid-big crater

[0164] X3 inelastic interaction, crack

[0165] X4 inelastic interaction, complete penetration Ml material, rubber

[0166] M2 material, plastic

[0167] M3 material, steel M4 material, wood

Claims

Claims1. A sensor system (1) comprising one or more sensor elements (2) for providing data in response to an interaction (13) with a surface (4) of a surface element (3) , wherein the sensor system (1) is configured for analyzing the data by a machine-learning classification model and / or a machine-learning localization algorithm.

2. The sensor system (1) of claim 1, wherein the machine-learning classification model is configured to classify a type of the interaction (13) , the type including at least one of an elastic interaction (Y) , an inelastic interaction (X, XI, X2, X3, X4) and a scratching interaction .

3. The sensor system (1) of any of the preceding claims, wherein the machine-learning classification model is configured to classify a type of the interaction (13) , wherein the type includes at least a scratching interaction in a class separate from other inelastic interactions.

4. The sensor system (1) of any of the preceding claims, wherein the classification model is configured to classify (CL) the material (Ml, M2, M3, M4 ) of an object (8, 9, 10) generating the interaction (13) .

5. The sensor system (1) of any of the preceding claims, wherein the classification model is configured to classify an intensity and / or a frequency of interactions (13) .

6. The sensor system (1) of any of the preceding claims,wherein the classification model is based on training data comprising interactions (13) generated by different materials (Ml, M2, M3, M4 ) of objects (8, 9, 10) and / or different sizes of objects (8, 9, 10) and / or different weights of objects (8, 9, 10) and / or different impact energy and / or different types of interaction (13) .

7. The sensor system (1) of any of the preceding claims, being configured for determining a point of interaction (C) by a machine-learning localization algorithm.

8. The sensor system (1) of any of the preceding claims, being configured for determining a point of interaction (C) by an analytical algorithm.

9. The sensor system of any of the preceding claims, comprising a processor (14) configured for carrying out the analysis and a memory (15) configured for storing the classification model and / or localization algorithm.

10. The sensor system of any of the preceding claims, comprising an alert system (17) for alerting a user when the interaction (13) is classified as an alertable interaction (13) .

11. The sensor system of claim 10, wherein the alert system (17) is configured for alerting a user depending on the classification (CL) .

12. The sensor system of claim 10, wherein the alert system (17) is configured for alerting a user when the interaction (13) is inelastic (X, XI, X2, X3,X4) or exceeds an allowed level of inelastic deformation.

13. A method for analyzing an interaction (13) with a surface (4) , comprising the steps of: receiving (R) data in response to an interaction (13) with a surface (4) of a surface element (3) , analyzing (C) the data by a machine-learning classification model and / or a machine-learning localization algorithm.

14. The method of claim 13, comprising a step of classifying (CL) a type of the interaction (13) , the type including at least one of an elastic interaction (Y) , an inelastic interaction (X, XI, X2, X3, X4) and a scratching interaction.

15. The method of any of claims 13 and 14, comprising a step of classifying (CL) a type of the interaction (13) , the type including at least a scratching interaction .

16. The method of any of claims 13 to 15, comprising a step of localizing (L) the point of interaction (13) .

17. The method of any of claims 13 to 16, wherein the method is computer-implemented.

18. A computer program product comprising instructions which, when executed on a computing device, implements the method according to any of claims 13 to 17.

19. A method for generating a training data set for the machine-learning classification model and / or machine-learning localization algorithm of any of the preceding claims,comprising generating interactions (13) of at least one of different materials (Ml, M2, M3, M4 ) of objects (8, 9, 10) and / or different sizes of objects (8, 9, 10) and / or different weights of objects (8, 9, 10) and / or different impact energy and / or different types of interaction (13) .

20. The method of claim 19, wherein the types of interaction (13) comprises objects (8, 9, 10) falling on the surface (4) , punching on the surface (4) or scratching on the surface (4) .

21. The method of claim 20, wherein the types of interaction (13) comprises objects (8, 9, 10) scratching on the surface (4) .

22. The method of any of claims 18 to 21, wherein the types of interaction (13) comprise a human touch.

23. A software package, wherein the software package is configured for carrying out the method of any of claims 13 to 17 and / or 19 to 22.

Images