Classification systems and methods for external impacts on windows or entrances of enclosed structures.

Abstract

Various embodiments of this disclosure relate to classification systems and methods for classifying external impacts on windows or entrances of enclosed structures. A classification system and method for classifying external impacts on windows or entrances of enclosed structures are disclosed. The classification system includes: a microphone; a barometric pressure sensor; an audio signal preprocessor; a pressure signal preprocessor; an audio signal feature processor; a pressure signal feature processor; a feature combination processor; and a classification processor configured to classify external impacts on windows or entrances by classifying audio features and pressure feature vectors, thereby generating a classification output; wherein the classification processor is configured to execute a first machine learning algorithm, wherein the audio features and pressure feature vectors are fed into the input layer of the first machine learning algorithm, and wherein the classification output is based on the output of the first machine learning algorithm.

Description

Technical Field

[0001] The embodiments relate to a classification system for classifying external impacts on windows or entrances of enclosed structures. Further embodiments relate to a method for classifying external impacts on windows or entrances of enclosed structures. More specifically, this disclosure uses audio signals from a microphone and pressure signals from a barometric pressure sensor to process the classification of external impacts on windows or entrances of enclosed structures. Background Technology

[0002] Several types of systems exist that can detect the presence of intruders in an indoor environment. These are primarily based on motion sensors (e.g., infrared or magnetic sensors) that can monitor multiple windows in a room, as well as contact sensors or glass breakage sensors mounted on doors and windows (based on vibration and applied directly to each glass pane). Image sensors and cameras are also options.

[0003] Signals measured at barometric pressure sensors or microphone sensors can be considered a simpler (less installation work) and cheaper alternative to the methods described above, but alternative methods utilize advanced machine learning techniques and neural network architectures to improve performance compared to many conventional methods.

[0004] Regarding specific detection algorithms, a simple prior art method based on a combination of sensor signals (including barometer readings) and thresholds is disclosed in US 9,940,825 B2. Due to its simplicity, this method suffers from the drawback of being unable to distinguish between different types of events (such as a window opening or tilting, or a window crashing) and is prone to generating excessive false alarms.

[0005] A more complex approach involves applying a Hidden Markov Model to the collected pressure sensor signals. This method requires the identification of different states and the transition probabilities from one state to another, which becomes very challenging if different types of opening events need to be classified (window open / tilt, door open, etc.), and therefore makes the use of simpler probabilistic models more attractive.

[0006] Regarding sensor fusion, a simple combination of pressure sensor and microphone sensor is proposed in EP 3 573 031 A1, in which most rules are applied to the final output of a basic threshold-based algorithm running on the signals from the pressure sensor and the microphone.

[0007] Although this approach is simple, it is not efficient and wise to use two available streams and will impair classification performance. Summary of the Invention

[0008] A classification system for classifying external impacts on windows or entrances of enclosed structures is disclosed. The classification system includes:

[0009] The microphone is configured to produce an audio signal corresponding to the sound within the enclosed structure;

[0010] A barometric sensor is configured to generate a pressure signal corresponding to the air pressure within an enclosed structure.

[0011] An audio signal preprocessor is configured to derive a preprocessed audio signal from an audio signal;

[0012] A pressure signal preprocessor is configured to derive a preprocessed pressure signal from a pressure signal;

[0013] An audio signal feature processor is configured to extract audio features from a preprocessed audio signal to generate an audio feature vector;

[0014] A pressure signal feature processor is configured to extract pressure features from a preprocessed pressure signal to generate a pressure feature vector;

[0015] A feature combination processor is configured to combine audio feature vectors and pressure feature vectors to generate audio feature and pressure feature vectors; and

[0016] The classification processor is configured to classify external impacts on a window or entrance by classifying audio features and pressure feature vectors, thereby producing a classification output;

[0017] The classification processor is configured to execute a first machine learning algorithm, wherein audio features and pressure feature vectors are fed into the input layer of the first machine learning algorithm, and wherein the classification output is based on the output of the first machine learning algorithm.

[0018] The classification system disclosed herein is based on a multi-stream approach, which combines data from sensors of different natures within a common network architecture to enhance each stream with knowledge from another stream early in the processing of the streams.

[0019] The term "classification of external impacts on windows or entrances of enclosed structures" refers to the detection of the nature of external events that affect windows or entrances of enclosed structures. Categories may include:

[0020] a) A window or entrance is being opened;

[0021] b) The window or entry point is closed:

[0022] c) The window or entrance has been hit:

[0023] d) The window or entrance / exit has been damaged, for example, the glass has broken;

[0024] e) The window or entrance has been attacked.

[0025] An enclosed structure can be, for example, a building, a part of a building (e.g., a room), a vehicle, or a part of a vehicle (e.g., a passenger compartment). An entrance can be a door, a luggage compartment cover, etc.

[0026] The term processor refers to an electronic device configured for a specific task. A processor can include hardware or a combination of hardware and software. Different processors may share hardware and / or software components.

[0027] An audio signal feature processor can be configured to transform an audio signal into an audio feature vector, wherein the audio feature vector includes eigenvalues ​​that have characteristics specific to the audio signal. Some features may correspond to the absolute value of the mean deviation of the corresponding audio signal, the phase of the mean deviation of the corresponding audio signal, the absolute value of the standard deviation of the corresponding audio signal, or the phase of the standard deviation of the corresponding audio signal. The maximum value, minimum value, dynamic moment, dynamic moment of the first derivative, or meta-features (e.g., kurtosis, skewness, or quantiles) of the audio signal are also considered.

[0028] Similarly, a pressure signal feature processor can be configured to transform a pressure signal into a pressure feature vector, wherein the pressure feature vector includes eigenvalues ​​that have characteristics specific to the pressure signal. Some features may correspond to the absolute value of the mean deviation of the corresponding pressure signal, the phase of the mean deviation of the corresponding pressure signal, the absolute value of the standard deviation of the corresponding pressure signal, or the phase of the standard deviation of the corresponding pressure signal. Maximum values, minimum values, dynamic torque, dynamic torque of the first derivative, or meta-features (e.g., kurtosis, skewness, or quantiles) of the pressure signal may also be considered.

[0029] The feature combination processor combines the audio feature vector and the pressure feature vector to generate audio feature vectors and pressure feature vectors, which are then fed into the classification processor.

[0030] Specifically, audio features and pressure feature vectors are fed into the input layer of a first machine learning algorithm, and the classification output is based on the output of the first machine learning algorithm.

[0031] The term "machine learning algorithm" refers to all statistical models by which computer systems perform specific tasks without explicit instructions, relying instead on patterns and inference. The structure of a machine learning algorithm is typically predefined. However, the parameters of a machine learning algorithm are usually learned during the pre-operational learning phase and / or the operational phase.

[0032] It should be noted that the proposed technique is flexible enough to incorporate other sensor signals besides those from barometric pressure sensors and microphones, such as those from radar or environmental sensors.

[0033] The core of this disclosure is a mechanism for classifying signals from multiple sources by fusing different streams early in the processing flow. Specifically, signals are measured at microphones and pressure sensors, with the ultimate goal of identifying impacts on windows or entrances of enclosed structures. The proposed classification system includes:

[0034] Multiple feature extraction processors, each of which is implemented to extract or learn the discriminative properties of a signal in the signal based on the specific properties of the corresponding signal, and can effectively reuse subsequent network layers;

[0035] A feature combination processor intelligently combines features from multiple streams, and its parameters can optionally be optimized along with other network parameters; and

[0036] The classification processor includes a first machine learning algorithm that outputs a supervised judgment on the observed events.

[0037] Compared to solutions where the features of the audio signal and the features of the pressure signal are classified independently, combining the features of the audio signal and the features of the pressure signal before they are fed into the first machine learning algorithm provides higher classification accuracy.

[0038] According to some embodiments, the audio signal preprocessor includes an audio signal normalization processor configured to normalize the audio signal, and / or the pressure signal preprocessor includes a pressure signal normalization processor configured to normalize the pressure signal. These features further improve classification accuracy.

[0039] According to some embodiments, the audio signal preprocessor includes an audio signal filter configured to filter the audio signal, and / or the pressure signal preprocessor includes a pressure signal filter configured to filter the pressure signal. Each filter can be implemented, for example, as a high-pass filter or a noise filter. These features further improve classification accuracy.

[0040] According to some embodiments, the audio signal preprocessor includes an audio signal outlier removal processor configured to remove outliers from the audio signal, and / or the pressure signal preprocessor includes a pressure signal outlier removal processor configured to remove outliers from the pressure signal. These features further improve classification accuracy.

[0041] According to some embodiments, the audio signal preprocessor includes an audio signal conversion processor configured to provide the preprocessed audio signal as a two-dimensional preprocessed audio signal time-frequency plot, and / or the pressure signal preprocessor includes a pressure signal conversion processor configured to provide the preprocessed pressure signal as a two-dimensional preprocessed pressure signal time-frequency plot. Each of the signal conversion processors can be implemented as a Mel-spectrum algorithm. These features simplify the subsequent determination of features and thus improve classification accuracy.

[0042] According to some embodiments, an audio signal feature processor is configured to calculate at least a first portion of an audio feature vector using one or more first explicit formulas, and / or a pressure signal feature processor is configured to calculate at least a first portion of a pressure feature vector using one or more second explicit formulas. In such embodiments, at least a first portion of each feature vector is calculated by a predefined formula, which limits the computational workload, compared to embodiments that use only an adaptive algorithm to calculate each feature vector.

[0043] According to some embodiments, the first machine learning algorithm is implemented as a first artificial neural network, wherein the classification processor includes a softmax transfer function (softmax) processor, the softmax processor is arranged at the output side of the first artificial neural network, and the output side of the first artificial neural network outputs a classification output.

[0044] Artificial neural networks have been found to be better suited for classification processors than other machine learning algorithms such as decision trees. Softmax processors are capable of executing the softmax function. The softmax function is used as the final layer in neural network-based classification processors. Specifically, the combination of artificial neural networks and softmax processors limits computational workload and provides high accuracy.

[0045] According to some embodiments, a first artificial neural network includes at least one first convolution processor. A convolution processor is a processor that acts as a convolutional neural network. The name "convolutional neural network" indicates that the network employs a mathematical operation called convolution. Convolution is a special type of linear operation. A convolutional network is simply a neural network that uses convolution in at least one layer instead of general matrix multiplication.

[0046] According to some embodiments, the first artificial neural network includes at least one first convolutional batch normalization processor. The convolutional batch normalization processor is a processor capable of performing a batch normalization algorithm. Batch normalization is a technique used to train very deep neural networks that normalizes the layer inputs for each micro-batch. This has the effect of stabilizing the learning process and significantly reducing the training epochs required to train deep networks.

[0047] The first convolutional processor and the first convolutional batch normalization processor can be arranged in parallel, wherein the outputs of the processors are summed to provide the output of the first artificial neural network. These features limit the computational workload and provide high accuracy.

[0048] According to some embodiments, the first artificial neural network includes at least one first rectified linear unit processor. In the context of an artificial neural network, a rectifier is an activation function defined as the positive part of its argument. It has been shown that rectifiers enable better training of deeper networks compared to other activation functions.

[0049] According to some embodiments, the classification processor includes at least one first pooling layer processor disposed at the output side of a first artificial neural network. The pooling layer processor shapes its input into a 1D array, which is then applied to a subsequent softmax layer.

[0050] According to some embodiments, the classification processor includes at least one second rectified linear unit processor, which is disposed at the output side of the first artificial neural network. The configuration of the second rectified linear unit processor is similar to that of the first rectified linear unit processor.

[0051] According to some embodiments, an audio signal feature processor is configured to execute a second machine learning algorithm, wherein at least a second portion of the audio feature vector is provided at the output layer of the second machine learning algorithm, and / or a pressure signal feature processor is configured to execute a third machine learning algorithm, wherein at least a second portion of the pressure feature vector is provided at the output layer of the third machine learning algorithm. In such embodiments, the computation of each feature vector depends not only on a predefined algorithm, but also at least partially on the machine learning algorithm. Through these features, the computation of the feature vectors can be adapted to the specific environment of the classification system, thereby improving classification accuracy.

[0052] According to some embodiments, the second machine learning algorithm is implemented as a second artificial neural network, and / or the third machine learning algorithm is implemented as a third artificial neural network. The second and / or third artificial networks can be implemented similarly to the first artificial network.

[0053] According to some embodiments, the second artificial neural network includes at least one second convolutional processor, and / or the third artificial neural network includes at least one third convolutional processor. The second and / or third convolutional processors can be implemented similarly to the first convolutional processor.

[0054] According to some embodiments, the second artificial neural network includes at least one second convolutional batch normalization processor, and / or the third artificial neural network includes at least one third convolutional batch normalization processor. The second and / or third convolutional batch normalization processors can be implemented similarly to the first convolutional batch normalization processor.

[0055] According to some embodiments, the audio signal feature processor includes at least one second pooling layer processor at the output side of the second artificial neural network, and / or the pressure signal feature processor includes at least one third pooling layer processor at the output side of the third artificial neural network. The second pooling layer processor and / or the third pooling layer processor can be implemented similarly to the first pooling layer processor.

[0056] According to some embodiments, the audio signal feature processor includes at least one third rectified linear unit processor at the output side of the second artificial neural network, and / or the pressure signal feature processor includes at least one fourth rectified linear unit processor at the output side of the third artificial neural network. The third and / or fourth rectified linear unit processors can be implemented similarly to the first rectified linear unit processor.

[0057] According to some embodiments, the audio signal feature processor includes at least one fourth convolutional processor at the output side of the second artificial neural network, and / or the pressure signal feature processor includes at least one fifth convolutional processor at the output side of the third artificial neural network. The fourth and / or fifth convolutional processors can be implemented similarly to the first convolutional processor.

[0058] According to some embodiments, the feature combination processor is configured to generate audio features and pressure feature vectors by cascading, summing, or element-wise multiplying audio feature vectors and pressure feature vectors.

[0059] Where F a Let F be the audio feature vector, and F p Given pressure feature vectors from two sensor streams (learned or explicitly extracted), and assuming they have been reduced to a compatible size in preprocessing and feature extraction steps, we can have various combinations. Possible options for generating audio and pressure feature vectors are:

[0060] 1) Cascade [F a F p ];

[0061] 2) Total [F] a +F p ];

[0062] 3) Element-wise multiplication [F] a ]*[F p].

[0063] According to some embodiments, a feature combination processor is configured to generate audio features and pressure feature vectors by generating a weighted concatenation, weighted sum, or weighted element-wise multiplication of audio feature vectors and pressure feature vectors, wherein the weights for the audio feature vectors and the weights for the pressure feature vectors are determined based on the audio feature vectors and / or pressure feature vectors by a weight calculation processor.

[0064] Where F a It is an audio feature vector, and F p Given pressure feature vectors from two sensor streams (learned or explicitly extracted), and assuming they have been reduced to a compatible size in preprocessing and feature extraction steps, we can have various combinations. Possible options are:

[0065] 4) Weighted cascade [α] a F a ,β p F p ], where α a β is the weight of the audio feature vector, and β p These are the weights of the pressure characteristic vector;

[0066] 5) Weighted sum [α] a F a +β p F p ], where α a β is the weight of the audio feature vector, and β p These are the weights of the pressure characteristic vector;

[0067] 6) Weighted element-wise multiplication [α] a F a ]*[β p F p ], where α a β is the weight of the audio feature vector, and β p It is the weight of the pressure characteristic vector.

[0068] The weighting takes into account that in some cases, the features of the audio signal may be more unique (characteristic) than those of the pressure signal, and vice versa. Therefore, when the features of the audio signal are more characteristic than those of the pressure signal, classification accuracy can be improved by assigning higher weights to the features of the audio signal than to those of the pressure signal. Conversely, when the features of the audio signal are not more characteristic than those of the pressure signal, classification accuracy can be improved by assigning higher weights to the features of the pressure signal than to those of the audio signal. In this case, the weights are determined at least in part using a predefined algorithm, which limits the computational workload.

[0069] According to some embodiments, a feature combination processor is configured to generate audio and pressure feature vectors by producing a weighted concatenation, weighted sum, or weighted element-wise multiplication of audio and pressure feature vectors, wherein the weights for the audio feature vectors and the weights for the pressure feature vectors are determined based on the hidden states of a first artificial neural network by a weight calculation processor. In this case, the weights are determined at least in part using learning data from the first artificial network, which further improves classification accuracy.

[0070] According to some embodiments, the feature combination processor is configured to add a first offset vector to the audio feature vector and / or add a second offset vector to the pressure feature vector before generating the audio feature and pressure feature vectors, wherein the first offset vector and / or the second offset vector are determined based on the audio feature vector and / or the pressure feature vector. In this case, the offset is determined at least in part using a predefined algorithm that limits the computational workload. Such features improve classification accuracy.

[0071] According to some embodiments, the feature combination processor is configured to add a first offset vector to the audio feature vector and / or add a second offset vector to the pressure feature vector before generating the audio feature and pressure feature vectors, wherein the first offset vector and / or the second offset vector is determined based on the hidden state of a first artificial neural network. In this case, the offset is determined at least in part using learning data from the first artificial network, which further improves classification accuracy. Such features improve classification accuracy.

[0072] A method for classifying external impacts on windows or entrances of enclosed structures is also disclosed. The method includes the following steps:

[0073] By using a microphone, an audio signal corresponding to the sound within the enclosed structure is generated;

[0074] By using a barometric pressure sensor, a pressure signal corresponding to the air pressure inside the enclosed structure is generated;

[0075] By using an audio signal preprocessor, a preprocessed audio signal is derived from the audio signal;

[0076] By using a pressure signal preprocessor, a preprocessed pressure signal is derived from the pressure signal;

[0077] By using an audio signal feature processor, audio features are extracted from the preprocessed audio signal to generate an audio feature vector;

[0078] Pressure features are extracted from the preprocessed pressure signal using a pressure signal feature processor to generate a pressure feature vector.

[0079] Audio feature vectors and pressure feature vectors are combined using a feature combination processor to generate audio feature vectors and pressure feature vectors; and

[0080] By using a classification processor, audio features and pressure feature vectors are classified to classify external impacts on a window or entrance, thereby producing a classification output;

[0081] The classification processor is executing a first machine learning algorithm, in which audio features and pressure feature vectors are fed into the input layer of the first machine learning algorithm, and the classification output is based on the output of the first machine learning algorithm.

[0082] Computer programs for implementing the methods disclosed herein when run on a computer are also disclosed. Attached Figure Description

[0083] Preferred embodiments of the invention will then be discussed with reference to the accompanying drawings, in which:

[0084] Figure 1 A schematic diagram of a first embodiment of a classification system for classifying external impacts on windows or entrances of enclosed structures according to the present disclosure is shown.

[0085] Figure 2 A schematic diagram of a second embodiment of a classification system for classifying external impacts on windows or entrances of enclosed structures according to the present disclosure is shown.

[0086] Figure 3 A schematic diagram of a third embodiment of a classification system for classifying external impacts on windows or entrances of enclosed structures according to the present disclosure is shown.

[0087] Figure 4 A schematic diagram of a fourth embodiment of a classification system for classifying external impacts on windows or entrances of enclosed structures according to the present disclosure is shown.

[0088] Figure 5 A schematic diagram of a fifth embodiment of a classification system for classifying external impacts on windows or entrances of enclosed structures according to the present disclosure is shown.

[0089] Figure 6 A schematic diagram of a sixth embodiment of a classification system for classifying external impacts on windows or entrances of enclosed structures according to the present disclosure is shown.

[0090] Figure 7 The typical pressure signal and typical audio signal that change over time are shown when the glass window is opened;

[0091] Figure 8The typical pressure signal and typical audio signal that change over time are shown when the glass window is closed.

[0092] Figure 9 The typical pressure signal and typical audio signal that change over time when a glass window is hit are shown.

[0093] Figure 10 The typical pressure signal and typical audio signal that change over time when a glass window breaks are shown.

[0094] Figure 11 The diagram shows typical pressure signals and typical audio signals over time when a glass window is struck; and...

[0095] Figure 12 This illustrates possible implementations of the convolutional processor described in this paper.

[0096] In the following description, identical or equivalent elements or elements having the same or equivalent functions are indicated by identical or equivalent reference numerals. Detailed Implementation

[0097] In the following description, numerous details are set forth to provide a more thorough explanation of embodiments of the present disclosure. However, it will be apparent to those skilled in the art that embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present disclosure. Additionally, unless otherwise specifically indicated, features of the different embodiments described below can be combined with each other.

[0098] Figure 1 A schematic diagram of a first embodiment of a classification system 1 according to the present disclosure is shown. The classification system 1 is used to classify external impacts on windows or entrances of enclosed structures. The classification system includes:

[0099] Microphone 2 is configured to generate an audio signal AS corresponding to the sound within the enclosed structure;

[0100] The bar pressure sensor 3 is configured to generate a pressure signal PS corresponding to the air pressure within the enclosed structure;

[0101] Audio signal preprocessor 4 is configured to derive a preprocessed audio signal PAS from the audio signal AS;

[0102] Pressure signal preprocessor 5 is configured to derive a preprocessed pressure signal PPS from the pressure signal PS;

[0103] The audio signal feature processor 6 is configured to extract audio features from the preprocessed audio signal PAS to generate an audio feature vector AFV;

[0104] The pressure signal feature processor 7 is configured to extract pressure features from the preprocessed pressure signal PPS to generate a pressure feature vector PFV.

[0105] Feature combination processor 8 is configured to combine audio feature vector AFV and pressure feature vector PVF to generate audio feature and pressure feature vector APV; and

[0106] Classification processor 9 is configured to classify external impacts on a window or entrance by classifying audio features and pressure feature vectors (APV) to produce a classification output CO.

[0107] The classification processor 9 is configured to execute a first machine learning algorithm, wherein audio features and pressure feature vectors (APVs) are fed into the input layer of the first machine learning algorithm, and wherein the classification output CO is based on the output of the first machine learning algorithm.

[0108] According to some embodiments, the audio signal preprocessor 4 includes an audio signal filter configured to filter the audio signal AS, and / or the pressure signal preprocessor 5 includes a pressure signal filter configured to filter the pressure signal PS.

[0109] According to some embodiments, the audio signal preprocessor 4 includes an audio signal outlier removal processor configured to remove outliers from the audio signal AS, and / or the pressure signal preprocessor 5 includes a pressure signal outlier removal processor configured to remove outliers from the pressure signal PS.

[0110] According to some embodiments, the audio signal feature processor 6 is configured to calculate at least a first portion of the audio feature vector AFV using one or more first explicit formulas, and / or the pressure signal feature processor 7 is configured to calculate at least a first portion of the pressure feature vector PFV using one or more second explicit formulas.

[0111] According to some embodiments, the feature combination processor 8 is configured to generate an audio feature vector and a pressure feature vector APV by cascading, summing, or element-wise multiplying an audio feature vector AFV and a pressure feature vector PFV.

[0112] On the other hand, this disclosure relates to a method for classifying external impacts on windows or entrances of enclosed structures. The method includes the following steps:

[0113] By using microphone 2, an audio signal AS corresponding to the sound within the enclosed structure is generated;

[0114] By using the air pressure sensor 3, a pressure signal PS corresponding to the air pressure inside the enclosed structure is generated;

[0115] The preprocessed audio signal PAS is derived from the audio signal AS by using the audio signal preprocessor 4;

[0116] The preprocessed pressure signal PPS is derived from the pressure signal PS by using the pressure signal preprocessor 5.

[0117] Audio features are extracted from the preprocessed audio signal PAS by using the audio signal feature processor 6 to generate the audio feature vector AFV;

[0118] Pressure features are extracted from the preprocessed pressure signal PPS using the pressure signal feature processor 7 to generate the pressure feature vector PFV.

[0119] By using the feature combination processor 8, the audio feature vector AFV and the pressure feature vector PFV are combined to generate the audio feature and pressure feature vector APV; and

[0120] External impacts on a window or entrance are classified by using classification processor 9 to classify audio features and pressure feature vectors (APV) to produce classification output CO.

[0121] The classification processor 9 executes a first machine learning algorithm, wherein audio features and pressure feature vectors (APVs) are fed into the input layer of the first machine learning algorithm, and wherein the classification output CO is based on the output of the first machine learning algorithm.

[0122] On the other hand, this disclosure relates to a computer program for implementing the methods discussed herein when run on a computer.

[0123] This disclosure relates to a smart combination—sensor fusion—of a pressure signal PS from a barometric pressure sensor 3 and an audio signal AS from a microphone 2, used to identify specific types of events (such as a window or house door being opened or closed, or a window glass breaking). In the context of smart homes and home scene analytics, the identification of such events can be used to detect when a burglar is entering a residence or house. Similar events are also associated with other enclosed environments (e.g., a car).

[0124] It has been observed that the pressure signal PS measured at the pressure sensor 3 exhibits a characteristic pattern depending on whether a window or door in the house where the pressure sensor 3 is placed is open, tilted, broken, etc. As long as the doors in the house are not tightly closed, the pattern of the pressure signal PS remains unique not only in the affected room but also throughout the entire house. Similarly, when events such as a slam, impact, or window strike occur near the microphone 2, the audio signal AS also exhibits very unique behavior. However, depending on the event, the specific patterns recorded at the microphone 2 and the pressure sensor 3 have different characteristics, which can be more or less obvious.

[0125] according to Figure 1 The audio signal AS and the pressure signal PS are first preprocessed separately by the audio signal preprocessor 4 and the pressure signal preprocessor 5, respectively. This includes high-pass filtering or noise filtering, signal normalization, outlier removal, etc., as needed, and can also generate a two-dimensional time-frequency plot (e.g., a feature map).

[0126] In the second step, the preprocessed audio signal PAS is fed to the audio signal feature processor 6, and the preprocessed pressure signal PPS is fed to the pressure signal feature processor 7. The signals PAS and PPS remain separate to allow for more appropriate and signal-specific learning of the discriminative attributes of the multi-stream data. The extracted audio feature vector AFV and the extracted pressure feature vector PFV can be explicitly calculated from a certain indicator (e.g., derivative, phase, mean, etc.) or can be implicitly learned using a separate neural network.

[0127] At this point, the extracted audio feature vector (AFV) and the extracted pressure feature vector (PFV) can be fused or combined to compute the audio feature vector and the pressure feature vector (APV). This operation can be implemented in various ways, ranging from simple feature concatenation to more complex weighted combinations, where weights (and offsets) are learned using separate attention mechanisms and separate backpropagation processes. The choice between different combination methods will depend on the specific architecture, subsequent architectures, their convergence properties, and the nature of the data (and the extracted features).

[0128] Now, the calculated audio features and pressure feature vectors (APVs) are sent to classification processor 9, which can be implemented as a general machine learning classification algorithm or, more conveniently, as a neural network followed by a softmax layer. Depending on which features were extracted in the previous steps and how they were combined, the specific network can be a fully combined network or one with additional convolutional blocks.

[0129] Figure 2 A schematic diagram of a second embodiment of a classification system 1 according to the present invention is shown. The classification system 1 is used to classify external impacts on windows or entrances of enclosed structures.

[0130] According to some embodiments, the audio signal preprocessor 4 includes an audio signal normalization processor 10 configured to normalize the audio signal AS, and / or the pressure signal preprocessor 5 includes a pressure signal normalization processor 11 configured to normalize the pressure signal PS.

[0131] According to some embodiments, the audio signal preprocessor 4 includes an audio signal conversion processor 12, which is configured to provide a preprocessed audio signal PAS as a two-dimensional preprocessed audio signal time-frequency diagram.

[0132] According to some embodiments, the first machine learning algorithm is implemented as a first artificial neural network 13, wherein the classification processor 9 includes a softmax processor 14, the softmax processor 14 is arranged at the output side of the first artificial neural network 13 and the output side of the first artificial neural network 13 outputs a classification output CO.

[0133] According to some embodiments, the audio signal feature processor 4 is configured to execute a second machine learning algorithm, wherein at least a second portion of the audio feature vector AFV is provided at the output layer of the second machine learning algorithm.

[0134] According to some embodiments, the second machine learning algorithm is implemented as a second artificial neural network 15.

[0135] According to some embodiments, the second artificial neural network 15 includes at least one second convolution processor 16.

[0136] According to some embodiments, the second artificial neural network 15 includes at least one second convolutional batch normalization processor 17.

[0137] According to some embodiments, the audio signal feature processor 6 includes at least one second pooling layer processor 18 at the output side of the second artificial neural network 15.

[0138] In data normalization, the first preprocessing step may include calculations for the audio signal AS and the pressure signal PS. Subsequently, a Mel-scaled spectrogram can be calculated for the audio signal AS. With a window size of 40 milliseconds and an overlap of 20 milliseconds, the audio spectrogram may include 40 Mel coefficients. Clearly, other combinations of window size, overlap, and the number of Mel coefficients are also possible.

[0139] For a pressure signal PS, the maximum, minimum, and average values, as well as the dynamic torque and the dynamic torque of the first derivative, can be extracted as indicators. Other indicators or meta-features should also be considered, such as kurtosis, skewness, and quantiles.

[0140] Regarding the audio signal, features are learned from the preprocessed audio signal PAS in the form of an audio time-frequency representation using a second neural network 15. This includes a combination of a convolutional processor 16 and a convolutional batch normalization processor 17. Here, the convolutional batch normalization processor 16 indicates a shortcut where the same preprocessed audio signal PAS entering the convolutional processor 15 is used as input to the convolutional batch normalization processor 17. This improves the convergence of the second artificial network 15. A second pooling layer processor 18 shapes the output of the second neural network 15 into a 1D array.

[0141] The calculated stress features and trained audio features are then cascaded and fed into a feedforward neural network 13, followed by a softmax processor 14, which ultimately passes the classification probability.

[0142] Figure 3 A schematic diagram of a third embodiment of a classification system 1 according to the present invention is shown. The classification system 1 is used to classify external impacts on windows or entrances of enclosed structures.

[0143] According to some embodiments, the audio signal feature processor 6 includes at least one third rectified linear unit processor 19 at the output side of the second artificial neural network 15.

[0144] According to some embodiments, the audio signal feature processor 6 includes at least one fourth convolution processor 20 at the output side of the second artificial neural network 15.

[0145] exist Figure 3 In the middle, the combination of convolution processor 16 and convolution batch normalization processor 16 is followed by a third rectified linear unit processor 19 and a fourth convolution processor 20. The second pooling layer processor 18 shapes the output of the second convolution processor 16 into a 1D array.

[0146] Figure 4 A schematic diagram of a fourth embodiment of a classification system 1 according to the present disclosure is shown. The classification system 1 is used to classify external impacts on windows or entrances of enclosed structures.

[0147] According to some embodiments, the pressure signal feature processor 6 is configured to execute a third machine learning algorithm, wherein at least a second portion of the pressure feature vector PFV is provided at the output layer of the third machine learning algorithm.

[0148] According to some embodiments, the third machine learning algorithm is implemented as a third artificial neural network 21.

[0149] According to some embodiments, the third artificial neural network 21 includes at least one third convolution processor.

[0150] According to some embodiments, the pressure signal feature processor 7 includes at least one third pooling layer processor at the output side of the third artificial neural network 21.

[0151] According to some embodiments, the pressure signal feature processor 7 includes at least one fourth rectified linear unit processor at the output side of the third artificial neural network 21.

[0152] According to some embodiments, the pressure signal feature processor 7 includes at least one fifth convolution processor at the output side of the third artificial neural network.

[0153] According to some embodiments, the pressure signal feature processor 7 includes at least one fifth convolution processor at the output side of the third artificial neural network.

[0154] According to some embodiments, the first artificial neural network 13 includes at least one first convolution processor 22.

[0155] According to some embodiments, the first artificial neural network 13 includes at least one first convolutional batch normalization processor 23.

[0156] According to some embodiments, the first artificial neural network 13 includes at least one first rectified linear unit processor 24.

[0157] According to some embodiments, the classification processor 9 includes at least one first pooling layer processor 25 disposed on the output side of the first artificial neural network 13.

[0158] According to some embodiments, the classification processor 9 includes at least one second rectified linear unit processor 26 disposed at the output side of the first artificial neural network 13.

[0159] exist Figure 4 In this process, the pressure signal PS is normalized to create a preprocessed pressure signal PPS. The pressure feature vector PVF is learned from the preprocessed pressure signal PPS through a third artificial neural network 21. For the audio signal AS, the same method as in the previous embodiment is implemented.

[0160] The outputs of the audio signal feature processor 6 and the pressure signal feature processor 7 are summed and then fed into a first neural network 13, which includes a second rectified linear unit processor 26, a first convolutional processor 22, and a first convolutional batch normalization processor 23. The sum of the outputs of the first convolutional processor 22 and the first convolutional batch normalization processor 23 is input to the second rectified linear unit processor 26, followed by a first pooling layer processor 25 and a softmax processor 14, for generating a classification output CO.

[0161] Figure 5A schematic diagram of a fifth embodiment of classification system 1 is shown. Classification system 1 is used to classify external impacts on windows or entrances of the closed structure of this disclosure.

[0162] According to some embodiments, the pressure signal preprocessor 5 includes a pressure signal conversion processor 27, which is configured to provide a preprocessed pressure signal PPS as a two-dimensional preprocessed pressure signal time-frequency diagram.

[0163] exist Figure 5 Another possible embodiment is depicted. The pressure signal PS and the audio signal AS are normalized and spectra are calculated. The spectra from the pressure signal PS are calculated using FFT, which is cascaded over a shifted window. In this embodiment, the convolutional batch normalization processor is not used in the audio signal feature processor 6 only after the feature combination processor 8. Additionally, a second pooling layer processor 18 is required after the fourth convolution processor 20 to be able to combine the audio feature vector AFV and the pressure feature vector PFV learned from the pressure signal feature processor 7. After combining the learned feature vectors AFV and PFV, the classification processor 9 employs a method similar to... Figure 4 The same network architecture is used to obtain the classification output CO.

[0164] Figure 6 A schematic diagram of a sixth embodiment of classification system 1 is shown. Classification system 1 is used to classify external impacts on windows or entrances of closed structures of the present disclosure.

[0165] According to some embodiments, the feature combination processor 8 is configured to generate audio feature vectors and pressure feature vectors APV by generating a weighted concatenation, weighted summation, or weighted element-wise multiplication of audio feature vector AFV and pressure feature vector PVF, wherein the weights AFW of audio feature vector AFV and PFV of pressure feature vector PFV are determined by a weight calculation processor 28 based on the audio feature vector AFV and / or pressure feature vector PFV.

[0166] According to some embodiments, the feature combination processor 8 is configured to generate audio feature vectors and pressure feature vectors APV by generating a weighted concatenation, weighted summation, or weighted element-wise multiplication of audio feature vector AFV and pressure feature vector PFV, wherein the weights of the audio feature vector AFV and the pressure feature vector PFV are determined by the weight calculation processor 28 based on the hidden state HS of the first artificial neural network 13.

[0167] According to some embodiments, the feature combination processor 8 is configured to add a first offset vector to the audio feature vector AFV and / or add a second offset vector to the pressure feature vector PFV before generating the audio feature vector and the pressure feature vector APV, wherein the first offset vector and / or the second offset vector are determined based on the audio feature vector AFV and / or the pressure feature vector PFV.

[0168] According to some embodiments, the feature combination processor 8 is configured to add a first offset vector to the audio feature vector AFV and / or add a second offset vector to the pressure feature vector PFV before generating the audio feature and pressure feature vector APV, wherein the first offset vector and / or the second offset vector are determined based on the hidden state HS of the first artificial neural network 13.

[0169] like Figure 6 As shown, weights and biases can be learned using the standard backpropagation algorithm and gradient descent implemented as a multilayer perceptron network, or by using a more sophisticated attention mechanism that takes features (and, if necessary, the hidden states of the classification network below) as input.

[0170] Regarding the weights (and offsets) of the feature combination processor 8, they can be learned directly, or they can be obtained through an attention-like mechanism, in which we first compute the score e. i Then, softmax is used for normalization.

[0171] [e1, ..., e, N] = tanh(V[F a (1), ..., F b (N), H]+b) (1)

[0172] a t =softmax(e t (2)

[0173] Where H is the hidden state of the global network (if available), V and b are the parameter matrix and vector to be learned, and a i It is the final weight associated with the preprocessed features.

[0174] For alternative locations, the following can be optimized directly:

[0175] [a1, ..., a] N ] = softmax(V[F o (1), ..., F b (N), H]+b) (3)

[0176] It should be noted that the individual networks and their hyperparameters in (1) and (2) are jointly trained using their respective global network architectures, so that the overall loss function is minimized.

[0177] Additionally, it should be noted that H in equations (1) and (3) is optional and becomes part of the optimization when the previous hidden state is available. This occurs, for example, when a recurrent neural network is deployed instead of a convolutional neural network as the first neural network 13.

[0178] Figure 7 The diagram shows a typical pressure signal PS and a typical audio signal AS that vary over time when a glass window is opened.

[0179] Figure 8 The typical pressure signal PS and the typical audio signal AS as they change over time when the glass window is closed are shown.

[0180] Figure 9 The diagram shows the typical pressure signal PS and the typical audio signal AS as they change over time when a glass window is hit.

[0181] Figure 10 The diagram shows a typical pressure signal PS and a typical audio signal AS that change over time when a glass window breaks.

[0182] Figure 11 The diagram shows the typical pressure signal PS and the typical audio signal AS as they change over time when a glass window is struck.

[0183] Figures 7 to 11 The diagram shows some typical signals measured at the pressure sensor and microphone when different types of events occur. It can be seen that the following signals exhibit very distinct patterns depending on whether the glass window is opened, closed, struck, slammed, or broken. Furthermore, it can be observed that the audio signal AS and the pressure signal PS differ significantly in properties, with the audio signal AS exhibiting very specific patterns for slamming, striking, and glass breakage, and the pressure signal PS showing more distinct behavior when attempting to separate open / close events. Similar considerations can be made by observing the spectra of different categories.

[0184] Figure 12Possible implementations of the convolutional processors 16, 20, 21, and 22 described herein are shown. Each of the convolutional processors 16, 20, 21, and 22 explained in this disclosure may (in this order) include the following sub-units: a first convolutional layer 29 comprising eight two-dimensional kernels, a first batch normalization layer 30, a first rectified linear layer (ReLU) 31, a second convolutional layer 32 comprising five two-dimensional kernels, a second batch normalization layer 33, a second rectified linear layer 34, a third convolutional layer 35 comprising three two-dimensional kernels, and a third batch normalization layer 36.

[0185] Table 1 compares the single-stream and multi-stream methods item by item. It can be seen that the simplicity and combination of the stress algorithm and the audio algorithm already provides better performance than each algorithm alone. However, in addition to saving computational resources by sharing processing blocks across different streams, the intelligent multi-stream method described in this disclosure also provides better results than using majority rules. (See reference...) Figure 4 The described embodiment using the classification system has achieved 98% accuracy. However, other embodiments disclosed herein provide similar accuracy.

[0186]

[0187]

[0188] Table 1: Overview and Algorithm Comparison

[0189] Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method, where a box or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of the corresponding box or item or feature of the corresponding apparatus.

[0190] The above description is merely exemplary, and it should be understood that modifications and variations of the arrangements and details described herein will be readily apparent to those skilled in the art. Therefore, the intent of the invention is limited only by the scope of the appended claims, and not by the specific details presented in the foregoing description and explanation.

Claims

1. A classification system for classifying external impacts on windows or entrances of enclosed structures, the classification system comprising: The microphone (2) is configured to generate an audio signal (AS) corresponding to the sound within the enclosed structure; A barometric pressure sensor (3) is configured to generate a pressure signal (PS) corresponding to the air pressure within the enclosed structure; An audio signal preprocessor (4) is configured to derive a preprocessed audio signal (PAS) from the audio signal (AS); The pressure signal preprocessor (5) is configured to derive a preprocessed pressure signal (PPS) from the pressure signal (PS); An audio signal feature processor (6) is configured to extract audio features from the preprocessed audio signal (PAS) to generate an audio feature vector (AFV); The pressure signal feature processor (7) is configured to extract pressure features from the preprocessed pressure signal (PPS) to generate a pressure feature vector (PFV); The feature combination processor (8) is configured to combine the audio feature vector (AFV) and the pressure feature vector (PFV) to generate an audio feature and a pressure feature vector (APV); and The classification processor (9) is configured to classify the external impact on the window or the entrance by classifying the audio features and the pressure feature vector (APV) to produce a classification output (CO); The classification processor (9) is configured to execute a first machine learning algorithm, wherein the audio features and pressure feature vectors (APV) are fed into the input layer of the first machine learning algorithm, and wherein the classification output (CO) is the output based on the first machine learning algorithm.

2. The classification system according to claim 1, wherein the audio signal preprocessor (4) includes an audio signal normalization processor (10) configured to normalize the audio signal (AS), and / or wherein the pressure signal preprocessor (5) includes a pressure signal normalization processor (11) configured to normalize the pressure signal (PS).

3. The classification system according to claim 1 or 2, wherein the audio signal preprocessor (4) includes an audio signal filter configured to filter the audio signal (AS), and / or wherein the pressure signal preprocessor (5) includes a pressure signal filter configured to filter the pressure signal (PS).

4. The classification system according to claim 1 or 2, wherein the audio signal preprocessor (4) includes an audio signal outlier removal processor configured to remove outliers from the audio signal (AS), and / or wherein the pressure signal preprocessor (5) includes a pressure signal outlier removal processor configured to remove outliers from the pressure signal (PS).

5. The classification system according to claim 1 or 2, wherein the audio signal preprocessor (4) includes an audio signal conversion processor (12) configured to provide the preprocessed audio signal (PAS) as a two-dimensional preprocessed audio signal time-frequency diagram, and / or wherein the pressure signal preprocessor (5) includes a pressure signal conversion processor (27) configured to provide the preprocessed pressure signal (PPS) as a two-dimensional preprocessed pressure signal time-frequency diagram.

6. The classification system according to claim 1 or 2, wherein the audio signal feature processor (6) is configured to calculate at least a first portion of the audio feature vector (AFV) by using one or more first explicit formulas, and / or wherein the pressure signal feature processor (7) is configured to calculate at least a first portion of the pressure feature vector (PFV) by using one or more second explicit formulas.

7. The classification system according to claim 1, wherein the first machine learning algorithm is implemented as a first artificial neural network (13), wherein the classification processor (9) includes a flexible maximum transfer function processor (14), the flexible maximum transfer function processor (14) is arranged at the output side of the first artificial neural network (13) and the output side of the first artificial neural network (13) outputs the classification output (CO).

8. The classification system according to claim 7, wherein the first artificial neural network (13) comprises at least one first convolutional processor (22).

9. The classification system according to claim 7 or 8, wherein the first artificial neural network (13) includes at least one first convolutional batch normalization processor (23).

10. The classification system according to claim 7 or 8, wherein the first artificial neural network (13) comprises at least one first rectified linear unit processor (24).

11. The classification system according to claim 7 or 8, wherein the classification processor (9) includes at least one first pooling layer processor (25) arranged on the output side of the first artificial neural network (13).

12. The classification system according to claim 7 or 8, wherein the classification processor (9) includes at least one second rectified linear unit processor (26) arranged on the output side of the first artificial neural network (13).

13. The classification system according to claim 1, wherein the audio signal feature processor (4) is configured to execute a second machine learning algorithm, wherein at least a second portion of the audio feature vector (AFV) is provided at the output layer of the second machine learning algorithm, and / or wherein the pressure signal feature processor (6) is configured to execute a third machine learning algorithm, wherein at least a second portion of the pressure feature vector (PFV) is provided at the output layer of the third machine learning algorithm.

14. The classification system according to claim 13, wherein the second machine learning algorithm is implemented as a second artificial neural network (15), and / or wherein the third machine learning algorithm is implemented as a third artificial neural network (21).

15. The classification system according to claim 14, wherein the second artificial neural network (15) includes at least one second convolutional processor (16), and / or the third artificial neural network (21) includes at least one third convolutional processor.

16. The classification system according to claim 14 or 15, wherein the second artificial neural network (15) includes at least one second convolutional batch normalization processor (17), and / or wherein the third artificial neural network (21) includes at least one third convolutional batch normalization processor.

17. The classification system according to claim 14 or 15, wherein the audio signal feature processor (6) includes at least one second pooling layer processor (18) at the output side of the second artificial neural network (15), and / or wherein the pressure signal feature processor (7) includes at least one third pooling layer processor at the output side of the third artificial neural network (21).

18. The classification system according to claim 14 or 15, wherein the audio signal feature processor (6) includes at least one third rectified linear unit processor (19) at the output side of the second artificial neural network (15), and / or wherein the pressure signal feature processor (7) includes at least one fourth rectified linear unit processor at the output side of the third artificial neural network (21).

19. The classification system according to claim 14 or 15, wherein the audio signal feature processor (6) includes at least one fourth convolution processor (20) at the output side of the second artificial neural network (15), and / or wherein the pressure signal feature processor (7) includes at least one fifth convolution processor at the output side of the third artificial neural network (21).

20. The classification system according to any one of claims 1, 2, 7, 8, 13 and 14, wherein the feature combination processor (8) is configured to generate the audio feature and pressure feature vector (APV) by generating a concatenation, summation or element-wise multiplication of the audio feature vector (AFV) and the pressure feature vector (PFV).

21. The classification system according to any one of claims 1, 2, 7, 8, 13 and 14, wherein the feature combination processor (8) is configured to generate the audio feature and pressure feature vector (APV) by generating a weighted concatenation, weighted sum or weighted element-wise multiplication of the audio feature vector (AFV) and the pressure feature vector (PFV), wherein the weights (AFW) for the audio feature vector (AFV) and the weights (PFW) for the pressure feature vector (PFV) are determined by the weight calculation processor (28) based on the audio feature vector (AFV) and / or the pressure feature vector (PFV).

22. The classification system according to claim 7 or 8, wherein the feature combination processor (8) is configured to generate the audio feature and the pressure feature vector (APV) by generating a weighted concatenation, a weighted sum, or a weighted element-wise multiplication of the audio feature vector (AFV) and the pressure feature vector (PFV), wherein the weights (AFW) for the audio feature vector (AFV) and the weights (PFW) for the pressure feature vector (PFV) are determined by the weight calculation processor (28) based on the hidden states (HS) of the first artificial neural network (13).

23. The classification system according to any one of claims 1, 2, 7, 8, 13 and 14, wherein the feature combination processor (8) is configured to add a first offset vector to the audio feature vector (AFV) and / or add a second offset vector to the pressure feature vector (PFV) before generating the audio feature and pressure feature vector (APV), wherein the first offset vector and / or the second offset vector are determined based on the audio feature vector (AFV) and / or the pressure feature vector (PFV).

24. The classification system according to claim 7 or 8, wherein the feature combination processor (8) is configured to add a first offset vector to the audio feature vector (AFV) and / or add a second offset vector to the pressure feature vector (PFV) before generating the audio feature and pressure feature vector (APV), wherein the first offset vector and / or the second offset vector is determined based on the hidden state (HS) of the first artificial neural network (13).

25. A method for classifying external impacts on windows or entrances of enclosed structures, the method comprising the steps of: By using a microphone (2), an audio signal (AS) corresponding to the sound within the enclosed structure is generated; By using a pressure sensor (3), a pressure signal (PS) corresponding to the air pressure inside the enclosed structure is generated; A preprocessed audio signal (PAS) is derived from the audio signal (AS) using an audio signal preprocessor (4); A preprocessed pressure signal (PPS) is derived from the pressure signal (PS) using a pressure signal preprocessor (5); Audio features are extracted from the preprocessed audio signal (PAS) using an audio signal feature processor (6) to generate an audio feature vector (AFV); Pressure features are extracted from the preprocessed pressure signal (PPS) using a pressure signal feature processor (7) to generate a pressure feature vector (PFV); The audio feature vector (AFV) and the pressure feature vector (PFV) are combined using a feature combination processor (8) to generate an audio feature and a pressure feature vector (APV); and The external impact on the window or the entrance is classified by classifying the audio features and pressure feature vector (APV) using a classification processor (9) to generate a classification output (CO). The classification processor (9) executes a first machine learning algorithm, wherein the audio features and pressure feature vectors (APV) are fed into the input layer of the first machine learning algorithm, and wherein the classification output (CO) is based on the output of the first machine learning algorithm.

Images