Non-intrusive dual-channel smart home security monitoring method

Abstract

The invention discloses a non-intrusive dual-channel smart home security monitoring method, and belongs to the technical field of network security. Aiming at the problem that the existing smart home privacy security monitoring method has defects, an audio channel takes a pre-training model Vgish as an audio feature extractor, and a pixel spectrogram corresponding to each section of 960ms frame length audio obtained through discrete Fourier framing processing is vectorized into 128-dimensional feature representation; then, a support vector machine (SVM-RBF) classifier identifies and obtains an audio event category sequence Ea; five-tuple information contained in event traffic is extracted in a traffic channel to be matched with a predefined event-rule base mode, a traffic event category sequence Et is recognized, and then event alignment, verification and fusion are carried out on the sequences Es and Et so as to monitor the security problem in the smart home. According to the method, the accuracy of event identification of the two-channel model is proved through an experiment carried out on a self-built experiment platform, and meanwhile, the necessity of safety monitoring by adopting two-channel fusion is also proved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a non-intrusive dual-channel smart home security monitoring method. Background Art

[0002] The smart home sector is experiencing significant growth as Internet of Things (IoT) technology continues to advance and consumers are increasingly demanding a higher standard of living. According to Grand View, the market is expected to grow at a compound annual growth rate (CAGR) of 27.07% between 2023 and 2030, reaching a market value of $537.01 billion by 2030. The penetration of smart homes in households is expected to rise from 18.9% in 2024 to 33.2% in 2028. A smart home is a residence equipped with smart IoT devices that integrates Internet connectivity, automation, and remote control capabilities through an interconnected platform to improve comfort, convenience, energy efficiency, and other aspects of the experience. Users of smart homes can remotely monitor and control devices through mobile applications or voice assistants, or set automation rules to enable devices to operate automatically based on predefined conditions (so-called "home automation"). However, despite the many advantages that smart homes bring, the complex interconnectivity and interactivity between IoT devices, platforms, and applications also expands the attack surface, making smart homes more vulnerable to various security and safety vulnerabilities.

[0003] Anomaly detection is generally regarded as a fundamental technology for detecting vulnerabilities, faults, misoperations, and potential attacks in smart homes by identifying behaviors that deviate from normal patterns in the system. Fahad et al. used a probabilistic neural network (PNN) to pre-segment activity instances to extract features, and further used an H2O autoencoder to detect abnormal activities in smart home environments. Zamani et al. proposed an autoencoder (AE) based on two one-dimensional convolutional neural networks (1D-CNN) and a temporal convolutional network (TCN) to detect anomalies in the energy consumption of smart home devices. Ramapatruni et al. used a hidden Markov model (HMM) trained on normal smart home activities to detect abnormal behaviors. However, existing solutions generally have at least one of the following limitations, which significantly weakens their overall effectiveness:

[0004] 1. Most existing solutions rely on IoT event sequences extracted from activity notification messages reported by devices to learn semantic association patterns between events and identify abnormal events that deviate from these patterns. The effectiveness and reliability of such methods rely heavily on the integrity of input events. However, false, erroneous, lost, or delayed events may occur due to a variety of factors such as system failures, misoperation or configuration errors, and attacks. In the absence of accurate input, anomaly detection models cannot make precise or reliable judgments.

[0005] 2. Smart homes may contain a variety of non-intelligent (i.e. "dumb") devices, whose activities are not recorded as events and reported, and therefore cannot be used for anomaly detection. If the activities of these non-intelligent devices can be included in the input data of anomaly detection, it is expected that the detection performance will be significantly improved.

[0006] 3. Existing methods lack versatility in fragmented smart home environments, especially in the absence of reference monitors. In modern smart home systems, multiple heterogeneous IoT devices are connected to platforms, but they exhibit the following characteristics: (1) each device is only connected to some platforms; (2) each platform only supports some devices. Therefore, no single platform can provide a comprehensive global view of all devices. Although a standalone host with access to all devices may solve this problem, this requires that all platforms provide device access APIs. However, due to the heterogeneity of the platforms, this requirement is often difficult to achieve. Summary of the invention

[0007] The VGGish pre-trained model was trained by Google based on the YouTube-8M video dataset. It was originally designed for visual recognition tasks. Since it has good generalization ability in the scene-based audio feature extraction process, the present invention chooses to migrate the model and apply it to the audio data feature extraction part in the smart home scene. The deep learning architecture model consists of 7 layers of Conv2d layers and Max_Pool2d layers. By using convolution and pooling operations to extract the hierarchical and potential feature representations of the spectrogram, a 128-dimensional feature representation of each 960ms short-time frame audio is finally obtained. In order to realize that the nonlinear indivisible high-dimensional vectorized representation becomes linearly separable, the support vector machine-radial basis function kernel classifier (SVM-RBF) will map the 128-dimensional audio feature representation to the high-dimensional feature space to find the optimal classification hyperplane, and finally map the output probability distribution of event_nums dimensions, and predict the physical event category by calculating the probability maximum index in the probability distribution, completing the audio channel smart device physical event recognition task.

[0008] In view of the shortcomings of the existing smart home privacy and security monitoring methods, the present invention proposes a non-intrusive dual-channel smart home security monitoring method, which improves the monitoring and detection effect of smart home privacy and security by performing event consistency verification with the help of two non-intrusive channel information, audio and traffic, so as to improve the privacy and security monitoring and detection effect of smart home.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A non-intrusive dual-channel smart home security monitoring method, the method comprising the following steps:

[0011] Step 1: Audio channel data preprocessing: Perform mean normalization, short-time discrete Fourier transform, 64-order Mel filter bank spectrum mapping and logarithmic offset processing on the two-channel WAV audio format signal collected by the smart microphone device from the indoor scene to obtain the corresponding log-Mel spectrum graph, input the log-Mel spectrum graph into the audio feature extractor composed of the pre-trained model Vggish, and obtain the 128-dimensional audio feature representation through PCA whitening operation;

[0012] Furthermore, the specific operations of step 1 are:

[0013] Step 1.1: Mean normalization operation: merge and add the two-channel WAV audio format signals and take the mean value to convert them into a continuous mono audio signal;

[0014] Step 1.2: Short-time discrete Fourier transform: The continuous monophonic audio signal obtained in step 1.1 is framed to obtain a 960ms short-time frame. Then, the obtained short-time frame is resampled at a target sampling rate of 44.1kHz and a sampling bit number of 32bit to generate discrete audio sampling points. Then, a 25ms frame length Hann window is moved on the discrete audio sampling points at a sliding interval of 10ms, and a short-time Fourier transform analysis is performed on the discrete audio sampling points in the window, and finally a linear spectrum diagram with a size of 96×257 pixels is obtained;

[0015] Step 1.3: 64-order Mel filter bank spectrum mapping and logarithmic offset processing: The linear spectrum of each short-time frame obtained in step 1.2 is mapped through a filter bank containing 64 Mel filters to generate a Mel spectrum, and then the Mel spectrum is smoothed by a logarithmic offset operation to finally obtain a log-Mel spectrum with a size of 96×64 pixels;

[0016] Step 1.4: Spectrum graph vectorization operation: Input the spectrum graph obtained in step 1.3 into the audio feature extractor, perform PCA whitening processing with the pre-trained parameter matrices pca_matrix and pca_means, and obtain a 128-dimensional audio feature representation without feature correlation.

[0017] Step 2: Construction, training and performance evaluation of audio channel event classifier; Use the support vector machine classifier based on the radial basis function kernel in the Python machine learning library scikits-learn as the audio channel event classifier; Input the 128-dimensional audio feature representation obtained in step 1 into the audio channel event classifier to predict the physical event category of smart home devices; Use the cross entropy loss between the predicted physical event category and the true label as the target optimization function, and then use the gradient descent optimization strategy based on the back-propagation algorithm to adjust the model parameters to achieve the goal of minimizing the loss. When the change in the loss function value between two adjacent training cycles drops to 0.0001, the classifier model has converged and the model training is terminated; Calculate the accuracy, precision, recall, and F1 score of the audio channel event classifier in the test audio data to evaluate the effectiveness of the audio channel classifier in identifying physical events of the device;

[0018] Furthermore, the specific operations of step 2 are:

[0019] Step 2.1: Audio channel event classifier construction: Use the support vector machine classifier based on radial basis function kernel in Python machine learning library scikits-learn as SVM-RBF audio channel event classifier;

[0020] Step 2.2: Audio channel event classifier training: Input the 128-dimensional audio feature representation obtained in step 1.4 into the audio channel event classifier, map the 128-dimensional audio feature vector to a high-dimensional feature space that makes nonlinear inseparable data linearly separable, and find a hyperplane in this space that can maximize the separation of audio events of different categories. Then get the probability distribution vector of event_nums dimension, and apply the argmax function to select the maximum value index in the probability distribution as the most likely event category. Finally, calculate the cross entropy loss between the predicted physical event category and the true label as the target loss function. The cross entropy loss calculation formula is:

[0021]

[0022] Among them, i represents the i-th event category, A real class representing a physical event of a device, represents the probability distribution of the audio channel classifier output after the softmax function, and all represents the number of event categories;

[0023] Step 2.3: Performance evaluation of the audio channel event classifier: Based on the test data in the audio dataset HomeSound-13 collected by the self-built experimental platform, the 128-dimensional audio feature representation obtained in step 1.4 is input into the audio channel event classifier that has been trained in step 2.2 to calculate the four evaluation indicators of accuracy, precision, recall rate, and F1 score, and it is concluded that the audio channel event classifier has excellent audio event recognition effect.

[0024] Step 3: Traffic metadata preprocessing, traffic channel rule base construction, and traffic channel event recognition performance evaluation; First, use Tshark to filter the network field of the collected Pcap format network metadata traffic, and organize and summarize the five-tuple traffic feature representation extracted from each physical event to obtain the event-rule base Set rule ; Then re-trigger the smart home device and extract the five-tuple feature representation of each physical event reported to the platform traffic, and compare it with the predefined event-rule library Set rule Perform pattern matching to predict event categories, and finally organize all triggered physical events into a flow channel event category sequence E in chronological order. t , and calculate the four evaluation indicators of accuracy, precision, recall and F1 score of the predicted traffic channel event category to evaluate the effectiveness of traffic channel identification of device physical events;

[0025] Furthermore, the specific operations of step 3 are:

[0026] Step 3.1: Traffic metadata preprocessing: Use the command line tool Tshark to monitor the network traffic flowing through the Raspberry Pi wlan0 interface of the router and parse and extract the five specific field information of the network data packets, namely frame length, source IP and source MAC address, destination IP and destination MAC address, to form a 5tuple. i , as the only representation of the traffic channel information of the physical events of smart devices;

[0027] Step 3.2: Traffic channel rule base construction: Based on the 5tuple representation of various equipment physical events extracted by filtering in step 3.1 i Perform inductive analysis and finally obtain an event-rule base Set containing various physical events rule ;

[0028] Step 3.3: Traffic channel event recognition performance evaluation: Re-trigger various physical events in real smart home scenarios, and extract the 5-tuple obtained by collecting the network traffic information flowing through the router during the communication with the cloud platform in real time in step 3.1 i Feature representation, and then with the predefined event-rule base Set constructed in step 3.2 rulePerform pattern matching to predict the event category sequence E corresponding to all traffic t ; The effectiveness of traffic channel identification of physical events is evaluated by calculating the accuracy, precision, recall and F1 score of the predicted sequence events.

[0029] Step 4: Smart home security monitoring: The WAV format audio signal collected in real time by triggering various physical events of indoor devices is vectorized into a 128-dimensional feature representation through step 1, and then the 128-dimensional audio feature representation is input into the audio channel event classifier that has been trained in step 2 to predict the audio channel event category sequence E. a , and then the audio channel event category sequence E a and traffic channel event category sequence E t Perform event alignment, verification, and fusion to monitor and detect various privacy and security anomalies in smart homes;

[0030] Furthermore, the specific operations of step 4 are:

[0031] Step 4.1: Predict the audio and traffic channel event category sequence: Re-trigger the indoor smart device physical event in real time and vectorize the collected WAV format audio signal into a 128-dimensional feature representation through step 1, and input it into the audio channel event classifier trained in step 2 to predict the audio channel event category sequence E a At the same time, the Pcap format traffic flowing through the programmable router when the device event is triggered is obtained through step 3.2 to obtain the traffic channel event category sequence E t ;

[0032] Step 4.2: Perform event alignment, verification, and fusion operations on the audio and traffic event category sequences obtained in step 4.1 to monitor and detect various privacy and security anomalies in smart homes;

[0033] Furthermore, the calculation method of the four evaluation indicators is:

[0034] The calculation formula for accuracy is:

[0035] The calculation formula for recall is:

[0036] The calculation formula of F1 score is:

[0037] The accuracy is calculated as follows:

[0038] Where i represents the i-th event category, all represents the number of event categories, TP represents true positive examples, FN represents false negative examples, FP represents false positive examples, and TN represents true negative examples.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1. This invention proposes a more practical and robust smart home privacy security monitoring method. This method uses two universal and non-intrusive physical channel information to identify device activity events, so that the entire process does not require cooperation with smart home manufacturers or modification of the device;

[0041] 2. The present invention implements a method for efficient dual-channel privacy-safe monitoring. By extending the existing audio-based and traffic-based event recognition technologies, an efficient process for jointly monitoring multi-channel activities in smart homes is designed, which can well address the challenges of data scarcity and heterogeneity in specific deployment scenarios;

[0042] 3. The present invention is evaluated on a self-built real-scene test platform. Based on an experimental platform containing 9 smart devices and 11 non-smart devices, the invention achieves an overall accuracy of 92.31% and 96% when identifying smart home event categories based on traffic channels and audio channels, respectively, and also achieves an accuracy of 94.04% for all device events by integrating dual-channel information for monitoring, which fully verifies the effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the non-intrusive audio and traffic dual-channel smart home security anomaly monitoring method of the present invention;

[0044] Figure 2 This is a diagram of the audio channel event recognition classification structure in the present invention;

[0045] Figure 3 It is a flow channel event identification classification structure diagram in the present invention;

[0046] Figure 4 This is a diagram of the event range of the audio and traffic channels in the present invention;

[0047] Figure 5 This is the equipment diagram of the self-built smart home experimental platform in the present invention. DETAILED DESCRIPTION

[0048] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0049] In order to better understand the present invention, the content of the present invention is described in detail below in combination with two data sets HomeSound-13 and HomeTraffic-10 collected based on a self-built experimental platform, but the content of the present invention is not limited to the following embodiments.

[0050] As attached Figure 1 As shown, a non-intrusive dual-channel smart home security monitoring method includes the following steps:

[0051] Step 1: Audio channel data preprocessing: Perform mean normalization, short-time discrete Fourier transform, 64-order Mel filter bank spectrum mapping and logarithmic offset processing on the two-channel WAV audio format signal collected by the smart microphone device from the indoor scene to obtain the corresponding log-Mel spectrum graph, input the log-Mel spectrum graph into the audio feature extractor composed of the pre-trained model Vggish, and obtain the 128-dimensional audio feature representation through PCA whitening operation;

[0052] Further, the specific operation of step 1 is as shown in the attached Figure 2 As shown:

[0053] Step 1.1: Mean normalization operation: merge and add the two-channel WAV audio format signals and convert them into continuous mono audio;

[0054] Step 1.2: Short-time discrete Fourier transform: The continuous monophonic audio signal obtained in step 1.1 is framed to obtain a 960ms short-time frame. Then, the obtained short-time frame is resampled at a target sampling rate of 44.1kHz and a sampling bit number of 32bit to generate discrete audio sampling points. Then, a 25ms frame length Hann window is moved on the discrete audio sampling points at a sliding interval of 10ms, and a short-time Fourier transform analysis is performed on the discrete audio sampling points in the window, and finally a linear spectrum diagram with a size of 96×257 pixels is obtained;

[0055] Step 1.3: 64-order Mel filter bank spectrum mapping and logarithmic offset processing: The linear spectrum of each short-time frame obtained in step 1.2 is mapped through a filter bank containing 64 Mel filters to generate a Mel spectrum, and then the Mel spectrum is smoothed by a logarithmic offset operation to finally obtain a log-Mel spectrum with a size of 96×64 pixels;

[0056] Step 1.4: Spectrum graph vectorization operation: Input the spectrum graph obtained in step 1.3 into the audio feature extractor, perform PCA whitening with the pre-trained parameter matrices pca_matrix and pca_means, and obtain a 128-dimensional audio feature representation without feature correlation. The number of channels or dimensions of the vector representation of each component of the feature extractor are shown in Table 1.

[0057] Table 1. Channel number / dimensionality change table of each component of VGGish feature extractor

[0058]

[0059] Step 2: Construction, training and performance evaluation of audio channel event classifier; Use the support vector machine classifier based on radial basis function kernel in Python machine learning library scikits-learn as the audio channel event classifier; Input the 128-dimensional audio feature representation obtained in step 1 into the audio channel event classifier based on radial basis function kernel support vector machine classifier to predict the physical event category of smart home devices; Use the cross entropy loss between the predicted physical event category and the true label as the target optimization function, and then use the gradient descent optimization strategy based on the back propagation algorithm to adjust the model parameters to achieve the goal of minimizing the loss. When the change in the loss function value between two adjacent training cycles drops to 0.0001, the classifier model has converged, and the model training is terminated at this time; Calculate the accuracy, precision, recall rate, and F1 score of the audio channel event classifier in the test audio data to evaluate the effectiveness of the audio channel classifier in identifying physical events of the device;

[0060] Furthermore, the specific operations of step 2 are:

[0061] Step 2.1: Audio channel event classifier construction: Use the support vector machine classifier based on radial basis function kernel in Python machine learning library scikits-learn as SVM-RBF audio channel event classifier;

[0062] Step 2.2: Audio channel event classifier training: Input the 128-dimensional audio feature representation obtained in step 1.4 into the support vector machine-radial basis function kernel audio channel event classifier, and map the 128-dimensional audio feature vector to a high-dimensional feature space that makes nonlinear inseparable data linearly separable. In this space, find a hyperplane that can maximize the separation of audio events of different categories, then obtain the probability distribution vector of event_nums dimension and apply the argmax function to select the maximum value index in the probability distribution as the most likely event category. Finally, calculate the cross entropy loss between the predicted category and the true event label as the target loss function. The cross entropy loss calculation formula is:

[0063]

[0064] Among them, i represents the i-th event category, A real class representing a physical event of a device, represents the probability distribution of the audio channel classifier output after the softmax function, and all represents the number of event categories;

[0065] Step 2.3: Performance evaluation of audio channel event classifier: Based on the test data from the audio dataset HomeSound-13 (collection duration of various physical events is shown in Table 2) collected by the self-built experimental platform, the 128-dimensional audio feature representation obtained in step 1.4 is input into the SVM-RBF audio channel event classifier trained in step 2.2 to calculate the accuracy, precision, recall, and F1 score. Four evaluation indicators are calculated, and it is concluded that the audio channel event classifier SVM-RBF has excellent audio event recognition effect. The "AER" column in Table 6 shows the recognition effect of the audio channel event classifier.

[0066] Table 2. Duration of each event audio in the HomeSound-13 audio dataset collected based on the self-built experimental platform

[0067]

[0068] Step 3: Traffic metadata preprocessing, traffic channel rule base construction, and traffic channel event recognition performance evaluation; First, use Tshark to filter the network field of the collected Pcap format network metadata traffic, and organize and summarize the five-tuple traffic feature representation extracted from each physical event to obtain the event-rule base Set rule ; Then re-trigger the smart home device and extract the five-tuple feature representation of each physical event reported to the platform traffic, and compare it with the predefined event-rule library Set rule Perform pattern matching to predict event categories, and finally organize all triggered physical events into a flow channel event category sequence E in chronological order. t , and calculate the four evaluation indicators of accuracy, precision, recall and F1 score of the predicted traffic channel event category to evaluate the effectiveness of traffic channel identification of device physical events;

[0069] Further, the specific operation of step 3 is as shown in the attached Figure 3 As shown:

[0070] Step 3.1: Traffic metadata preprocessing: Use the command line tool Tshark to monitor the network traffic flowing through the Raspberry Pi wlan0 interface of the router and parse and extract the five specific field information of the network data packets, namely frame length, source IP and source MAC address, destination IP and destination MAC address, to form a 5tuple. i , which is used as the only representation of the traffic channel information of the physical event of the smart device; Table 3 shows the communication protocols supported by each device in the experimental platform, and it can be concluded that which types of smart home device event quintuple features can be collected and filtered by monitoring the wireless wlan0 interface of the router;

[0071] Table 3. Communication protocols supported by different devices in the self-built experimental platform

[0072]

[0073] Step 3.2: Traffic channel rule base construction: Based on the 5tuple representation of various equipment physical events extracted by filtering in step 3.1 i Perform inductive analysis and finally obtain an event-rule base Set containing various physical events rule ; Table 4 summarizes the “directional frame length” rule features corresponding to different physical events of each type of smart home device contained in the predefined event-rule library;

[0074] Table 4. Predefined events - The rule base contains a set of rules corresponding to different activity events of various smart devices

[0075]

[0076] Step 3.3: Traffic channel event recognition performance evaluation: Re-trigger various physical events in real smart home scenarios, and extract the 5-tuple obtained by collecting the network traffic information flowing through the router during the communication with the cloud platform in real time in step 3.1 i Feature representation, and then with the predefined event-rule base Set constructed in step 3.2 rule Perform pattern matching to predict the event category sequence Et corresponding to all traffic flows; evaluate the effectiveness of traffic channel identification of physical events by calculating the accuracy, precision, recall, and F1 score of the predicted sequence events;

[0077] Furthermore, the calculation method of the four evaluation indicators is:

[0078] The calculation formula for accuracy is:

[0079] The calculation formula for recall is:

[0080] The calculation formula of F1 score is:

[0081] The accuracy is calculated as follows:

[0082] Where i represents the i-th event category, all represents the number of event categories, TP represents true positive examples, FN represents false negative examples, FP represents false positive examples, and TN represents true negative examples.

[0083] Step 4: Smart home security monitoring: The WAV format audio signal collected in real time by triggering various physical events of indoor devices is vectorized into a 128-dimensional feature representation through step 1, and then the 128-dimensional audio feature representation is input into the audio channel event classifier that has been trained in step 2 to predict the audio channel event category sequence E. a , and then the audio channel event category sequence E a With the traffic channel event category sequence E t Perform event alignment, verification, and fusion to monitor and detect various privacy and security anomalies in smart homes;

[0084] Furthermore, the specific operations of step 4 are:

[0085] Step 4.1: Predict the audio and traffic channel event category sequence: Re-trigger the indoor smart device physical event in real time and vectorize the collected WAV format audio signal into a 128-dimensional feature representation through step 1, and input it into the audio channel event classifier trained in step 2 to predict the audio channel event category sequence E a At the same time, the Pcap format traffic flowing through the programmable router when the device event is triggered is obtained through step 3.2 to obtain the traffic channel event category sequence E t ; Table 5 shows the time cost of these two event recognition based on single channel information on two computing platforms: laptop and Raspberry Pi;

[0086] Table 5. Computational overhead of event recognition based on audio and traffic single-channel information on a laptop and Raspberry Pi

[0087]

[0088] Step 4.2: Perform event alignment, verification and fusion operations on the audio and traffic event category sequences obtained in step 4.1 to monitor and detect various privacy and security anomalies in smart homes; Table 5 shows in detail the recognition efficiency of the audio and traffic channel event classifiers on high (laptop) and low (Raspberry Pi) performance computing platforms, and the "TER" and "AER" columns in Table 6 respectively show in detail the recognition effects of the network traffic channel and audio channel event recognition classifiers, which respectively achieve an overall recognition accuracy of 92.31% and 96% for smart home events and non-smart home events in the self-built experimental platform, and the dual-channel fusion event recognition classifier proposed in the present invention better expands the single-channel model event recognition range, as shown in the "DCA" column in Table 6, the recognition accuracy of all events of all smart and non-smart devices in the experimental platform reaches 94.04%.

[0089] Table 6. Physical event recognition effects based on audio or traffic single-channel information and dual-channel information

[0090]

[0091] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A non-intrusive dual-channel smart home security monitoring method, characterized in that: The method comprises the following steps: Step 1: Audio channel data preprocessing: Perform mean normalization, short-time discrete Fourier transform, 64-order Mel filter bank spectrum mapping and logarithmic offset processing on the two-channel WAV audio format signal collected by the smart microphone device from the indoor scene to obtain the corresponding log-Mel spectrum graph, input the log-Mel spectrum graph into the audio feature extractor composed of the pre-trained model Vggish, and obtain the 128-dimensional audio feature representation through PCA whitening operation; Step 2: Construction, training and performance evaluation of audio channel event classifier; Use the support vector machine classifier based on the radial basis function kernel in the Python machine learning library scikits-learn as the audio channel event classifier; Input the 128-dimensional audio feature representation obtained in step 1 into the audio channel event classifier to predict the physical event category of smart home devices; Use the cross entropy loss between the predicted physical event category and the true label as the target optimization function, and then use the gradient descent optimization strategy based on the back-propagation algorithm to adjust the model parameters. When the change in the loss function value between two adjacent training cycles drops to 0.0001, the classifier model has converged and the model training is terminated; Calculate the accuracy, precision, recall, and F1 score of the audio channel event classifier in the test audio data to evaluate the effectiveness of the audio channel classifier in identifying physical events of the device; Step 3: Traffic metadata preprocessing, traffic channel rule base construction, and traffic channel event recognition performance evaluation; First, use Tshark to filter the network field of the collected Pcap format network metadata traffic, and organize and summarize the five-tuple traffic feature representation extracted from each physical event to obtain the event-rule base Set rule ; Then re-trigger the smart home device and extract the five-tuple feature representation of each physical event reported to the platform traffic, and compare it with the predefined event-rule library Set rule Perform pattern matching to predict event categories, and finally organize all triggered physical events into a flow channel event category sequence E in chronological order. t , and calculate the four evaluation indicators of accuracy, precision, recall and F1 score of the predicted traffic channel event category to evaluate the effectiveness of traffic channel identification of device physical events; Step 4: Smart home security monitoring: The WAV format audio signal collected in real time by triggering various physical events of indoor devices is vectorized into a 128-dimensional feature representation through step 1, and then the 128-dimensional audio feature representation is input into the audio channel event classifier that has been trained in step 2 to predict the audio channel event category sequence E. a , and then the audio channel event category sequence E a and traffic channel event category sequence E t Event alignment, verification, and fusion are performed to monitor and detect various privacy and security anomalies in smart homes.

2. A non-intrusive dual-channel smart home security monitoring method according to claim 1, characterized in that: The specific operations of step 1 are: Step 1.1: Mean normalization operation: merge and add the two-channel WAV audio format signals and take the mean value to convert them into a continuous mono audio signal; Step 1.2: Short-time discrete Fourier transform: The continuous monophonic audio signal obtained in step 1.1 is framed to obtain a 960ms short-time frame. Then, the obtained short-time frame is resampled at a target sampling rate of 44.1kHz and a sampling bit number of 32bit to generate discrete audio sampling points. Then, a 25ms frame length Hann window is moved on the discrete audio sampling points at a sliding interval of 10ms, and a short-time Fourier transform analysis is performed on the discrete audio sampling points in the window, and finally a linear spectrum diagram with a size of 96×257 pixels is obtained; Step 1.3: 64-order Mel filter bank spectrum mapping and logarithmic offset processing: The linear spectrum of each short-time frame obtained in step 1.2 is mapped through a filter bank containing 64 Mel filters to generate a Mel spectrum, and then the Mel spectrum is smoothed by a logarithmic offset operation to finally obtain a log-Mel spectrum with a size of 96×64 pixels; Step 1.4: Spectrum graph vectorization operation: Input the spectrum graph obtained in step 1.3 into the audio feature extractor, perform PCA whitening processing with the pre-trained parameter matrices pca_matrix and pca_means, and obtain a 128-dimensional audio feature representation without feature correlation.

3. A non-intrusive dual-channel smart home security monitoring method according to claim 2, characterized in that: The specific operations of step 2 are: Step 2.1: Audio channel event classifier construction: Use the support vector machine classifier based on radial basis function kernel in Python machine learning library scikits-learn as SVM-RBF audio channel event classifier; Step 2.2: Audio channel event classifier training: Input the 128-dimensional audio feature representation obtained in step 1.4 into the audio channel event classifier, map the 128-dimensional audio feature vector to a high-dimensional feature space, and find the hyperplane that maximizes the separation of audio events of different categories in this space. Then, obtain the probability distribution vector of the event_nums dimension and apply the argmax function to select the maximum value index in the probability distribution as the most likely event category. Finally, calculate the cross entropy loss between the predicted category and the true event label as the target loss function. The cross entropy loss calculation formula is: Among them, i represents the i-th event category, A real class representing a physical event of a device, represents the probability distribution of the audio channel classifier output after the softmax function, and all represents the number of event categories; Step 2.3: Performance evaluation of audio channel event classifier: Based on the test data in the audio dataset HomeSound-13 collected by the self-built experimental platform, the 128-dimensional audio feature representation obtained in step 1.4 is input into the audio channel event classifier that has been trained in step 2.2 to calculate the four evaluation indicators of accuracy, precision, recall, and F1 score.

4. A non-intrusive dual-channel smart home security monitoring method according to claim 3, characterized in that: The specific operations of step 3 are: Step 3.1: Traffic metadata preprocessing: Use the command line tool Tshark to monitor the network traffic flowing through the Raspberry Pi wlan0 interface of the router and parse and extract the five specific field information of the network data packets, namely frame length, source IP and source MAC address, destination IP and destination MAC address, to form a 5tuple. i , as the only representation of the traffic channel information of the physical events of smart devices; Step 3.2: Traffic channel rule base construction: Based on the 5tuple representation of various equipment physical events extracted by filtering in step 3.1 i Perform inductive analysis and finally obtain an event-rule base Set containing various physical events rule ; Step 3.3: Traffic channel event recognition performance evaluation: Re-trigger various physical events in real smart home scenarios, and extract the 5-tuple obtained by collecting the network traffic information flowing through the router during the communication with the cloud platform in real time in step 3.1 i Feature representation, and then with the predefined event-rule base Set constructed in step 3.2 rule Perform pattern matching to predict the event category sequence E corresponding to all traffic t ; The effectiveness of traffic channel identification of physical events is evaluated by calculating the accuracy, precision, recall and F1 score of the predicted sequence events.

5. A non-intrusive dual-channel smart home security monitoring method according to claim 4, characterized in that: The specific operations of step 4 are: Step 4.1: Predict the audio and traffic channel event category sequence: Re-trigger the indoor smart device physical event in real time and vectorize the collected WAV format audio signal into a 128-dimensional feature representation through step 1, and input it into the audio channel event classifier trained in step 2 to predict the audio channel event category sequence E a At the same time, the Pcap format traffic flowing through the programmable router when the device event is triggered is obtained through step 3.2 to obtain the traffic channel event category sequence E t ; Step 4.2: Perform event alignment, verification, and fusion operations on the audio and traffic event category sequences obtained in step 4.1 to monitor and detect various privacy and security anomalies in smart homes.

6. A non-intrusive dual-channel smart home security monitoring method according to claim 4, characterized in that: The calculation methods of the four evaluation indicators are as follows: The calculation formula for accuracy is: The calculation formula for recall is: The calculation formula of F1 score is: The accuracy is calculated as follows: Where i represents the i-th event category, all represents the number of event categories, TP represents true positive examples, FN represents false negative examples, FP represents false positive examples, and TN represents true negative examples.

Images