A non-invasive dual-channel intelligent home security monitoring method

Abstract

The application discloses a non-invasive dual-channel intelligent home security monitoring method and belongs to the technical field of network security. In view of the problems of the existing intelligent home privacy security monitoring method, the audio channel takes a pre-trained model Vggish as an audio feature extractor, converts a pixel spectrum graph corresponding to each 960 ms frame length audio obtained through discrete Fourier frame processing into a 128-dimensional feature representation, and then identifies an audio event category sequence E a by a support vector machine (SVM-RBF) classifier; in the flow channel, five-tuple information contained in an event flow is matched with a pre-defined event-rule library mode, a flow event category sequence E t is identified, and then the sequences E s and E t are subjected to event alignment, verification and fusion to monitor the security problem in the intelligent home. The application proves the accuracy of event identification of the dual-channel model through experiments on a self-built experimental platform, and also proves the necessity of security monitoring by using dual-channel fusion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network security, and particularly relates to a non-intrusive dual-channel intelligent home security monitoring method. BACKGROUND

[0002] With the continuous progress of Internet of Things (IoT) technology and the growing demand of consumers for a higher standard of living, the smart home field is experiencing significant growth. According to Grand View, the market is expected to have a compound annual growth rate (CAGR) of 27.07% from 2023 to 2030, with a market value of $5370.1 billion expected by 2030. The penetration rate of smart homes in households is expected to rise from 18.9% in 2024 to 33.2% in 2028. Smart homes are residences equipped with smart IoT devices that integrate internet connectivity, automation, and remote control functions through an interconnected platform to enhance experiences in comfort, convenience, energy efficiency, and more. Users of smart homes can remotely monitor and control devices through mobile applications or voice assistants, or set up automation rules to enable devices to run automatically based on predefined conditions (i.e., so-called "home automation"). However, despite the many advantages of smart homes, the complex interconnectivity and interaction between IoT devices, platforms, and applications have expanded the attack surface, making smart homes more vulnerable to various security and safety vulnerabilities.

[0003] Anomaly detection, which identifies behaviors deviating from normal patterns in a system, is generally considered a fundamental technique for detecting vulnerabilities, malfunctions, misoperations, and potential attacks in smart homes. Fahad et al. utilized a Probabilistic Neural Network (PNN) to pre-segment activity instances to extract features and further utilized an H2O autoencoder to detect abnormal activities in a smart home environment. Zamani et al. proposed an autoencoder (AE) based on two one-dimensional convolutional neural networks (1D-CNN) and temporal convolution networks (TCN) for detecting anomalies in smart home device energy consumption. Ramapatruni et al. utilized a Hidden Markov Model (HMM) trained on normal smart home activities to detect abnormal behavior. However, existing solutions generally have at least one of the following limitations, which significantly weakens their overall effectiveness:

[0004] 1. Most existing solutions rely on the sequence of IoT events extracted from the activity notification messages reported by devices to learn the semantic association patterns between events and identify abnormal events deviating from these patterns. The effectiveness and reliability of such methods largely depend on the integrity of the input events. However, due to various factors such as system failures, misoperations or configuration errors, and attacks, false, erroneous, missing, or delayed events can occur. In the absence of accurate inputs, the anomaly detection model cannot make accurate or reliable judgments.

[0005] 2. Smart homes can contain a variety of non-intelligent (i.e., "dumb") devices whose activities are not recorded as events and reported, and thus cannot be used for anomaly detection. If the activities of these non-intelligent devices can be included in the input data for anomaly detection, it is expected that the detection performance will be significantly improved.

[0006] 3. Existing methods lack universality in fragmented smart home environments, especially in the absence of reference monitors. In modern smart home systems, a variety of heterogeneous IoT devices are connected to platforms, but exhibit the following characteristics: (1) each device is connected to only a portion of the platforms; (2) each platform supports only a portion of the devices. Therefore, any single platform cannot provide a comprehensive global view of all devices. Although an independent host that can access all devices can solve this problem, it requires all platforms to provide APIs for device access. However, due to the heterogeneity of platforms, this requirement is often difficult to achieve. SUMMARY

[0007] The VGGish pre-trained model is trained by Google based on the YouTube-8M video dataset and is originally designed for visual recognition tasks. Due to its good generalization ability in the process of scene-class audio feature extraction, the present invention selects this model to be applied to the audio data feature extraction part in the smart home scene. This deep learning architecture model is composed of 7 layers of Conv2d and Max_Pool2d, which extracts hierarchical and latent feature representations of the spectrogram by using convolution and pooling operations, and finally obtains a 128-dimensional feature representation of each 960ms short-time frame audio. The support vector machine-radial basis function kernel classifier (SVM-RBF) maps the 128-dimensional audio feature representation to a high-dimensional feature space to find the optimal classification hyperplane in order to make the nonlinear and non-separable high-dimensional vector representation linearly separable. Finally, the output probability distribution of event_nums dimensions is obtained, and the physical event category is predicted by calculating the maximum probability index in the probability distribution, completing the audio channel intelligent device physical event identification task.

[0008] In view of the deficiencies of the existing smart home privacy security monitoring method, the application provides a non-intrusive dual-channel smart home security monitoring method, which verifies the event consistency by means of audio and flow, two non-intrusive channels, to improve the monitoring and detection effect of smart home privacy security.

[0009] In order to achieve the above purpose, the application adopts the following technical solutions:

[0010] A non-intrusive dual-channel smart home security monitoring method, the method comprising the following steps:

[0011] Step 1: audio channel data preprocessing; the dual-channel WAV audio format signal collected by the smart microphone device from the indoor scene is subjected to mean normalization, short-time discrete Fourier transform, 64-order Mel filter bank spectrum mapping and logarithmic bias processing to obtain the corresponding log-Mel spectrum graph, the log-Mel spectrum graph is input into the audio feature extractor composed of the pre-trained model Vggish, and the 128-dimensional audio feature representation is obtained through PCA whitening operation;

[0012] Further, the specific operation of step 1 is:

[0013] Step 1.1: mean normalization operation: the dual-channel WAV audio format signal is fused and added to obtain the mean value to convert into a continuous single-channel audio signal;

[0014] Step 1.2: short-time discrete Fourier transform: the continuous single-channel audio signal obtained in step 1.1 is subjected to frame processing to obtain a 960ms short-time frame, then the short-time frame is resampled at a target sampling rate of 44.1kHz and a sampling bit number of 32bit to generate discrete audio sampling points, then a hann window with a frame length of 25ms is moved on the discrete audio sampling points at a sliding interval of 10ms, and the discrete audio sampling points in the window are subjected to short-time Fourier transform analysis, finally a linear spectrum graph with a size of 96x257 pixels is obtained;

[0015] Step 1.3: 64-order Mel filter bank spectrum mapping and logarithmic bias processing: each short-time frame linear spectrum graph obtained in step 1.2 is mapped to generate a Mel spectrum graph through a filter bank containing 64 Mel filters, and then the Mel spectrum graph is subjected to logarithmic bias operation for smoothing processing, finally a log-Mel spectrum graph with a size of 96x64 pixels is obtained;

[0016] Step 1.4: spectrum graph vectorization operation: the spectrum graph obtained in step 1.3 is input into the audio feature extractor to perform PCA whitening processing with the pre-trained parameter matrix pca_matrix and pca_means, and a de-correlated 128-dimensional audio feature representation is obtained.

[0017] Step 2: Construction, training and performance evaluation of the audio channel event classifier; a support vector machine classifier based on a radial basis function kernel in the Python machine learning library scikits-learn is used as the audio channel event classifier; the 128-dimensional audio feature representation obtained in step 1 is input into the audio channel event classifier to predict the physical event class of the smart home device; the cross-entropy loss of the predicted physical event class and the true label is used as the target optimization function, and then a gradient descent optimization strategy based on the back propagation algorithm is used to adjust the model parameters to minimize the loss. When the change amplitude of the loss function value between adjacent two training periods decreases to 0.0001, the classifier model has converged, and the model training is terminated; the accuracy, precision, recall and F1 score of the audio channel event classifier in the test audio data are calculated to evaluate the effectiveness of the audio channel classifier in identifying device physical events;

[0018] Further, the specific operation of step 2 is:

[0019] Step 2.1: Construction of the audio channel event classifier: a support vector machine classifier based on a radial basis function kernel in the Python machine learning library scikits-learn is used as the SVM-RBF audio channel event classifier;

[0020] Step 2.2: Training of the audio channel event classifier: the 128-dimensional audio feature representation obtained in step 1.4 is input into the audio channel event classifier, the 128-dimensional audio feature vector is mapped to a high-dimensional feature space that makes nonlinear and non-separable data linearly separable, and a hyperplane that can maximize the separation of different class event audios in the space is found. Then an event_nums-dimensional probability distribution vector is obtained, and the argmax function is applied to select the maximum value index in the probability distribution as the most likely event class. Finally, the cross-entropy loss between the predicted physical event class and the true label is calculated as the target loss function, and the cross-entropy loss calculation formula is:

[0021]

[0022] Where i represents the i-th event class, represents the true class of the device physical event, represents the probability distribution output by the audio channel classifier after the softmax function, and all represents the number of event classes.

[0023] Step 2.3: Performance evaluation of audio channel event classifier: The test data in the audio data set HomeSound-13 collected based on the self-built experimental platform is input into the audio channel event classifier trained in step 2.2 to calculate the accuracy, precision, recall and F1 score, and it is found that the audio channel event classifier has excellent audio event recognition effect.

[0024] Step 3: Flow metadata preprocessing, flow channel rule base construction, and flow channel event recognition performance evaluation; first, the Pcap format network field of the collected network metadata is filtered by Tshark, and the five tuple flow feature representation extracted from each physical event is sorted and summarized to obtain the event-rule base Set rule ; then the intelligent home device is triggered again and the five tuple feature representation of the flow reported to the platform by each physical event is extracted, and the event-rule base Set rule is matched to predict the event category, and finally all triggered physical events are sorted in time sequence to form a flow channel event category sequence E t , and the accuracy, precision, recall and F1 score of the predicted flow channel event category are calculated to evaluate the effectiveness of the flow channel recognition device in identifying physical events.

[0025] Further, the specific operation of step 3 is as follows:

[0026] Step 3.1: Flow metadata preprocessing: The network flow through the wlan0 interface of the router Raspberry Pi is monitored by the command line tool Tshark, and the frame length, source IP and source MAC address, destination IP and destination MAC address five specific field information in these network packets are extracted to form the five tuple 5tuple i , which is used as the unique representation of the physical event flow channel information of the intelligent device.

[0027] Step 3.2: Flow channel rule base construction: Based on the five tuple representation 5tuple i of various device physical events filtered and extracted in step 3.1, the event-rule base Set rule containing various physical events is finally obtained through induction analysis.

[0028] Step 3.3: Flow channel event recognition performance evaluation: In the real smart home scene, various physical events are triggered again, and the 5tuple i feature representation extracted from the network flow information flowing through the router during the communication with the cloud platform is obtained in real time by step 3.1, and then matched with the pre-defined event-rule base Set rulePerform pattern matching to predict the event category sequence E corresponding to all traffic. t The effectiveness of traffic channels in identifying physical events is evaluated by calculating four evaluation metrics: accuracy, precision, recall, and F1 score.

[0029] Step 4: Smart Home Security Monitoring: The WAV format audio signals collected in real time from various physical events triggered by indoor devices are vectorized into 128-dimensional feature representations using the methods described in Step 1. These 128-dimensional audio feature representations are then input into the audio channel event classifier trained in Step 2 to predict the audio channel event category sequence E. a Then, the audio channel event category sequence E a With traffic channel event category sequence E t Perform event alignment, verification, and fusion to monitor and detect various privacy and security anomalies in smart homes;

[0030] Furthermore, the specific operation of step 4 is as follows:

[0031] Step 4.1: Predict the audio and traffic channel event category sequence: Re-trigger the physical events of the indoor smart device in real time and vectorize the collected WAV format audio signal into a 128-dimensional feature representation through Step 1. Input the vector representation into the audio channel event classifier trained in Step 2 to predict the audio channel event category sequence E. a Simultaneously, the Pcap format traffic flowing through the programmable router when the device event is triggered is used to obtain the traffic channel event category sequence E through step 3.2. t ;

[0032] Step 4.2: Perform event alignment, verification, and fusion operations on the audio and traffic event category sequences obtained in Step 4.1 to monitor and detect various privacy and security anomalies in smart homes;

[0033] Furthermore, the calculation methods for the four evaluation indicators are as follows:

[0034] The formula for calculating accuracy is:

[0035] The formula for calculating recall rate is:

[0036] The formula for calculating the F1 score is:

[0037] The formula for calculating accuracy is:

[0038] In the formula: i represents the i-th event category, all represents the number of event categories, TP represents a true negative example, FN represents a false negative example, FP represents a false positive example, and TN represents a true negative example.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] 1. The present application proposes a more practical and robust smart home privacy security monitoring method. The method uses two ubiquitous and non-intrusive physical channel information to identify device activity events, so that the entire process does not need to rely on cooperation of smart home manufacturers or modification of devices;

[0041] 2. The present application realizes a method of dual-channel privacy security efficient monitoring. By extending existing audio-based and traffic-based event identification technologies, a high-efficiency process for joint monitoring of smart home multi-channel activities is designed, which can well solve the challenges of data scarcity and heterogeneity in specific deployment scenarios;

[0042] 3. The present application performs performance evaluation on a self-built real scene test experiment platform. Based on an experiment platform containing 9 smart devices and 11 non-smart devices, the present application achieves an overall accuracy of 92.31% and 96% respectively in identifying smart home event categories based on traffic channel and audio channel, and through fusion of dual-channel information for monitoring, an accuracy of 94.04% for all device events is achieved, fully verifying the effectiveness of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flow chart of the non-intrusive audio and traffic dual-channel smart home security anomaly monitoring method in the present application;

[0044] Figure 2 is a structure diagram of audio channel event identification classification in the present application;

[0045] Figure 3 is a structure diagram of traffic channel event identification classification in the present application;

[0046] Figure 4 is a diagram of audio and traffic two-channel sensing event range in the present application;

[0047] Figure 5 is a diagram of self-built smart home experiment platform equipment in the present application. DETAILED DESCRIPTION

[0048] In order to better understand the present application, it will be described in detail. However, the present application has multiple implementation ways and is not limited to the specific examples listed in this paper. The presentation of these examples aims to deepen the overall understanding of the disclosed content of the present application.

[0049] In order to better understand the present application, the following will be described in detail based on two data sets HomeSound-13 and HomeTraffic-10 collected from self-built experimental platform, but the content of the present application is not limited to the following examples.

[0050] As shown in the accompanying Figure 1 A non-invasive two-channel intelligent home safety monitoring method, comprising the following steps:

[0051] Step 1: audio channel data preprocessing; the double-channel WAV audio format signal collected by the intelligent microphone device from the indoor scene is subjected to mean normalization, short-time discrete Fourier transform, 64-order Mel filter bank spectrum mapping and logarithmic bias processing to obtain the corresponding log-Mel spectrum graph, and the log-Mel spectrum graph is input into the audio feature extractor composed of the pre-trained model Vggish to obtain a 128-dimensional audio feature representation through PCA whitening operation;

[0052] Further, the specific operation of step 1 is as shown in the accompanying Figure 2 :

[0053] Step 1.1: mean normalization operation: the double-channel WAV audio format signal is fused and added to obtain the mean value to convert into a continuous single-channel audio;

[0054] Step 1.2: short-time discrete Fourier transform: the continuous single-channel audio signal obtained in step 1.1 is subjected to frame processing to obtain a 960ms short-time frame, then the short-time frame is resampled at a target sampling rate of 44.1kHz and a sampling bit number of 32bit to generate discrete audio sampling points, then a hann window with a frame length of 25ms is moved on the discrete audio sampling points at a sliding interval of 10ms, and the discrete audio sampling points in the window are subjected to short-time Fourier transform analysis, finally a 96x257 pixel size linear spectrum graph is obtained;

[0055] Step 1.3: 64-order Mel filter bank spectrum mapping and logarithmic bias processing: each short-time frame linear spectrum graph obtained in step 1.2 is mapped to generate a Mel spectrum graph through a filter bank containing 64 Mel filters, and then the Mel spectrum graph is subjected to logarithmic bias operation for smoothing processing, finally a 96x64 pixel size log-Mel spectrum graph is obtained;

[0056] Step 1.4: spectrum graph vectorization operation: the spectrum graph obtained in step 1.3 is input into the audio feature extractor to perform PCA whitening processing with the pre-trained parameter matrix pca_matrix and pca_means, to obtain a 128-dimensional audio feature representation without feature correlation. The channel number or dimension change of each component of the feature extractor is shown in Table 1.

[0057] Table 1. Channel number / dimension changes for each component of the VGGish feature extractor

[0058]

[0059] Step 2: Construction, training, and performance evaluation of the audio channel event classifier; using a support vector machine classifier based on a radial basis function kernel in the Python machine learning library scikits-learn as the audio channel event classifier; inputting the 128-dimensional audio feature representation obtained in step 1 into the audio channel event classifier based on the support vector machine classifier based on the radial basis function kernel, predicting the physical event class of the smart home device; taking the cross-entropy loss of the predicted physical event class and the true label as the target optimization function, then using the gradient descent optimization strategy based on the backpropagation algorithm to adjust the model parameters to achieve the goal of minimizing the loss, when the change amplitude of the loss function value between adjacent two training periods decreases to 0.0001, the classifier model has converged, at this time the model training is terminated; calculating the accuracy, precision, recall, and F1 score of the audio channel event classifier in the test audio data, evaluating the effectiveness of the audio channel classifier in identifying device physical events;

[0060] Further, the specific operation of step 2 is:

[0061] Step 2.1: Construction of the audio channel event classifier: using a support vector machine classifier based on a radial basis function kernel in the Python machine learning library scikits-learn as the SVM-RBF audio channel event classifier;

[0062] Step 2.2: Training of the audio channel event classifier: inputting the 128-dimensional audio feature representation obtained in step 1.4 into the support vector machine-radial basis function kernel audio channel event classifier, mapping the 128-dimensional audio feature vector to a high-dimensional feature space that makes nonlinearly inseparable data linearly separable, finding a hyperplane in this space that can maximize the separation of different class event audio, then obtaining an event_nums-dimensional probability distribution vector and applying the argmax function to select the maximum value index in the probability distribution as the most likely event class, and finally calculating the cross-entropy loss between the predicted class and the true event label as the target loss function, the cross-entropy loss calculation formula is:

[0063]

[0064] where i represents the i-th event class, represents the true class of the device physical event, represents the probability distribution output by the audio channel classifier after the softmax function, all represents the number of event classes;

[0065] Step 2.3: Audio channel event classifier performance evaluation: The test data in the audio data set HomeSound-13 (the duration of each physical event is shown in Table 2) collected based on the self-built experimental platform is input into the SVM-RBF audio channel event classifier trained in step 2.2 to calculate the accuracy, precision, recall, and F1 score four evaluation indexes, and it is concluded that the audio channel event classifier SVM-RBF has excellent audio event recognition effect. The recognition effect of the audio channel event classifier is shown in the "AER" column in Table 6.

[0066] Table 2. Audio duration of each event in the audio data set HomeSound-13 collected based on the self-built experimental platform

[0067]

[0068] Step 3: Flow metadata preprocessing, flow channel rule base construction, and flow channel event recognition performance evaluation; first, the Pcap format network field of the collected network metadata is filtered by Tshark, and the five-tuple flow feature representation extracted from each physical event is sorted and summarized to obtain the event-rule base Set rule ; then the intelligent home device is triggered again and the five-tuple feature representation of the flow reported to the platform by each physical event is extracted, and the event-rule base Set rule is matched to predict the event category, and finally all triggered physical events are sorted in time sequence to form the flow channel event category sequence E t , and the accuracy, precision, recall, and F1 score four evaluation indexes of the predicted flow channel event category are calculated to evaluate the effectiveness of the flow channel recognition device in identifying physical events;

[0069] Further, the specific operation of step 3 is as shown in the accompanying Figure 3 ;

[0070] Step 3.1: Flow metadata preprocessing: the network flow through the wlan0 interface of the router Raspberry Pi is monitored and parsed by the command line tool Tshark to extract the frame length, source IP and source MAC address, destination IP and destination MAC address five specific field information to form the five-tuple 5tuple i , which is used as the unique representation of the physical event flow channel information of the intelligent device; Table 3 shows the communication protocols supported by each device in the experimental platform, and it can be concluded that by monitoring the wireless wlan0 interface of the router, the five-tuple features of which smart home device events can be collected and filtered;

[0071] Table 3. Communication protocols supported by different devices in the self-built experimental platform

[0072]

[0073] Step 3.2: Flow channel rule base construction: Based on the filtering extraction of step 3.1, the 5-tuple representation of various device physical events i Inductive analysis, finally get the event-rule base Set containing various physical events rule Table 4 summarizes the "directional frame length" rule characteristics corresponding to different physical events of each smart home device included in the pre-defined event-rule base;

[0074] Table 4. The pre-defined event-rule base contains a set of rules corresponding to different activity events of various smart devices

[0075]

[0076] Step 3.3: Performance evaluation of flow channel event identification: re-trigger various physical events in real smart home scenarios, and extract 5-tuple through real-time collection of network traffic information flowing through the router during communication with the cloud platform in step 3.1 i Feature representation, and then match the pre-defined event-rule base Set constructed in step 3.2 rule Predict the event category sequence Et corresponding to all traffic; By calculating the accuracy, precision, recall and F1 score of the predicted sequence of events, evaluate the effectiveness of the flow channel in identifying physical events;

[0077] Further, the calculation method of the four evaluation indexes is:

[0078] The calculation formula of precision is:

[0079] The calculation formula of recall is:

[0080] The calculation formula of F1 score is:

[0081] The calculation formula of accuracy is:

[0082] In the formula: i represents the i-th event category, all represents the number of event categories, TP represents the true positive, FN represents the false negative, FP represents the false positive, and TN represents the true negative.

[0083] Step 4: Smart home security monitoring: The WAV format audio signals collected in real time by triggering various physical events of indoor devices are vectorized into 128-dimensional feature representations through step 1, and then the 128-dimensional audio feature representations are input into the audio channel event classifier trained in step 2 to predict the audio channel event class sequence E a Then the audio channel event class sequence E a is aligned, verified and fused with the traffic channel event class sequence E t to monitor and detect various privacy security anomalies existing in the smart home;

[0084] Further, the specific operation of step 4 is:

[0085] Step 4.1: Predicting audio and traffic channel event class sequences: Re-triggering indoor smart device physical events in real time and collecting WAV format audio signals, which are vectorized into 128-dimensional feature representations through step 1, are input into the audio channel event classifier trained in step 2 to predict the audio channel event class sequence E a At the same time, the Pcap format traffic flowing through the programmable router when the device event is triggered is obtained through step 3.2 to obtain the traffic channel event class sequence E t Table 5 shows the time overhead of event recognition based on single channel information on notebook computers and Raspberry Pi platforms;

[0086] Table 5. Computational overhead of event recognition based on audio and traffic single channel information on notebook computers and Raspberry Pi platforms

[0087]

[0088] Step 4.2: Aligning, verifying and fusing the audio and traffic event class sequences obtained in step 4.1 to monitor and detect various privacy security anomalies existing in the smart home; Table 5 shows the recognition efficiency of the audio and traffic channel event classifiers on high (notebook computer) and low (Raspberry Pi) performance platforms, and Table 6 shows the recognition accuracy of the network traffic channel and audio channel event recognition classifiers, which are 92.31% and 96% for smart home events and non-smart home events on the self-built experimental platform, respectively. The dual-channel fusion event recognition classifier proposed in the present application better expands the event recognition range of the single-channel model, as shown in Table 6, and the overall event recognition accuracy of all smart and non-smart devices on the experimental platform is 94.04%.

[0089] Table 6. Event recognition effect based on only audio or traffic single channel information and based on dual-channel information

[0090]

[0091] The description herein of any aspect of the application using terms such as "processing" or "comparing" or "determining" or "displaying" or "transmitting" or "receiving" or "identifying" or "selecting" or "extracting" or "generating" or "calculating" or "associating" or "determining" or "recommending" or "providing" or "outputting" or "receiving" or "retrieving" or "accessing" or "updating" or "monitoring" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving" or "receiving

Claims

1. A non-intrusive dual-channel smart home security monitoring method, characterized in that, The method includes the following steps: Step 1: Audio channel data preprocessing; The dual-channel WAV audio format signal collected by the smart microphone device from the indoor scene is subjected to mean normalization, short-time discrete Fourier transform, 64th order Mel filter bank spectrum mapping and logarithmic bias processing to obtain the corresponding log-Mel spectrum. The log-Mel spectrum is input into the audio feature extractor composed of the pre-trained model Vggish, and a 128-dimensional audio feature representation is obtained through PCA whitening operation. Step 2: Construction, training, and performance evaluation of the audio channel event classifier; The support vector machine classifier based on radial basis function kernels from the Python machine learning library scikit-learn is used as the audio channel event classifier; the 128-dimensional audio feature representation obtained in Step 1 is input into the audio channel event classifier to predict the physical event categories of smart home devices; the cross-entropy loss between the predicted physical event category and the true label is used as the objective optimization function, and then a gradient descent optimization strategy based on backpropagation is used to adjust the model parameters. When the change in the loss function value between two adjacent training cycles drops to 0.0001, the classifier model has converged, and model training is terminated; the accuracy, precision, recall, and F1 score of the audio channel event classifier in the test audio data are calculated to evaluate the effectiveness of the audio channel classifier in identifying physical events of devices; Step 3: Traffic metadata preprocessing, traffic channel rule base construction, and traffic channel event identification performance evaluation; First, the collected Pcap format network metadata traffic is filtered by network fields using Tshark. The five-tuple traffic feature representations extracted from each physical event are then organized and summarized to obtain the event-rule base. Then, the smart home devices are retried and the five-tuple feature representation of the traffic reported to the platform for each physical event is extracted and compared with a predefined event-rule base. Pattern matching is used to predict event categories, and finally, all triggered physical events are arranged in chronological order to form a traffic channel event category sequence. It calculates four evaluation metrics—accuracy, precision, recall, and F1 score—to predict the event categories of the traffic channel and evaluates the effectiveness of the traffic channel identification device for identifying physical events. Step 3.1: Traffic Metadata Preprocessing: Monitor network traffic flowing through the Raspberry Pi wlan0 interface of the router using the command-line tool Tshark, and extract five specific fields from these network packets: frame length, source IP and source MAC address, destination IP and destination MAC address, forming a 5-tuple. This serves as the unique representation of the physical event traffic channel information of smart devices; Step 3.2: Construction of the traffic channel rule base: Based on the five-tuple representation of various device physical events extracted in Step 3.

1. Through inductive analysis, an event-rule base containing various physical events was ultimately obtained. ; Step 3.3: Performance Evaluation of Traffic Channel Event Recognition: Various physical events are re-triggered in a real smart home scenario. This is achieved by extracting network traffic information flowing through the router during real-time communication with the cloud platform, as described in Step 3.

1. Feature representation, then combined with the predefined event-rule base constructed in step 3.2 Perform pattern matching to predict the event category sequence corresponding to all traffic. The effectiveness of traffic channels in identifying physical events is evaluated by calculating four evaluation metrics: accuracy, precision, recall, and F1 score. Step 4: Smart Home Security Monitoring: The WAV format audio signals collected in real time from various physical events triggered by indoor devices are vectorized into 128-dimensional feature representations using the methods described in Step 1. These 128-dimensional audio feature representations are then input into the audio channel event classifier trained in Step 2 to predict the audio channel event category sequence. Then, the audio channel event category sequence With traffic channel event category sequence Perform event alignment, verification, and fusion to monitor and detect various privacy and security anomalies in smart homes; Step 4.1: Predict the audio and traffic channel event category sequence: Re-trigger the physical events of the indoor smart devices in real time and vectorize the collected WAV format audio signals into a 128-dimensional feature representation through Step 1. Input the vector representation into the audio channel event classifier trained in Step 2 to predict the audio channel event category sequence. Simultaneously, the Pcap format traffic flowing through the programmable router when the device event is triggered is used to obtain the traffic channel event category sequence through step 3.

2. ; Step 4.2: Perform event alignment, verification, and fusion operations on the audio and traffic event category sequences obtained in Step 4.1 to monitor and detect various privacy and security anomalies in smart homes.

2. The non-intrusive dual-channel smart home security monitoring method according to claim 1, characterized in that, The specific operation of step 1 is as follows: Step 1.1: Mean normalization operation: fuse and add the two-channel WAV audio format signals, take the mean, and convert them into a continuous mono audio signal; Step 1.2: Short-time Discrete Fourier Transform: The continuous mono audio signal obtained in Step 1.1 is processed into frames to obtain 960ms short-time frames. Then, the short-time frames are resampled at a target sampling rate of 44.1 kHz and a sampling bit depth of 32 bits to generate discrete audio sampling points. Then, a 25ms frame-long Hann window is moved on the discrete audio sampling points with a sliding interval of 10ms, and the discrete audio sampling points within the window are analyzed by short-time Fourier transform to finally obtain a linear spectrum of 96×257 pixels. Step 1.3: Spectrum mapping and logarithmic bias processing of 64-order Mel filter bank: The linear spectrum of each short frame obtained in Step 1.2 is mapped to generate a Mel spectrum through a filter bank containing 64 Mel filters. Then, the Mel spectrum is smoothed by logarithmic bias operation to finally obtain a log-Mel spectrum of size 96×64 pixels. Step 1.4: Spectrum vectorization operation: Input the spectrum obtained in step 1.3 into the audio feature extractor, and perform PCA whitening with the pre-trained parameter matrices pca_matrix and pca_means to obtain a 128-dimensional audio feature representation without feature correlation.

3. The non-intrusive dual-channel smart home security monitoring method according to claim 2, characterized in that, The specific operation of step 2 is as follows: Step 2.1: Audio Channel Event Classifier Construction: The Support Vector Machine classifier based on radial basis function kernels from the Python machine learning library scikit-learn is used as the SVM-RBF audio channel event classifier; Step 2.2: Audio Channel Event Classifier Training: Input the 128-dimensional audio feature representation obtained in Step 1.4 into the audio channel event classifier. By mapping the 128-dimensional audio feature vector to a high-dimensional feature space, find the hyperplane that maximizes the separation of different categories of audio events in this space. Then, obtain the probability distribution vector of dimension event_nums and apply the argmax function to select the maximum value index in the probability distribution as the most likely event category. Finally, calculate the cross-entropy loss between the predicted category and the true event label as the target loss function. The formula for calculating the cross-entropy loss is: ,in, Indicates the first Event categories, Indicates the true category of physical events of the device. This represents the probability distribution of the audio channel classifier output after passing through the softmax function. Indicates the number of event categories; Step 2.3: Performance evaluation of the audio channel event classifier: The test data from the HomeSound-13 audio dataset collected based on the self-built experimental platform is used. The 128-dimensional audio feature representation obtained in step 1.4 is input into the audio channel event classifier that has been trained in step 2.2 to calculate four evaluation metrics: accuracy, precision, recall, and F1 score.

4. The non-intrusive dual-channel smart home security monitoring method according to claim 3, characterized in that, The calculation methods for the four evaluation indicators are as follows: The formula for calculating accuracy is: , The formula for calculating recall rate is: , The formula for calculating the F1 score is: , The formula for calculating accuracy is: , In the formula: Indicates the first Event categories, Indicates the number of event categories. Representing a real example, Indicates a false counterexample. Indicates a false positive example. This indicates a true counterexample.

Images