Acoustic side channel attack detection and identification method and device based on multi-dimensional information fusion

Abstract

This application relates to the field of industrial manufacturing technology, specifically, to a method and device for detecting and identifying acoustic side-channel attacks using multi-dimensional information fusion. This method can, to a certain extent, address the problem of acceleration and audio sensors in industrial environments being attacked by acoustic side-channels, leading to false alarms in predictive maintenance systems, as well as the scarcity of fault sample data. The method first dynamically adjusts the trust score through a target detection algorithm to monitor physical intrusions to defend against non-physical contact attacks, enhancing the system's security protection capabilities. It then performs multimodal data fusion on acceleration and audio data to generate a new dataset to train the fusion algorithm. The algorithm uses a deep separable convolutional network and a residual network architecture to perform in-depth analysis of the acceleration and audio data, respectively, and inputs the extracted relevant feature information into an autoencoder and a convolutional attention mechanism to derive a relevant score, thereby improving the detection capability of acoustic side-channel attacks.

Description

Technical Field

[0001] The present application relates to the field of industrial manufacturing technology, and more specifically, to a method and device for detecting and identifying acoustic side-channel attacks using multi-dimensional information fusion. Background Art

[0002] With the rapid development of industry, a large number of low-cost acceleration sensors and acoustic sensors have been introduced into the industrial manufacturing field. These sensors obtain a large number of operating status parameters of industrial equipment at different locations. Moreover, acceleration and audio data are both waveform data, and the characteristics of waveform data at different locations vary greatly. In this context, predictive maintenance systems that rely on a single security layer are more vulnerable to complex attacks.

[0003] Bearing failure is the most common type of equipment failure. With the rapid growth of multi-sensor waveform data, the field of industrial equipment fault diagnosis has ushered in new development opportunities, but also faces new security threats, especially acoustic side-channel attacks targeting the physical characteristics of the equipment.

[0004] Nowadays, multimodal fusion is developing rapidly. As a technology that integrates multiple data sources, multimodal fusion has shown great potential in the system's perception and analysis capabilities. Currently, in the fields of image recognition, speech processing, etc., multimodal fusion has successfully improved the performance and robustness of the system.

[0005] However, traditional solutions rely solely on accelerometers or audio sensors, which have obvious limitations in dealing with side-channel attacks that are highly complex and diverse. In addition, in practical applications, most mechanical equipment operates in normal conditions for a long time, resulting in a scarcity of fault samples and data imbalance. Summary of the Invention

[0006] In order to solve the problem that traditional solutions rely solely on accelerometers or audio sensors, which have obvious limitations in dealing with side-channel attacks with high complexity and diversity, and in practical applications, most mechanical equipment runs in normal state for a long time, resulting in a scarcity of fault samples and data imbalance, the present application provides an acoustic side-channel attack detection and identification method and device with multi-dimensional information fusion.

[0007] The embodiment of the present application is implemented as follows:

[0008] In a first aspect, the present application provides a multi-dimensional information fusion acoustic side channel attack detection and identification method, including:

[0009] Use a target algorithm to identify specific targets in the scene and define a warning area for the specific target. At the same time, based on the target detection results and its interaction with surrounding non-target objects, a relevant confidence score is calculated;

[0010] Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode;

[0011] The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model is trained in the fused feature space;

[0012] The generated adversarial samples are mixed with the original labeled samples for secondary training to enhance the model's detection ability against human-made adversarial attacks.

[0013] In a possible implementation, obtaining the relevant credibility score by calculation further includes:

[0014] The relevant trust score is calculated by dynamically adjusting the function. That is, the trust score is obtained by using target detection technology to judge the intrusion of others and the impact of non-target objects on the target object. The formula is as follows:

[0015]

[0016] Among them, a i Is to calculate the adjustment factor, through the nonlinear function for each risk score r i The coefficient k of the dynamic selection of the risk score range is calculated by a i Calculate the cumulative adjustment factor A;

[0017] The following formula is obtained by calculating the above formula:

[0018]

[0019] Where n is the total number of risk factors;

[0020] The final confidence score is:

[0021] S confidence =yolod(min(max(w·C+(1-w)·A,0),1));

[0022] Among them, S confidence Represents the state credibility score, C represents the original credibility and the weight w of the original credibility, and yolod represents all objects and risk factors detected by the Yolo algorithm over a period of time.

[0023] In a possible implementation, obtaining the relevant credibility score by calculation further includes:

[0024] The sensor data is divided into vibration acceleration data and audio data fusion to obtain a certain fault state of the equipment, as shown in the formula:

[0025]

[0026] Among them, S status Represents the device status, fusion represents the multimodal fusion algorithm, Represents the value of vibration acceleration, The value of the audio signal represented;

[0027] When an attacker attacks, he will use data to attack, which will cause the acceleration and audio to change. At the same time, k will also change. Then the data fusion will change to the following formula:

[0028]

[0029] in, and They represent the vibration acceleration and audio signal values ​​after being affected by the attack.

[0030] In one possible implementation, performing modal feature extraction on the collected acceleration data and audio data further includes extracting features from the acceleration data:

[0031] Extract standard deviation, skewness, and kurtosis from the time domain;

[0032] From the frequency domain perspective, FFT and power spectrum density (PSD) are used to extract frequency domain features;

[0033] WT is used to extract features from acceleration data.

[0034] In one possible implementation, performing modal feature extraction on the collected acceleration data and audio data further includes extracting features from the audio data:

[0035] The continuous audio signal S(t) is divided into several short frames Sf(t), and each frame signal is Fourier transformed to obtain the spectrum F(K);

[0036] The spectrum F(K) is mapped to the Mel scale m and processed by the Mel filter bank:

[0037] After discrete cosine transform (DCT), 20 cepstral coefficients are extracted to generate MFCC feature maps.

[0038] In a possible implementation, extracting features from the audio data further includes:

[0039] For long audio stream files, we cut the audio files into segments every five seconds to ensure that they match the acquisition period of vibration acceleration data, thereby achieving data synchronization. MFCC is extracted from each segment and visualized.

[0040] In a possible implementation, the fusion network includes:

[0041] In the acceleration sub-network module, the input layer receives the combined features of the six features of the acceleration data. The processed features are sent to the acceleration network for classification;

[0042] The audio sub-network module converts audio data into MFCC feature maps as input. It uses a pre-trained ResNet18 model to perform deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms, and fully connected layers.

[0043] The fusion network module combines the dynamic features of the acceleration data and the MFCC features of the audio extracted by the two sub-networks into a unified feature representation; this fused feature is used through a convolutional self-attention mechanism to enhance key information and obtain a new feature representation; this enhanced feature representation is then fed into the encoder part of the autoencoder for further feature extraction and compression to reduce the feature dimension; the decoder part of the autoencoder then attempts to reconstruct the features to ensure that important information is effectively captured; the compressed features are processed and transformed through a series of fully connected layers for final classification, outputting category labels corresponding to different device states.

[0044] In one possible implementation, the acceleration sub-network module adopts a depthwise separable convolutional layer, which extracts spatial features through depthwise convolution and reduces the number of model parameters through point-by-point convolution. After two depthwise separable convolutions and maximum pooling, the dimension of the data is reduced while retaining key feature information.

[0045] The acceleration sub-network module also includes a Dropout layer to reduce the risk of overfitting and enhance the generalization ability of the model on unseen data;

[0046] The acceleration sub-network module performs final processing and classification decisions on the features through three fully connected layers. The three fully connected layers gradually refine the features and ultimately output classification results for specific tasks.

[0047] In one possible implementation, the audio subnetwork module includes a custom convolutional block, which includes a convolutional layer, batch normalization, and a ReLU activation function. The convolutional block performs preliminary feature extraction on the input audio data. The audio feature map is passed through the second and third blocks composed of ResNet18 residual units. These residual units enhance the depth and learning ability of the network while avoiding the vanishing gradient problem common in deep networks.

[0048] The audio sub-network module also introduces a spatial attention mechanism. The key feature extracted from the acceleration data is FA. The audio sub-network based on the ResNet18 architecture processes the audio signal converted into MFCC feature maps and extracts complex audio features FM through residual connections and the spatial attention mechanism.

[0049] In the final classification stage, the audio sub-network module uses a fully connected layer to process the pooled features and make the final classification judgment. The network also includes a Dropout layer to reduce overfitting and enhance the generalization ability of the model.

[0050] In a second aspect, the present application provides a multi-dimensional information fusion acoustic side channel attack detection and identification device, comprising:

[0051] The target recognition module is used to identify specific targets in the scene using a target algorithm, define a warning area for the specific target, and calculate a relevant confidence score based on the target detection results and its interaction with surrounding non-target objects;

[0052] A feature extraction module is used to perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality;

[0053] The integrated training module is used to integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space;

[0054] The hybrid training module is used to mix the generated adversarial samples with the original labeled samples for secondary training to enhance the model's detection ability against human adversarial attacks.

[0055] The technical solution provided by this application can achieve at least the following beneficial effects:

[0056] This application proposes a multi-dimensional information fusion method for detecting and identifying acoustic side-channel attacks. By proposing a multi-dimensional information fusion solution for acoustic side-channel attacks on sensors, the method cascades two layers of technology to combat attacks: a multimodal fusion network and target detection technology. Compared to traditional single-modal networks, the multimodal fusion network demonstrates higher accuracy and robustness when processing complex data. Target detection technology enables efficient and accurate identification of unauthorized intrusions in industrial environments, and the target detection algorithm implements dynamic trust score adjustment to ensure the quadratic accuracy of the multimodal fusion network.

[0057] In addition, in response to the limitations of insufficient fault samples and data imbalance in the original dataset, this application proposes a method to improve the dataset, which is to use FGSM to generate new adversarial samples for the original dataset, mix the adversarial samples into the original dataset to form a new dataset, and the new dataset can enhance the network model's resistance to attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 This is a flowchart of a multi-dimensional information fusion acoustic side channel attack detection and identification method shown in an exemplary embodiment of the present application;

[0060] Figure 2 This is a flowchart of a specific implementation process of the identification method shown in an exemplary embodiment of the present application;

[0061] Figure 3 1 is a schematic diagram of an MFCC feature extraction process shown in an exemplary embodiment of the present application;

[0062] Figure 4 1 is a schematic diagram of an acceleration network architecture with depthwise separable convolution, shown in an exemplary embodiment of the present application;

[0063] Figure 5 is a schematic diagram of a network architecture in the convergence stage shown in an exemplary embodiment of the present application;

[0064] Figure 6 It is a structural diagram of an acoustic side-channel attack detection and identification device with multi-dimensional information fusion shown in an exemplary embodiment of the present application.

[0065] Reference numerals:

[0066] 1. Target recognition module; 2. Feature extraction module; 3. Integrated training module; 4. Hybrid training module. DETAILED DESCRIPTION

[0067] In order to make the purpose, implementation methods and advantages of the present application clearer, the exemplary implementation methods of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0068] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0069] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0070] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0071] Before explaining the multi-dimensional information fusion acoustic side-channel attack detection and identification method provided in the embodiment of the present application, the application scenario and implementation environment of the embodiment of the present application are first introduced.

[0072] With the rapid development of industry, a large number of low-cost acceleration sensors and acoustic sensors have been introduced into the industrial manufacturing field. These sensors obtain a large number of operating status parameters of industrial equipment at different locations. Moreover, acceleration and audio data are both waveform data, and the characteristics of waveform data at different locations vary greatly. In this context, predictive maintenance systems that rely on a single security layer are more vulnerable to complex attacks.

[0073] Bearing failure is the most common type of equipment failure. With the rapid growth of multi-sensor waveform data, the field of industrial equipment fault diagnosis has ushered in new development opportunities, but also faces new security threats, especially acoustic side-channel attacks targeting the physical characteristics of the equipment.

[0074] Nowadays, multimodal fusion is developing rapidly. As a technology that integrates multiple data sources, multimodal fusion has shown great potential in the system's perception and analysis capabilities. Currently, in the fields of image recognition, speech processing, etc., multimodal fusion has successfully improved the performance and robustness of the system.

[0075] However, traditional solutions rely solely on accelerometers or audio sensors, which have obvious limitations in dealing with side-channel attacks that are highly complex and diverse. In addition, in practical applications, most mechanical equipment operates in normal conditions for a long time, resulting in a scarcity of fault samples and data imbalance.

[0076] Based on this, this application provides a method and device for detecting and identifying acoustic side-channel attacks using multi-dimensional information fusion. This multi-dimensional information fusion-based defense mechanism cascades two defense layers, using target detection technology and a multimodal data fusion detection algorithm. Through this cascaded approach, the security system can monitor and respond to potential security threats from different perspectives, providing more comprehensive protection for the system.

[0077] Next, the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems will be described in detail through embodiments and in conjunction with the accompanying drawings. The various embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them.

[0078] Figure 1 It is a flowchart of an acoustic side-channel attack detection and identification method based on multi-dimensional information fusion shown in an exemplary embodiment of the present application.

[0079] In an exemplary embodiment, Figure 1 As shown, a multi-dimensional information fusion acoustic side channel attack detection and identification method is provided, which may include the following steps:

[0080] Step 100: Use a target algorithm to identify a specific target in the scene and define a warning area for the specific target. At the same time, based on the target detection results and its interaction with surrounding non-target objects, a relevant credibility score is calculated.

[0081] Step 200: Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality.

[0082] Step 300: The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model training is performed in the fused feature space.

[0083] Step 400: Mix the generated adversarial samples with the original labeled samples and perform secondary training to enhance the model's ability to detect human adversarial attacks.

[0084] Figure 2 It is a flowchart of a specific implementation process of the identification method shown in an exemplary embodiment of the present application.

[0085] In one possible implementation, Figure 2 As shown, the specific steps of the identification method are as follows:

[0086] 1. Use the target algorithm to identify specific targets in the scene and define warning areas for these targets. Based on the target detection results and their interactions with surrounding non-target objects, the relevant trust scores are calculated by dynamically adjusting the function using formulas (2), (3), and (4).

[0087] 2. Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode.

[0088] 3. The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model training is performed in the fused feature space.

[0089] 4. Mix the generated adversarial samples with the original labeled samples and perform secondary training to enhance the model's ability to detect human adversarial attacks.

[0090] In one possible implementation, the dynamic adjustment of the trust score is specifically achieved through the following steps:

[0091] By using target detection technology to judge the intrusion of others, the credibility score of the impact of non-target objects on the target object is obtained, as shown in formula (2) and formula (3):

[0092]

[0093] Among them, a i Is to calculate the adjustment factor, through the nonlinear function for each risk score r i The coefficient k of the dynamic selection of the risk score range is calculated by a i Calculate the cumulative adjustment factor A, that is, calculate formula (3) through formula (2).

[0094]

[0095] Where n is the total number of risk factors. The final confidence score is shown in formula (4):

[0096] S confidence=yolod(min(max(w·C+(1-w)·A,0),1)) (4)

[0097] Among them, S confidence Represents the state credibility score, C represents the original credibility and the weight w of the original credibility, and yolod represents all objects and risk factors detected by the Yolo algorithm over a period of time.

[0098] The sensor data is divided into vibration acceleration data and audio data fusion to obtain a certain fault state of the equipment. As shown in formula (5):

[0099]

[0100] Among them, S status Represents the device status, fusion represents the multimodal fusion algorithm, Represents the value of vibration acceleration, represents the value of the audio signal. If an attacker wants to attack, he will start with the data, which will cause the acceleration and audio to change, and k will also change. Then the data fusion will become as shown in the following formula (6):

[0101]

[0102] in, and They represent the vibration acceleration and audio signal values ​​after being affected by the attack.

[0103] Figure 3 FIG. 1 is a schematic diagram of an MFCC feature extraction process shown in an exemplary embodiment of the present application.

[0104] In one possible implementation, the feature extraction of the fusion network is specifically implemented by the following steps:

[0105] 1. Extraction of acceleration features

[0106] Acceleration data, as a key time series data, reflects the dynamic characteristics of objects over time. Here, a multidimensional feature extraction method is employed to deeply analyze acceleration data. From a basic statistical perspective, standard deviation, skewness, and kurtosis are first extracted from the time domain. Next, frequency domain features are extracted using FFT and power spectral density (PSD). Finally, WT is used to extract features from the acceleration data. Compared to traditional time and frequency domain methods, wavelet transform features provide the signal's time-frequency characteristics.

[0107] A total of six eigenvectors were extracted from the acceleration data, and these six eigenvectors were concatenated to form a multi-dimensional feature vector, as shown in Table 1 below, which shows the six features of the multi-dimensional feature set and their descriptions.

[0108] Table 1 Manual feature extraction for acceleration data

[0109]

[0110]

[0111] 2. Audio Feature Extraction

[0112] In audio signal processing, MFCC is the core technology for extracting sound features. It effectively extracts the main features of audio signals by simulating the auditory characteristics of the human ear. The MFCC feature extraction process is as follows: Figure 3 shown.

[0113] (1) The continuous audio signal S(t) is divided into several short frames Sf(t). Each frame signal is Fourier transformed to obtain the spectrum F(K). Then, these spectrum F(K) are mapped to the Mel scale m and processed by the Mel filter bank:

[0114]

[0115] Finally, 20 cepstral coefficients are extracted through discrete cosine transform (DCT) to generate the MFCC feature map.

[0116] The implementation process is shown in formula (7):

[0117] MFCC=DCT(log(MelFilterBank(F(k))))(7)

[0118] (2) For long audio stream files, this paper adopts the method of cutting the audio files into segments every five seconds to ensure that it matches the acquisition period of vibration acceleration data, thereby achieving data synchronization, extracting MFCC from each segment and visualizing it.

[0119] Figure 4 This is a schematic diagram of an acceleration network architecture with depthwise separable convolution shown in an exemplary embodiment of the present application. Figure 5 FIG. 1 is a schematic diagram of a network architecture in the fusion stage shown in an exemplary embodiment of the present application.

[0120] In a possible implementation, the construction of the feature fusion network is specifically implemented through the following steps:

[0121] The multidimensional features extracted from acceleration and audio data are a combination of six acceleration features and MFCC multidimensional feature maps of audio. The main components of the overall fusion network are as follows:

[0122] 1. Acceleration sub-network module

[0123] The input layer of the acceleration sub-network model is designed to receive the combined features of the six features of acceleration data. The entire acceleration network architecture is as follows: Figure 4 As shown, the processed features are fed into the acceleration network for classification.

[0124] The network uses a depth-wise separable convolution layer, which first performs depth-wise convolution to extract spatial features and then performs point-by-point convolution to reduce the number of model parameters. After two depth-wise separable convolutions and maximum pooling, the dimension of the data is reduced while retaining key feature information.

[0125] The network also includes a dropout layer to reduce the risk of overfitting and enhance the model's generalization ability on unseen data. After this, the network performs final feature processing and classification decisions through three fully connected layers. These fully connected layers gradually refine the features and ultimately output classification results for specific tasks. Through this layered approach, the model can learn complex representations from acceleration data and effectively perform classification.

[0126] 2. Audio Sub-Network Module

[0127] The input of the audio sub-network model is the audio data converted into MFCC feature maps. The network uses the pre-trained ResNet18 as the basis and performs deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms and fully connected layers.

[0128] The audio network module begins with a custom convolutional block, including convolutional layers, batch normalization, and ReLU activation functions, to perform preliminary feature extraction on the input audio data. Next, the audio feature map is passed through the second and third blocks consisting of ResNet18 residual units. These residual units enhance the network's depth and learning capabilities while avoiding the vanishing gradient problem common in deep networks.

[0129] To further focus on important feature areas, the network introduces a spatial attention mechanism. The key feature extracted from the acceleration data is FA, while the audio subnetwork based on the ResNet18 architecture processes the audio signal converted into MFCC feature maps. Through residual connections and the spatial attention mechanism, it extracts complex audio features FM, preserving key features while reducing the computational burden.

[0130] In the final classification stage, the network uses fully connected layers to process the pooled features and make the final classification decision. The network also includes a dropout layer to reduce overfitting and enhance the model's generalization ability.

[0131] 3. Converged Network Module

[0132] In this fusion network, the early fusion method is adopted. The fusion network model is shown in the figure below. Figure 5 shown.

[0133] First, the dynamic features of acceleration data and MFCC features of audio are extracted through two sub-networks respectively.

[0134] Then, these features are concatenated into a unified feature representation F = (F A ; F M ), which combines key information from acceleration and audio.

[0135] This fused feature is then passed through the convolutional self-attention mechanism to strengthen the key information and obtain a new feature representation F = CA(F′).

[0136] Next, this enhanced feature representation is fed into the encoder part of the autoencoder for further feature extraction and compression, thereby reducing the feature dimension to Z = Encoder(F′).

[0137] The decoder part of the autoencoder then tries to reconstruct the features To ensure that important information is captured effectively.

[0138] Finally, the compressed features are processed and transformed through a series of fully connected layers for final classification, outputting category labels corresponding to different device states. This process also incorporates nonlinear activation functions and Dropout regularization strategies to enhance the network's learning capabilities and prevent overfitting.

[0139] It should be understood that, although the various steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the instructions, these steps are not necessarily performed in sequence according to the order of the instructions. Unless otherwise specified in the present application, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0140] Corresponding to the embodiment of the aforementioned multi-dimensional information fusion acoustic side-channel attack detection and identification method, adopting the same technical concept, the present application also provides an embodiment of the multi-dimensional information fusion acoustic side-channel attack detection and identification device.

[0141] Figure 6 It is a structural diagram of an acoustic side-channel attack detection and identification device with multi-dimensional information fusion shown in an exemplary embodiment of the present application.

[0142] In an exemplary embodiment, Figure 6 As shown, the multi-dimensional information fusion acoustic side channel attack detection and identification device includes:

[0143] Target recognition module 1 is used to identify specific targets in the scene using a target algorithm, demarcate a warning area for the specific target, and calculate a relevant confidence score based on the target detection results and its interaction with surrounding non-target objects;

[0144] Feature extraction module 2 is used to perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality;

[0145] Integration training module 3 is used to integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space;

[0146] The hybrid training module 4 is used to mix the generated adversarial samples with the original labeled samples for secondary training to enhance the model's detection ability against human adversarial attacks.

[0147] It can be seen that:

[0148] 1. This application proposes a multi-dimensional information fusion solution for acoustic side-channel attacks on sensors. This solution cascades two layers of technology to combat attacks: a multimodal fusion network and target detection. Compared to traditional single-modal networks, this multimodal fusion network demonstrates higher accuracy and robustness when processing complex data. Target detection technology enables efficient and accurate identification of unauthorized intrusions in industrial environments. Furthermore, the target detection algorithm implements dynamic trust score adjustment to ensure the quadratic accuracy of the multimodal fusion network.

[0149] 2. To address the limitations of the original dataset, which suffers from insufficient fault samples and data imbalance, this application proposes a method for improving the dataset. This method uses FGSM to generate new adversarial samples from the original dataset. These adversarial samples are then mixed into the original dataset to form a new dataset. This new dataset can enhance the network model's resistance to attacks.

[0150] 3. This application compares and analyzes the performance of the defense mechanism, and compares in detail the performance of the fusion algorithms under different circumstances and different fusion architectures. For the specific definition of the acoustic side channel attack detection and identification device with multi-dimensional information fusion, please refer to the definition of the acoustic side channel attack detection and identification method with multi-dimensional information fusion above, which will not be repeated here. Each module in the above-mentioned acoustic side channel attack detection and identification device with multi-dimensional information fusion can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-dimensional information fusion acoustic side channel attack detection and identification method, characterized by: include: Use a target algorithm to identify specific targets in the scene and define a warning area for the specific target. At the same time, based on the target detection results and its interaction with surrounding non-target objects, a relevant confidence score is calculated; Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode; The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model is trained in the fused feature space; The generated adversarial samples are mixed with the original labeled samples for secondary training to enhance the model's ability to detect human adversarial attacks; The fusion network includes: The acceleration sub-network module has an input layer that receives a combination of six features of acceleration data. The processed features are then fed into the acceleration sub-network for classification. The audio sub-network module converts audio data into MFCC feature maps as input. It uses a pre-trained ResNet18 model to perform deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms, and fully connected layers. The fusion network module combines the dynamic features of the acceleration data and the MFCC features of the audio extracted by the two sub-networks into a unified feature representation; this fused feature is used through a convolutional self-attention mechanism to enhance key information and obtain a new feature representation; this enhanced feature representation is then fed into the encoder part of the autoencoder for further feature extraction and compression to reduce the feature dimension; the decoder part of the autoencoder then attempts to reconstruct the features to ensure that important information is effectively captured; the compressed features are processed and transformed through a series of fully connected layers for final classification, outputting category labels corresponding to different device states.

2. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1 is characterized in that: The step of obtaining the relevant credibility score by calculation further includes: The relevant trust score is calculated by dynamically adjusting the function. That is, the trust score is obtained by using target detection technology to judge the intrusion of others and the impact of non-target objects on the target object. The formula is as follows: Among them, a i Is to calculate the adjustment factor, through the nonlinear function for each risk score r i and the coefficient k of the dynamic selection of the risk score range; by a i Calculate the cumulative adjustment factor A; Where n is the total number of risk factors; The final confidence score is: S confidence =yolod(min(max(w C+(1-w) A,0),1)); Among them, S confidence Represents the state credibility score, C represents the original credibility and the weight w of the original credibility, and yolod represents all objects and risk factors detected by the Yolo algorithm over a period of time.

3. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The calculation of the relevant credibility score also includes: The fusion of vibration acceleration data and audio data can be used to determine a certain fault state of the equipment, as shown in the formula: Among them, S status Represents the device status, fusion represents the multimodal fusion algorithm, Represents the value of vibration acceleration, The value of the audio signal represented; When the attacker attacks, the data fusion will change to the following: in, and They respectively represent the values ​​of vibration acceleration and audio signal after being affected by the attack.

4. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The performing of modal feature extraction on the collected acceleration data and audio data further includes extracting features from the acceleration data: Extract standard deviation, skewness, and kurtosis from the time domain; From the perspective of frequency domain, FFT and power spectrum density are used to extract frequency domain features; WT is used to extract features from acceleration data.

5. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The performing of modal feature extraction on the collected acceleration data and audio data further includes extracting features from the audio data: The continuous audio signal S(t) is divided into several short frames Sf(t), and each frame signal is Fourier transformed to obtain the spectrum F(K); The spectrum F(K) is mapped to the Mel scale m and processed by the Mel filter bank: After discrete cosine transform, 20 cepstral coefficients are extracted to generate MFCC feature maps.

6. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 5, characterized in that: The feature extraction of the audio data further includes: For long audio stream files, we cut the audio files into segments every five seconds to ensure that they match the acquisition period of vibration acceleration data, thereby achieving data synchronization. MFCC is extracted from each segment and visualized.

7. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The acceleration sub-network module uses a depthwise separable convolutional layer, which extracts spatial features through depthwise convolution and reduces the number of model parameters through point-by-point convolution. After two depthwise separable convolutions and maximum pooling, the dimension of the data is reduced while retaining key feature information. The acceleration sub-network module also includes a Dropout layer; The acceleration sub-network module performs final processing and classification decisions on the features through three fully connected layers. The three fully connected layers gradually refine the features and ultimately output classification results for specific tasks.

8. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The audio sub-network module includes a custom convolutional block, which includes a convolutional layer, batch normalization, and ReLU activation function to perform preliminary feature extraction on the input audio data; The audio sub-network module also introduces a spatial attention mechanism. The key feature extracted from the acceleration data is FA. The audio sub-network based on the ResNet18 architecture processes the audio signal converted into MFCC feature maps and extracts complex audio features FM through residual connections and the spatial attention mechanism. In the final classification stage, the audio sub-network module uses a fully connected layer to process the pooled features and make a final classification judgment. The audio sub-network module also includes a Dropout layer.

9. A multi-dimensional information fusion acoustic side channel attack detection and identification device, characterized in that: include: The target recognition module is used to identify specific targets in the scene using a target algorithm, define a warning area for the specific target, and calculate a relevant confidence score based on the target detection results and its interaction with surrounding non-target objects; A feature extraction module is used to perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality; The integrated training module is used to integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space; A hybrid training module is used to mix the generated adversarial samples with the original labeled samples for secondary training to enhance the model's ability to detect human adversarial attacks; The fusion network includes: The acceleration sub-network module has an input layer that receives a combination of six features of acceleration data. The processed features are then fed into the acceleration sub-network for classification. The audio sub-network module converts audio data into MFCC feature maps as input. It uses a pre-trained ResNet18 model to perform deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms, and fully connected layers. The fusion network module combines the dynamic features of the acceleration data and the MFCC features of the audio extracted by the two sub-networks into a unified feature representation; this fused feature is used through a convolutional self-attention mechanism to enhance key information and obtain a new feature representation; this enhanced feature representation is then fed into the encoder part of the autoencoder for further feature extraction and compression to reduce the feature dimension; the decoder part of the autoencoder then attempts to reconstruct the features to ensure that important information is effectively captured; the compressed features are processed and transformed through a series of fully connected layers for final classification, outputting category labels corresponding to different device states.

Images