Acoustic side channel attack detection and identification method and device based on multi-dimensional information fusion

Abstract

The invention relates to the technical field of industrial manufacturing, in particular to an acoustic side channel attack detection and recognition method and device based on multi-dimensional information fusion, and can solve the problem that a predictive maintenance system has false alarms due to the fact that an acceleration and an audio sensor in an industrial environment are attacked by an acoustic side channel to cause the failure of the detection and recognition to a certain extent. And meanwhile, the problem of fault sample data scarcity exists. According to the method, firstly, a credible score is dynamically adjusted through a target detection algorithm, physical intrusion monitoring is achieved to defend non-physical contact attacks, the safety protection capability of a system is enhanced, then multi-modal data fusion is carried out on acceleration and audio data, a new data set is generated to train a fusion algorithm, and the safety of the system is improved. According to the algorithm, acceleration and audio data are deeply analyzed through a deep separable convolutional network and a residual network architecture, and extracted related feature information is input into an auto-encoder and a convolutional attention mechanism to obtain a related score, so that the detection capability of acoustic side channel attacks is improved.

Description

Technical Field

[0001] The present application relates to the field of industrial manufacturing technology, and more specifically, to a method and device for detecting and identifying acoustic side-channel attacks using multi-dimensional information fusion. Background Art

[0002] With the rapid development of industry, a large number of low-cost acceleration sensors and acoustic sensors have been covered in the industrial manufacturing field. These sensors obtain a large number of operating status parameters of industrial equipment at different locations. Moreover, acceleration and audio data are both waveform data. The characteristics of waveform data at different locations are quite different. In this context, predictive maintenance systems that rely on a single security layer are more vulnerable to complex attacks.

[0003] Bearing failure is the most common type of equipment failure. Today, with the rapid growth of multi-sensor waveform data, the field of industrial equipment fault diagnosis has ushered in new development opportunities, but also faces new security threats, especially acoustic side-channel attacks targeting the physical characteristics of the equipment.

[0004] Nowadays, multimodal fusion is developing rapidly. As a technology that integrates multiple data sources, multimodal fusion has shown great potential in the perception and analysis capabilities of the system. Currently, in the fields of image recognition and speech processing, multimodal fusion has successfully improved the performance and robustness of the system.

[0005] However, traditional solutions only rely on accelerometers or audio sensors, which have obvious limitations in dealing with side-channel attacks with high complexity and diversity. In addition, in practical applications, most mechanical equipment runs in normal state for a long time, resulting in a scarcity of fault samples and data imbalance. Summary of the invention

[0006] In order to solve the problem that traditional solutions only rely on accelerometers or audio sensors, which have obvious limitations in dealing with side channel attacks with high complexity and diversity, and in practical applications, most mechanical equipment runs in normal state for a long time, resulting in scarcity of fault samples and data imbalance, the present application provides an acoustic side channel attack detection and identification method and device with multi-dimensional information fusion.

[0007] The embodiment of the present application is implemented as follows:

[0008] In a first aspect, the present application provides a method for detecting and identifying acoustic side channel attacks using multi-dimensional information fusion, including:

[0009] Use the target algorithm to identify specific targets in the scene and define a warning area for the specific targets. At the same time, according to the target detection results and its interaction with surrounding non-target objects, the relevant trust score is calculated;

[0010] Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode;

[0011] The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model is trained in the fused feature space;

[0012] The generated adversarial samples are mixed with the original labeled samples for secondary training to enhance the model's detection ability against human adversarial attacks.

[0013] In a possible implementation, obtaining the relevant credibility score by calculation further includes:

[0014] The relevant trust score is calculated by dynamically adjusting the function, that is, using the target detection technology to judge the intrusion of others, and the trust score obtained for the impact of non-target objects on the target object is as follows:

[0015]

[0016] Among them, a i is to calculate the adjustment factor, which is a nonlinear function for each risk score r i and the coefficient k dynamically selected by the risk score range. i Calculate the cumulative adjustment factor A;

[0017] The following formula is obtained by calculating the above formula:

[0018]

[0019] Where n is the total number of risk factors;

[0020] The final confidence score is:

[0021] S confidence =yolod(min(max(w·C+(1-w)·A,0),1));

[0022] Among them, S confidence represents the state credibility score, C represents the original credibility and the weight w of the original credibility, and yolod represents all objects and risk factors detected by the Yolo algorithm over a period of time.

[0023] In a possible implementation, obtaining the relevant credibility score by calculation further includes:

[0024] The sensor data is divided into vibration acceleration data and audio data. The fusion results in a certain fault state of the equipment, as shown in the formula:

[0025]

[0026] Among them, S status represents the device status, fusion represents the multimodal fusion algorithm, Represents the value of vibration acceleration, The value of the audio signal represented;

[0027] When an attacker attacks, he will start with data, which will cause the acceleration and audio to change, and k will also change. Then the data fusion will change to the following formula:

[0028]

[0029] in, and They respectively represent the vibration acceleration and audio signal values ​​after being affected by the attack.

[0030] In a possible implementation, performing modal feature extraction on the collected acceleration data and audio data further includes extracting features from the acceleration data:

[0031] Extract standard deviation, skewness, and kurtosis from the time domain;

[0032] From the frequency domain perspective, FFT and Power Spectrum Density (PSD) are used to extract frequency domain features;

[0033] Use WT to extract features from acceleration data.

[0034] In a possible implementation, performing modal feature extraction on the collected acceleration data and audio data further includes extracting features from the audio data:

[0035] The continuous audio signal S(t) is divided into several short frames Sf(t), and each frame signal is Fourier transformed to obtain the spectrum F(K);

[0036] The spectrum F(K) is mapped to the Mel scale m and processed by the Mel filter bank:

[0037] After discrete cosine transform (DCT), 20 cepstral coefficients are extracted to generate MFCC feature maps.

[0038] In a possible implementation, the extracting features from the audio data further includes:

[0039] For long audio stream files, the method of cutting the audio files into segments every five seconds is adopted to ensure that it matches the acquisition cycle of vibration acceleration data, thereby achieving data synchronization, extracting MFCC from each segment and visualizing it.

[0040] In a possible implementation, the fusion network includes:

[0041] In the acceleration sub-network module, the input layer receives the combined features of the six features of the acceleration data, and the processed features are sent to the acceleration network for classification;

[0042] The audio sub-network module converts audio data into MFCC feature maps as input, uses the pre-trained ResNet18 foundation, and performs deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms, and fully connected layers;

[0043] The fusion network module concatenates the dynamic features of the acceleration data and the MFCC features of the audio extracted by the two sub-networks into a unified feature representation; the fused features are used through the convolutional self-attention mechanism to strengthen the key information and obtain a new feature representation; the strengthened feature representation is then sent to the encoder part of the autoencoder for further feature extraction and compression to reduce the feature dimension; the decoder part of the autoencoder attempts to reconstruct the features to ensure that important information is effectively captured; the compressed features are processed and transformed through a series of fully connected layers for final classification, and the category labels corresponding to different device states are output.

[0044] In one possible implementation, the acceleration sub-network module adopts a depthwise separable convolutional layer, which extracts spatial features through depthwise convolution and reduces the number of parameters of the model through point-by-point convolution. After two depthwise separable convolutions and maximum pooling, the dimension of the data is reduced while retaining key feature information.

[0045] The acceleration sub-network module also includes a Dropout layer to reduce the risk of overfitting and enhance the generalization ability of the model on unseen data;

[0046] The acceleration sub-network module performs final processing and classification decisions on the features through three fully connected layers. The three fully connected layers gradually refine the features and finally output classification results for specific tasks.

[0047] In a possible implementation, the audio subnetwork module includes a custom convolutional block, which includes a convolutional layer, batch normalization and a ReLU activation function, performs preliminary feature extraction on the input audio data, and the audio feature map is passed through the second and third blocks composed of residual units of ResNet18. These residual units enhance the depth and learning ability of the network, while avoiding the common gradient vanishing problem of deep networks.

[0048] At the same time, the audio sub-network module introduces a spatial attention mechanism. The key feature extracted from the acceleration data is FA, and the audio sub-network based on the ResNet18 architecture processes the audio signal converted into the MFCC feature map, and extracts the complex audio feature FM through residual connection and spatial attention mechanism;

[0049] In the final classification stage, the audio subnetwork module uses a fully connected layer to process the pooled features and make the final classification judgment. The network also includes a Dropout layer to reduce overfitting and enhance the generalization ability of the model.

[0050] In a second aspect, the present application provides a multi-dimensional information fusion acoustic side channel attack detection and identification device, comprising:

[0051] A target recognition module is used to use a target algorithm to identify a specific target in the scene and define a warning area for the specific target. At the same time, according to the target detection result and its interaction with surrounding non-target objects, a relevant credibility score is calculated;

[0052] A feature extraction module is used to perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality;

[0053] The integrated training module is used to integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space;

[0054] The hybrid training module is used to mix the generated adversarial samples with the original labeled samples for secondary training to enhance the model's detection ability against artificial adversarial attacks.

[0055] The technical solution provided by this application can at least achieve the following beneficial effects:

[0056] The multi-dimensional information fusion acoustic side channel attack detection and identification method provided by this application proposes a multi-dimensional information fusion solution for acoustic side channel attacks on sensors, cascading two-layer technology to resist attacks, namely multi-modal fusion network and target detection technology. Compared with the traditional single-modal network, the multi-modal fusion network shows higher accuracy and robustness in processing complex data. The target detection technology realizes efficient and accurate identification of unauthorized intrusion behaviors in industrial environments, and the target detection algorithm realizes dynamic trust score adjustment to ensure the secondary accuracy of the multi-modal fusion network.

[0057] In view of the limitations of insufficient fault samples and data imbalance in the original data set, this application proposes a method to improve the data set, which is to use FGSM to generate new adversarial samples for the original data set, mix the adversarial samples into the original data set to form a new data set, and the new data set can enhance the network model's resistance to attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 It is a flowchart of an acoustic side channel attack detection and identification method using multi-dimensional information fusion as shown in an exemplary embodiment of the present application;

[0060] Figure 2 It is a flowchart of a specific implementation process of the identification method shown in an exemplary embodiment of the present application;

[0061] Figure 3 is a schematic diagram of an MFCC feature extraction process shown in an exemplary embodiment of the present application;

[0062] Figure 4 is a schematic diagram of an acceleration network architecture with depthwise separable convolution shown in an exemplary embodiment of the present application;

[0063] Figure 5 is a schematic diagram of a network architecture in the convergence stage shown in an exemplary embodiment of the present application;

[0064] Figure 6 It is a structural diagram of an acoustic side-channel attack detection and identification device with multi-dimensional information fusion shown in an exemplary embodiment of the present application.

[0065] Reference numerals:

[0066] 1. Target recognition module; 2. Feature extraction module; 3. Integrated training module; 4. Hybrid training module. DETAILED DESCRIPTION

[0067] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0068] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0069] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0070] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0071] Before explaining the acoustic side-channel attack detection and identification method with multi-dimensional information fusion provided in the embodiment of the present application, the application scenario and implementation environment of the embodiment of the present application are first introduced.

[0072] With the rapid development of industry, a large number of low-cost acceleration sensors and acoustic sensors have been covered in the industrial manufacturing field. These sensors obtain a large number of operating status parameters of industrial equipment at different locations. Moreover, acceleration and audio data are both waveform data. The characteristics of waveform data at different locations are quite different. In this context, predictive maintenance systems that rely on a single security layer are more vulnerable to complex attacks.

[0073] Bearing failure is the most common type of equipment failure. Today, with the rapid growth of multi-sensor waveform data, the field of industrial equipment fault diagnosis has ushered in new development opportunities, but also faces new security threats, especially acoustic side-channel attacks targeting the physical characteristics of the equipment.

[0074] Nowadays, multimodal fusion is developing rapidly. As a technology that integrates multiple data sources, multimodal fusion has shown great potential in the perception and analysis capabilities of the system. Currently, in the fields of image recognition and speech processing, multimodal fusion has successfully improved the performance and robustness of the system.

[0075] However, traditional solutions only rely on accelerometers or audio sensors, which have obvious limitations in dealing with side-channel attacks with high complexity and diversity. In addition, in practical applications, most mechanical equipment runs in normal state for a long time, resulting in a scarcity of fault samples and data imbalance.

[0076] Based on this, the present application provides a method and device for detecting and identifying acoustic side channel attacks based on multi-dimensional information fusion, and a defense mechanism based on multi-dimensional information fusion, which cascades two defense layers, using target detection technology and multi-modal data fusion detection algorithm respectively. Through this cascade method, the security system can monitor and respond to potential security threats from different angles, thereby providing more comprehensive protection for the system.

[0077] Next, the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems will be described in detail through embodiments and in combination with the accompanying drawings. The embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all of them.

[0078] Figure 1 It is a flowchart of an acoustic side-channel attack detection and identification method based on multi-dimensional information fusion shown in an exemplary embodiment of the present application.

[0079] In an exemplary embodiment, Figure 1 As shown, a method for detecting and identifying acoustic side channel attacks by multi-dimensional information fusion is provided, and the method may include the following steps:

[0080] Step 100: Use a target algorithm to identify a specific target in the scene, and define a warning area for the specific target. At the same time, according to the target detection result and its interaction with surrounding non-target objects, a relevant credibility score is calculated.

[0081] Step 200: Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality.

[0082] Step 300: Integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space.

[0083] Step 400: Mix the generated adversarial samples with the original labeled samples and perform secondary training to enhance the model's detection capability against artificial adversarial attacks.

[0084] Figure 2 It is a flowchart of a specific implementation process of the identification method shown in an exemplary embodiment of the present application.

[0085] In one possible implementation, Figure 2 As shown, the specific steps of the identification method are as follows:

[0086] 1. Use the target algorithm to identify specific targets in the scene and define warning areas for these targets. Based on the target detection results and their interaction with surrounding non-target objects, the relevant trust scores are calculated by dynamically adjusting the function through formulas (2), (3), and (4).

[0087] 2. Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode.

[0088] 3. The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model is trained in the fused feature space.

[0089] 4. Mix the generated adversarial samples with the original labeled samples and perform secondary training to enhance the model's ability to detect artificial adversarial attacks.

[0090] In a possible implementation, the dynamic adjustment of the trust score is specifically achieved through the following steps:

[0091] By using target detection technology to judge the intrusion of others, the credibility score of the impact of non-target objects on the target object is obtained, as shown in formula (2) and formula (3):

[0092]

[0093] Among them, a i is to calculate the adjustment factor, which is a nonlinear function for each risk score r i and the coefficient k dynamically selected by the risk score range. i Calculate the cumulative adjustment factor A, that is, calculate formula (3) through formula (2).

[0094]

[0095] Where n is the total number of risk factors. The final credibility score is shown in formula (4):

[0096] S confidence=yolod(min(max(w·C+(1-w)·A,0),1)) (4)

[0097] Among them, S confidence represents the state credibility score, C represents the original credibility and the weight w of the original credibility, and yolod represents all objects and risk factors detected by the Yolo algorithm over a period of time.

[0098] The sensor data is divided into vibration acceleration data and audio data fusion to obtain a certain fault state of the equipment. As shown in formula (5):

[0099]

[0100] Among them, S status represents the device status, fusion represents the multimodal fusion algorithm, Represents the value of vibration acceleration, represents the value of the audio signal. If an attacker wants to attack, he will start with the data, which will cause the acceleration and audio to change, and k will also change. Then the data fusion will become as shown in the following formula (6):

[0101]

[0102] in, and They respectively represent the vibration acceleration and audio signal values ​​after being affected by the attack.

[0103] Figure 3 It is a schematic diagram of the MFCC feature extraction process shown in an exemplary embodiment of the present application.

[0104] In a possible implementation, the feature extraction of the fusion network is specifically implemented by the following steps:

[0105] 1. Extraction of acceleration features

[0106] Acceleration data, as a key time series data, reflects the dynamic characteristics of objects changing over time. A multi-dimensional feature extraction method is used here to deeply analyze acceleration data. From a basic statistical perspective, standard deviation, skewness and kurtosis are first extracted from the time domain. Secondly, from the frequency domain perspective, FFT and power spectrum density (PSD) are used to extract frequency domain features. Finally, WT is used to extract features from acceleration data. Compared with traditional time domain and frequency domain methods, wavelet transform features provide the time-frequency characteristics of the signal.

[0107] A total of six feature vectors are extracted from the acceleration data, and these six feature vectors are concatenated to form a multi-dimensional feature vector, as shown in Table 1 below, which shows the six features of the multi-dimensional feature set and their descriptions.

[0108] Table 1 Manual feature extraction for acceleration data

[0109]

[0110]

[0111] 2. Extraction of Audio Features

[0112] In audio signal processing, MFCC is the core technology for extracting sound features. It effectively extracts the main features of audio signals by simulating the auditory characteristics of the human ear. The MFCC feature extraction process is as follows: Figure 3 shown.

[0113] (1) The continuous audio signal S(t) is divided into several short frames Sf(t), and each frame signal is Fourier transformed to obtain the spectrum F(K). Then, the spectrum F(K) is mapped to the Mel scale m and processed by the Mel filter bank:

[0114]

[0115] Finally, 20 cepstral coefficients are extracted through discrete cosine transform (DCT) to generate the MFCC feature map.

[0116] The implementation process is shown in formula (7):

[0117] MFCC=DCT(log(MelFilterBank(F(k))))(7)

[0118] (2) For long audio stream files, this paper adopts the method of cutting the audio files into segments every five seconds to ensure that it matches the acquisition period of vibration acceleration data, thereby achieving data synchronization, extracting MFCC from each segment and visualizing it.

[0119] Figure 4 It is a schematic diagram of an acceleration network architecture with depthwise separable convolution shown in an exemplary embodiment of the present application. Figure 5 It is a schematic diagram of a network architecture in the fusion stage shown in an exemplary embodiment of the present application.

[0120] In a possible implementation, the construction of the feature fusion network is specifically implemented by the following steps:

[0121] The multi-dimensional features extracted from acceleration and audio data are the combination of six features of acceleration and the MFCC multi-dimensional feature map of audio. The main components of the overall fusion network are as follows:

[0122] 1. Acceleration sub-network module

[0123] The input layer of the acceleration sub-network model is designed to receive the combined features of the six features of acceleration data. The entire acceleration network architecture is as follows: Figure 4 As shown, the processed features are fed into the acceleration network for classification.

[0124] The network uses a depthwise separable convolutional layer, which first performs depthwise convolution to extract spatial features and then performs pointwise convolution to reduce the number of parameters of the model. After two depthwise separable convolutions and maximum pooling, the dimension of the data is reduced while retaining key feature information.

[0125] The network also includes a Dropout layer to reduce the risk of overfitting and enhance the model's generalization ability on unseen data. After that, the network performs final processing and classification decisions on the features through three fully connected layers. These fully connected layers gradually refine the features and ultimately output classification results for specific tasks. Through this layered approach, the model can learn complex representations from acceleration data and effectively classify them.

[0126] 2. Audio Sub-Network Module

[0127] The input of the audio subnetwork model is the audio data converted into MFCC feature maps. The network uses the pre-trained ResNet18 as the basis and performs deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms, and fully connected layers.

[0128] The audio network module starts with a custom convolutional block, including convolutional layers, batch normalization, and ReLU activation functions, to perform preliminary feature extraction on the input audio data. Next, the audio feature map is passed through the second and third blocks consisting of ResNet18 residual units. These residual units enhance the depth and learning ability of the network while avoiding the gradient vanishing problem common in deep networks.

[0129] In order to further focus on important feature areas, the network introduces a spatial attention mechanism. The key feature extracted from the acceleration data is FA, while the audio sub-network based on the ResNet18 architecture processes the audio signal converted into MFCC feature maps, and extracts complex audio features FM through residual connections and spatial attention mechanisms, which not only retains the key features but also reduces the computational burden.

[0130] In the final classification stage, the network uses a fully connected layer to process the pooled features and make the final classification judgment. The network also includes a Dropout layer to reduce overfitting and enhance the generalization ability of the model.

[0131] 3. Converged Network Module

[0132] In this fusion network, the early fusion method is adopted. The fusion network model is shown in the figure below. Figure 5 shown.

[0133] First, the dynamic features of acceleration data and MFCC features of audio are extracted through two sub-networks respectively.

[0134] Then, these features are concatenated into a unified feature representation F = (F A ; F M ), which combines the key information of acceleration and audio.

[0135] This fused feature is then passed through the convolutional self-attention mechanism to strengthen the key information and obtain a new feature representation F=CA(F′).

[0136] Next, this enhanced feature representation is fed into the encoder part of the autoencoder for further feature extraction and compression, thereby reducing the feature dimension to Z = Encoder(F′).

[0137] The decoder part of the autoencoder then tries to reconstruct the features To ensure that important information is captured effectively.

[0138] Finally, the compressed features are processed and transformed through a series of fully connected layers for final classification, and finally the category labels corresponding to different device states are output. Nonlinear activation functions and Dropout regularization strategies are also embedded in this process to enhance the learning ability of the network and prevent overfitting.

[0139] It should be understood that, although each step in the flow chart involved in the above-described embodiment is displayed in sequence according to the instructions, these steps are not necessarily executed in sequence according to the order of the instructions. Unless there is a clear explanation in the present application, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flow chart involved in the above-described embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0140] Corresponding to the embodiment of the aforementioned multi-dimensional information fusion acoustic side-channel attack detection and identification method, adopting the same technical concept, the present application also provides an embodiment of the multi-dimensional information fusion acoustic side-channel attack detection and identification device.

[0141] Figure 6 It is a structural diagram of an acoustic side-channel attack detection and identification device with multi-dimensional information fusion shown in an exemplary embodiment of the present application.

[0142] In an exemplary embodiment, Figure 6 As shown, the multi-dimensional information fusion acoustic side channel attack detection and identification device includes:

[0143] Target recognition module 1, used to use a target algorithm to identify a specific target in the scene, and to define a warning area for the specific target, and to calculate a relevant credibility score based on the target detection result and its interaction with surrounding non-target objects;

[0144] Feature extraction module 2, used to perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each modality;

[0145] Integration training module 3 is used to integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space;

[0146] The hybrid training module 4 is used to mix the generated adversarial samples with the original labeled samples for secondary training to enhance the model's detection capability against artificial adversarial attacks.

[0147] It can be seen that:

[0148] 1. This application proposes for the first time a multi-dimensional information fusion solution for acoustic side channel attacks on sensors. The solution cascades two layers of technology to resist attacks, namely multimodal fusion network and target detection technology. Compared with traditional single-modal networks, the multimodal fusion network shows higher accuracy and robustness in processing complex data. Target detection technology enables efficient and accurate identification of unauthorized intrusion behaviors in industrial environments. And the target detection algorithm implements dynamic trust score adjustment to ensure the secondary accuracy of the multimodal fusion network.

[0149] 2. In view of the limitations of insufficient fault samples and data imbalance in the original data set, this application proposes a method to improve the data set, which is to use FGSM to generate new adversarial samples for the original data set, and mix the adversarial samples into the original data set to form a new data set. The new data set can enhance the network model's ability to resist attacks.

[0150] 3. This application compares and analyzes the performance of defense mechanisms, and compares in detail the performance of fusion algorithms under different circumstances and with different fusion architectures. For the specific limitations of the acoustic side channel attack detection and identification device with multi-dimensional information fusion, please refer to the limitations of the acoustic side channel attack detection and identification method with multi-dimensional information fusion mentioned above, which will not be repeated here. Each module in the above-mentioned acoustic side channel attack detection and identification device with multi-dimensional information fusion can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-dimensional information fusion acoustic side channel attack detection and identification method, characterized in that: include: Use the target algorithm to identify specific targets in the scene and define a warning area for the specific targets. At the same time, according to the target detection results and its interaction with surrounding non-target objects, the relevant trust score is calculated; Perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode; The extracted features of different modalities are integrated through the autoencoder and convolutional attention mechanism fusion network, and then the model is trained in the fused feature space; The generated adversarial samples are mixed with the original labeled samples for secondary training to enhance the model's detection ability against human adversarial attacks.

2. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The step of obtaining the relevant credibility score by calculation further includes: The relevant trust score is calculated by dynamically adjusting the function, that is, using the target detection technology to judge the intrusion of others, and the trust score obtained for the impact of non-target objects on the target object is as follows: Where ai is the calculation adjustment factor, which is calculated by a nonlinear function for each risk score ri and the coefficient k dynamically selected for the risk score range. The cumulative adjustment factor A is calculated by ai; The following formula is obtained by calculating the above formula: Where n is the total number of risk factors; The final confidence score is: S confidence =yolod(min(max(w C+(1-w) A,0),1)); Among them, S confidence represents the state credibility score, C represents the original credibility and the weight w of the original credibility, and yolod represents all objects and risk factors detected by the Yolo algorithm over a period of time.

3. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The obtaining of the relevant credibility score by calculation also includes: The sensor data is divided into vibration acceleration data and audio data. The fusion results in a certain fault state of the equipment, as shown in the formula: Among them, S status represents the device status, fusion represents the multimodal fusion algorithm, Represents the value of vibration acceleration, The value of the audio signal represented; When an attacker attacks, he will start with data, which will cause the acceleration and audio to change, and k will also change. Then the data fusion will change to the following formula: in, and They respectively represent the vibration acceleration and audio signal values ​​after being affected by the attack.

4. The acoustic side channel attack detection and identification method based on multi-dimensional information fusion as claimed in claim 1, characterized in that: The performing of modal feature extraction on the collected acceleration data and audio data further includes extracting features from the acceleration data: Extract standard deviation, skewness, and kurtosis from the time domain; From the frequency domain perspective, FFT and Power Spectrum Density (PSD) are used to extract frequency domain features; Use WT to extract features from acceleration data.

5. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The performing of modal feature extraction on the collected acceleration data and audio data further includes extracting features from the audio data: The continuous audio signal S(t) is divided into several short frames Sf(t), and each frame signal is Fourier transformed to obtain the spectrum F(K); The spectrum F(K) is mapped to the Mel scale m and processed by the Mel filter bank: After discrete cosine transform (DCT), 20 cepstral coefficients are extracted to generate MFCC feature maps.

6. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 5, characterized in that: The extracting of features from the audio data also includes: For long audio stream files, the method of cutting the audio files into segments every five seconds is adopted to ensure that it matches the acquisition cycle of vibration acceleration data, thereby achieving data synchronization, extracting MFCC from each segment and visualizing it.

7. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The fusion network includes: In the acceleration sub-network module, the input layer receives the combined features of the six features of the acceleration data, and the processed features are sent to the acceleration network for classification; The audio sub-network module converts audio data into MFCC feature maps as input, uses the pre-trained ResNet18 foundation, and performs deep feature extraction and classification of audio data through a series of convolutional layers, residual units, spatial attention mechanisms, and fully connected layers; The fusion network module concatenates the dynamic features of the acceleration data and the MFCC features of the audio extracted by the two sub-networks into a unified feature representation; the fused features are used through the convolutional self-attention mechanism to strengthen the key information and obtain a new feature representation; the strengthened feature representation is then sent to the encoder part of the autoencoder for further feature extraction and compression to reduce the feature dimension; the decoder part of the autoencoder attempts to reconstruct the features to ensure that important information is effectively captured; the compressed features are processed and transformed through a series of fully connected layers for final classification, and the category labels corresponding to different device states are output.

8. The multi-dimensional information fusion acoustic side channel attack detection and identification method according to claim 1, characterized in that: The acceleration sub-network module adopts a depthwise separable convolutional layer, which extracts spatial features through depthwise convolution and reduces the number of model parameters through point-by-point convolution. After two depthwise separable convolutions and maximum pooling, the dimension of the data is reduced while retaining key feature information. The acceleration sub-network module also includes a Dropout layer to reduce the risk of overfitting and enhance the generalization ability of the model on unseen data; The acceleration sub-network module performs final processing and classification decisions on the features through three fully connected layers. The three fully connected layers gradually refine the features and finally output classification results for specific tasks.

9. The acoustic side channel attack detection and identification method based on multi-dimensional information fusion as claimed in claim 1, characterized in that: The audio subnetwork module includes a custom convolutional block, which includes a convolutional layer, batch normalization and ReLU activation function, and performs preliminary feature extraction on the input audio data. The audio feature map is passed through the second and third blocks composed of ResNet18 residual units. These residual units enhance the depth and learning ability of the network, while avoiding the common gradient vanishing problem of deep networks. At the same time, the audio sub-network module introduces a spatial attention mechanism. The key feature extracted from the acceleration data is FA, and the audio sub-network based on the ResNet18 architecture processes the audio signal converted into the MFCC feature map, and extracts the complex audio feature FM through residual connection and spatial attention mechanism; In the final classification stage, the audio subnetwork module uses a fully connected layer to process the pooled features and make the final classification judgment. The network also includes a Dropout layer to reduce overfitting and enhance the generalization ability of the model.

10. A multi-dimensional information fusion acoustic side channel attack detection and identification device, characterized in that: include: A target recognition module is used to use a target algorithm to identify a specific target in the scene and define a warning area for the specific target. At the same time, according to the target detection result and its interaction with surrounding non-target objects, a relevant credibility score is calculated; A feature extraction module is used to perform modal feature extraction on the collected acceleration data and audio data to accurately capture the key information contained in each mode; The integrated training module is used to integrate the extracted features of different modalities through the autoencoder and convolutional attention mechanism fusion network, and then perform model training in the fused feature space; The hybrid training module is used to mix the generated adversarial samples with the original labeled samples for secondary training to enhance the model's detection ability against artificial adversarial attacks.

Images