A multi-modal artificial intelligence based cyber security threat detection method and system

Abstract

The application provides a network security threat detection method and system based on multi-modal artificial intelligence, acquires physical layer vibration signals of an Internet of Things device group, converts the physical layer vibration signals into continuous waveform data, aligns the continuous waveform data with network communication messages of target devices in time stamp, generates time-synchronized vibration data and time-synchronized network message data, extracts discrete features from the time-synchronized network message data, associates and encodes the time-synchronized vibration data and the discrete features, and generates a multi-modal data sequence, analyzes the multi-modal data sequence based on a neural network model to generate a cross-modal time sequence association relationship, and when the cross-modal time sequence association relationship simultaneously contains an abnormal fluctuation mode of the physical layer vibration signals and a protocol conflict feature of the network communication messages within a preset time window, generates a threat detection signal to perform network security threat detection, and the application realizes cross-modal collaborative detection of physical intrusion behaviors and network attack behaviors, and improves the accuracy and timeliness of threat detection.

Description

Technical Field

[0001] This invention relates to the field of cybersecurity threat detection technology, and in particular to a cybersecurity threat detection method and system based on multimodal artificial intelligence. Background Technology

[0002] With the rapid development of IoT technology, an increasing number of physical devices are being connected to the network, forming a large and complex device cluster. These IoT devices also face dual security threats from both the physical and network layers. Therefore, there is an urgent need for a real-time threat detection mechanism that can integrate physical layer signals and network communication behavior to improve the overall system's security capabilities.

[0003] Current solutions attempt to jointly analyze physical layer vibration data and network communication traffic from IoT devices using deep learning methods to identify abnormal behavior. These solutions employ multi-channel convolutional neural networks to extract time-series features from vibration signals and network packets, respectively, and then fuse the information through an attention mechanism to determine the presence of potential security threats. This improves the ability to identify complex attack patterns and reduces reliance on manual rules. However, existing solutions have some inherent limitations. These include the difficulty of accurately capturing fine-grained co-variations between vibration signals and network packets due to their different time scales and data structures; and the lack of modeling capabilities for dynamic dependencies between discontinuous events, resulting in high false positive and false negative rates when facing highly concealed and diverse physical layer intrusion behaviors. Summary of the Invention

[0004] This invention provides a network security threat detection method and system based on multimodal artificial intelligence, which addresses the problems in existing technologies, such as the difficulty of accurately capturing the fine-grained collaborative changes between vibration signals and network packets due to their different time scales and data structures; and the lack of modeling ability for dynamic dependencies between discontinuous events based on static feature extraction and fusion strategies, resulting in high false alarm and false negative rates when facing highly concealed and diverse physical layer intrusion behaviors.

[0005] In a first aspect, the present invention provides a network security threat detection method based on multimodal artificial intelligence, comprising:

[0006] The physical layer vibration signal of an IoT device group is acquired, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase change;

[0007] The continuous waveform data is timestamped and the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data.

[0008] Discrete features containing protocol type and load length are extracted from the time synchronization network message data, and the time synchronization vibration data and the discrete features are cross-modal correlated and encoded to generate a multimodal data sequence.

[0009] The multimodal data sequence is analyzed using a neural network model based on the neuronal pulse time coding mechanism to generate cross-modal temporal correlation relationships.

[0010] When the cross-modal temporal correlation simultaneously includes the abnormal fluctuation pattern of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window, a threat detection signal is generated for network security threat detection.

[0011] Optionally, physical layer vibration signals of an IoT device group are acquired, wherein the physical layer vibration signals are converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase changes, including:

[0012] Deploy an optical fiber sensor array containing multiple sensing units in the IoT device group;

[0013] A constant wavelength probe light signal is emitted to the sensing unit of the fiber optic sensor array to obtain a modulated light signal with a phase shift caused by the surface vibration of the fiber optic sensor array.

[0014] The instantaneous vibration displacement of the monitoring area where each sensing unit is located is determined based on the change in interference fringes between the modulated optical signal and the probe optical signal.

[0015] The instantaneous vibration displacement of multiple sensing units at the same moment is superimposed according to their spatial distribution to generate continuous waveform data characterizing the vibration intensity of the equipment surface.

[0016] Optionally, the continuous waveform data is timestamped and aligned with the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data, including:

[0017] The starting time point at which the first sudden change in vibration intensity occurs in the vibration signal of the physical layer is detected is used to generate the initial synchronization reference time point;

[0018] In the network communication messages of the target device in the IoT device group, match the first message start boundary whose time deviation from the initial synchronization reference time point is less than a preset threshold, and generate the message start boundary to be corrected.

[0019] The time of the start boundary of the message to be corrected is adjusted until the time of the start boundary of the message to be corrected is consistent with the initial synchronization reference time point, and the corrected message start boundary is generated.

[0020] Based on the sampling interval of the continuous waveform data and the corrected message start boundary, the time axis of the network communication message is segmented and adjusted to generate a target message data segment whose start time is consistent with the start time of the waveform data segment;

[0021] The vibration intensity feature sequence of each waveform data segment in the continuous waveform data is time-correspondingly bound to the original byte stream of the target message data segment to generate time-synchronized vibration data and time-synchronized network message data.

[0022] Optionally, discrete features containing protocol type and payload length are extracted from the time synchronization network message data, and the time synchronization vibration data and the discrete features are cross-modal correlated and encoded to generate a multimodal data sequence, including:

[0023] Each segment of the time synchronization network message data is parsed to generate a discrete feature set containing the protocol type code and payload length value.

[0024] Based on the time range corresponding to each protocol type code in the discrete feature set, the time-synchronized vibration data is cut into multiple vibration waveform segments to generate a vibration intensity change curve associated with the protocol type code;

[0025] Calculate the ratio of the load length value to the maximum vibration amplitude of the vibration intensity variation curve, and generate a mapping table between the protocol type code, the load length value, and the vibration intensity variation curve;

[0026] The waveform morphology characteristics of the vibration intensity change curve are combined and encoded with the same protocol type code in the mapping table to generate multiple cross-modal associated data blocks;

[0027] Arrange the cross-modal associated data blocks in chronological order to generate a multimodal data sequence.

[0028] Optionally, the waveform morphology characteristics of the vibration intensity variation curve are combined and encoded with the same protocol type code in the mapping table to generate multiple cross-modal association data blocks, including:

[0029] Based on the numerical identifier of the protocol type code, the corresponding waveform feature extraction rule is selected from the preset encoding rule set;

[0030] Based on the ratio of the load length value to the maximum vibration amplitude, the feature sensitivity parameters of the waveform feature extraction rule are adjusted to generate the adjusted feature sensitivity parameters.

[0031] According to the adjusted feature sensitivity parameters, the waveform morphology features of the vibration intensity change curve are subjected to feature quantization processing to generate a physical vibration feature code bound to the protocol type code.

[0032] The physical vibration feature code and the protocol type code are combined by bit sequence cross-combination to generate multiple cross-modal associated data blocks.

[0033] Optionally, the multimodal data sequence is analyzed using a neural network model based on a neuronal pulse time coding mechanism to generate cross-modal temporal correlations, including:

[0034] Receive each cross-modal associated data block in the multimodal data sequence, and divide the cross-modal associated data block into multiple time segments according to a preset time window length;

[0035] The pulse emission time interval is calculated for the physical vibration feature encoding and network protocol feature encoding within each time segment to generate a pulse sequence;

[0036] Based on the time interval difference between the physical vibration feature pulses and the network protocol feature pulses in the pulse sequence, the connection strength parameters between the physical vibration feature nodes and the network protocol feature nodes in the neural network model constructed based on the neuronal pulse time coding mechanism are adjusted to generate the adjusted connection strength parameters.

[0037] The adjusted connection strength parameters are superimposed with the pulse firing patterns of consecutive time segments in the pulse sequence to generate the cross-modal temporal correlation.

[0038] Optionally, when the cross-modal temporal correlation simultaneously includes the abnormal fluctuation pattern of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window, a threat detection signal is generated for network security threat detection, including:

[0039] The cross-modal temporal correlation is divided into multiple consecutive time segments according to the length of a preset time window;

[0040] Within each time segment, the number of consecutive changes in the vibration intensity value exceeding a preset fluctuation threshold in the physical layer vibration signal is detected, and periodic spike characteristics of the abnormal fluctuation pattern of the physical layer vibration signal are generated.

[0041] The system synchronously detects the matching status between the protocol type code in the network communication message and the preset protocol whitelist, as well as the proportion of the payload length value in the network communication message that exceeds the preset range, in order to generate an abnormal protocol behavior identifier for the protocol conflict characteristics of the network communication message.

[0042] When the periodic spike feature and the abnormal protocol behavior identifier exist simultaneously within the same time segment, the time segment is marked as a threat-associated segment;

[0043] The cumulative density of threat-related segments in a continuous time segment is statistically analyzed. When the cumulative density exceeds a preset security threshold, a threat detection signal is generated.

[0044] Secondly, the present invention provides a network security threat detection system based on multimodal artificial intelligence, comprising:

[0045] The acquisition module is used to acquire the physical layer vibration signal of the IoT device group, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase change;

[0046] The alignment module is used to perform timestamp alignment processing on the continuous waveform data and the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data.

[0047] The encoding module is used to extract discrete features containing protocol type and load length from the time synchronization network message data, and to perform cross-modal correlation encoding between the time synchronization vibration data and the discrete features to generate a multimodal data sequence;

[0048] The analysis module is used to analyze the multimodal data sequence through a neural network model built based on the neuronal pulse time coding mechanism, and generate cross-modal temporal correlation relationships.

[0049] The generation module is used to generate a threat detection signal for network security threat detection when the cross-modal temporal correlation simultaneously contains the abnormal fluctuation mode of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window.

[0050] Thirdly, the present invention provides a computing device including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a network security threat detection method based on multimodal artificial intelligence as described in any of the first aspects.

[0051] Fourthly, the present invention provides a computer storage medium storing computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement a network security threat detection method based on multimodal artificial intelligence as described in any one of the first aspects.

[0052] This invention achieves cross-modal collaborative detection of physical intrusion and network attack behaviors by integrating physical layer vibration signals from IoT devices with network communication message data. Fiber optic sensing technology is used to convert device surface vibrations into continuous waveform data, which is then timestamped and analyzed using pulse neural networks with network protocol characteristics. This effectively solves the problem of missed detection in traditional single-modal detection methods when dealing with physical-network collaborative attacks. By identifying vibration anomalies and protocol conflict characteristics that coexist in cross-modal temporal correlations, the accuracy and timeliness of threat detection are significantly improved. It can identify associated network penetration behaviors before the device is physically damaged, providing proactive security protection for IoT devices.

[0053] Furthermore, by deploying a fiber optic sensor array on the equipment surface and employing optical phase interferometry, millimeter-level displacement sensing and full-surface coverage monitoring of equipment vibration were achieved. Continuous waveform data was generated by spatially superimposing the vibration displacement data from multiple sensing units, which not only improved the accuracy of vibration intensity detection but also effectively distinguished between normal operational vibration and malicious physical intrusion vibration. This approach overcomes the shortcomings of traditional accelerometers, such as susceptibility to electromagnetic interference and insufficient deployment density, providing a high-fidelity physical-layer vibration data foundation for subsequent multimodal correlation analysis and ensuring the reliable operation of the threat detection system in complex industrial environments.

[0054] These or other aspects of the invention will become more apparent from the following description of the embodiments. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a network security threat detection method based on multimodal artificial intelligence, provided as an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a network security threat detection system based on multimodal artificial intelligence provided in an embodiment of the present invention;

[0058] Figure 3This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0060] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Figure 1 A flowchart of a network security threat detection method based on multimodal artificial intelligence is provided in this embodiment of the invention, as follows: Figure 1 As shown, the method includes:

[0063] To address the pain point of traditional single-modal detection methods in physical layer intrusion detection scenarios involving IoT device clusters, which are unable to effectively identify the coordinated behavior of physical intrusion and network attacks, this invention designs a cross-modal temporal correlation analysis mechanism by integrating high-precision physical vibration signals captured by fiber optic sensing technology with network communication protocol characteristics. Existing technologies mainly suffer from the following drawbacks: First, physical sensors and network traffic detection systems operate independently, making it difficult to capture network protocol anomalies caused by physical device tampering; second, conventional neural networks cannot effectively handle the temporal correlation between continuous time-domain features of vibration signals and discrete network protocol events; third, the insufficient accuracy of traditional time synchronization methods leads to misjudgments of the temporal coupling relationship between physical and network behaviors. This solution uses optical phase changes to convert vibration signals into continuous waveform data in real time, and combines this with the unique biological temporal coding mechanism of pulse neural networks to establish cross-modal correlation rules between vibration waveform mutations and protocol conflicts under a unified time reference. This solves the technical challenge of co-detecting physical layer intrusion behavior and network layer attack signals, improving the ability to identify new composite attacks. Based on this, this invention provides a network security threat detection method based on multimodal artificial intelligence, such as... Figure 1 ,include:

[0064] Step 101: Obtain the physical layer vibration signal of the IoT device group, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase change.

[0065] In this step, the physical layer vibration signal refers to the mechanical vibration signal of the equipment surface collected by the fiber optic sensor array, including vibration waveforms caused by physical contact or impact; the optical phase change refers to the phase shift caused by the optical path difference between the probe light and the reflected light due to equipment vibration, which is used to quantify vibration displacement; the conversion operation refers to the process of converting the optical phase change into vibration displacement and generating continuous waveform data by superimposing the displacements of multiple sensors; the equipment surface vibration intensity refers to the characteristics of the vibration energy distribution on the equipment surface reflected by the continuous waveform data, including amplitude and frequency information; the continuous waveform data refers to the sequence of quantized vibration intensity values ​​arranged in time order, characterizing the time-domain variation law of the vibration signal.

[0066] In this embodiment of the invention, physical layer vibration signals are first collected by an array of fiber optic sensors deployed on the surface of an IoT device group. Then, a constant wavelength probe light signal is emitted to the sensors, and the phase shift signal of the reflected light caused by device vibration is received. Next, the instantaneous vibration displacement of each sensor's monitoring area is calculated based on the change in the interference fringes of the reflected and probe light. Finally, the displacements of all sensors at the same moment are superimposed according to their spatial distribution to generate continuous waveform data characterizing the vibration intensity on the device surface.

[0067] Step 102: The continuous waveform data is timestamped with the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data.

[0068] In this step, network communication messages refer to data packets transmitted between IoT devices, including protocol headers and payload content; timestamp alignment processing refers to adjusting the time base based on the matching of vibration intensity abrupt events and message start boundaries to achieve physical and network data synchronization; time-synchronized vibration data refers to vibration waveform data segments that are strictly aligned with network messages after time axis remapping; time-synchronized network message data refers to message data segments whose start time is aligned with the vibration data segments.

[0069] In this embodiment of the invention, the starting time point at which the first sudden change in vibration intensity occurs in the continuous waveform data is detected first as the initial synchronization reference. Then, the first message start boundary in the network communication messages with a time deviation less than a preset threshold from this reference is searched. Next, the time of the message start boundary is adjusted to coincide with the initial synchronization reference. Finally, the messages are remapped along the time axis according to the sampling interval of the continuous waveform data to generate time-synchronized vibration data and network message data.

[0070] Step 103: Extract discrete features containing protocol type and payload length from the time synchronization network message data, and perform cross-modal correlation encoding on the time synchronization vibration data and the discrete features to generate a multimodal data sequence.

[0071] In this step, the protocol type refers to the encoding of the communication rules in the message header; the payload length refers to the number of bytes of data actually transmitted in the message, reflecting the data capacity characteristics; discrete features refer to the non-continuous feature set composed of the protocol type code and payload length value extracted from the message; cross-modal correlation coding operation refers to the operation of dynamically combining vibration waveform features and protocol type codes according to the mapping relationship table to generate joint characterization data; multimodal data sequence refers to cross-modal correlation data blocks arranged in chronological order, containing the fusion features of physical vibration and network protocol.

[0072] In this embodiment of the invention, the header of the time synchronization network message data is first parsed to extract the protocol type code and load length value to form a discrete feature set. Then, the time synchronization vibration data is segmented according to the time range corresponding to the protocol type code to generate a vibration intensity variation curve. Next, the ratio of the load length value to the maximum amplitude of the vibration curve is calculated to establish a mapping relationship table. Finally, the waveform morphology features of the vibration curve are dynamically combined and encoded with the protocol type code to generate a multimodal data sequence.

[0073] Step 104: Analyze the multimodal data sequence using a neural network model built based on the neuronal pulse time coding mechanism to generate cross-modal temporal correlation relationships.

[0074] In this step, the neuronal pulse time-coding mechanism refers to the signal processing mechanism that simulates the biological neuron's adjustment of connection strength based on the pulse trigger time difference; the construction operation refers to the process of configuring event processing nodes for physical vibration and network protocol characteristics in the neural network model and initializing connection parameters; the neural network model refers to the computational model designed using the pulse time-coding mechanism for analyzing multimodal temporal correlations; the analysis operation refers to the process of dynamically adjusting model parameters by calculating the pulse trigger time difference and superimposing temporal patterns to generate correlations; and the cross-modal temporal correlation refers to the quantitative characteristics that reflect the coordinated changes of physical vibration and network protocol behavior in the time dimension.

[0075] In this embodiment of the invention, the multimodal data sequence is first divided into time segments according to a preset time window. Then, the pulse triggering time difference between the physical vibration and the network protocol encoding within each segment is calculated to generate a pulse sequence. Next, the connection strength parameters of the two types of feature nodes in the neural network model are dynamically adjusted according to the triggering time difference. Finally, the adjusted parameters are superimposed with the temporal pattern of the pulse sequence to generate a cross-modal temporal correlation.

[0076] Step 105: When the cross-modal temporal correlation simultaneously contains the abnormal fluctuation mode of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window, a threat detection signal is generated to perform network security threat detection.

[0077] In this step, the abnormal fluctuation mode refers to the periodic spike characteristic of the vibration intensity value continuously exceeding the preset threshold within a short period of time; the protocol conflict characteristic refers to the combined abnormal state of the protocol type code not matching the whitelist and the payload length continuously exceeding the limit; the threat detection signal refers to the network security alarm command generated based on the cumulative density of threat-related fragments exceeding the security threshold.

[0078] In this embodiment of the invention, the cross-modal temporal correlation is first segmented into continuous segments according to a preset time window. Then, within each segment, the number of times the vibration intensity value continuously exceeds a threshold is counted to generate periodic spike features. Simultaneously, the matching status of the protocol type code and the whitelist, as well as the abnormal proportion of load length, are detected to generate abnormal protocol identifiers. Next, segments that simultaneously exhibit both features are marked as threat-associated segments. Finally, the threat density of continuous segments is counted, and a threat detection signal is generated when it exceeds a safety threshold.

[0079] For example, firstly, fiber optic sensor arrays are deployed at 20cm intervals on the metal casing of industrial IoT devices. By emitting 1550nm probe light and receiving the phase shift signal of the reflected light, the vibration displacement of each sensor monitoring point is calculated in real time. Secondly, the displacement values ​​are superimposed to generate continuous waveform data of the overall vibration intensity of the device, while simultaneously capturing Modbus-TCP protocol messages from the target device. Then, the starting time of the first vibration abrupt change in the waveform data is detected, and the time axis is remapped by matching the message start boundary, generating time-synchronized vibration and message data. Next, the protocol type code and load length value are extracted from the message, and the vibration data is segmented into time periods according to the protocol type to generate intensity change curves, establishing a mapping table between load length and vibration peak value. Then, the morphological characteristics of the vibration curve and the protocol type code are dynamically encoded to generate a multimodal data sequence, which is input into a spiking neural network model to analyze cross-modal temporal correlations. Finally, if the vibration intensity exceeds the threshold three times consecutively within a 5-second time window and the protocol type code matches abnormally, and the statistical threat segment density exceeds 80%, a device isolation command is triggered.

[0080] This invention utilizes fiber optic sensing technology to capture the physical vibration characteristics of devices with high precision. Combined with deep network protocol analysis and pulse neural network time-series analysis, it achieves cross-modal collaborative detection of physical intrusions and network attacks. It overcomes the technical bottlenecks of traditional detection methods in areas such as time synchronization accuracy, heterogeneous data fusion, and time-series correlation modeling. This significantly improves the accuracy and timeliness of identifying novel composite attacks such as network penetration triggered by physical device manipulation, providing proactive defense capabilities for IoT devices.

[0081] To address the issues of electromagnetic interference and insufficient accuracy of traditional vibration sensors in detecting surface vibrations of IoT devices, this step utilizes a fiber optic sensor array and optical phase interferometry to achieve high-precision vibration signal acquisition. A specific embodiment of this invention is provided: Step 101, acquiring the physical layer vibration signal of an IoT device group, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase changes, specifically including the following steps:

[0082] Step 111: Deploy an optical fiber sensor array containing multiple sensing units in the IoT device group.

[0083] In this step, the fiber optic sensor array refers to a monitoring network consisting of multiple distributed fiber optic sensing units. Each sensing unit is connected by an optical fiber and fixed to the surface of the device, and is used to synchronously collect vibration signals from multiple areas.

[0084] In this embodiment of the invention, firstly, multiple sensing units of a fiber optic sensor array are deployed at preset intervals based on the surface shape and vibration-sensitive area distribution of the IoT device group. Secondly, the sensing units are connected to a signal processing terminal via fiber optic lines. Finally, the power supply and communication configuration of the sensor array are completed to ensure that each sensing unit covers the key monitoring areas on the device surface.

[0085] Step 112: Transmit a probe light signal of constant wavelength to the sensing unit of the fiber optic sensor array to obtain a modulated light signal with phase shift of the reflected light signal caused by the surface vibration of the fiber optic sensor array.

[0086] In this step, the constant wavelength probe light signal refers to the light wave signal with a fixed optical frequency emitted by the laser generator, which serves as the reference light signal for vibration detection; the modulated light signal refers to the light wave signal after the phase shift of the reflected light signal caused by the deformation of the optical fiber, and the amount of phase change is proportional to the amount of vibration displacement.

[0087] In this embodiment of the invention, a fixed-wavelength probe light signal is first emitted to the sensing unit of the fiber optic sensor array via a laser generator. Then, reflected light signals generated by fiber deformation due to vibrations on the device surface are received. Next, the phase shift of the reflected light signal relative to the probe light signal is detected. Finally, a modulated light signal containing vibration information is generated.

[0088] Step 113: Determine the instantaneous vibration displacement of the monitoring area where each sensing unit is located based on the change in interference fringes between the modulated optical signal and the probe optical signal.

[0089] In this step, the change in interference fringes refers to the degree of shift in the position or spacing of the fringes in the interference pattern formed after the probe light and the modulation light are superimposed, which is used to infer the change in optical path difference; the instantaneous vibration displacement refers to the small displacement value caused by the vibration of the device surface detected by a single sensing unit at a specific time point, which is obtained by calculating the optical path difference.

[0090] In this embodiment of the invention, the probe light signal and the modulated light signal are first superimposed on the interferometer, and then the movement direction and spacing change of the interference fringes are observed. Next, based on the proportional relationship between the fringe movement and the light wavelength, the instantaneous vibration displacement of the monitoring area of ​​each sensing unit is calculated, and finally, the displacement quantization data of each sensing unit is output.

[0091] Step 114: Superimpose the instantaneous vibration displacement of multiple sensing units at the same moment according to their spatial distribution to generate continuous waveform data characterizing the vibration intensity of the equipment surface.

[0092] In this step, the superposition operation refers to the data fusion process of weighted summation of the displacements of multiple sensing units according to their spatial distribution to generate the overall vibration intensity of the equipment.

[0093] In this embodiment of the invention, the instantaneous vibration displacement of all sensing units at the same moment is first obtained, and then the displacements are superimposed according to the spatial coordinate arrangement of the sensing units on the surface of the device. Next, the superimposed displacements are normalized, and finally, a continuous waveform data sequence reflecting the overall vibration intensity of the device is generated.

[0094] This invention achieves millimeter-level precise monitoring of device surface vibrations using a fiber optic sensor array. It utilizes optical phase interferometry to eliminate electromagnetic interference and combines displacement superposition of multiple sensing units to enhance detection sensitivity. Compared to traditional piezoelectric sensors, it significantly improves the ability to distinguish between localized weak vibrations and global vibration modes, providing a high-fidelity physical layer data foundation for cross-modal threat detection.

[0095] To improve the time synchronization accuracy between physical vibration data and network communication messages, this step employs a time reference alignment mechanism triggered by a sudden vibration intensity event to eliminate cross-layer data timing deviations. This invention provides a specific embodiment: Step 102 involves timestamping the continuous waveform data with the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data. This specifically includes the following steps:

[0096] Step 201: Detect the starting time point of the first sudden change in vibration intensity in the physical layer vibration signal, and generate the initial synchronization reference time point.

[0097] In this step, the initial synchronization reference time point refers to the precise time marker at which a sudden change in vibration intensity is first detected in the physical layer vibration signal, and is used to establish a unified time reference for physical and network data.

[0098] In this embodiment of the invention, the continuous waveform data of the physical layer vibration signal is first monitored in real time to detect the starting time point when the vibration intensity value first exceeds a preset fluctuation threshold. This time point is then recorded as the initial synchronization reference time point, and finally, the reference time point is transmitted to the network data processing module.

[0099] Step 202: In the network communication messages of the target device in the IoT device group, match the first message start boundary whose time deviation from the initial synchronization reference time point is less than a preset threshold, and generate the message start boundary to be corrected.

[0100] In this step, the matching operation refers to the calculation process of searching for the message start boundary with the smallest time deviation from the physical event in the network message, including time difference calculation and threshold judgment; the first message start boundary refers to the start time identifier of the first data packet in the network communication message, reflecting the initial time of message transmission; the message start boundary to be corrected refers to the message boundary with a time deviation from the initial synchronization reference within the allowable range, which needs to be timestamped to achieve accurate alignment.

[0101] In this embodiment of the invention, the first message start boundary is first searched in the timing data of network communication messages, and the absolute time difference between the boundary timestamp and the initial synchronization reference time is calculated. Next, it is determined whether the time difference is less than a preset threshold. If the condition is met, the message start boundary is marked as a message start boundary to be corrected. Finally, the message data segment corresponding to the boundary is extracted.

[0102] Step 203: Adjust the time of the start boundary of the message to be corrected until the time of the start boundary of the message to be corrected is consistent with the initial synchronization reference time point, and generate the corrected start boundary of the message.

[0103] In this step, the adjustment operation refers to the calibration process of modifying the message start boundary timestamp in a stepwise approximation manner to make it consistent with the physical event reference time; the corrected message start boundary refers to the message start time identifier that is completely aligned with the physical reference time after the timestamp adjustment.

[0104] In this embodiment of the invention, the original timestamp of the start boundary of the message to be corrected is first obtained. Then, the timestamp is gradually increased or decreased until it completely coincides with the initial synchronization reference time. Subsequently, the corrected start boundary of the message is generated, and finally, the timestamp metadata of the message data segment is updated.

[0105] Step 204: Based on the sampling interval of the continuous waveform data and the corrected message start boundary, the time axis of the network communication message is segmented and adjusted to generate a target message data segment whose start time is consistent with the start time of the waveform data segment.

[0106] In this step, the segmented adjustment operation refers to the process of dividing the message time axis into equal-length time periods according to the vibration data sampling interval and realigning them; the target message data segment refers to the message data unit whose start time corresponds to the start time of the vibration waveform data segment after the time axis adjustment.

[0107] In this embodiment of the invention, the sampling interval parameter of the continuous waveform data is first read. Then, starting from the corrected message start boundary, the time axis of the network communication message is divided into equal-length time periods according to the sampling interval. Subsequently, the message data in each time period is re-aligned to the start time of the corresponding waveform data segment, and finally, a target message data segment with a strictly synchronized start time is generated.

[0108] Step 205: Bind the vibration intensity feature sequence of each waveform data segment in the continuous waveform data to the original byte stream of the target message data segment in time correspondence to generate time-synchronized vibration data and time-synchronized network message data.

[0109] In this step, the vibration intensity feature sequence refers to the set of vibration intensity quantization values ​​arranged in chronological order; the raw byte stream refers to the unparsed network packet binary data, which contains complete information about the protocol header and payload content; the time-correspondence binding operation refers to establishing a mapping relationship between vibration data and packet data under the same timestamp, ensuring the time consistency of cross-modal data.

[0110] In this embodiment of the invention, firstly, the vibration intensity value sequence of each waveform data segment in the continuous waveform data is extracted; secondly, the original byte stream data within the same time range in the target message data segment is obtained. Subsequently, a correspondence between the vibration intensity sequence and the timestamp of the byte stream data is established; finally, time-synchronized vibration data and network message data are generated through data binding.

[0111] This invention addresses the millisecond-level time deviation problem in cross-layer detection of physical networks by employing a time base synchronization mechanism triggered by physical vibration events. By using dynamic adjustment of message boundaries and time axis segmentation alignment technology, it achieves millisecond-level precise synchronization between vibration waveforms and network messages, providing a strictly time-aligned data foundation for subsequent cross-modal correlation analysis and significantly reducing the risk of erroneous correlations caused by time inaccuracies.

[0112] To address the insufficient correlation between protocol type and vibration characteristics in multimodal data fusion, this step establishes a load-vibration ratio mapping relationship to achieve cross-modal semantic-level coding. This invention provides a specific embodiment: Step 103, extracting discrete features containing protocol type and load length from the time-synchronized network message data, and performing cross-modal correlation coding between the time-synchronized vibration data and the discrete features to generate a multimodal data sequence, specifically including the following steps:

[0113] Step 301: Parse the header structure of each segment of the time synchronization network message data to generate a discrete feature set containing the protocol type code and payload length value.

[0114] In this step, the header structure parsing operation refers to the process of parsing the network packet header fields byte by byte to extract the protocol type code and payload length value; the protocol type code refers to the digital code in the packet header that identifies the communication protocol type; the payload length value refers to the number of bytes of data actually transmitted in the packet, reflecting the size of the data segment; the discrete feature set refers to the non-continuous feature set composed of the protocol type code and payload length value, which characterizes the discrete event attributes of network behavior.

[0115] In this embodiment of the invention, firstly, the header fields of each segment of the time synchronization network message data are parsed to extract the protocol type code that identifies the communication rules and the payload length value that represents the data capacity. Secondly, the protocol type code and the payload length value are combined into a discrete feature set, and finally, the discrete feature set is transmitted to the vibration data processing module.

[0116] Step 302: Based on the time range corresponding to each protocol type code in the discrete feature set, the time-synchronized vibration data is cut into multiple vibration waveform segments to generate a vibration intensity change curve associated with the protocol type code.

[0117] In this step, the vibration waveform segment refers to a continuous waveform data segment generated by cutting vibration data according to the time period corresponding to the protocol type code; the vibration intensity change curve refers to a time-domain change trend graph generated by calculating the average vibration intensity within the waveform segment.

[0118] In this embodiment of the invention, firstly, the time interval for cutting the time-synchronized vibration data is determined based on the message transmission time period corresponding to each protocol type code in the discrete feature set. Secondly, the vibration data is cut into multiple waveform segments according to the time interval. Subsequently, the average vibration intensity of each segment is calculated to generate a variation curve. Finally, the correlation between the variation curve and the protocol type code is established.

[0119] Step 303: Calculate the ratio of the load length value to the maximum vibration amplitude of the vibration intensity change curve, and generate a mapping table of the protocol type code, the load length value, and the vibration intensity change curve.

[0120] In this step, the mapping table refers to a structured database table that stores the protocol type code, load length value, and vibration amplitude ratio.

[0121] In this embodiment of the invention, the maximum amplitude value of the vibration intensity variation curve is first obtained, and then the ratio of the load length value to the maximum amplitude value is calculated. Subsequently, the protocol type code, load length value, and ratio are stored according to their corresponding relationship, and finally a structured data table containing the mapping relationship between the three is generated.

[0122] Step 304: Combine and encode the waveform morphology features of the vibration intensity change curve with the same protocol type code in the mapping table to generate multiple cross-modal associated data blocks.

[0123] In this step, waveform morphology features refer to quantitative parameters such as rise rate and fluctuation period extracted from the vibration curve that describe the waveform shape; combined coding operation refers to the data fusion process of generating joint codes by combining waveform morphology features and protocol type codes according to preset rules; cross-modal associated data block refers to a binary data unit containing the fusion features of physical vibration and network protocol, and has a timestamp identifier.

[0124] In this embodiment of the invention, the waveform rise slope and fluctuation frequency of the vibration intensity change curve are first extracted as morphological features. Then, the encoding weights are adjusted according to the ratios corresponding to the protocol type codes in the mapping table. Subsequently, the morphological features and protocol type codes are combined according to their weights to generate binary encoded blocks, and finally, cross-modal correlation data blocks are output.

[0125] Step 305: Arrange the cross-modal associated data blocks in chronological order to generate a multimodal data sequence.

[0126] In this step, the permutation operation refers to sorting and concatenating data blocks in chronological order to form a time-complete multimodal data stream.

[0127] In this embodiment of the invention, the data blocks are first sorted according to their timestamps, and then the sorted data blocks are concatenated in chronological order. The temporal continuity between the data blocks is then verified, and finally, a multimodal data sequence containing time dimension information is generated.

[0128] This invention achieves refined correlation encoding between network protocol behavior and physical vibration characteristics through a protocol-type-driven vibration data segmentation and load-vibration ratio mapping mechanism. It overcomes the technical bottlenecks of feature dimension mismatch and semantic association loss in traditional multimodal fusion methods, generating multimodal data sequences with physical network interaction semantics, providing high-information-density input data for subsequent time-series analysis.

[0129] To improve the weight adaptation between vibration features and protocol type encoding, this step dynamically adjusts the feature sensitivity parameter based on the load vibration ratio to generate bit-level fused data. This invention provides a specific embodiment: Step 304, combining the waveform morphology features of the vibration intensity change curve with the same protocol type code in the mapping table to generate multiple cross-modal associated data blocks, specifically including the following steps:

[0130] Step 341: Select the corresponding waveform feature extraction rule from the preset encoding rule set according to the numerical identifier of the protocol type code.

[0131] In this step, the numerical identifier refers to the specific numerical representation of the protocol type code; the waveform feature extraction rule refers to the waveform parameter extraction strategy predefined according to the protocol type, including the rise rate threshold, fluctuation period range, etc.

[0132] In this embodiment of the invention, the numerical identifier of the protocol type code is first read, and then the corresponding waveform feature extraction rule is matched in a predefined set of encoding rules based on the numerical identifier. Next, the matched rule is loaded into the feature processing module, and finally, the binding between the protocol type code and the waveform feature extraction rule is completed.

[0133] Step 342: Based on the ratio of the load length value to the maximum vibration amplitude, adjust the feature sensitivity parameters of the waveform feature extraction rule to generate the adjusted feature sensitivity parameters.

[0134] In this step, the adjustment operation refers to the calculation process of dynamically modifying the feature extraction parameters based on the load-vibration scaling factor; the feature sensitivity parameter refers to the adjustment parameter that controls the accuracy of waveform feature quantization and determines the granularity of feature level division; the adjusted feature sensitivity parameter refers to the feature quantization parameter after scaling by the load-vibration scaling factor, which is used to adapt to different protocol scenarios.

[0135] In this embodiment of the invention, the ratio of the load length value to the maximum amplitude of the vibration intensity change curve is first obtained. Then, this ratio is multiplied by the feature sensitivity baseline to obtain an adjustment coefficient. Subsequently, the parameter thresholds in the waveform feature extraction rules are scaled according to the adjustment coefficient, and finally, the adjusted feature sensitivity parameters are generated.

[0136] Step 343: According to the adjusted feature sensitivity parameters, perform feature quantization processing on the waveform morphology features of the vibration intensity change curve to generate a physical vibration feature code bound to the protocol type code.

[0137] In this step, feature quantization refers to the data conversion process that maps waveform morphological features to discrete coding levels; physical vibration feature coding refers to the binary sequence generated by waveform morphological feature quantization, which reflects the binding relationship between vibration mode and protocol type.

[0138] In this embodiment of the invention, firstly, the quantization range of the waveform morphology features is set according to the adjusted feature sensitivity parameters. Then, the waveform rise rate and fluctuation period of the vibration intensity change curve are divided into different levels. Next, a binary encoding sequence is generated according to the level. Finally, the encoding sequence is bound to the protocol type code to generate a physical vibration feature code.

[0139] Step 344: Combine the physical vibration feature code with the protocol type code by bit sequence cross-combination to generate multiple cross-modal associated data blocks.

[0140] In this step, the bit sequence cross-combination operation refers to a data fusion method that generates joint codes by alternating the bits of vibration codes and protocol type codes.

[0141] In this embodiment of the invention, the physical vibration feature code is first converted into a binary bit sequence, and then the binary representation of the protocol type code is alternately inserted into the vibration feature code sequence according to odd and even bits. Next, the integrity of the cross-combined bit sequence is checked, and finally, a cross-modal association data block is generated.

[0142] This invention employs a protocol-type-driven dynamic feature quantization mechanism to achieve adaptive matching between vibration feature extraction rules and network protocol behavior. By combining load-vibration ratio adjustment of feature sensitivity, it effectively improves coding discriminability across different protocol scenarios. The generated cross-modal correlation data blocks retain bit-level interaction information between physical and network features, providing highly discriminative input data for temporal correlation analysis and significantly improving the accuracy of cooperative attack detection.

[0143] To address the insufficient ability of neural network models to model temporal correlations in physical networks, this step employs a dynamic connection strength adjustment mechanism driven by pulse firing time intervals. This invention provides a specific embodiment where step 104 involves analyzing the multimodal data sequence using a neural network model constructed based on a neuronal pulse timing encoding mechanism to generate cross-modal temporal correlations. This specifically includes the following steps:

[0144] Step 401: Receive each cross-modal associated data block in the multimodal data sequence, and divide the cross-modal associated data block into multiple time segments according to a preset time window length.

[0145] In this step, the preset time window length refers to the pre-defined time segment cutting length, which is used to unify the analysis granularity of multimodal data.

[0146] In this embodiment of the invention, cross-modal correlated data blocks from a multimodal data sequence are first received. Then, each data block is divided into multiple equal-length segments according to a preset time window length. Next, the start and end times of the time segments are marked, and finally, the segmented time segments are transmitted to the pulse processing module.

[0147] Step 402: Calculate the pulse emission time interval for the physical vibration feature encoding and network protocol feature encoding within each time segment to generate a pulse sequence.

[0148] In this step, network protocol feature encoding refers to the feature representation of the protocol type code extracted from the network message after binary conversion; pulse emission time interval calculation operation refers to the process of calculating the time difference between the physical vibration feature pulse and the network protocol feature pulse at the first trigger time; pulse sequence refers to the set of pulse triggering events arranged in chronological order, with each pulse corresponding to the activation state of a feature encoding.

[0149] In this embodiment of the invention, the physical vibration feature codes and network protocol feature codes within a time segment are first extracted, and then the time interval between the first triggering of their pulse signals is calculated. Next, the time interval is converted into the timestamp difference of the pulse sequence, and finally, a pulse sequence reflecting the temporal relationship between physical vibration and network protocol behavior is generated.

[0150] Step 403: Based on the time interval difference between the physical vibration feature pulses and the network protocol feature pulses in the pulse sequence, adjust the connection strength parameters between the physical vibration feature nodes and the network protocol feature nodes in the neural network model constructed based on the neuron pulse time coding mechanism, and generate the adjusted connection strength parameters.

[0151] In this step, the physical vibration feature pulse refers to the pulse signal triggered by the vibration intensity feature encoding, reflecting the temporal activation event of physical vibration; the network protocol feature pulse refers to the pulse signal triggered by the protocol type code, reflecting the temporal activation event of network protocol behavior; the transmission time interval difference refers to the absolute time difference between the triggering times of the physical vibration pulse and the network protocol pulse, used to quantify the temporal correlation between the two types of features; the adjustment operation refers to the calculation process of scaling the connection strength parameter proportionally according to the time interval difference; the physical vibration feature node refers to the neuron node in the neural network model that specifically processes vibration feature encoding; the network protocol feature node refers to the neuron node in the neural network model that specifically processes protocol feature encoding; the connection strength parameter refers to the weight value of the connection between the two types of feature nodes in the neural network model, which determines the signal transmission strength; the adjusted connection strength parameter refers to the node connection weight value dynamically updated according to the pulse time difference.

[0152] In this embodiment of the invention, the time interval difference between the physical vibration characteristic pulse and the network protocol characteristic pulse is first obtained from the pulse sequence. Then, a connection strength adjustment coefficient is calculated based on the magnitude of the difference. Next, the adjustment coefficient is multiplied by the current connection strength parameter to generate a new parameter. Finally, the connection strength parameter between the two types of feature nodes in the neural network model is updated.

[0153] Step 404: The adjusted connection strength parameters are superimposed with the pulse firing modes of continuous time segments in the pulse sequence to generate the cross-modal temporal correlation.

[0154] In this step, the pulse firing pattern of continuous time segments refers to the temporal pattern of pulse triggering within multiple adjacent time segments, such as periodic triggering or sudden triggering; the overlay processing operation refers to the data fusion process of weighting and accumulating the connection strength parameter and the pulse firing pattern according to the time dimension.

[0155] In this embodiment of the invention, the firing patterns of pulse sequences within continuous time segments are first extracted. Then, the adjusted connection strength parameters are superimposed on the pulse firing patterns in chronological order. Next, the superposition result is normalized, and finally, a quantitative feature reflecting the cross-modal temporal correlation between physical vibration and network protocols is generated.

[0156] This invention employs a dynamic connection strength adjustment mechanism driven by pulse firing time intervals to achieve accurate temporal correlation modeling of physical vibration and network protocol behavior. By superimposing firing patterns of pulse sequences, the saliency of cross-modal temporal features is enhanced, overcoming the limitation of traditional neural network static weights in adapting to temporal changes, and significantly improving the sensitivity for detecting complex threats such as network attacks triggered by physical intrusions.

[0157] To reduce the impact of single-modal false alarms on threat detection, this step improves alarm accuracy through spatiotemporal coupling verification of dual-modal anomaly features and cumulative density statistics. This invention provides a specific embodiment: Step 105, when the cross-modal temporal correlation simultaneously includes the abnormal fluctuation pattern of the physical layer vibration signal and the protocol conflict characteristics of the network communication packet within a preset time window, a threat detection signal is generated for network security threat detection, specifically including the following steps:

[0158] Step 501: Divide the cross-modal temporal correlation into multiple consecutive time segments according to the length of a preset time window.

[0159] In this step, the segmentation operation refers to the process of dividing the time-series data stream into multiple analysis units according to a fixed time length, which is used to unify the time granularity of threat detection.

[0160] In this embodiment of the invention, a cross-modal temporal correlation data stream is first acquired. Then, the data stream is divided into multiple equal-length and time-continuous time segments according to a preset time window length. Next, the start and end times of each time segment are marked. Finally, the segmented time segments are transmitted to the anomaly detection module.

[0161] Step 502: Within each time segment, detect the number of consecutive changes in the vibration intensity value exceeding the preset fluctuation threshold in the physical layer vibration signal, and generate the periodic spike characteristics of the abnormal fluctuation pattern of the physical layer vibration signal.

[0162] In this step, the periodic spike feature refers to the regular abnormal waveform feature formed by the vibration intensity value exceeding the threshold multiple times in a short period of time, reflecting the repetitive actions of physical intrusion.

[0163] In this embodiment of the invention, physical layer vibration signal data within a time segment is first read, and then the number of times the vibration intensity value continuously exceeds a preset fluctuation threshold is detected. Next, the number of consecutive fluctuations exceeding the threshold per unit time is counted, and finally, a spike feature identifier reflecting the periodic characteristics of vibration anomalies is generated.

[0164] Step 503: Synchronously detect the matching status of the protocol type code in the network communication message with the preset protocol whitelist and the proportion of the payload length value in the network communication message that exceeds the preset range, so as to generate an abnormal protocol behavior identifier of the protocol conflict characteristics of the network communication message.

[0165] In this step, the preset protocol whitelist refers to a predefined set of legal communication protocol types used to verify whether the message protocol type is compliant; the abnormal protocol behavior identifier refers to a combination of abnormal status identifiers where the protocol type does not match the whitelist and the payload length exceeds the limit.

[0166] In this embodiment of the invention, a preset protocol whitelist database is first loaded, and then the protocol type codes in network communication packets within a time segment are matched with whitelist entries. Simultaneously, the percentage of packets with payload length values ​​exceeding a preset range is calculated, and finally, an abnormal protocol behavior identifier containing both protocol type mismatch and payload over-limit status is generated.

[0167] Step 504: When the periodic spike feature and the abnormal protocol behavior identifier exist simultaneously in the same time segment, mark the time segment as a threat-associated segment.

[0168] In this step, the threat-associated segment refers to a time segment where both physical vibration anomalies and network protocol anomalies exist simultaneously, characterizing a potential risk period for cross-modal coordinated attacks.

[0169] In this embodiment of the invention, a logical AND operation is first performed on the periodic spike characteristics and abnormal protocol behavior identifiers within the same time segment, and then it is determined whether the two exist simultaneously. If the condition is met, the time segment is marked as a threat-associated segment, and finally the time range and characteristic type of the segment are recorded.

[0170] Step 505: Calculate the cumulative density of threat-related segments in a continuous time segment. When the cumulative density exceeds a preset security threshold, generate a threat detection signal.

[0171] In this step, cumulative density refers to the proportion of threat-related fragments within a continuous time period, and is calculated by dividing the number of threat fragments by the total number of fragments.

[0172] In this embodiment of the invention, the number of threat-related segments in a continuous time segment is first counted, and then the ratio of the number of threat segments to the total number of segments is calculated as the cumulative density. Next, the cumulative density is compared with a preset security threshold, and finally, a network security threat detection signal is generated when the density exceeds the limit.

[0173] This invention employs a spatiotemporal synchronous verification mechanism based on dual-modal anomaly features to effectively filter out false alarms of a single modality and accurately identify the coordinated threat of physical intrusion and network attacks. It utilizes a cumulative density statistical method to enhance the reliability of detection results, avoiding false triggers caused by occasional anomalies and providing high-confidence proactive security protection capabilities for IoT devices.

[0174] Figure 2 This invention provides a schematic diagram of the structure of a network security threat detection system based on multimodal artificial intelligence, as shown in the embodiment of the invention. Figure 2 As shown, the system includes:

[0175] The acquisition module 21 is used to acquire the physical layer vibration signal of the Internet of Things device group, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase change;

[0176] Alignment module 22 is used to perform timestamp alignment processing on the continuous waveform data and the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data;

[0177] Encoding module 23 is used to extract discrete features containing protocol type and load length from the time synchronization network message data, and to perform cross-modal correlation encoding between the time synchronization vibration data and the discrete features to generate a multimodal data sequence;

[0178] Analysis module 24 is used to analyze the multimodal data sequence through a neural network model constructed based on the neuronal pulse time coding mechanism, and generate cross-modal temporal correlation relationships;

[0179] The generation module 25 is used to generate a threat detection signal for network security threat detection when the cross-modal temporal correlation simultaneously contains the abnormal fluctuation mode of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window.

[0180] Figure 2 The aforementioned network security threat detection system based on multimodal artificial intelligence can perform... Figure 1The implementation principle and technical effects of the network security threat detection method based on multimodal artificial intelligence described in the illustrated embodiment will not be repeated here. The specific methods by which each module and unit of the network security threat detection system based on multimodal artificial intelligence in the above embodiments perform operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0181] In one possible design, Figure 2 The network security threat detection system based on multimodal artificial intelligence shown in the embodiment can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0182] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0183] The processing component 32 is configured to: acquire physical layer vibration signals of an IoT device group, wherein the physical layer vibration signals are converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase changes; perform timestamp alignment processing on the continuous waveform data and the network communication messages of the target device in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data; extract discrete features containing protocol type and payload length from the time-synchronized network message data, and perform cross-modal correlation encoding on the time-synchronized vibration data and the discrete features to generate a multimodal data sequence; analyze the multimodal data sequence through a neural network model based on a neuronal pulse time coding mechanism to generate a cross-modal temporal correlation relationship; when the cross-modal temporal correlation relationship simultaneously contains the abnormal fluctuation mode of the physical layer vibration signal and the protocol conflict feature of the network communication message within a preset time window, generate a threat detection signal for network security threat detection.

[0184] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0185] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0186] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0187] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0188] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0189] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0190] This invention also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The embodiment shown is a network security threat detection method based on multimodal artificial intelligence.

[0191] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network security threat detection method based on multimodal artificial intelligence, characterized in that, include: The physical layer vibration signal of an IoT device group is acquired, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase change; The continuous waveform data is timestamped and the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data. Extracting discrete features containing protocol type and payload length from the time-synchronized network message data, and performing cross-modal correlation encoding on the time-synchronized vibration data and the discrete features to generate a multimodal data sequence, includes: parsing the header structure of each segment of the time-synchronized network message data to generate a discrete feature set containing protocol type code and payload length value; dividing the time-synchronized vibration data into multiple vibration waveform segments according to the time range corresponding to each protocol type code in the discrete feature set, generating a vibration intensity variation curve associated with the protocol type code; calculating the ratio of the payload length value to the maximum vibration amplitude of the vibration intensity variation curve, generating a mapping table of the protocol type code, the payload length value, and the vibration intensity variation curve; combining and encoding the waveform morphology features of the vibration intensity variation curve with the same protocol type code in the mapping table to generate multiple cross-modal correlation data blocks; arranging each of the cross-modal correlation data blocks in chronological order to generate a multimodal data sequence. The multimodal data sequence is analyzed using a neural network model based on the neuronal pulse time coding mechanism to generate cross-modal temporal correlation relationships. When the cross-modal temporal correlation simultaneously includes the abnormal fluctuation pattern of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window, a threat detection signal is generated for network security threat detection.

2. The method according to claim 1, characterized in that, Acquire physical layer vibration signals from a group of IoT devices, wherein the physical layer vibration signals are converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase changes, including: Deploy an optical fiber sensor array containing multiple sensing units in the IoT device group; A constant wavelength probe light signal is emitted to the sensing unit of the fiber optic sensor array to obtain a modulated light signal with phase shift of the reflected light signal caused by the surface vibration of the fiber optic sensor array. The instantaneous vibration displacement of the monitoring area where each sensing unit is located is determined based on the change in interference fringes between the modulated optical signal and the probe optical signal. The instantaneous vibration displacement of multiple sensing units at the same moment is superimposed according to their spatial distribution to generate continuous waveform data characterizing the vibration intensity of the equipment surface.

3. The method according to claim 1, characterized in that, The continuous waveform data is timestamped and aligned with the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data, including: The starting time point at which the first sudden change in vibration intensity occurs in the vibration signal of the physical layer is detected is used to generate the initial synchronization reference time point; In the network communication messages of the target device in the IoT device group, match the first message start boundary whose time deviation from the initial synchronization reference time point is less than a preset threshold, and generate the message start boundary to be corrected. The time of the start boundary of the message to be corrected is adjusted until the time of the start boundary of the message to be corrected is consistent with the initial synchronization reference time point, and the corrected message start boundary is generated. Based on the sampling interval of the continuous waveform data and the corrected message start boundary, the time axis of the network communication message is segmented and adjusted to generate a target message data segment whose start time is consistent with the start time of the waveform data segment; The vibration intensity feature sequence of each waveform data segment in the continuous waveform data is time-correspondingly bound to the original byte stream of the target message data segment to generate time-synchronized vibration data and time-synchronized network message data.

4. The method according to claim 1, characterized in that, The waveform morphology characteristics of the vibration intensity variation curve are combined and encoded with the protocol type codes that are identical in the mapping table to generate multiple cross-modal associated data blocks, including: Based on the numerical identifier of the protocol type code, the corresponding waveform feature extraction rule is selected from the preset encoding rule set; Based on the ratio of the load length value to the maximum vibration amplitude, the feature sensitivity parameters of the waveform feature extraction rule are adjusted to generate the adjusted feature sensitivity parameters. According to the adjusted feature sensitivity parameters, the waveform morphology features of the vibration intensity change curve are subjected to feature quantization processing to generate a physical vibration feature code bound to the protocol type code. The physical vibration feature code and the protocol type code are combined by bit sequence cross-combination to generate multiple cross-modal associated data blocks.

5. The method according to claim 1, characterized in that, The multimodal data sequences are analyzed using a neural network model based on a neuronal pulse time coding mechanism to generate cross-modal temporal correlations, including: Receive each cross-modal associated data block in the multimodal data sequence, and divide the cross-modal associated data block into multiple time segments according to a preset time window length; The pulse emission time interval is calculated for the physical vibration feature encoding and network protocol feature encoding within each time segment to generate a pulse sequence; Based on the time interval difference between the physical vibration feature pulses and the network protocol feature pulses in the pulse sequence, the connection strength parameters between the physical vibration feature nodes and the network protocol feature nodes in the neural network model constructed based on the neuronal pulse time coding mechanism are adjusted to generate the adjusted connection strength parameters. The adjusted connection strength parameters are superimposed with the pulse firing patterns of consecutive time segments in the pulse sequence to generate the cross-modal temporal correlation.

6. The method according to claim 1, characterized in that, When the cross-modal temporal correlation simultaneously contains the abnormal fluctuation pattern of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window, a threat detection signal is generated for network security threat detection, including: The cross-modal temporal correlation is divided into multiple consecutive time segments according to the length of a preset time window; Within each time segment, the number of consecutive changes in the vibration intensity value exceeding a preset fluctuation threshold in the physical layer vibration signal is detected, and periodic spike characteristics of the abnormal fluctuation pattern of the physical layer vibration signal are generated. The system synchronously detects the matching status between the protocol type code in the network communication message and the preset protocol whitelist, as well as the proportion of the payload length value in the network communication message that exceeds the preset range, in order to generate an abnormal protocol behavior identifier for the protocol conflict characteristics of the network communication message. When the periodic spike feature and the abnormal protocol behavior identifier exist simultaneously within the same time segment, the time segment is marked as a threat-associated segment; The cumulative density of threat-related segments in a continuous time segment is statistically analyzed. When the cumulative density exceeds a preset security threshold, a threat detection signal is generated.

7. A network security threat detection system based on multimodal artificial intelligence, characterized in that, include: The acquisition module is used to acquire the physical layer vibration signal of the IoT device group, wherein the physical layer vibration signal is converted into continuous waveform data characterizing the vibration intensity of the device surface through optical phase change; The alignment module is used to perform timestamp alignment processing on the continuous waveform data and the network communication messages of the target devices in the IoT device group to generate time-synchronized vibration data and time-synchronized network message data. An encoding module is used to extract discrete features containing protocol type and payload length from the time-synchronized network message data, and to perform cross-modal correlation encoding on the time-synchronized vibration data and the discrete features to generate a multimodal data sequence. This includes: parsing the header structure of each segment of the time-synchronized network message data to generate a discrete feature set containing protocol type code and payload length value; dividing the time-synchronized vibration data into multiple vibration waveform segments according to the time range corresponding to each protocol type code in the discrete feature set, generating a vibration intensity variation curve associated with the protocol type code; calculating the ratio of the payload length value to the maximum vibration amplitude of the vibration intensity variation curve, generating a mapping table between the protocol type code, the payload length value, and the vibration intensity variation curve; combining and encoding the waveform morphology features of the vibration intensity variation curve with the same protocol type code in the mapping table to generate multiple cross-modal correlation data blocks; and arranging each cross-modal correlation data block in chronological order to generate a multimodal data sequence. The analysis module is used to analyze the multimodal data sequence through a neural network model built based on the neuronal pulse time coding mechanism, and generate cross-modal temporal correlation relationships. The generation module is used to generate a threat detection signal for network security threat detection when the cross-modal temporal correlation simultaneously contains the abnormal fluctuation mode of the physical layer vibration signal and the protocol conflict characteristics of the network communication message within a preset time window.

8. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement a network security threat detection method based on multimodal artificial intelligence as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The device contains a computer program that, when executed by a computer, implements a network security threat detection method based on multimodal artificial intelligence as described in any one of claims 1 to 6.

Images