Intelligent safety early warning method, device and equipment based on multi-modal data fusion and storage medium

Abstract

The invention discloses an intelligent security early warning method, device and equipment based on multi-modal data fusion and a storage medium, relates to the field of information security, is applied to user terminal equipment carrying a security early warning model, and comprises the following steps: acquiring target data by using target data acquisition equipment corresponding to the user terminal equipment, processing the target data to obtain corresponding processed data; performing space-time alignment on the processed data based on the timestamp of the target data by using an attention mechanism to obtain corresponding aligned data, and obtaining target fusion data based on the aligned data by using a graph neural network; and judging whether the target fusion data meets a preset early warning condition by using a safety early warning model to obtain an early warning result, and when the early warning result represents that the target fusion data meets the preset early warning condition, reporting the target fusion data and the early warning result to a central server through an edge server corresponding to the user terminal equipment. According to the invention, the safety early warning response speed is improved.

Description

Technical Field

[0001] This invention relates to the field of information security, and in particular to an intelligent security early warning method, apparatus, device, and storage medium based on multimodal data fusion. Background Technology

[0002] Currently, relying solely on a single modality (such as text or image) for risk feature extraction makes it difficult to comprehensively capture multi-dimensional information in complex scenarios. For example, in financial risk control, simply analyzing transaction text records may overlook abnormal operation patterns in user behavior images, while relying solely on image recognition cannot parse the implicit semantic relationships in text, resulting in blind spots in risk assessment. Existing security early warning systems mostly adopt a centralized cloud processing model, with a long chain from data collection and transmission to model inference. In addition, traditional deep learning models have large parameters and time-consuming inference, making it difficult to meet the millisecond-level response requirements in scenarios such as industrial fault early warning and network attack detection.

[0003] Therefore, how to comprehensively capture multi-dimensional information in complex scenarios for security early warning and improve the real-time response speed of security early warning is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an intelligent security early warning method, device, equipment, and storage medium based on multimodal data fusion, which can comprehensively capture multi-dimensional information in complex scenarios for security early warning and improve the real-time response speed of security early warning. The specific solution is as follows:

[0005] Firstly, this application discloses an intelligent security early warning method based on multimodal data fusion, applied to a user terminal device equipped with a security early warning model, comprising:

[0006] The target data is acquired using the target data acquisition devices corresponding to the user terminal device, and the target data is processed to obtain corresponding processed data; the target data acquisition devices include a log server, sensors, and cameras; the target data includes target text data, target sensor data, and target image data.

[0007] An attention mechanism is used to perform spatiotemporal alignment of each of the processed data based on the timestamp of the target data to obtain the corresponding aligned data, and a graph neural network is used to obtain the target fused data based on the aligned data;

[0008] The security early warning model is used to determine whether the target fused data meets the preset early warning conditions to obtain the corresponding early warning results. When the early warning results indicate that the target fused data meets the preset early warning conditions, the target fused data and the early warning results are reported to the central server through the edge server corresponding to the user terminal device.

[0009] Optionally, processing the target data to obtain corresponding processed data includes:

[0010] Identify the target sensitive fields and the target ID field in the target text data;

[0011] The target sensitive field is replaced with a target fixed character to obtain the processed sensitive field;

[0012] The target ID field is processed by irreversible hashing and salting to obtain the processed ID field;

[0013] Based on the processed sensitive field and the processed ID field, determine the processed text data corresponding to the target text data;

[0014] The target image data is reduced in sharpness based on a preset sharpness threshold to obtain the first image data;

[0015] The target image replacement model is used to identify and replace sensitive objects in the first image data to obtain the second image data;

[0016] The sensitive regions in the second image data are identified based on the target sensitive region detection algorithm, and the sensitive regions are subjected to Gaussian blurring to obtain the processed image data.

[0017] Random noise is added to the target sensor data based on a preset error range to obtain the first sensor data;

[0018] The timestamp data in the first sensor data is matched based on a preset time period to replace the timestamp data in the first sensor data with time period data. The preset regional range is matched with the location data in the first sensor data to replace the location data in the first sensor data with regional data, thus obtaining processed sensor data.

[0019] Optionally, the step of using an attention mechanism to perform spatiotemporal alignment of each of the processed data based on the timestamp of the target data to obtain corresponding aligned data includes:

[0020] The semantic feature data corresponding to each of the processed data is extracted using the target word vector representation model;

[0021] Based on the timestamp of the target data, the corresponding semantic feature data are aligned to a unified time resolution to obtain the first aligned data;

[0022] The missing data timestamp is determined based on the timestamp of the first aligned data, and the missing data is filled in based on the missing data timestamp using a preset missing value filling method to obtain the second aligned data;

[0023] The second aligned data is spatiotemporally aligned using an attention mechanism to obtain the corresponding aligned data.

[0024] Optionally, the step of using a graph neural network to obtain target fused data based on the aligned data includes:

[0025] A graph neural network is used to determine the target graph corresponding to the aligned data of each modality based on the aligned data, and cross-modal node connections are established between the target graphs.

[0026] The target feature information corresponding to each target image is determined using a target feature extraction model.

[0027] The target feature information is fused based on the modal weight adjustment mechanism to obtain target fusion data.

[0028] Optionally, after reporting the target fusion data and the early warning result to the central server through the edge server corresponding to the user terminal device, the method further includes:

[0029] The central server determines whether there is an error in the warning result based on the target fusion data, and if there is an error in the warning result, it returns the corresponding warning analysis result to the user terminal device through the edge server corresponding to the user terminal device.

[0030] Optionally, the intelligent security early warning method based on multimodal data fusion further includes:

[0031] If the warning analysis result obtained by the user terminal device meets the model update conditions, the security warning model on the local side of the user terminal device is updated based on the preset model iteration rules to obtain the updated security warning model.

[0032] The updated local model parameters of the security early warning model are uploaded to the central server through the edge server corresponding to the user terminal device using the federated learning framework.

[0033] Optionally, the intelligent security early warning method based on multimodal data fusion further includes:

[0034] If the local model parameters uploaded by all the edge servers to the central server meet the global model parameter update conditions, then the central server generates new global model parameters based on all the local model parameters, and distributes them to the user terminal devices corresponding to each edge server through all the edge servers, so that each user terminal device can update its local security warning model using the federated learning framework after obtaining the new global model parameters.

[0035] Secondly, this application discloses an intelligent security early warning device based on multimodal data fusion, applied to a user terminal device equipped with a security early warning model, comprising:

[0036] The data processing module is used to acquire target data using the target data acquisition devices corresponding to the user terminal device, and to process the target data to obtain corresponding processed data; the target data acquisition devices include a log server, sensors, and cameras; the target data includes target text data, target sensor data, and target image data;

[0037] The data fusion module is used to perform spatiotemporal alignment of each of the processed data based on the timestamp of the target data using an attention mechanism to obtain the corresponding aligned data, and to use a graph neural network to obtain the target fused data based on the aligned data;

[0038] The early warning result reporting module is used to determine whether the target fused data meets the preset early warning conditions using the security early warning model to obtain the corresponding early warning result, and when the early warning result indicates that the target fused data meets the preset early warning conditions, the target fused data and the early warning result are reported to the central server through the edge server corresponding to the user terminal device.

[0039] Thirdly, this application discloses an electronic device, including:

[0040] Memory, used to store computer programs;

[0041] A processor is used to execute the computer program to implement the aforementioned intelligent security early warning method based on multimodal data fusion.

[0042] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned intelligent security early warning method based on multimodal data fusion.

[0043] In this application, when a user terminal device equipped with a security warning model issues a security warning, it acquires target data using target data acquisition devices corresponding to the user terminal device, and processes the target data to obtain corresponding processed data. The target data acquisition devices include a log server, sensors, and cameras. The target data includes target text data, target sensor data, and target image data. An attention mechanism is used to perform spatiotemporal alignment of each processed data based on the timestamp of the target data to obtain corresponding aligned data. A graph neural network is used to acquire target fusion data based on the aligned data. The security warning model is used to determine whether the target fusion data meets preset warning conditions to obtain corresponding warning results. When the warning results indicate that the target fusion data meets the preset warning conditions, the target fusion data and the warning results are reported to the central server through the edge server corresponding to the user terminal device. As can be seen, this application utilizes various target data acquisition devices corresponding to user terminal devices to acquire target data, and then processes the target data to obtain processed data. This target data includes multi-source heterogeneous data such as target text data collected by log servers, target sensor data collected by sensors, and target image data collected by cameras. Then, based on the attention mechanism, the spatiotemporal dimension features of the processed data are aligned, and the intermodal correlation is modeled by graph neural networks to obtain target fusion data corresponding to the target data. Finally, the security early warning model locally mounted on the user terminal device is used to determine whether the target fusion data meets the preset early warning conditions. When the early warning result indicates that the target fusion data meets the preset early warning conditions, the target fusion data and the early warning result are reported to the central server through the edge server corresponding to the user terminal device. The central server can then perform subsequent early warning operations based on the early warning result and the target fusion data. By acquiring data, processing data, and using the security early warning model for risk analysis to obtain early warning results at the edge where the user terminal device is located, the link from data acquisition and transmission to model inference and the inference time are significantly shortened. Moreover, it can automatically identify potential security threats from complex multimodal data and improve the real-time response speed of security early warning. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 This is a flowchart of an intelligent security early warning method based on multimodal data fusion disclosed in this application;

[0046] Figure 2 This is a schematic diagram of the structure of an intelligent safety early warning device based on multimodal data fusion disclosed in this application;

[0047] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Currently, relying solely on a single modality (such as text or image) for risk feature extraction makes it difficult to comprehensively capture multi-dimensional information in complex scenarios. For example, in financial risk control, simply analyzing transaction text records may overlook abnormal operation patterns in user behavior images, while relying solely on image recognition cannot parse the implicit semantic relationships in text, resulting in blind spots in risk assessment. Existing security early warning systems mostly adopt a cloud-based centralized processing model, with long links from data collection and transmission to model inference. In addition, traditional deep learning models have large parameters and time-consuming inference, making it difficult to meet the millisecond-level response requirements in scenarios such as industrial fault early warning and network attack detection. To solve the above technical problems, this application discloses an intelligent security early warning method based on multi-modal data fusion, which can comprehensively capture multi-dimensional information in complex scenarios for security early warning and improve the real-time response speed of security early warning.

[0050] See Figure 1 As shown, this embodiment of the invention discloses an intelligent security early warning method based on multimodal data fusion, applied to a user terminal device equipped with a security early warning model, including:

[0051] Step S11: Acquire target data using the target data acquisition devices corresponding to the user terminal device, and process the target data to obtain corresponding processed data; the target data acquisition devices include a log server, sensors, and cameras; the target data includes target text data, target sensor data, and target image data.

[0052] In this embodiment, user terminal devices equipped with a security warning model are divided into multiple groups based on geographical location, network topology, or function. Each group corresponds to an edge server. The central server distributes global model parameters (such as neural network weights) to all edge servers, which then distribute them to the user terminal devices within the group. Aggregation rules (such as weighting methods and communication frequencies) and privacy protection mechanisms (such as differential privacy and encrypted transmission) are defined for each layer, thereby completing the initialization and layered architecture construction of the security warning system.

[0053] In this embodiment, the user terminal device is equipped with a security warning model. When conducting intelligent security warnings, it first uses the target data acquisition device corresponding to the user terminal device to acquire target data. The target data acquisition process of each target data acquisition device is performed synchronously. In one specific implementation, an NTP (Network Time Protocol) server is deployed, and all target data acquisition devices (cameras, sensors, log servers) are synchronized to the same standard time. For devices that cannot connect to the network (such as local cameras), millisecond-level synchronization is achieved through PTP (Precision Time Protocol) or hardware clock calibration. After time synchronization is achieved, log collection tools (such as Fluentd, Logstash) are used to capture log files in real time. Structured reports (such as JSON, CSV format) are obtained through API (Application Programming Interface) interfaces or database queries. Video streams are pulled through RTSP / RTMP (Real Time Streaming Protocol / Real Time Messaging Protocol) protocols or SDK (Software Development Kit) (such as FFmpeg, OpenCV). For video data, keyframe extraction and frequency alignment are also required. This involves saving keyframes as needed (e.g., 1 frame per second) to reduce storage pressure, then pushing data to the edge gateway via MQTT (Message Queuing Telemetry Transport), HTTP (Hypertext Transfer Protocol), or OPC UA (Open Platform Communications Unified Architecture). Simultaneously, the sensor sampling frequency (e.g., 10Hz) is set to match or be an integer multiple of the video frame rate (e.g., 25fps) to achieve frequency alignment. After data acquisition, tools like Kafka and RabbitMQ are used to cache data in time windows (e.g., 1 second) to ensure data from the same time period enters the same processing flow. Data is also cached locally on the device (e.g., cameras, sensors) and uploaded in batches after network recovery, enabling data storage and indexing. Finally, alignment verification is performed again. For example, using sensor timestamps as a baseline, text and image timestamps are aligned by interpolation or truncation, and time windows are randomly selected to check if the timestamps of text, images, and sensor data are within the allowable error range (e.g., ±100ms).

[0054] In this embodiment, processing these target data yields the corresponding processed data. The specific data processing steps may include: determining the target sensitive field and target ID field in the target text data; replacing the target sensitive field with a target fixed character to obtain the processed sensitive field; processing the target ID field using irreversible hashing and salting to obtain the processed ID field; determining the processed text data corresponding to the target text data based on the processed sensitive field and processed ID field; reducing the clarity of the target image data based on a preset clarity threshold to obtain the first image data; identifying and replacing sensitive objects in the first image data using a target image replacement model to obtain the second image data; identifying sensitive regions in the second image data based on a target sensitive region detection algorithm and performing Gaussian blur processing on the sensitive regions to obtain the processed image data; adding random noise to the target sensor data based on a preset error range to obtain the first sensor data; matching the timestamp data in the first sensor data based on a preset time period to replace the timestamp data in the first sensor data with time period data, and matching a preset regional range with the location data in the first sensor data to replace the location data in the first sensor data with regional data to obtain the processed sensor data.

[0055] In other words, in this embodiment, since the target data, including target text data, target sensor data, and target image data, correspond to three types of target data acquisition devices—log servers, sensors, and cameras—the processing of the target data is divided into text desensitization, image / video desensitization, and sensor data desensitization. Specifically, text desensitization can involve replacing sensitive fields with fixed values ​​(e.g., using the string "****" to replace the middle four digits of a phone number), and using irreversible hashing (e.g., SHA-256) with salt for ID-type fields. Image / video desensitization can involve reducing image resolution or frame rate to make details unrecognizable, such as using deep learning models (e.g., YOLO, You Only Look Once) to identify and replace sensitive objects, and using OpenCV to detect and Gaussian blur sensitive areas. Sensor data desensitization can be achieved by adding random noise to numerical values ​​(with controlled error range), replacing precise timestamps with time periods (e.g., replacing 12:00:00 with 12:00-12:15), and blurring GPS (Global Positioning System) coordinates to regions (e.g., city level).

[0056] Step S12: Use an attention mechanism to perform spatiotemporal alignment on each of the processed data based on the timestamp of the target data to obtain the corresponding aligned data, and use a graph neural network to obtain the target fusion data based on the aligned data.

[0057] In this embodiment, after processing the target data, an attention mechanism can be used to perform spatiotemporal alignment of each processed data based on the timestamp of the target data to obtain the corresponding aligned data. The specific process may include: extracting semantic feature data corresponding to each processed data using the target word vector representation model BERT (Bidirectional Encoder Representations from Transformers) or Word2Vec; aligning the corresponding semantic feature data to a uniform time resolution (e.g., second-level) based on the timestamp of the target data to obtain the first aligned data; determining the timestamp of missing data based on the timestamp of the first aligned data, and using a preset missing value completion method (e.g., interpolation or imputation) to fill in the missing data based on the timestamp of the missing data to obtain the second aligned data; and performing multimodal spatiotemporal alignment of the second aligned data based on an attention mechanism that can simultaneously capture temporal dependence and spatial correlation, thereby using attention weights to fuse the spatiotemporal features of the multimodal data to obtain the corresponding aligned data.

[0058] In this embodiment, the acquisition of target fusion data based on aligned data using a graph neural network can specifically include: using a graph neural network to determine the target graph corresponding to the aligned data of each modality, and establishing cross-modal node connections between the target graphs; using a target feature extraction model to determine the target feature information corresponding to each target graph; and fusing all target feature information based on a modality weight adjustment mechanism to obtain target fusion data. In a specific implementation, nodes are defined according to modal characteristics to construct edges based on intra-modal relationships. Cross-modal node connections are established through semantic matching (such as the correspondence between text descriptions and image targets) or external knowledge (such as entity links) to achieve multimodal data processing and graph construction. Then, word vector representation models (such as Word2Vec, BERT) or TF-IDF (Term Frequency-Inverse Document Frequency, a common weighting technique for information retrieval and data mining) are used to obtain corresponding word vectors. Based on these word vectors, CNN (Convolutional Neural Network) is used to extract features (such as ResNet, VGG). For the graph of each modality, GNN (Graph Neural Network) (such as GCN (Graph Convolutional Networks), GAT (Graph Attention Network), GraphSAGE (Graph Sample and Aggregate)) is applied to aggregate neighborhood information, thereby obtaining target feature information and completing intra-modal feature extraction. Next, inter-modal attention weights are designed to dynamically evaluate the contribution of other modalities to the current modality, merging multimodal nodes into the same graph. Heterogeneous graphs (such as knowledge graphs) are designed, and heterogeneous GNNs (such as RGCN, Relational Graph Convolutional Network) are used to model the aggregation stage of GNNs, allowing nodes of different modalities to exchange information and realizing inter-modal association modeling. Finally, the node representations of each modality are concatenated and fused through a fully connected layer or Transformer to learn modality weights (e.g., through gating mechanisms or attention), dynamically balancing the contributions of different modalities. Global pooling of the graph (e.g., Graph Pooling) is then performed to generate a multimodal joint representation (i.e., target fusion data), completing the multimodal feature fusion. In one specific implementation, the system achieves spatiotemporal feature alignment through an attention mechanism and combines graph neural networks to model the intermodal relationships, resulting in a risk identification F1-score of 92.5% (a 28% improvement over single-modal methods).For example, in industrial equipment monitoring, sensor time-series data, equipment operation videos and maintenance logs can be analyzed simultaneously to accurately capture the associated risks of abnormal temperature and vibration frequency with visual characteristics (such as sparks and leaks), avoiding missed detections of a single mode.

[0059] Step S13: Use the security early warning model to determine whether the target fused data meets the preset early warning conditions to obtain the corresponding early warning results. When the early warning results indicate that the target fused data meets the preset early warning conditions, report the target fused data and the early warning results to the central server through the edge server corresponding to the user terminal device.

[0060] In this embodiment, a security early warning model is used to determine whether the target fused data meets preset early warning conditions, thereby obtaining the corresponding early warning result. The specific preset early warning conditions are not limited here and can be adjusted as needed in actual use. If the early warning result indicates that the target fused data meets the preset early warning conditions, the user terminal device will report the target fused data and the early warning result to the central server through the edge server corresponding to the user terminal device.

[0061] In this embodiment, after the user terminal device reports the target fusion data and warning results to the central server through the edge server corresponding to the user terminal device, the central server will determine whether there are any errors in the warning results based on the target fusion data. If there are errors in the warning results, the central server will return the corresponding warning analysis results to the user terminal device through the edge server corresponding to the user terminal device. If the warning analysis results obtained by the user terminal device meet the model update conditions, the local security warning model of the user terminal device will be updated according to the preset model iteration rules to obtain the updated security warning model. The local model parameters of the updated security warning model will be uploaded to the central server through the edge server corresponding to the user terminal device using the federated learning framework. In addition, if the local model parameters uploaded to the central server by all edge servers meet the global model parameter update conditions, the central server will generate new global model parameters based on all local model parameters and distribute them to the user terminal devices corresponding to each edge server through all edge servers, so that each user terminal device can update its local security warning model using the federated learning framework after obtaining the new global model parameters.

[0062] In other words, in this embodiment, user terminal devices do not share raw data. Instead, they use local private data to run gradient descent or local epoch training based on the received model parameters to update the local security warning model. The locally updated local model parameters (such as gradients or model differences) are encrypted and sent to the corresponding edge server. The central server receives the local model parameters from each edge server, generates new global model parameters, and distributes the global parameters to the edge servers. The edge servers then distribute the updated model to the user terminal devices within the group. The user terminal devices replace their local models with the newly received security warning model and enter the next training cycle. It should be noted that the edge servers continuously receive parameter updates from the user terminal devices within the group and support asynchronous uploads (without waiting for all user terminal devices to complete training), realizing dynamic parameter aggregation of edge nodes. By repeating the process of local training → upload → aggregation → update, continuous iteration is performed, and the aggregation weights are dynamically adjusted according to changes in the data distribution of user terminal devices (such as the FedAvg algorithm), thereby achieving iterative and feedback optimization of the security warning model.

[0063] In this embodiment, a federated learning framework and edge node real-time fine-tuning technology are employed to achieve an early warning latency of <200ms. The system aggregates model parameters hierarchically, performs lightweight processing of multimodal data at the edge (e.g., knowledge distillation and model compression), and performs global optimization in the cloud, ensuring millisecond-level response to risks such as production line equipment failures and supply chain logistics anomalies, meeting the real-time control requirements of industry. Based on federated learning and differential privacy technology, data is trained across modalities under encrypted and anonymized conditions, achieving ISO 27001 certification. Simultaneously, the system supports dynamic model updates: edge nodes fine-tune the model based on local real-time data, and the cloud integrates global feature distribution changes, effectively adapting to the evolution of risk patterns such as equipment aging and seasonal environmental fluctuations, reducing false alarm rates. In supply chain management, the system integrates multimodal data such as market demand forecasts, supplier delivery cycles, and logistics trajectories, using graph neural networks to identify supply chain bottleneck nodes (e.g., shortages of key components or congested transportation routes), generating resilience optimization solutions in advance. In industrial scenarios, by combining historical equipment failure maps with real-time multimodal data, interpretable risk root cause analysis is output to guide precise maintenance decisions. By breaking through the limitations of traditional single-point monitoring through multimodal data fusion, breakthroughs are achieved in accuracy, real-time performance, compliance, and adaptability, providing intelligent solutions for the entire chain, from risk perception and rapid early warning to strategy optimization, for scenarios such as industrial safety and supply chain management.

[0064] As can be seen, this application utilizes various target data acquisition devices corresponding to user terminal devices to acquire target data, and then processes the target data to obtain processed data. This target data includes multi-source heterogeneous data such as target text data collected by log servers, target sensor data collected by sensors, and target image data collected by cameras. Then, based on the attention mechanism, the spatiotemporal dimension features of the processed data are aligned, and the intermodal correlation is modeled by graph neural networks to obtain target fusion data corresponding to the target data. Finally, the security early warning model locally mounted on the user terminal device is used to determine whether the target fusion data meets the preset early warning conditions. When the early warning result indicates that the target fusion data meets the preset early warning conditions, the target fusion data and the early warning result are reported to the central server through the edge server corresponding to the user terminal device. The central server can then perform subsequent early warning operations based on the early warning result and the target fusion data. By acquiring data, processing data, and using the security early warning model for risk analysis to obtain early warning results at the edge where the user terminal device is located, the link from data acquisition and transmission to model inference and the inference time are significantly shortened. Moreover, it can automatically identify potential security threats from complex multimodal data and improve the real-time response speed of security early warning.

[0065] See Figure 2 As shown, this application discloses an intelligent security early warning device based on multimodal data fusion, applied to a user terminal device equipped with a security early warning model, comprising:

[0066] The data processing module 11 is used to acquire target data using the target data acquisition devices corresponding to the user terminal device, and to process the target data to obtain corresponding processed data; the target data acquisition devices include a log server, sensors, and cameras; the target data includes target text data, target sensor data, and target image data.

[0067] The data fusion module 12 is used to perform spatiotemporal alignment of each of the processed data based on the timestamp of the target data using an attention mechanism to obtain the corresponding aligned data, and to use a graph neural network to obtain the target fused data based on the aligned data.

[0068] The early warning result reporting module 13 is used to determine whether the target fused data meets the preset early warning conditions using the security early warning model to obtain the corresponding early warning result, and when the early warning result indicates that the target fused data meets the preset early warning conditions, the target fused data and the early warning result are reported to the central server through the edge server corresponding to the user terminal device.

[0069] As can be seen, this application utilizes various target data acquisition devices corresponding to user terminal devices to acquire target data, and then processes the target data to obtain processed data. This target data includes multi-source heterogeneous data such as target text data collected by log servers, target sensor data collected by sensors, and target image data collected by cameras. Then, based on the attention mechanism, the spatiotemporal dimension features of the processed data are aligned, and the intermodal correlation is modeled by graph neural networks to obtain target fusion data corresponding to the target data. Finally, the security early warning model locally mounted on the user terminal device is used to determine whether the target fusion data meets the preset early warning conditions. When the early warning result indicates that the target fusion data meets the preset early warning conditions, the target fusion data and the early warning result are reported to the central server through the edge server corresponding to the user terminal device. The central server can then perform subsequent early warning operations based on the early warning result and the target fusion data. By acquiring data, processing data, and using the security early warning model for risk analysis to obtain early warning results at the edge where the user terminal device is located, the link from data acquisition and transmission to model inference and the inference time are significantly shortened. Moreover, it can automatically identify potential security threats from complex multimodal data and improve the real-time response speed of security early warning.

[0070] In one specific embodiment, the data processing module 11 may include:

[0071] A field determination unit is used to determine the target sensitive field and the target ID field in the target text data;

[0072] The first field processing unit is used to replace the target sensitive field with a target fixed character to obtain the processed sensitive field;

[0073] The second field processing unit is used to process the target ID field through irreversible hashing and salting to obtain the processed ID field;

[0074] The text data processing unit is used to determine the processed text data corresponding to the target text data based on the processed sensitive field and the processed ID field.

[0075] The first image data acquisition unit is used to perform a sharpness reduction operation on the target image data based on a preset sharpness threshold to obtain the first image data;

[0076] The second image data acquisition unit is used to identify and replace sensitive objects in the first image data using a target image replacement model to obtain the second image data.

[0077] The processed image data acquisition unit is used to identify the sensitive regions in the second image data based on the target sensitive region detection algorithm, and to perform Gaussian blur processing on the sensitive regions to obtain the processed image data.

[0078] The first sensor data acquisition unit is used to add random noise to the target sensor data based on a preset error range to obtain the first sensor data;

[0079] The processed sensor data acquisition unit is used to match the timestamp data in the first sensor data based on a preset time period to replace the timestamp data in the first sensor data with time period data, and to match the preset regional range with the location data in the first sensor data to replace the location data in the first sensor data with regional data, thereby obtaining processed sensor data.

[0080] In one specific embodiment, the data fusion module 12 may include:

[0081] The semantic feature data acquisition unit is used to extract the semantic feature data corresponding to each of the processed data using the target word vector representation model;

[0082] The first alignment data acquisition unit is used to align the corresponding semantic feature data to a unified time resolution based on the timestamp of the target data to obtain the first alignment data.

[0083] The second alignment data acquisition unit is used to determine the missing data timestamp based on the timestamp of the first alignment data, and use a preset missing value completion method to complete the missing data based on the missing data timestamp to obtain the second alignment data;

[0084] The aligned data acquisition unit is used to perform multimodal spatiotemporal alignment on the second aligned data based on an attention mechanism to obtain the corresponding aligned data.

[0085] In one specific embodiment, the data fusion module 12 may include:

[0086] A node connection establishment unit is used to determine the target graph corresponding to the aligned data of each modality based on the aligned data using a graph neural network, and to establish cross-modal node connections between the target graphs.

[0087] The target feature information acquisition unit is used to determine the target feature information corresponding to each target image using a target feature extraction model.

[0088] The target fusion data acquisition unit is used to fuse all the target feature information based on the modal weight adjustment mechanism to obtain target fusion data.

[0089] In one specific embodiment, the device may further include:

[0090] The early warning analysis result acquisition module is used to determine whether there is an error in the early warning result based on the target fusion data through the central server, and to return the corresponding early warning analysis result to the user terminal device through the edge server corresponding to the user terminal device when there is an error in the early warning result.

[0091] In one specific embodiment, the device may further include:

[0092] The first model update module is used to update the local security warning model of the user terminal device based on a preset model iteration rule to obtain the updated security warning model if the warning analysis result obtained by the user terminal device meets the model update conditions.

[0093] The local model parameter upload module is used to upload the updated local model parameters of the security warning model to the central server through the edge server corresponding to the user terminal device using the federated learning framework.

[0094] In one specific embodiment, the device may further include:

[0095] The second model update module is used to generate new global model parameters based on all the local model parameters if the local model parameters uploaded by all the edge servers to the central server meet the global model parameter update conditions, and then distribute the new global model parameters to the user terminal devices corresponding to each edge server through all the edge servers, so that each user terminal device can update its local security warning model using a federated learning framework after obtaining the new global model parameters.

[0096] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0097] Figure 3This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the intelligent security early warning method based on multimodal data fusion disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the intelligent security early warning method based on multimodal data fusion executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0101] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned intelligent security early warning method based on multimodal data fusion. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0103] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A smart security early warning method based on multimodal data fusion, characterized in that, Applied to user terminal devices equipped with security early warning models, including: The target data is acquired using the target data acquisition devices corresponding to the user terminal device, and the target data is processed to obtain corresponding processed data; the target data acquisition devices include a log server, sensors, and cameras; the target data includes target text data, target sensor data, and target image data. An attention mechanism is used to perform spatiotemporal alignment of each of the processed data based on the timestamp of the target data to obtain the corresponding aligned data, and a graph neural network is used to obtain the target fused data based on the aligned data; The security early warning model is used to determine whether the target fused data meets the preset early warning conditions to obtain the corresponding early warning results. When the early warning results indicate that the target fused data meets the preset early warning conditions, the target fused data and the early warning results are reported to the central server through the edge server corresponding to the user terminal device.

2. The intelligent security early warning method based on multimodal data fusion according to claim 1, characterized in that, The process of processing the target data to obtain the corresponding processed data includes: Identify the target sensitive fields and the target ID field in the target text data; The target sensitive field is replaced with a target fixed character to obtain the processed sensitive field; The target ID field is processed by irreversible hashing and salting to obtain the processed ID field; Based on the processed sensitive field and the processed ID field, determine the processed text data corresponding to the target text data; The target image data is reduced in sharpness based on a preset sharpness threshold to obtain the first image data; The target image replacement model is used to identify and replace sensitive objects in the first image data to obtain the second image data; The sensitive regions in the second image data are identified based on the target sensitive region detection algorithm, and the sensitive regions are subjected to Gaussian blurring to obtain the processed image data. Random noise is added to the target sensor data based on a preset error range to obtain the first sensor data; The timestamp data in the first sensor data is matched based on a preset time period to replace the timestamp data in the first sensor data with time period data. The preset regional range is matched with the location data in the first sensor data to replace the location data in the first sensor data with regional data, thus obtaining processed sensor data.

3. The intelligent security early warning method based on multimodal data fusion according to claim 1, characterized in that, The step of using an attention mechanism to perform spatiotemporal alignment of each processed data based on the timestamp of the target data to obtain corresponding aligned data includes: The semantic feature data corresponding to each of the processed data is extracted using the target word vector representation model; Based on the timestamp of the target data, the corresponding semantic feature data are aligned to a unified time resolution to obtain the first aligned data; The missing data timestamp is determined based on the timestamp of the first aligned data, and the missing data is filled in based on the missing data timestamp using a preset missing value filling method to obtain the second aligned data; The second aligned data is spatiotemporally aligned using an attention mechanism to obtain the corresponding aligned data.

4. The intelligent security early warning method based on multimodal data fusion according to claim 1, characterized in that, The step of using a graph neural network to obtain target fused data based on the aligned data includes: A graph neural network is used to determine the target graph corresponding to the aligned data of each modality based on the aligned data, and cross-modal node connections are established between the target graphs. The target feature information corresponding to each target image is determined using a target feature extraction model. The target feature information is fused based on the modal weight adjustment mechanism to obtain target fusion data.

5. The intelligent security early warning method based on multimodal data fusion according to claim 1, characterized in that, After reporting the target fusion data and the early warning result to the central server through the edge server corresponding to the user terminal device, the method further includes: The central server determines whether there is an error in the warning result based on the target fusion data, and if there is an error in the warning result, it returns the corresponding warning analysis result to the user terminal device through the edge server corresponding to the user terminal device.

6. The intelligent security early warning method based on multimodal data fusion according to claim 5, characterized in that, Also includes: If the warning analysis result obtained by the user terminal device meets the model update conditions, the security warning model on the local side of the user terminal device is updated based on the preset model iteration rules to obtain the updated security warning model. The updated local model parameters of the security early warning model are uploaded to the central server through the edge server corresponding to the user terminal device using the federated learning framework.

7. The intelligent security early warning method based on multimodal data fusion according to claim 6, characterized in that, Also includes: If the local model parameters uploaded by all the edge servers to the central server meet the global model parameter update conditions, then the central server generates new global model parameters based on all the local model parameters, and distributes them to the user terminal devices corresponding to each edge server through all the edge servers, so that each user terminal device can update its local security warning model using the federated learning framework after obtaining the new global model parameters.

8. An intelligent safety early warning device based on multimodal data fusion, characterized in that, Applied to user terminal devices equipped with security early warning models, including: The data processing module is used to acquire target data using the target data acquisition devices corresponding to the user terminal device, and to process the target data to obtain corresponding processed data; the target data acquisition devices include a log server, sensors, and cameras; the target data includes target text data, target sensor data, and target image data; The data fusion module is used to perform spatiotemporal alignment of each of the processed data based on the timestamp of the target data using an attention mechanism to obtain the corresponding aligned data, and to use a graph neural network to obtain the target fused data based on the aligned data; The early warning result reporting module is used to determine whether the target fused data meets the preset early warning conditions using the security early warning model to obtain the corresponding early warning result, and when the early warning result indicates that the target fused data meets the preset early warning conditions, the target fused data and the early warning result are reported to the central server through the edge server corresponding to the user terminal device.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the intelligent security early warning method based on multimodal data fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the intelligent security early warning method based on multimodal data fusion as described in any one of claims 1 to 7.

Images