Safety monitoring method and system in satellite signal denial environment

Abstract

The application discloses a kind of satellite signal rejection environment under safe monitoring method and system, it is related to safety control technical field, including: by collecting multi-modal data and equipment text data, and it is time-space synchronization, encryption processing and structured analysis, constructs target scene multi-modal knowledge graph.Self-supervised learning is used for feature fusion, and a unified feature is generated for equipment state classification.The state information is stored after being hashed and encrypted, achieving safe data management and real-time monitoring.When detecting abnormal equipment state information, different geological risk information is obtained through analysis to provide early warning, achieving real-time monitoring of equipment state under satellite signal rejection environment to provide geological risk early warning and safety monitoring of construction personnel.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of safety management, and in particular to a safety monitoring method and system in a satellite signal denial environment. BACKGROUND

[0002] In a satellite signal denial environment (including mine tunnels, underground caverns, hydropower plant workshops and underground pipe galleries and the like), satellite signals are easily affected by terrain shielding, building shielding or electromagnetic interference, resulting in signal attenuation, loss of lock or even complete interruption, and traditional monitoring schemes relying on satellites face significant technical bottlenecks. In a satellite-restricted environment, there is a lack of satellite time and positioning reference such as Beidou and GPS, and multi-device collected multi-modal data is prone to time deviation and spatial coordinate confusion, resulting in invalid data correlation and making it impossible to accurately restore the device operating state and the environment scene. At the same time, there are structured data such as device tables and text descriptions, as well as unstructured multi-modal data such as images and videos in the monitoring scene, and existing technologies are difficult to achieve deep semantic correlation of cross-modal data, which is prone to data islands and cannot provide comprehensive data support for device state evaluation. Therefore, there is an urgent need for a safety monitoring method to realize real-time monitoring of device state for geological risk warning and safety monitoring of construction personnel in a satellite signal denial environment. SUMMARY

[0003] The main purpose of the present application is to provide a safety monitoring method and system in a satellite signal denial environment, aiming to solve the technical problem of how to realize real-time monitoring of device state for geological risk warning and safety monitoring of construction personnel in a satellite signal denial environment.

[0004] To achieve the above-mentioned purpose, the present application provides a safety monitoring method in a satellite signal denial environment, comprising:

[0005] Obtaining multi-modal data and device text data of a target device, the multi-modal data comprising image data, video data and audio data, and the device text data comprising device table data and device text description data;

[0006] Performing spatio-temporal synchronization processing on the multi-modal data to generate spatio-temporal consistent data;

[0007] Performing end-side encryption processing on the spatio-temporal consistent data to obtain encrypted multi-modal data, wherein the encrypted multi-modal data is uploaded to a data lake for storage through a secure transmission protocol;

[0008] Analyzing the device text data through multi-scale morphological enhancement and hierarchical structure perception coding to obtain structured data and generate corresponding triples, the triples corresponding to device entities and attributes;

[0009] Perform semantic association based on the structured data and the encrypted multi-modal data, and construct a target scene multi-modal knowledge graph;

[0010] Perform feature fusion on the features in the target scene multi-modal knowledge graph using a self-supervised learning strategy, and generate unified fusion features;

[0011] Process the unified fusion features through a preset device state classification model to obtain device state information, which is hashed and stored on-chain;

[0012] Compare the state parameter values of the device state information with the device state threshold to obtain a comparison result, wherein the state parameter values are the operating environment temperature, the operating environment pollutant concentration, and the operating time of the device;

[0013] When the comparison result is that the state parameter values exceed the preset state threshold, generate an alarm trigger instruction and send it to the target device to make the target device feedback geological risk information;

[0014] Receive the geological risk information and perform corresponding warning operations according to the geological risk information.

[0015] In an embodiment, the step of performing spatio-temporal synchronization processing on the multi-modal data to generate spatio-temporally consistent data includes:

[0016] Obtain the spatial coordinate information and the reference time information of the device, wherein the spatial coordinate information is obtained through UWB positioning, and the reference time information is generated by a PTP master clock server in the mine network;

[0017] Establish a spatio-temporal mapping relationship table based on the spatial coordinate information and the reference time information, wherein the spatio-temporal mapping relationship table includes a time deviation compensation parameter and a spatial coordinate conversion matrix;

[0018] Perform spatio-temporal correction on the multi-modal data based on the spatio-temporal mapping relationship table to obtain spatio-temporally consistent data, wherein the spatio-temporally consistent data is added with a unified spatio-temporal label, and the unified spatio-temporal label includes a GPS timestamp and three-dimensional spatial coordinates.

[0019] In an embodiment, the step of parsing the device text data through multi-scale morphological enhancement and hierarchical structure perception encoding to obtain structured data and generate corresponding triples includes:

[0020] Convert the device text data into image information;

[0021] Perform grayscale processing on the image information to obtain a grayscale image;

[0022] performing enhancement processing on the gray-scale image by using a multi-scale morphological operation to obtain an enhanced image, wherein the multi-scale morphological operation comprises an opening operation, a closing operation, a top-hat transformation, and a bottom-hat transformation;

[0023] performing feature extraction on the enhanced image to generate basic visual features;

[0024] projecting the basic visual features into a query, key, and value space in a row direction and a column direction, respectively, to obtain a row attention matrix and a column attention matrix;

[0025] performing structure-aware encoding based on the basic visual features, the row attention matrix, and the column attention matrix to obtain structure-aware features;

[0026] generating cell coordinates and content information of a table based on the structure-aware features;

[0027] converting the cell coordinates and the content information into structured data;

[0028] generating corresponding triples based on the structured data, wherein the structure of the triples is {device entity, attribute name, attribute value}.

[0029] In an embodiment, the step of performing semantic association based on the structured data and the encrypted multi-modal data to construct a multi-modal knowledge graph of a target scene comprises:

[0030] performing modality-specific preprocessing on the encrypted multi-modal data to output a standardized multi-modal data set, wherein the modality-specific preprocessing comprises image denoising and enhancement, audio framing and noise reduction, video key frame extraction, and text content recognition;

[0031] performing entity extraction based on the structured data to obtain basic triple data of corresponding entities, wherein the entities are components of a target device;

[0032] generating corresponding semantic text descriptions based on the basic triple data to output a structured semantic description set;

[0033] performing cross-modal semantic retrieval in the multi-modal data set based on the structured semantic description set to output a multi-modal association result set;

[0034] generating extended triples that describe entity relationships based on the multi-modal association result set;

[0035] constructing an initial knowledge graph based on the extended triples, wherein the initial knowledge graph is constructed with entities as nodes and triple relationships as edges;

[0036] The initial knowledge graph is subjected to semantic consistency verification and conflict resolution, and a target scene multi-modal knowledge graph is output.

[0037] In an embodiment, the step of employing a self-supervised learning strategy to perform feature fusion on the features in the target scene multi-modal knowledge graph to generate unified fusion features comprises:

[0038] Based on the target scene multi-modal knowledge graph, a multi-modal corresponding node feature vector is extracted.

[0039] According to the multi-modal corresponding node feature vector, a heterogeneous graph neural network is constructed as a node.

[0040] The node feature vector is subjected to intra-modal contrast learning using a self-supervised learning strategy to obtain intra-modal neighborhood node information.

[0041] The node feature vector is subjected to inter-modal contrast learning using a self-supervised learning strategy to obtain inter-modal neighborhood node information.

[0042] The intra-modal neighborhood node information and the inter-modal neighborhood node information are input into the heterogeneous graph neural network for aggregation to obtain updated multi-modal node features.

[0043] The updated multi-modal node features are subjected to weighted fusion and normalization processing to obtain unified fusion features.

[0044] In an embodiment, the step of processing the unified fusion features through a preset device state classification model to obtain device state information, and hashing and storing the device state information, comprises:

[0045] The unified fusion features are calculated through a preset device state classification model to obtain a probability distribution of the device state.

[0046] Based on the probability distribution, a state category corresponding to a preset probability is selected to obtain device state information.

[0047] After obtaining the device state information, the following steps are further included:

[0048] A blockchain network is obtained.

[0049] According to the device state information, a hash digest is obtained through IPFS distributed storage and calculation.

[0050] According to the hash digest and the blockchain network, a node identity public key is obtained.

[0051] According to the node identity public key, a node identity is obtained.

[0052] The node identity and the hash digest are aggregated and stored in the blockchain network in combination with a timestamp to complete the storage of the device state information.

[0053] In an embodiment, the step of receiving the geological risk information and performing corresponding early warning operations according to the geological risk information comprises:

[0054] The geological risk information is matched with a preset geological risk level division rule to determine a current geological risk level and geographical location information of a risk occurrence;

[0055] An early warning strategy library is called according to the geological risk level and the geographical location information to generate corresponding early warning operation instructions, wherein the early warning operation instructions include early warning time, response measures, execution personnel, and processing results;

[0056] The early warning operation instructions, the geological risk level, and the geographical location information are sent to an early warning information publishing platform to realize safety management and control.

[0057] In addition, to achieve the above-mentioned purpose, the application further provides a safety monitoring device in a satellite signal denial environment, which comprises:

[0058] An acquisition module is configured to acquire multi-modal data and device text data of a target device, wherein the multi-modal data includes image data, video data, and audio data, and the device text data includes device table data and device text description data;

[0059] A processing module is configured to perform spatio-temporal synchronization processing on the multi-modal data to generate spatio-temporal consistent data;

[0060] An encryption module is configured to perform end-side encryption processing on the spatio-temporal consistent data to obtain encrypted multi-modal data, wherein the encrypted multi-modal data is uploaded to a data lake for storage through a secure transmission protocol;

[0061] An analysis module is configured to analyze the device text data through multi-scale morphological enhancement and hierarchical structure perception coding to obtain structured data and generate corresponding triples, wherein the triples correspond to device entities and attributes;

[0062] A construction module is configured to perform semantic association based on the structured data and the encrypted multi-modal data to construct a target scene multi-modal knowledge graph;

[0063] A fusion module is configured to perform feature fusion on features in the target scene multi-modal knowledge graph by using a self-supervised learning strategy to generate unified fusion features;

[0064] A result module is configured to process the unified fusion feature through a preset device state classification model to obtain device state information, and the device state information is stored in a chain after being encrypted by hashing.

[0065] A comparison module is configured to compare a state parameter value of the device state information with a device state threshold value to obtain a comparison result, the state parameter value being an operating environment temperature, an operating environment pollutant concentration, and an operating time of the device.

[0066] An instruction generation module is configured to generate an alarm triggering instruction and send the alarm triggering instruction to a target device to make the target device feed back geological risk information when the comparison result is that the state parameter value exceeds the preset state threshold value.

[0067] An execution module is configured to receive the geological risk information and perform a corresponding early warning operation according to the geological risk information.

[0068] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the safety monitoring method in a satellite signal denial environment as described above.

[0069] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the safety monitoring method in a satellite signal denial environment as described above.

[0070] The present application collects multi-modal data and device text data, and performs spatio-temporal synchronization, encryption processing and structured analysis on the data, and constructs a multi-modal knowledge graph of a target scene. Self-supervised learning is used for feature fusion to generate unified features for device state classification. The state information is stored in a chain after being encrypted by hashing, realizing safe data management and real-time monitoring. When the device state information is detected to be abnormal, different geological risk information is obtained through analysis to perform early warning, realizing geological risk early warning and safety monitoring of construction personnel through real-time monitoring of device state in a satellite signal denial environment. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0072] Figure 1 It is a flowchart of a first embodiment of the safety monitoring method in a satellite signal denial environment of the present application.

[0073] Figure 2 A flowchart of a second embodiment of the safety monitoring method in a satellite signal denial environment of the present application;

[0074] Figure 3 A flowchart of a third embodiment of the safety monitoring method in a satellite signal denial environment of the present application;

[0075] Figure 4 A flowchart of a fourth embodiment of the safety monitoring method in a satellite signal denial environment of the present application;

[0076] Figure 5 A module structure diagram of a safety monitoring device in a satellite signal denial environment of the first embodiment of the safety monitoring method in a satellite signal denial environment of the present application;

[0077] Figure 6 A device structure diagram of a hardware operating environment involved in the safety monitoring method in a satellite signal denial environment of the embodiments of the present application.

[0078] The purpose implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0079] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0080] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0081] In a satellite signal denial environment (including mine tunnel, underground chamber, hydropower plant and underground pipe gallery and other scenes), satellite signals are easily affected by terrain obstruction, building shielding or electromagnetic interference, resulting in signal attenuation, loss of lock or even complete interruption. The traditional satellite-dependent monitoring scheme faces significant technical bottlenecks. In the satellite-restricted environment, there is a lack of satellite time and positioning reference such as Beidou, and the multi-device collected multi-modal data is prone to time deviation and spatial coordinate confusion, resulting in invalid data correlation, which cannot accurately restore the device operating state and environmental scene. At the same time, there are structured data such as device table and text description, as well as unstructured multi-modal data such as image and video in the monitoring scene, and the existing technology is difficult to realize the deep semantic correlation of cross-modal data, which is easy to form a data island and cannot provide comprehensive data support for device state evaluation.

[0082] Therefore, the application provides a safety monitoring method in a satellite signal denial environment to solve the above problems. The main solution of the embodiment of the application is: obtaining multi-modal data and device text data of a target device, the multi-modal data including image data, video data and audio data, and the device text data including device table data and device text description data; performing spatio-temporal synchronization processing on the multi-modal data to generate spatio-temporal consistent data; performing end-side encryption processing on the spatio-temporal consistent data to obtain encrypted multi-modal data, wherein the encrypted multi-modal data is uploaded to a data lake for storage through a secure transmission protocol; analyzing the device text data through multi-scale morphological enhancement and hierarchical structure perception coding to obtain structured data and generate corresponding triples; performing semantic association based on the structured data and the encrypted multi-modal data to construct a target scene multi-modal knowledge graph; using a self-supervised learning strategy to perform feature fusion on the features in the target scene multi-modal knowledge graph to generate unified fusion features; processing the unified fusion features through a preset device state classification model to obtain device state information; comparing the state parameter value of the device state information with a device state threshold value to obtain a comparison result; when the comparison result is that the state parameter value exceeds the preset state threshold value, generating an alarm triggering instruction and sending it to the target device to make the target device feedback geological risk information; receiving the geological risk information and performing a corresponding early warning operation according to the geological risk information.

[0083] Based on the above, the embodiment of the application also provides a safety monitoring method in a satellite signal denial environment. Referring to Figure 1 , Figure 1 is a flowchart of the first embodiment of the safety monitoring method in a satellite signal denial environment of the application. In this embodiment, the safety monitoring method in a satellite signal denial environment includes steps S10-S100:

[0084] Step S10, obtaining multi-modal data and device text data of a target device.

[0085] It should be noted that the multi-modal data includes image data, video data and audio data, the image data is collected in real time by high-definition cameras and infrared thermal imagers deployed in key areas of the target scene, covering information such as device appearance, running state, temperature distribution, etc., the target scene is a satellite signal denial environment (including mine tunnel, underground chamber, hydropower plant and underground pipe gallery, etc.), and the target device refers to key mechanical and electrical equipment, infrastructure or mobile platform deployed in or running in the satellite signal denial environment, whose running state needs to be monitored and intelligently managed in real time under the condition of no global navigation satellite system (GNSS) signal; the video data is recorded by fixed and mobile camera equipment, recording the device running process, personnel operation behavior and environmental changes; the audio data is collected by high-sensitivity microphones, capturing sound signals such as abnormal sound, vibration sound, etc., for anomaly detection. The device text data includes device table data and device text description data, the device table data covers structured information such as device parameters, maintenance records, fault codes, running logs, etc., and the device text description data includes non-structured documents such as device operation manual, technical specification, fault analysis report, etc. The above data is collected in real time by industrial Internet of Things sensors, monitoring equipment and information systems, forming a comprehensive target device running data system, providing a rich data basis for subsequent data processing, analysis and decision-making.

[0086] In step S20, the multi-modal data is processed in time and space synchronization to generate time and space consistent data.

[0087] It should be noted that, due to the wide distribution of target devices and the large number of data collection points, there may be time deviation and spatial position difference in data collection by different sensors, cameras and microphones, therefore, high-precision time and space synchronization technology is needed to unify and calibrate the multi-modal data.

[0088] Further, step S20 further includes obtaining spatial coordinate information and reference time information of the device, wherein the spatial coordinate information is obtained by UWB positioning, and the reference time information is generated by a PTP master clock server in a mine area network. Specifically, the UWB positioning technology has the advantages of high positioning accuracy, strong anti-interference ability and good penetration, etc., and can achieve centimeter-level or even millimeter-level positioning accuracy in complex target scene environment. By deploying UWB positioning base stations in key areas of the target scene, and providing UWB positioning tags for mobile devices and sensor terminals, the system can obtain accurate coordinate information of the device in three-dimensional space in real time, including longitude, latitude and height. At the same time, the reference time information is generated by a precision time protocol (PTP) master clock server in the mine area network. The PTP master clock server serves as the time reference source of the entire mine area, obtains standard time signals through a high-precision atomic clock or a GPS time service module, and distributes time synchronization signals to each sensor terminal and edge computing node through the network, ensuring that the entire system has a unified time reference.

[0089] Then, a space-time mapping relationship table is established according to the space coordinate information and the reference time information. Specifically, the space-time mapping relationship table is the core basis for data synchronization correction. The space-time mapping relationship table includes a time deviation compensation parameter and a space coordinate conversion matrix. The time deviation compensation parameter is used to correct the time deviation between different sensors due to factors such as clock drift and network delay, ensuring the consistency of all data in time. The space coordinate conversion matrix is used to convert the data collected by different sensors from the local coordinate system to the unified mine global coordinate system, eliminating the spatial position deviation and realizing the unification of data in the spatial dimension.

[0090] Finally, based on the space-time mapping relationship table, the multi-modal data is space-time corrected to obtain space-time consistent data. Specifically, first, the time stamp in the multi-modal data is corrected according to the time deviation compensation parameter, eliminating the time deviation and ensuring that all data is aligned under the unified time reference. Then, the spatial coordinates in the data are converted using the space coordinate conversion matrix, converting the coordinate information in the local coordinate system to the unified coordinate in the mine global coordinate system. Through the space-time correction process, space-time consistent data is generated, ensuring the consistency of all multi-modal data in time and space dimensions. In order to facilitate subsequent data management and analysis, the space-time consistent data adds a unified space-time label, which includes a GPS time stamp and a three-dimensional spatial coordinate. The GPS time stamp provides a global unified time reference, ensuring the time consistency of data when applied across systems and regions. The three-dimensional spatial coordinate provides accurate spatial position information, supporting subsequent spatial analysis, location association and visualization display. By adding the unified space-time label, the system realizes the space-time unified identification of multi-modal data, providing a high-quality data basis for subsequent data fusion, knowledge graph construction, intelligent analysis and decision support.

[0091] In step S30, the space-time consistent data is processed by end-side encryption to obtain encrypted multi-modal data.

[0092] It should be noted that due to the wide distribution of target devices and the complex data transmission path, there is a risk of being illegally intercepted, tampered with or leaked, so an end-side encryption mechanism is needed to perform local encryption processing on the multi-modal data at the data acquisition terminal or edge computing node, ensuring that the data is highly secure before transmission. End-side encryption adopts a hybrid encryption strategy, combining the advantages of symmetric encryption algorithms (such as AES-256) and asymmetric encryption algorithms (such as RSA) to achieve efficient and secure data protection.

[0093] Specifically, the multi-modal data is quickly encrypted using AES, and the encryption key is encapsulated using RSA, thereby balancing encryption efficiency and security of key management. In addition, lightweight compression encryption algorithms can also be used according to the format characteristics of different modal data to reduce bandwidth occupation and improve transmission efficiency. The encrypted multi-modal data is uploaded to the data lake for storage through the secure transmission protocol. The encrypted multi-modal data is uploaded to the data lake of the mine data center for unified storage through a secure transmission protocol (such as HTTPS, MQTT over TLS, SFTP, etc.). The data lake uses a distributed storage architecture to support efficient storage and management of multi-modal data, and the stored data is statically encrypted.

[0094] Through end-side encryption and secure transmission, the security of the data in the whole process of collection, transmission and storage is realized, and the confidentiality, integrity and availability of the multi-modal data in the target scene are ensured, providing a safe and reliable data basis for subsequent data analysis and intelligent decision-making.

[0095] In step S40, the device text data is parsed through multi-scale morphological enhancement and hierarchical structure perception coding to obtain structured data and generate corresponding triples.

[0096] It should be noted that the text data includes device standard specifications, operation manuals, fault logs, and maintenance records, etc., and usually exists in the form of unstructured or semi-structured, such as complex tables and free text descriptions in PDF documents. In order to extract useful knowledge from these rich information sources and convert them into machine understandable form, the embodiment adopts a method combining multi-scale morphological enhancement and hierarchical structure perception coding for parsing.

[0097] Specifically, for the equipment table data, a multi-scale morphological enhancement technique is adopted for preprocessing. Through morphological operations such as opening operation, closing operation, top-hat transformation, and bottom-hat transformation at different scales, the contrast and clarity of the table border and text area are enhanced, effectively improving the readability of low-quality scanned images. Subsequently, a hierarchical structure perception coding technique is used to analyze the layout of the enhanced image, identify the row and column structure, cell boundaries, and their logical relationships of the table, and accurately extract the text content and structural information in the table. Next, for the pure text description data, named entity recognition (NER) and relationship extraction (RE) techniques are used to obtain structured data, automatically identify key entities (such as "hydraulic pump", "bearing", "motor") and their attribute relationships (such as "temperature", "pressure", "state") from the structured data content, and generate standardized knowledge triples (such as <hydraulic pump, temperature, 85°C>), which correspond to equipment entities and attributes. Through the above processing, the system successfully converts unstructured equipment text data into structured data form, providing a high-quality semantic basis for subsequent knowledge graph construction and multi-modal data fusion.

[0098] Step S50, based on the structured data and encrypted multi-modal data, semantic association is performed to construct a multi-modal knowledge graph of the target scene.

[0099] It should be noted that this process first relies on the structured information extracted from the equipment text data, including equipment entities, attributes, relationships, and their corresponding triple data, such as {device name, attribute, attribute value}. These structured data constitute the basic skeleton of the knowledge graph, providing clear semantic units and logical structures for subsequent semantic association. On this basis, the encrypted multi-modal data is decrypted to obtain the original image, video, audio, and sensor data. Through multi-modal feature extraction techniques, feature vectors with semantic expression capabilities are extracted from these data. For example, visual features are extracted from image data through convolutional neural networks, spatio-temporal features are extracted from video data through three-dimensional convolutional networks, acoustic features are extracted from audio data through spectral analysis or deep models, and time series features are extracted from sensor data through trend analysis and anomaly detection.

[0100] Next, based on predefined semantic association rules, the multi-modal features are matched and linked with entities and attributes in structured data. Specifically, through methods such as semantic similarity calculation, entity alignment, and attribute mapping, the corresponding relationship between multi-modal data and device entities in the knowledge graph is established. For example, an image showing "hydraulic pump oil leakage" will be associated with the "hydraulic pump" entity and assigned the "fault status" attribute. To further enhance the expression and reasoning capabilities of the knowledge graph, domain ontology and rule engines are also introduced to perform semantic enhancement and logical verification on entity relationships, ensuring the accuracy and consistency of the graph. At the same time, the system supports multi-modal association at the triple level, i.e., binding triples representing complex states or events (such as <bearing, status, overheating>) with corresponding multi-modal data (such as infrared images, abnormal audio), achieving event-level semantic modeling.

[0101] Finally, a target scene multi-modal knowledge graph that integrates structured semantics and multi-modal features is constructed, supporting device state monitoring, fault diagnosis, predictive maintenance, and other intelligent applications.

[0102] Step S60, using a self-supervised learning strategy to fuse the features in the target scene multi-modal knowledge graph, generating unified fusion features.

[0103] It should be noted that the self-supervised learning strategy, as a method that can learn the internal structure of data without the need for a large amount of manually annotated data, has obvious advantages in handling massive and diverse data in the target scene environment.

[0104] For data of different modalities, appropriate pre-training models are used to extract initial feature representations. For example, SlowFast network can be used for video data, VGGish model for audio data, and BERT or RoBERTa model for text description to extract semantic features. These models can capture important features in their respective modalities, laying the foundation for subsequent cross-modal feature fusion.

[0105] Next, a heterogeneous graph neural network (HGNN) is constructed to explicitly model the cross-modal association within instances and the intra-modal similarity relationship between instances. Each node represents a specific modality of a specific instance, and the edge represents the similarity between instances or the intra-modal connection. Through the hierarchical graph attention mechanism (GAT), the network can aggregate intra-modal and inter-modal information while dynamically balancing the information contribution of the two through an adaptive mechanism. This design ensures that even in cases where some modal information is abundant or less noisy, it will not disproportionately dominate the final representation results.

[0106] Further, multiple contrast learning objectives and regularization terms are applied to optimize the learned representation, ensuring its discriminativeness, consistency, and information completeness. For example, cross-modal contrast learning maximizes the consistency between different modal representations of the same instance; intra-modal contrast learning encourages closer representations for semantically similar instances within the same modality; distribution alignment regularization ensures that the fused representation retains the main information of each single modality; and redundancy suppression reduces the linear correlation between dimensions of the final fused representation. In this way, a unified fused feature is generated by considering multiple optimization objectives comprehensively. In this way, the system not only improves the depth of understanding of the device operating state, but also enhances the accuracy and timeliness of fault detection, thereby improving the safety and efficiency of the target scenario operation.

[0107] At step S70, the unified fused feature is processed through a preset device state classification model to obtain device state information.

[0108] It should be noted that the above-mentioned preset device state classification model is a deep learning model (for example, a convolutional neural network CNN, a recurrent neural network RNN, or a Transformer architecture).

[0109] Further, step S70 includes: calculating the unified fused feature through the preset device state classification model to obtain a probability distribution of the device state. Specifically, the unified fused feature contains comprehensive information extracted from multi-modal data (such as images, videos, audio, and sensor data) and fused through a self-supervised learning strategy. This feature is input into the preset device state classification model, which outputs a probability value corresponding to each possible device state, indicating the likelihood of the current device being in each state. For example, for a given time point, the model outputs a probability of 0.85 for the "normal" state, a probability of 0.10 for the "minor fault" state, and a probability of 0.05 for the "serious fault" state. Then, based on the probability distribution, the state category corresponding to the preset probability is selected to obtain the device state information. In this embodiment, the preset probability is the highest probability value, and the state category corresponding to the highest probability is selected as the final device state information.

[0110] Further, step S70 is followed by: obtaining a blockchain network, calculating a hash digest based on the device status information through IPFS distributed storage, obtaining a node identity public key based on the hash digest combined with the blockchain network, calculating the node identity based on the node identity public key, and aggregating and storing the timestamp based on the node identity and the hash digest into the blockchain network to complete the device status information storage. Specifically, the device status information is converted into a unique hash digest by using a secure hash algorithm (such as SHA-256). The hash digest, as a digital fingerprint of data, has uniqueness and irreversibility, which can effectively prevent data tampering during transmission and storage. This process ensures the integrity and consistency of data from collection to storage. Secondly, the node identity public key is obtained based on the hash digest combined with the blockchain network. Specifically, each node generates a pair of keys, including a private key and a public key, through an asymmetric encryption algorithm (such as RSA or elliptic curve encryption algorithm) when joining the blockchain network. The private key is securely stored by the node for signature and decryption operations; the public key is publicly disclosed in the blockchain network for identity verification and data encryption. Through the combination of hash digest and blockchain network, a unique and trusted digital identity public key can be generated for each node. Then, the node identity is calculated based on the node identity public key, specifically, a unique node identity is generated based on the public key of the node through a hash function (such as SHA-256). The hash function converts the public key into a fixed-length hash value, which serves as the node identity, ensuring the uniqueness and tamper-proofing of the node identity. During the storage of device status information, it is crucial to ensure the secure access of data and the correct execution of contract terms. Therefore, the node identity is assigned access rights, thereby controlling which nodes have the right to operate specific data or execute certain contract terms. Finally, the timestamp is aggregated and stored into the blockchain network based on the node identity and the hash digest to complete the device status information storage. By utilizing the tamper-proofing and distributed characteristics of blockchain technology, a highly trusted and transparent storage environment can be provided for device status information.

[0111] Step S80: comparing the state parameter value of the device status information with the device status threshold to obtain a comparison result.

[0112] It should be noted that the state parameter value is the operating environment temperature, operating environment pollutant concentration (such as gas concentration, dust concentration) and operating time of the device. These parameter values are important indicators reflecting the running condition of the device and the safety of the environment. At the same time, the device state threshold is set, which is determined according to the running specification, safety standard and historical running data of the device. For example, the threshold of the device operating environment temperature may be set to not more than a certain specific temperature value to prevent the device from overheating and damaging; the threshold of the operating environment pollutant concentration is used to ensure the safety of the target scene working environment, to avoid accidents caused by high concentration of harmful gas or dust; the running time threshold is used to monitor the cumulative running time of the device, so as to arrange maintenance in time. The real-time acquired state parameter value is compared with the preset device state threshold one by one. If the state parameter value exceeds or is lower than the corresponding threshold, a comparison result will be generated, indicating that the parameter is in an abnormal state. For example, if the device operating environment temperature exceeds the preset maximum temperature threshold, the system will be marked as "temperature abnormality"; if the gas concentration exceeds the safety threshold, it will be marked as "gas concentration exceeds the standard"; if the device running time exceeds the specified maintenance period, it will be marked as "overrun".

[0113] Through this real-time parameter value and threshold comparison mechanism, potential risks in device operation can be quickly identified, providing accurate basis for subsequent alarm triggering, fault diagnosis and maintenance decision.

[0114] Step S90, when the comparison result is that the state parameter value exceeds the preset state threshold, an alarm triggering instruction is generated and sent to the target device to make the target device feedback geological risk information.

[0115] It should be noted that when the comparison result is that the state parameter value exceeds the preset state threshold, an alarm triggering instruction is generated and sent to the target device, so that the target device starts the local sensing and communication module and actively feedbacks the geological risk information related to its operating environment. The geological risk information includes the rock mass displacement rate, surrounding rock stress change, groundwater leakage condition, gas concentration anomaly (such as methane, carbon monoxide), structure vibration frequency spectrum characteristics and acoustic emission signal of the area where the device is located, and these data are collected by the special sensors carried by the device or deployed around.

[0116] Step S100, receiving the geological risk information and performing corresponding warning operation according to the geological risk information.

[0117] It should be noted that after the system receives the geological risk information, the pre-stored geological environment ontology and historical risk cases in the multi-modal knowledge graph are combined for semantic matching and risk level evaluation to identify potential collapse, water inrush, rock burst or foundation settlement and other disaster signs. Further, through the self-supervised fusion model, the correlation between multi-modal data is analyzed to improve the accuracy and robustness of risk discrimination. After confirming the existence of major risks, the system automatically triggers a hierarchical warning mechanism, pushing a structured warning report containing location, risk type, severity and recommended measures to the operation and maintenance personnel, and linking to on-site sound and light alarms, ventilation control or personnel evacuation guidance. In addition, the system will also send personalized safety warnings to construction personnel in the risk area through wireless communication devices such as smart watches or safety helmets. These warning information includes evacuation direction, safety assembly point location and necessary protective measures to ensure that every worker can receive accurate escape guidance in time. At the same time, the on-site management personnel will track the location and status of each construction personnel in real time through the command interface of the monitoring center to facilitate effective coordination and support in emergency situations.

[0118] Therefore, the above steps realize the closed-loop response from device state anomaly to geological disaster prediction and safety monitoring of construction personnel, improving the safety and emergency response capability in complex denial environments.

[0119] The embodiment collects multi-modal data and device text data, and performs spatio-temporal synchronization, encryption processing and structured analysis to construct a multi-modal knowledge graph of the target scene. Self-supervised learning is used for feature fusion to generate unified features for device state classification. The state information is hashed and stored on the chain to realize safe data management and real-time monitoring. When device state information anomaly is detected, different geological risk information is obtained for warning, realizing geological risk warning and safety monitoring of construction personnel through real-time monitoring of device state in satellite signal denial environment.

[0120] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 2 , the safety monitoring method in the satellite signal denial environment step S40 further includes steps S201-S209:

[0121] Step S201, converting the device text data into image information.

[0122] It should be noted that this process usually involves using optical character recognition (OCR) technology to capture the device table data and device text description data of the device text data and convert them into digital image format for subsequent processing. For example, text materials such as operation manuals, maintenance records, and fault logs are often in PDF or other non-editable formats, making it difficult to extract information directly. By scanning or digitizing these documents, high-resolution image files can be generated.

[0123] Step S202, the image information is grayed to obtain a gray image.

[0124] It should be noted that the gray processing is an important step in image preprocessing, which reduces the computational complexity and simplifies the subsequent image analysis task by converting the color image into a single-channel gray image. Each pixel point of the gray image only contains brightness information, usually represented as an integer value between 0 and 255, where 0 represents black and 255 represents white, and intermediate values represent different levels of gray. Grayscale processing can be achieved through various methods, the most common being the weighted average method based on the RGB color model. Graying not only helps reduce the demand for computing resources, but also facilitates the application of various image processing techniques such as edge detection, morphological operations, etc. For example, when processing the operation manual or maintenance record of the target device, the gray image can more clearly highlight the table lines and text content, thereby improving the accuracy of character recognition.

[0125] Step S203, a multi-scale morphological operation is used to enhance the gray image to obtain an enhanced image.

[0126] It should be noted that the multi-scale morphological operation includes opening operation, closing operation, top-hat transformation and bottom-hat transformation, which enhances the gray image (wherein is the height, is the width, and 3 is the number of RGB channels), the enhanced image can be calculated by the following formula:

[0127]

[0128]

[0129]

[0130] wherein, is a hyperparameter that controls the degree of preservation of the gray image, represents the th morphological operation, is the total number of morphological operations used, is the adaptive weight of the th morphological operation, The score function is used to evaluate the first kind of morphological operation The table structure enhancement effect of the gray image comprehensively considers the edge sharpness , contrast and network structure , is the corresponding weight coefficient of the edge sharpness, contrast and network structure.

[0131] In step S204, feature extraction is performed on the enhanced image to generate basic visual features.

[0132] It should be noted that a convolutional neural network (CNN) or Transformer architecture is used as the backbone network to extract the basic visual features of the image. For example, using models such as ResNet or Swin Transformer can effectively capture multi-level feature representations in the image. These models learn feature mappings from low-level edge detection to high-level semantic information through a series of convolutional layers, batch normalization and nonlinear activation functions (such as ReLU). The specific formula is:

[0133]

[0134] wherein, is the basic visual feature, is the long and wide of the down-sampling, is the number of feature channels, represents the backbone network.

[0135] In step S205, the basic visual features are projected into the query, key and value spaces in the row direction and column direction respectively to calculate the row attention matrix and the column attention matrix.

[0136] It should be noted that the basic visual features are mapped to the query, key and value spaces in the row direction and column direction through linear transformation. Specifically, for the position of each pixel point, we use a learnable projection matrix to convert it into three different vectors: query vector (Q), key vector (K) and value vector (V). These vectors are used to calculate attention weights to determine the correlation between different positions. The specific projection is:

[0137]

[0138]

[0139] wherein, is the learnable projection matrix. In the row direction, the query vector and the key vector are compressed in the column dimension through average pooling along the column dimension, while the value vector The row attention matrix is computed by keeping the full dimension of the value vector

[0140]

[0141]

[0142]

[0143] where represents the row-to-row relevance, is the down-sampled length, represents the matrix has a spatial dimension of , is the dimension of the key vector, which is used as a scaling factor, is the query vector after compressing the column dimension, is the key vector after compressing the column dimension, is the transpose key vector after compressing the column dimension, represents the average pooling along the column direction.

[0144] Similarly, in the column direction, the query vector and the key vector are compressed in the row dimension by the average pooling operation along the column dimension, while the value vector keeps the full dimension. This way, the column-to-column relevance score, i.e., the column attention matrix, is computed as follows:

[0145]

[0146]

[0147]

[0148] where represents the column-to-column relevance, is the down-sampled width, represents the matrix has a spatial dimension of , is the dimension of the key vector, which is used as a scaling factor, is the query vector after compressing the row dimension, is the key vector after compressing the row dimension, is the transpose key vector after compressing the row dimension, represents the average pooling along the row direction.

[0149] Step S206: Perform structure-aware encoding based on basic visual features, row attention matrix, and column attention matrix to obtain structure-aware features.

[0150] It should be noted that, in the row direction, the row attention matrix is ​​used. and row value vector An aggregation operation is then performed. Specifically, by multiplying the row attention matrix by the value vector, a structure-aware row feature representation can be obtained:

[0151]

[0152] In the column direction, using the column attention matrix and column value vector Aggregation operations are performed. Specifically, by multiplying the column attention matrix by the value vector, a structure-aware column feature representation can be obtained:

[0153]

[0154] To further enhance the model's understanding of table structure, it is necessary to incorporate the structure-aware features generated above ( and and basic visual features To merge them, this process can be achieved through a simple splicing operation:

[0155]

[0156] The importance of different features can also be adjusted through a weighted fusion strategy:

[0157]

[0158] in, These are learnable weight coefficients. Finally, after further processing by a feedforward neural network (FFN), the final structure-aware features are generated, with the specific formula as follows:

[0159]

[0160] This structure-aware feature not only includes rich local details but also possesses a deep understanding of the overall table structure. For example, when parsing complex tables, it can accurately identify the location and content of each cell and understand the logical relationships between different cells.

[0161] Step S207: Generate the cell coordinates and content information of the table based on structure-aware features.

[0162] It should be noted that, based on structure-aware features The accurate boundary and center point coordinates of each cell need to be predicted, which is usually modeled in an autoregressive manner, first predicting the center point coordinates of each cell, and once the center point of the cell is determined, the next task is to predict its accurate boundary. In order to adapt to cells of various shapes, an n-point polygon (such as a quadrilateral) can be used to represent the cell boundary. Specifically, the model will predict the offset from the center point to the polygon vertex, and combine RoIAlign or similar methods to extract the features of the cell region from the fused features using the predicted contour.

[0163] Step S208, converting the cell coordinates and content information into structured data.

[0164] It should be noted that the logical structure model of the table is constructed according to the cell coordinates, and the row and column positions of each cell in the table and its belonging relationship are determined. Then, the text content in the cell is semantically parsed to identify entities, attributes, and numerical information. Through rule matching and natural language processing technology, the system can map the cell content to standardized data fields such as device name, parameter type, and parameter value. Finally, the structured data is output in JSON or CSV format to ensure readability and scalability. This structured data not only facilitates storage and query, but also can be directly used for knowledge graph construction, data mining, and intelligent analysis.

[0165] Step S209, generating corresponding triples according to the structured data.

[0166] It should be noted that the triple structure is {device entity, attribute name, attribute value}, specifically, the system first identifies the correspondence between the header row and the content row in the table, such as "fault code", "fault description", "suggested measures" as attribute names, and the specific content of each row as the value of the attribute. Then, combined with the context to determine the belonging device entity, such as "hydraulic pump", "ventilator" or "conveyor motor", etc., and combine them with the corresponding attribute name and attribute value into complete triples. For example, when parsing the "fault code table" in the document, if a row shows that "fault code A01" corresponds to "insufficient pressure", and gives the suggestion of "check the oil circuit", the following triples will be generated: {hydraulic pump, fault code, A01}, {hydraulic pump, fault description, insufficient pressure}, {hydraulic pump, suggested measures, check the oil circuit}. These triples not only clearly express the semantic relationship between the device entity and its various attributes, but also can be uniformly included in the target scene multi-modal knowledge graph, realizing cross-modal, cross-document information association and reasoning.

[0167] The embodiment images device text data, combines gray processing, multi-scale morphological enhancement and attention mechanism for feature extraction and structure perception coding, finally generates structured data and knowledge triples, improves the analysis accuracy and automation level of unstructured information in the target scene document, and realizes efficient extraction and semantic modeling of device knowledge.

[0168] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 3 , the safety monitoring step S50 in the satellite signal denial environment further includes steps S301-S304:

[0169] Step S301, entity extraction based on structured data, to obtain basic triple data corresponding to the entity.

[0170] It should be noted that this process mainly identifies specific equipment component information such as "hydraulic pump", "conveyor belt", "crusher bearing" and other key component parts from equipment technical documents, maintenance records and fault report texts. By introducing the method of combining named entity recognition (NER) technology and domain dictionary matching, various equipment component entities can be accurately identified and extracted, and they can be bound with related attributes or relationships to form basic triple data.

[0171] Step S302, constructing an initial knowledge graph according to the basic triple data.

[0172] It should be noted that the knowledge graph takes equipment entities as core nodes, and organizes key information such as equipment components, state parameters and fault types through attribute relationships and interaction relationships to form a semantic knowledge network.

[0173] Step S303, according to the encrypted multi-modal data, the corresponding entity is obtained and a multi-modal data index table is constructed.

[0174] It should be noted that after decrypting and parsing the encrypted multi-modal data, through target detection, speech recognition, natural language processing and feature matching, etc. The entity information related to the target device is identified, such as device name, component number, state parameter, fault type, etc. Then, these entities are associated with the corresponding data sources (such as camera number, sensor ID, document path, etc.), and a structured multi-modal data index table is established. The modal data index table includes data modal type, collection time and spatial coordinates, and its fields usually include: entity identifier, data type, storage location, timestamp, spatial coordinates, data summary, etc. The data modal type indicates the data source form, such as image, video, audio, text, sensor value, etc.; the collection time records the specific timestamp of data generation or collection, which is used for time series analysis and historical backtracking; the spatial coordinates represent the physical location information of data collection, such as GPS coordinates, lane number, device number, etc., which are used for spatial positioning and regional analysis, while the entity identifier labels the core device entity or attribute involved in the data. The storage path / hash summary indicates the storage location or unique identifier of the data in the data lake, which facilitates quick retrieval and verification. For example, when parsing an encrypted text containing the description "abnormal temperature of hydraulic pump", the system can extract the entities "hydraulic pump", "temperature" and "abnormal", and link them to the corresponding sensor data, alarm records and related chapters in the maintenance manual.

[0175] Step S304, based on the multi-modal data index table and the initial knowledge graph, semantic association is performed to construct a multi-modal knowledge graph.

[0176] It should be noted that first, the entity identifier in the multi-modal data index table is used to map data from different modalities (such as device images, vibration signals, fault logs, etc.) to corresponding nodes in the initial knowledge graph. For example, a video about "abnormal vibration of the crusher" can be connected to the "crusher" entity in the graph, and its numerical data collected by the "vibration sensor" can be timestamped and semantically fused. On this basis, further introduce semantic association rules and context reasoning mechanism, to mine the implicit relationships between different modal data. For example, combining image recognition results with temperature sensor data, the potential causal relationship of "high temperature leading to bearing wear" is inferred, and the corresponding edge or attribute is added to the knowledge graph. In addition, by introducing deep learning methods such as graph neural network (GNN), the multi-modal knowledge graph can be dynamically updated and optimized, continuously improving its semantic expression ability and reasoning accuracy. Finally, the target scene multi-modal knowledge graph is constructed, providing strong knowledge support for device state prediction, fault diagnosis, intelligent retrieval and other applications, and promoting the evolution of the target scene system to a higher level of intelligence.

[0177] The embodiment constructs device base triples through structured data entity extraction, forms an initial knowledge graph, realizes cross-modal semantic association by combining encrypted multi-modal data analysis and index construction, and finally establishes a multi-modal knowledge graph. The method improves the organization and semantic expression ability of target device knowledge, enhances the intelligent analysis and decision-making level under data driving, and provides an efficient and unified knowledge support system for intelligent operation and maintenance of the target scene.

[0178] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 4 , the safety monitoring method under the satellite signal denial environment comprises steps S401-S406:

[0179] Step S401, extracting multi-modal corresponding node feature vectors based on the multi-modal knowledge graph of the target scene.

[0180] It should be noted that the multi-modal data nodes associated with the device entity are identified from the knowledge graph, including image, video, audio, text and sensor data, etc. Each modality of data is processed by a corresponding feature extraction model. The image data adopts a convolutional neural network (such as ResNet) to extract visual features, the video data extracts spatio-temporal features through a three-dimensional convolutional network (such as C3D), the audio data extracts voiceprint features using an acoustic model (such as VGGish), the text data extracts semantic features through a pre-trained language model (such as BERT), and the sensor data extracts trend features through a time series modeling method. The extracted feature vectors are stored as the attributes of the corresponding nodes in the knowledge graph, forming a multi-modal node feature set. Subsequently, the node features are aggregated and updated using a graph neural network (GNN), and the structure information in the graph is combined to further enhance the expression ability of the features. Finally, the node feature vectors that fuse the structure semantics and multi-modal content are output.

[0181] Step S402, constructing a heterogeneous graph neural network according to the node feature vectors of different modalities.

[0182] It should be noted that in order to more effectively model the complex relationship between these heterogeneous information, the system introduces a heterogeneous graph neural network (Heterogeneous Graph Neural Network, HGNN), which takes the node feature vectors of different modalities as input, and constructs a graph structure model with semantic differentiation capability. The heterogeneous graph neural network establishes a multi-level, multi-relationship graph structure by defining the relationship edges between different types of nodes (such as "device-contains-component", "component-monitors-state parameter", "state parameter-associated-failure type", etc.). Each relationship corresponds to a specific message passing mechanism, enabling the model to propagate and aggregate information along different types of relationship paths in the graph. For example, when processing the state prediction task of "hydraulic pump", the model can simultaneously consider multiple modal features such as associated vibration images, temperature sensor data, and maintenance record texts, and jointly model through the heterogeneous graph structure.

[0183] Specifically, first, the feature vector corresponding to each modality is taken as a node in the heterogeneous graph, and the node types include image nodes, video nodes, audio nodes, text nodes, and sensor nodes, each node having different feature dimensions and semantic attributes. To depict the semantic association between modalities, the system constructs two types of edges between nodes: cross-modal edges and intra-modal edges. Cross-modal edges connect different modality nodes under the same device entity, representing different perspectives of the same object; intra-modal edges are constructed based on node feature similarity, connecting semantically similar nodes, representing intra-modal semantic clustering relationships. Each edge type corresponds to a learnable relationship embedding, which enhances the expression ability of the graph neural network. The system uses a hierarchical graph attention mechanism to aggregate information in the heterogeneous graph, dynamically allocating attention weights to neighbor nodes, and achieving adaptive fusion of intra-modal and inter-modal information. Through iterative updating of the multi-layer graph neural network, the node features continuously fuse structural semantics and cross-modal contextual information, and finally output a fused node feature with high expressiveness and discriminability.

[0184] Step S403, using a self-supervised learning strategy to perform intra-modal contrast learning on the node feature vector, to obtain the neighborhood node information within the modality.

[0185] It should be noted that the self-supervised learning strategy is used to perform intra-modal contrast learning on the node feature vector, thereby obtaining a more discriminative node representation. Specifically, intra-modal contrast learning aims to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, to enhance the consistent expression between nodes and their neighborhood nodes. For a specific modality (such as image modality or text modality), we extract a center node and its first-order neighborhood nodes from the graph as positive sample pairs, and other unrelated nodes as negative samples.

[0186] At step S404, the inter-modal contrast learning strategy is used for inter-modal neighbor node information learning of the node feature vector, to obtain inter-modal neighbor node information.

[0187] It should be noted that the inter-modal contrast learning aims to make the node features from different modalities but semantically related as close as possible in the embedding space through the contrast learning mechanism, while the semantically unrelated nodes are pulled apart. For example, in the target device fault analysis scenario, the state parameter "high bearing temperature" (numerical modality) and "infrared thermal image" (image modality) should have high semantic similarity, while the audio modality node "normal voice alarm content" should have low similarity.

[0188] At step S405, the intra-modal neighbor node information and the inter-modal neighbor node information are input into the heterogeneous graph neural network for aggregation, to obtain updated multi-modal node features.

[0189] It should be noted that after completing the intra-modal contrast learning and the inter-modal contrast learning, each node not only obtains the semantic enhanced representation from its same-modal neighborhood (i.e., intra-modal neighborhood information), but also fuses the cross-modal association information with other related modal nodes (i.e., inter-modal neighborhood information). These information respectively reflect the role of the node in the local structure and the semantic consistency in different data modalities. The above two types of neighborhood information are input into the heterogeneous graph neural network, and the system defines different message passing mechanisms according to the node type and its relationship type. For example, the device node can aggregate information from its "image modality neighborhood" (such as the corresponding fault image), "text modality neighborhood" (such as the maintenance record), and "sensor modality neighborhood" (such as the vibration signal), and dynamically adjust the contribution weight of each modality by combining the attention mechanism.

[0190] At step S406, the updated multi-modal node features are weighted and fused and normalized, to obtain unified fusion features.

[0191] It should be noted that the updated multi-modal node features obtained by the heterogeneous graph neural network aggregation have fused the intra-modal and inter-modal neighborhood information, and have strong semantic expression ability and cross-modal consistency. However, due to the large difference in the distribution of different modal data (such as image, text, sensor signal, etc.), the corresponding node features may have significant differences in numerical range, representation space and semantic emphasis. Therefore, in order to further improve the uniformity and discriminability of the node features, the multi-modal node features need to be weighted and fused and normalized to generate unified fusion features.

[0192] The embodiment realizes modeling of neighborhood information within and between modalities by constructing a target scene multi-modal knowledge graph, combining a heterogeneous graph neural network and self-supervised contrast learning, and obtaining a unified representation through feature aggregation and fusion. The method effectively improves the semantic consistency and model expression ability of multi-modal data, and enhances the accuracy and robustness of the target scene system in device state recognition, fault warning and intelligent reasoning.

[0193] Based on the first embodiment of the present application, the present application also provides a safety monitoring device in a satellite signal denial environment, please refer to Figure 5 , the device comprises:

[0194] The acquisition module 10 is configured to acquire multi-modal data and device text data of a target device, wherein the multi-modal data includes image data, video data and audio data, and the device text data includes device table data and device text description data.

[0195] The processing module 20 is configured to perform spatio-temporal synchronization processing on the multi-modal data to generate spatio-temporal consistent data.

[0196] The encryption module 30 is configured to perform end-side encryption processing on the spatio-temporal consistent data to obtain encrypted multi-modal data, wherein the encrypted multi-modal data is uploaded to a data lake for storage through a secure transmission protocol.

[0197] The analysis module 40 is configured to analyze the device text data through multi-scale morphological enhancement and hierarchical structure perception coding to obtain structured data and generate corresponding triples, wherein the triples correspond to device entities and attributes.

[0198] The construction module 50 is configured to perform semantic association based on the structured data and the encrypted multi-modal data to construct a target scene multi-modal knowledge graph.

[0199] The fusion module 60 is configured to perform feature fusion on the features in the target scene multi-modal knowledge graph using a self-supervised learning strategy to generate unified fusion features.

[0200] The result module 70 is configured to process the unified fusion features through a preset device state classification model to obtain device state information, and the device state information is stored through hash encryption and chaining.

[0201] The comparison module 80 is configured to compare the state parameter values of the device state information with device state thresholds to obtain a comparison result, wherein the state parameter values include the operating environment temperature, the operating environment pollutant concentration and the operating time of the device.

[0202] The instruction generation module 90 is configured to generate an alarm triggering instruction when the comparison result is that the state parameter values exceed the preset state thresholds, and send the alarm triggering instruction to the target device to make the target device perform a warning operation.

[0203] The satellite signal denial environment safety monitoring device provided by the application adopts the satellite signal denial environment safety monitoring method in the above embodiment, and can solve the technical problem of how to realize geological risk warning and safety monitoring of construction personnel through real-time monitoring of the device state in a satellite signal denial environment. Compared with the prior art, the satellite signal denial environment safety monitoring device provided by the application has the same beneficial effects as the satellite signal denial environment safety monitoring method provided by the above embodiment, and the other technical features of the satellite signal denial environment safety monitoring device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0204] In an embodiment, the processing module 20 is further configured to obtain spatial coordinate information and reference time information of the device, wherein the spatial coordinate information is obtained through UWB positioning, and the reference time information is generated by a PTP master clock server in a mine network; a space-time mapping relationship table is established according to the spatial coordinate information and the reference time information, the space-time mapping relationship table includes a time deviation compensation parameter and a spatial coordinate conversion matrix; the multi-modal data is space-time corrected based on the space-time mapping relationship table to obtain space-time consistent data, and the space-time consistent data is added with a unified space-time label, and the unified space-time label includes a GPS time stamp and three-dimensional spatial coordinates.

[0205] In an embodiment, the analysis module 40 is further configured to convert the device text data into image information; perform grayscale processing on the image information to obtain a grayscale image; perform enhancement processing on the grayscale image by using a multi-scale morphological operation to obtain an enhanced image, wherein the multi-scale morphological operation includes opening operation, closing operation, top-hat transformation and bottom-hat transformation; perform feature extraction on the enhanced image to generate basic visual features; project the basic visual features into a row direction and a column direction query, key and value space respectively to calculate a row attention matrix and a column attention matrix; perform structure perception coding based on the basic visual features and the row attention matrix and the column attention matrix to obtain structure perception features; generate cell coordinates and content information of a table based on the structure perception features; convert the cell coordinates and the content information into structured data; and generate corresponding triples according to the structured data, and the triple structure is {device entity, attribute name, attribute value}.

[0206] In an embodiment, the construction module 50 is further configured to perform entity extraction based on the structured data to obtain basic triple data of corresponding entities, and the entity is a component of the target device; construct an initial knowledge graph according to the basic triple data; analyze the encrypted multi-modal data to obtain corresponding entities and construct a multi-modal data index table, and the multi-modal data index table includes a data modal type, a collection time and a spatial coordinate; perform semantic association based on the multi-modal data index table and the initial knowledge graph to construct a multi-modal knowledge graph.

[0207] In an embodiment, the fusion module 60 is further configured to extract multi-modal corresponding node feature vectors based on the target scene multi-modal knowledge graph; construct a heterogeneous graph neural network according to the multi-modal corresponding node feature vectors as nodes; perform intra-modal contrast learning on the node feature vectors using a self-supervised learning strategy to obtain intra-modal neighborhood node information; perform inter-modal contrast learning on the node feature vectors using the self-supervised learning strategy to obtain inter-modal neighborhood node information; input the intra-modal neighborhood node information and the inter-modal neighborhood node information into the heterogeneous graph neural network for aggregation to obtain updated multi-modal node features; and perform weighted fusion and normalization processing on the updated multi-modal node features to obtain unified fusion features.

[0208] In an embodiment, the result module 70 is further configured to calculate the unified fusion features through a preset device state classification model to obtain a probability distribution of the device state; select a state category corresponding to a preset probability based on the probability distribution to obtain device state information; after obtaining the device state information, further comprising: acquiring a blockchain network; calculating a hash digest by IPFS distributed storage according to the device state information; obtaining a node identity public key according to the hash digest in combination with the blockchain network; calculating a node identity according to the node identity public key; and aggregating the node identity and the hash digest in combination with a timestamp and storing them in the blockchain network to complete device state information storage.

[0209] In an embodiment, the execution module 100 is further configured to match the geological risk information with a preset geological risk level division rule to determine a current geological risk level and geographical location information of the risk occurrence; call an early warning strategy library according to the geological risk level and the geographical location information to generate corresponding early warning operation instructions, wherein the early warning operation instructions include early warning time, response measures, execution personnel, and processing results; and send the early warning operation instructions, the geological risk level, and the geographical location information to an early warning information publishing platform to realize safety management and control.

[0210] The present application provides a kind of satellite signal rejection environment under safe monitoring equipment, satellite signal rejection environment under safe monitoring equipment includes: at least one processor;And, with at least one processor communication connection's memory;Wherein, memory stores the instruction that can be executed by at least one processor, instruction is executed by at least one processor, to enable at least one processor to execute the satellite signal rejection environment under safe monitoring method in above-mentioned embodiment one.

[0211] Reference will be made to the following Figure 6This document illustrates a structural schematic diagram of a security monitoring device suitable for implementing the embodiments of this application under a satellite signal denial environment. The security monitoring device under a satellite signal denial environment in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The security monitoring equipment shown in the satellite signal denial environment is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0212] like Figure 6 As shown, the security monitoring device under satellite signal denial conditions may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the security monitoring device under satellite signal denial conditions. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows security monitoring equipment in satellite signal denial environments to exchange data wirelessly or via wired communication with other devices. Although various types of security monitoring equipment in satellite signal denial environments are shown in the figures, it should be understood that it is not required to implement or possess all of those shown. More or fewer of these devices may be implemented alternatively.

[0213] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0214] The satellite signal denial environment safety monitoring device provided by the present application adopts the satellite signal denial environment safety monitoring method in the above-mentioned embodiments, and can solve the technical problem of how to realize geological risk early warning and safety monitoring of construction personnel by real-time monitoring of the state of the device under the satellite signal denial environment. Compared with the prior art, the satellite signal denial environment safety monitoring device provided by the present application has the same beneficial effects as the satellite signal denial environment safety monitoring method provided by the above-mentioned embodiments, and other technical features in the satellite signal denial environment safety monitoring device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0215] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0216] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0217] The present application provides a computer-readable storage medium having stored thereon computer-readable program instructions (i.e. computer program) for executing the satellite signal denial environment safety monitoring method in the above-mentioned embodiments.

[0218] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible storage medium containing or storing a program that can be executed by instructions, used by a device, or used in conjunction with it. The program code contained on the computer-readable storage medium may be transmitted using any suitable storage medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0219] The aforementioned computer-readable storage medium may be included in a security monitoring device under satellite signal denial conditions; or it may exist independently and not be assembled into a security monitoring device under satellite signal denial conditions.

[0220] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a security monitoring device in a satellite signal denial environment, enable the security monitoring device to write computer program code for performing the operations of this application in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0221] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of various embodiments of the methods and computer program products according to the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0222] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the names of the modules do not limit the modules themselves.

[0223] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the safety monitoring method in the satellite signal denial environment. The computer readable storage medium provided by the present application can solve the technical problem of how to realize geological risk early warning and safety monitoring of construction personnel by real-time monitoring of the device state in the satellite signal denial environment. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the safety monitoring method in the satellite signal denial environment provided by the above-mentioned embodiments, and will not be described here.

[0224] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the safety monitoring method in the satellite signal denial environment as described above.

[0225] The computer program product provided by the present application can solve the technical problem of how to realize geological risk early warning and safety monitoring of construction personnel by real-time monitoring of the device state in the satellite signal denial environment. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the safety monitoring method in the satellite signal denial environment provided by the above-mentioned embodiments, and will not be described here.

[0226] The above merely describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like, which is made based on the technical concept of the present application and the content of the specification and drawings, is included in the patent protection scope of the present application.

Claims

1. A security monitoring method under satellite signal denial environment, characterized in that, include: Acquire multimodal data and device text data of the target device. The multimodal data includes image data, video data, and audio data, and the device text data includes device table data and device text description data. The multimodal data is subjected to spatiotemporal synchronization processing to generate spatiotemporally consistent data; The spatiotemporal consistent data is encrypted on the edge to obtain encrypted multimodal data, which is then uploaded to the data lake for storage via a secure transmission protocol. The device text data is parsed using multi-scale morphological enhancement and hierarchical structure-aware encoding to obtain structured data and generate corresponding triples, which correspond to device entities and attributes. Based on the structured data and the encrypted multimodal data, semantic association is performed to construct a multimodal knowledge graph of the target scene; A self-supervised learning strategy is used to fuse features in the multimodal knowledge graph of the target scene to generate unified fused features; The unified fusion features are processed through a preset device status classification model to obtain device status information, which is then encrypted by hash and stored on the blockchain. The comparison results are obtained by comparing the status parameter values ​​of the equipment status information with the equipment status threshold. The status parameter values ​​are the operating environment temperature, the concentration of pollutants in the operating environment, and the operating time of the equipment. When the comparison result indicates that the state parameter value exceeds a preset state threshold, an alarm trigger command is generated and sent to the target device to enable the target device to provide geological risk information. Receive the geological risk information and execute corresponding early warning operations based on the geological risk information; The step of constructing a multimodal knowledge graph of the target scene by performing semantic association based on the structured data and the encrypted multimodal data includes: The encrypted multimodal data is subjected to modality-specific preprocessing to output a standardized multimodal dataset. The modality-specific preprocessing includes image denoising and enhancement, audio framing and noise reduction, video keyframe extraction, and text content recognition. Based on the structured data, entity extraction is performed to obtain the basic triplet data of the corresponding entity, where the entity is a component of the target device; Generate corresponding semantic text descriptions based on the basic triplet data, and output a structured semantic description set; Based on the structured semantic description set, cross-modal semantic retrieval is performed in the multimodal dataset, and a multimodal association result set is output. Based on the modal data in the multimodal association result set, generate extended triples describing entity relationships; An initial knowledge graph is constructed based on the extended triples, wherein the initial knowledge graph is constructed with entities as nodes and triple relationships as edges. The initial knowledge graph is subjected to semantic consistency verification and conflict resolution, and a multimodal knowledge graph of the target scene is output.

2. The method as described in claim 1, characterized in that, The step of performing spatiotemporal synchronization processing on the multimodal data to generate spatiotemporally consistent data includes: The spatial coordinate information and reference time information of the device are obtained, wherein the spatial coordinate information is obtained through UWB positioning and the reference time information is generated by the PTP master clock server in the mining area network. A spatiotemporal mapping relationship table is established based on the spatial coordinate information and the reference time information. The spatiotemporal mapping relationship table includes time deviation compensation parameters and spatial coordinate transformation matrix. Based on the spatiotemporal mapping table, the multimodal data is spatiotemporally corrected to obtain spatiotemporally consistent data. The spatiotemporally consistent data is then labeled with a unified spatiotemporal tag, which includes a GPS timestamp and three-dimensional spatial coordinates.

3. The method as described in claim 1, characterized in that, The step of parsing the device text data through multi-scale morphological enhancement and hierarchical structure-aware encoding to obtain structured data and generate corresponding triples includes: Convert the device's text data into image information; The image information is converted to grayscale to obtain a grayscale image; The grayscale image is enhanced by multi-scale morphological operations to obtain an enhanced image, wherein the multi-scale morphological operations include opening, closing, top-hat transformation and bottom-hat transformation. Feature extraction is performed on the enhanced image to generate basic visual features; The basic visual features are projected onto the query, key, and value spaces in the row and column directions, respectively, and the row attention matrix and column attention matrix are obtained. Based on the basic visual features, the row attention matrix, and the column attention matrix, structure-aware encoding is performed to obtain structure-aware features; The table's cell coordinates and content information are generated based on the structure-aware features. Convert the cell coordinates and content information into structured data; The structured data is used to generate corresponding triples, and the triple structure is {device entity, attribute name, attribute value}.

4. The method as described in claim 1, characterized in that, The step of using a self-supervised learning strategy to fuse features in the multimodal knowledge graph of the target scene and generate unified fused features includes: Based on the multimodal knowledge graph of the target scene, extract the node feature vectors corresponding to the multimodalities; A heterogeneous graph neural network is constructed using the node feature vectors corresponding to the multimodalities as nodes. A self-supervised learning strategy is used to perform intra-modal contrastive learning on the node feature vectors to obtain intra-modal neighborhood node information; A self-supervised learning strategy is used to perform inter-modal comparison learning on the node feature vectors to obtain neighborhood node information between modalities. The neighborhood node information within the modality and the neighborhood node information between the modalities are input into the heterogeneous graph neural network for aggregation to obtain the updated multimodal node features; The updated multimodal node features are weighted, fused, and normalized to obtain unified fused features.

5. The method as described in claim 1, characterized in that, The step of processing the unified fusion features through a preset device status classification model to obtain device status information, and storing the device status information through hash encryption and on-chain storage, includes: The unified fusion features are calculated using a preset device status classification model to obtain the probability distribution of device status; Based on the probability distribution, the state category corresponding to the preset probability is selected to obtain the device state information; After obtaining the device status information, the following is also included: Access to the blockchain network; The hash digest is obtained by distributing the device status information through IPFS and calculating the hash digest. The node's public key is obtained by combining the hash digest with the blockchain network. The node identity is calculated based on the node's public key. The node identity and hash digest are aggregated with a timestamp and stored on the blockchain network to complete the storage of the device status information.

6. The method as described in claim 1, characterized in that, The steps of receiving the geological risk information and executing corresponding early warning operations based on the geological risk information include: The geological risk information is matched with the preset geological risk level classification rules to determine the current geological risk level and the geographical location information of the risk occurrence; Based on the geological risk level and the geographical location information, the early warning strategy library is invoked to generate corresponding early warning operation instructions, wherein the early warning operation instructions include the early warning time, response measures, executors, and processing results; The warning operation instructions, geological risk level, and geographical location information are sent to the warning information release platform to achieve safety management.

7. A security monitoring device for satellite signal denial environments, characterized in that, The device includes: The acquisition module is used to acquire multimodal data and device text data of the target device. The multimodal data includes image data, video data, and audio data, and the device text data includes device table data and device text description data. The processing module is used to perform spatiotemporal synchronization processing on the multimodal data to generate spatiotemporally consistent data; An encryption module is used to perform end-side encryption processing on the spatiotemporal consistent data to obtain encrypted multimodal data, wherein the encrypted multimodal data is uploaded to the data lake for storage through a secure transmission protocol; The parsing module is used to parse the device text data through multi-scale morphological enhancement and hierarchical structure-aware encoding to obtain structured data and generate corresponding triples, wherein the triples correspond to device entities and attributes; The module is configured to: construct a target scene multimodal knowledge graph by performing semantic association based on the structured data and the encrypted multimodal data; perform modality-specific preprocessing on the encrypted multimodal data to output a standardized multimodal dataset, wherein the modality-specific preprocessing includes image denoising and enhancement, audio framing and noise reduction, video keyframe extraction, and text content recognition; extract entities based on the structured data to obtain basic triplet data for corresponding entities, wherein the entities are components of the target device; generate corresponding semantic text descriptions based on the basic triplet data to output a structured semantic description set; perform cross-modal semantic retrieval on the multimodal dataset based on the structured semantic description set to output a multimodal association result set; generate extended triplets describing entity relationships based on each modality data in the multimodal association result set; construct an initial knowledge graph based on the extended triplets, wherein the initial knowledge graph is constructed with entities as nodes and triplet relationships as edges; and perform semantic consistency verification and conflict resolution on the initial knowledge graph to output the target scene multimodal knowledge graph. The fusion module is used to perform feature fusion on the features in the multimodal knowledge graph of the target scene using a self-supervised learning strategy to generate unified fused features; The result module is used to process the unified fusion features through a preset device status classification model to obtain device status information, which is encrypted by hash and stored on the blockchain. The comparison module is used to compare the status parameter values ​​of the device status information with the device status threshold to obtain a comparison result. The status parameter values ​​are the device's operating environment temperature, operating environment pollutant concentration, and operating time. The instruction generation module is used to generate an alarm trigger instruction and send it to the target device when the comparison result is that the state parameter value exceeds a preset state threshold, so that the target device can provide feedback on geological risk information. The execution module is used to receive the geological risk information and perform corresponding early warning operations based on the geological risk information.

8. A security monitoring device for satellite signal denial environments, characterized in that, The device includes: a memory, a processor, and a security monitoring program for a satellite signal denial environment stored in the memory and running on the processor, the security monitoring program for a satellite signal denial environment being configured to implement the steps of the security monitoring method for a satellite signal denial environment as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a security monitoring program for satellite signal denial environments. When the security monitoring program for satellite signal denial environments is executed by the processor, it implements the steps of the security monitoring method for satellite signal denial environments as described in any one of claims 1-6.

Images