Safety production detection data processing method and system based on digitization

Abstract

The application provides a kind of based on digitization safety production detection data processing method and system, method includes: obtaining the safety detection data under target scene, and receive the detection instruction for safety detection data, safety detection data includes the continuous time series data stream of multidimensional sensor acquisition;Detection instruction is carried out semantic analysis, and the detection instruction feature including task type and entity distribution relationship is generated;Safety detection data is carried out cross-modal fusion processing by the multi-modal feature extraction model of joint training, and the safety feature vector of each time series node is generated;Based on the dynamic matching result of detection instruction feature and the safety feature vector of each time series node, key data section is screened out from time series data stream;Call pre-trained safety decision model to carry out abnormal mode identification and risk assessment to key data section, and generate the structured safety report corresponding to detection instruction.Through the application, the accuracy and reliability of safety detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method and system for processing digital safety production inspection data. Background Technology

[0002] In modern safety production across various fields, the processing of safety production monitoring data faces numerous pressing issues. With the widespread application of advanced sensors and monitoring equipment, safety monitoring data sources are complex and diverse. Different sensors collect multi-dimensional data with varying formats, sampling frequencies, and meanings, making data integration and processing challenging. Due to the large volume of safety monitoring data and the presence of significant redundant information, existing data filtering methods lack effective utilization of instruction intent and data temporal characteristics, resulting in low accuracy and reliability of filtering results. Furthermore, for the selected key data segments, existing anomaly pattern recognition and risk assessment methods often rely on single features or models, making it difficult to comprehensively and accurately capture complex information, easily leading to missed or misjudged potential safety hazards. In summary, the accuracy of existing safety production monitoring data processing technologies needs improvement. Summary of the Invention

[0003] The purpose of this invention is to provide a digital-based method for processing safety production inspection data.

[0004] The embodiments of the present invention are implemented as follows:

[0005] In a first aspect, embodiments of the present invention provide a digital-based method for processing safety production inspection data, comprising: acquiring safety inspection data in a target scenario and receiving inspection instructions for the safety inspection data, wherein the safety inspection data includes a continuous time-series data stream collected by multi-dimensional sensors; performing semantic parsing on the inspection instructions to generate inspection instruction features containing task type and entity distribution relationship; performing cross-modal fusion processing on the safety inspection data through a jointly trained multi-modal feature extraction model to generate a safety feature vector for each time-series node; selecting key data segments from the time-series data stream based on the dynamic matching results of the inspection instruction features and the safety feature vectors of each time-series node; and calling a pre-trained safety decision model to perform abnormal pattern recognition and risk assessment on the key data segments to generate a structured safety report corresponding to the inspection instructions.

[0006] In a second aspect, the present invention provides a computer system comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, they implement the method described above.

[0007] The technical effects of this invention include at least the following: This invention acquires safety detection data from a target scenario, containing rich information related to safe production. Next, it performs semantic parsing on the detection instructions, transforming them into detection instruction features that include task type and entity distribution relationships, providing clear targets and directions for subsequent data screening and analysis. Then, it utilizes a jointly trained multimodal feature extraction model to perform cross-modal fusion processing on the safety detection data, mining the correlation and complementary information between different modalities of data, and generating a safety feature vector for each time-series node to comprehensively and accurately represent the safe production status. Subsequently, based on the dynamic matching results of the detection instruction features and the safety feature vectors, key data segments are selected from the time-series data stream, focusing on important data related to the detection instructions. Finally, a pre-trained safety decision model is invoked to perform anomaly pattern recognition and risk assessment on the key data segments, generating a structured safety report, providing a scientific basis for safe production management.

[0008] A multimodal feature extraction model is used to perform cross-modal fusion processing on safety detection data, integrating visual modal data, acoustic modal data, and equipment status modal data to fully explore the potential information between different modal data. This cross-modal fusion processing generates more comprehensive and accurate safety feature vectors, solving the problem of insufficient information from single-modal data and improving the ability to perceive the safety production status. Semantic parsing of detection commands generates detection command features containing task types and entity distribution relationships. Specific steps include syntax tree parsing, semantic expansion, intent classification, and feature fusion, which deeply mines the semantic information of detection commands, determines task type priorities, and generates optimized detection command feature vectors. Based on the dynamic matching results of detection command features and safety feature vectors at each time-series node, key data segments are selected from the time-series data stream. Through steps such as calculating similarity scores, time window sliding analysis, length weight adjustment, context dependency analysis, and density clustering, the task type priority of detection commands and the time-series characteristics of the data are comprehensively considered to select the most relevant key data segments. A pre-trained safety decision model is invoked to perform abnormal pattern recognition and risk assessment on the key data segments, generating a structured safety report.

[0009] This invention effectively solves key problems such as the complexity and diversity of data processing, the understanding and execution of detection instructions, the accuracy and efficiency of data screening, and the identification of abnormal patterns and risk assessment. It improves the accuracy and efficiency of safety production detection, promptly identifies potential safety hazards, provides strong support for safety production management, and ensures the safe and stable operation of the production process. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0011] Figure 1 This is a flowchart of a digital-based safety production inspection data processing method provided in an embodiment of the present invention.

[0012] Figure 2 This is a schematic diagram of the composition of a computer system provided in an embodiment of the present invention. Detailed Implementation

[0013] In this embodiment of the invention, the execution entity of the digital safety production detection data processing method is a computer system, including but not limited to servers, personal computers, laptops, tablets, and smartphones. The computer system can operate independently to implement this invention, or it can connect to a network and interact with other computer systems within the network to implement this invention. The network in which the computer system is located includes, but is not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and VPN networks.

[0014] This invention provides a digital-based method for processing safety production inspection data, such as... Figure 1 As shown, the method includes:

[0015] Step 100: Acquire security detection data in the target scenario and receive detection instructions for the security detection data. The security detection data includes a continuous time-series data stream collected by multi-dimensional sensors.

[0016] In practical operation, various target scenarios may be encountered, such as industrial production workshops, intelligent building environments, and transportation hubs. This invention does not limit the target scenario. Embodiments of this invention can utilize various multi-dimensional sensors to collect safety detection data. These sensors are distributed at various key locations within the workshop to comprehensively and in real-time acquire relevant information. For example, in terms of vision, high-definition cameras can be used to continuously monitor personnel operations, equipment operating status, and material stacking within the workshop. The collected video data constitutes visual modal safety detection data. In terms of acoustics, microphone sensors can be used to collect sound information within the workshop. Under normal circumstances, the operation of machines within the workshop exhibits specific sound frequencies and intensity ranges. Abnormal sharp noises or irregular sound fluctuations may indicate equipment malfunctions. Regarding equipment status, various sensors can be used to monitor parameters such as temperature, pressure, current, and voltage of the equipment in real time. This data collected by multi-dimensional sensors is a continuous time-series data stream, meaning the data is continuously generated over time, and each data point carries a corresponding timestamp to reflect the order in which the data was generated. The data collected by the sensors can be transmitted to a data processing center for storage and further analysis via wired or wireless networks.

[0017] Step 200: Perform semantic parsing on the detection instructions to generate detection instruction features that include the relationship between task type and entity distribution.

[0018] For received detection instructions, it is necessary to clearly define the type of task to be performed and the scope of the detection to ensure targeted processing of safety detection data. Taking an industrial production scenario as an example, suppose the received detection instruction is "Check whether the temperature and pressure of all production equipment in workshop A are abnormal in the past 24 hours." This instruction requires in-depth semantic analysis. Specifically, natural language processing (NLP) technology can be used to parse the detection instructions. At the grammatical level, analyze the individual words in the instruction and their grammatical relationships to identify key entity objects and operation predicates. In the example above, the entity objects are "workshop A" and "production equipment," and the operation predicates are "check" and "whether abnormal." Through this grammatical analysis, the basic structure and key elements of the instruction can be initially grasped. Next, semantic expansion of the entity objects is performed. For example, using a pre-trained domain knowledge graph containing various knowledge and information in the industrial production field, entity objects such as "workshop A" and "production equipment" can be associated and expanded with relevant knowledge in the graph. For example, information such as the specific layout of workshop A, the model and specifications of production equipment can be obtained from the knowledge graph, thereby forming a more comprehensive entity association network, which helps the system to understand the entity distribution relationship more accurately.

[0019] For operation predicates, intent classification can be performed to determine the task type priority of the detection instruction. In this example, "inspect" indicates the core intent of the operation. Based on preset rules and models, it is determined that this is a safety detection task, and further analysis is conducted on the urgency and importance of this task within the entire safety production process. Through intent classification, corresponding priorities can be assigned to different task types, ensuring that high-priority tasks are processed first when handling data.

[0020] After semantic expansion of entity objects and intent classification of operation predicates, relevant information can be fused to generate multi-dimensional detection instruction features. These features include entity distribution vectors and task weight matrices. The entity distribution vectors reflect the distribution of entity objects within the entire entity distribution relationship, while the task weight matrix reflects the priority and importance of different task types. For example, in this case, the entity distribution vectors might show the distribution of production equipment in different areas of workshop A, and the task weight matrix would highlight the high priority of the "temperature and pressure anomaly detection" task. Finally, an attention mechanism can be used to dynamically weight the multi-dimensional detection instruction features. The attention mechanism automatically adjusts the weights of different features based on the context and specific requirements of the instruction, making the system focus more on important information.

[0021] Step 300: Perform cross-modal fusion processing on the security detection data using a jointly trained multimodal feature extraction model to generate a security feature vector for each time-series node.

[0022] Safety detection data is collected from multi-dimensional sensors and possesses multiple modalities, such as visual, acoustic, and equipment status data. These different modalities are fused to extract more comprehensive and accurate safety features. First, the safety detection data is modally split into visual modal data, acoustic modal data, and equipment status modal data. For visual modal data, a dedicated visual encoder can be used for processing. The visual encoder can extract spatial features from video footage. For acoustic modal data, an acoustic encoder can be used for spectral decomposition and abnormal waveform detection. The acoustic encoder can analyze the frequency, intensity, and waveform characteristics of sound to determine the presence of abnormal sound signals. The status encoder can determine whether the equipment is operating normally based on parameters such as temperature, pressure, and current. For example, for a motor, the status encoder can determine whether the motor is overloaded or has a short circuit based on changes in current, generating an equipment operating status vector. After feature extraction from different modalities, these features are aligned and fused using a cross-modal attention network. The cross-modal attention network can automatically adjust the weights of different modal features, allowing the system to focus more on important information. For example, in some cases, visual features may better reflect the external damage to a device, while acoustic features may be better at detecting internal faults. Cross-modal attention networks can assign appropriate weights to features from different modalities based on specific circumstances, fusing visual features, acoustic energy distribution features, and device operating state vectors to generate a safety feature vector for each time-series node. This vector integrates information from multiple modalities, providing a more comprehensive and accurate reflection of the safety status of the target scene at each time point, and offering strong support for subsequent data analysis and decision-making. Deep learning algorithms can be used to jointly train the multimodal feature extraction model, continuously adjusting the model's parameters to improve its performance and accuracy.

[0023] Step 400: Based on the dynamic matching results of the detection instruction features and the security feature vectors of each time-series node, key data segments are selected from the time-series data stream.

[0024] Specifically, the feature vector of the detection instruction can be dynamically matched with the security feature vector of each time-series node, and a similarity score can be calculated between them. The higher the similarity score, the stronger the correlation between the data of that time-series node and the detection instruction. Next, a time window sliding analysis can be performed on the initial matching sequence. The time window is a fixed-length time period. By sliding this window across the time-series data stream, the similarity scores of multiple consecutive time-series nodes within the window can be observed. If the similarity scores within a certain window all exceed a dynamic threshold, it indicates that the data in this time period is highly correlated with the detection instruction, and the data in this time period will be identified as a candidate data segment. Then, the length weight of the candidate data segments can be adjusted according to the task type priority in the detection instruction features. If the task that the detection instruction focuses on is of high priority, the length weight of the corresponding candidate data segment will increase; conversely, the length weight of the candidate data segment corresponding to a low-priority task will decrease. For example, if the high-priority task in the detection instruction is a sudden failure of the detection equipment, the length weight of the candidate data segment related to this task will be increased, thus ensuring that critical data is not missed.

[0025] Next, the optimized candidate data segment set can be input into a pre-trained temporal correlation model to analyze the contextual dependencies of each candidate data segment in the temporal data stream. Isolated candidate data segments with a correlation degree less than the correlation threshold with preceding and following time nodes can be removed, as these isolated data may be interfering information. Simultaneously, candidate data segments with temporal overlap can be merged to make the data more compact and meaningful. Finally, density clustering can be performed based on the distribution density of each data segment in the optimized data segment set within the temporal data stream. If the distribution density of a data segment is higher than a preset clustering threshold, it is selected as the final set of key data segments. Key data segments contain a preset minimum number of consecutive time nodes and cover the set of entity objects specified in the detection instruction features, providing accurate and effective data support for subsequent anomaly pattern recognition and risk assessment. Machine learning algorithms can be used to perform similarity calculation, time window sliding analysis, temporal correlation analysis, and density clustering to ensure the accuracy and efficiency of key data segment selection.

[0026] Step 500: Call the pre-trained security decision model to perform abnormal pattern recognition and risk assessment on the key data segments, and generate a structured security report corresponding to the detection instructions.

[0027] In step 500, a pre-trained security decision model is invoked to perform anomaly pattern recognition and risk assessment on key data segments, generating a structured security report corresponding to the detection instructions. In a safety production detection scenario, after the key data segments are selected through the preceding steps, the hidden security information behind these data is further mined to ensure the safe operation of production activities.

[0028] For example, key data segments are sliced ​​into multi-granularity data blocks, each containing global state features and local anomaly features. These hierarchical data blocks are then input into the convolutional recurrent network of the safety decision model for spatiotemporal feature extraction. Next, the pattern matching layer in the safety decision model compares the extracted spatiotemporal features with a pre-defined anomaly pattern library. This library stores various known safety anomalies and their characteristics; by comparing these patterns, it can be determined whether the current data contains patterns similar to known anomalies. For instance, if the temperature and pressure change patterns of the reactor are similar to the "reaction runaway" pattern in the anomaly pattern library, then there may be a risk of reaction runaway.

[0029] Based on the comparison results, risk assessment indicators can be generated, including risk level, anomaly type, and scope of impact. Risk level can be divided into different levels such as high, medium, and low; anomaly type can be specified as such as equipment failure or operational error; and scope of impact can be clearly defined as whether it only affects the reactor itself or will affect surrounding equipment or personnel.

[0030] Finally, based on the task types in the risk assessment indicators and detection instructions, a structured safety report containing visual charts and handling recommendations is generated. The visual charts can intuitively display the risk situation, such as using line graphs to show the trends of temperature and pressure changes, and bar charts to compare risk levels over different time periods. The handling recommendations provide specific countermeasures based on the risk assessment results; for example, for the risk of runaway reaction, it is recommended to immediately stop feeding and activate the emergency cooling system. Deep learning algorithms can be used to train the safety decision-making model, image processing technology can be used to generate visual charts, and handling recommendations can be formulated by combining expert knowledge in the field of safety production, thereby providing a scientific and effective basis for safety production management.

[0031] As one implementation method, step 200 involves semantic parsing of the detection instructions to generate detection instruction features that include the relationship between task type and entity distribution, including:

[0032] Step 210: Perform syntax tree parsing on the detection instructions to extract the set of entity objects and the set of operation predicates from the instructions;

[0033] Step 220: Input the set of entity objects into the pre-trained domain knowledge graph for semantic expansion to generate an expanded entity association network;

[0034] Step 230: Classify the intent of the operation predicate set to determine the task type priority of the detection instructions;

[0035] Step 240: Integrate the topological features of the entity association network with the task type priority to generate multi-dimensional detection instruction features, wherein the detection instruction features include entity distribution vectors and task weight matrices;

[0036] Step 250: Dynamically weight the multi-dimensional detection instruction features using an attention mechanism to generate an optimized detection instruction feature vector.

[0037] In step 210, the detection instruction is parsed using a syntax tree to extract the set of entity objects and the set of operation predicates. Taking the detection instruction "Check whether the temperature and pressure of all production equipment in workshop A are normal" from an industrial production scenario as an example, this sentence can be syntactically analyzed to identify the entity objects and operation predicates. Entity objects refer to the specific things or concepts involved in the instruction; in this example, "workshop A" and "production equipment" are entity objects. Operation predicates represent the actions required by the instruction; here, "check" and "whether normal" reflect the core intent of the operation. Natural language processing toolkits such as NLTK (Natural Language Toolkit) or SpaCy can be used to extract entity objects and operation predicates.

[0038] In step 220, the set of entity objects is input into a pre-trained domain knowledge graph for semantic expansion, generating an expanded entity association network. In the industrial production field, the knowledge graph may contain information such as equipment models, specifications, performance parameters, workshop layout, and safety regulations. Graph databases, such as Neo4j, can be used to store and manage the domain knowledge graph, and entity information can be retrieved and expanded using a graph query language.

[0039] In step 230, the intent of the operation predicate set is classified to determine the task type priority of the detection instruction. Operation predicates embody the core intent of the detection instruction, but different operation predicates may have different priorities and importance. These operation predicates can be classified and prioritized to prioritize high-priority tasks in subsequent data processing. Taking the instruction "Check whether the temperature and pressure of all production equipment in workshop A are normal" as an example, the intent of the operation predicate "check" is to perform status detection. Based on preset rules and models, the urgency and importance of this detection task in the entire safe production process need to be determined. If the production equipment in workshop A is performing a critical production task, then abnormal temperature and pressure could lead to a serious production accident; in this case, the priority of this detection task would be higher. Machine learning algorithms, such as Support Vector Machines (SVM) or deep learning models, can be used to classify the intent of the operation predicates and set priority rules for task types based on domain knowledge and experience.

[0040] In step 240, the topological features of the entity association network and task type priority are integrated to generate multi-dimensional detection instruction features, which include entity distribution vectors and task weight matrices. The topological features of the entity association network reflect the relationships and structure between entity objects, while task type priority reflects the importance of different tasks. Combining the two generates a multi-dimensional vector and matrix that comprehensively reflects the detection instruction features. In the example above, the entity distribution vector can represent the distribution of production equipment in workshop A, such as which equipment is located in key positions in the workshop and which equipment is closely related to other equipment. The task weight matrix can assign corresponding weights to different detection tasks according to task type priority. If the temperature detection task has a higher priority, then the weight corresponding to temperature detection in the task weight matrix will be larger. The entity distribution vector and task weight matrix can be generated using a vector space model and matrix operations. By calculating the similarity between entities and the importance of tasks, this information is integrated into the detection instruction features.

[0041] In step 250, the multi-dimensional detection command features are dynamically weighted using an attention mechanism to generate an optimized detection command feature vector. An attention mechanism is a technique that automatically adjusts the weights of different features, allowing the system to focus more on important information based on the context and specific requirements of the command. In safety production inspection, different inspection tasks and entities may have different levels of importance under different circumstances. For example, in some cases, equipment temperature may be more critical than pressure; in this case, the attention mechanism can automatically assign higher weights to temperature-related features. By dynamically weighting the multi-dimensional detection command features, a more optimized detection command feature vector can be generated. This vector more accurately reflects the core requirements of the detection command, providing more precise guidance for subsequent data screening and analysis. Deep learning-based attention models, such as the multi-head attention mechanism in Transformer, can be used to achieve dynamic weighting of the detection command features.

[0042] Steps 210-250 provide a complete method and process for semantic parsing of detection instructions. Through syntactic analysis, entity expansion, intent classification, feature fusion, and dynamic weighting of instructions, the meaning of detection instructions can be accurately understood, generating detection instruction features that include task type and entity distribution relationships, providing strong support for data processing and analysis in safety production detection.

[0043] As one implementation method, step 230 involves classifying the set of operation predicates by intent and determining the task type priority of the detection instructions, including:

[0044] Step 231: Standardize the operation predicate set, remove redundant stop words and ambiguous expressions, and generate a standardized operation predicate set;

[0045] Step 232: Call the pre-trained intent recognition model to perform multi-granular semantic parsing on the standardized set of operation predicates, extract the logical dependencies and action triggering conditions between operation predicates, and generate an initial set of intent labels;

[0046] Step 233: Based on the preset safety production domain knowledge base, perform semantic verification on the initial intent tag set, verify the matching degree of each tag in the initial intent tag set with the standard operation scenario in the domain knowledge base, and generate the verified intent tag confidence score;

[0047] Step 234: Dynamically assign weights to the initial intent tag set based on the confidence level of the verified intent tags. Intent tags with a matching degree higher than the preset verification threshold of the standard operation scenario are given high confidence weights to generate a weighted intent tag set.

[0048] Step 235: Input the weighted intent tag set into the predefined task priority rule base, and map each intent tag to the corresponding priority weight according to the security response level and operation urgency mapping relationship defined in the rule base;

[0049] Step 236: Combine the contextual features of the detection command to extract the set of entity objects involved in the detection command and the timeliness constraints of the operation, and dynamically adjust the priority weight;

[0050] Step 237: Sort the weighted intent label set based on the corrected priority weights to generate a task type priority queue. The order of each element in the queue reflects the urgency of its corresponding operation predicate in safety production detection, and the head element of the queue is marked as the core task type of the detection instruction.

[0051] In step 231, the operation predicate set is standardized to remove redundant stop words and vague expressions, generating a standardized operation predicate set. In natural language expression, detection instructions may contain words that do not affect the core semantics, such as modal particles and auxiliary words, which can interfere with the accurate understanding of the operation predicates. For the instruction "Please carefully check whether the temperature and pressure of all production equipment in workshop A are normal," words such as "please," "carefully," and "check" are redundant stop words and can be removed. At the same time, some vague expressions, such as "probably" and "maybe," will also be processed. Through this standardization process, a more concise and clear operation predicate set can be obtained, for example, the above instruction can be processed into "Check whether the temperature and pressure of production equipment in workshop A are normal." Stop word lists and regular expressions can be used to remove redundant stop words and vague expressions. The stop word list contains common words without actual semantics, and regular expressions can be used to match and replace vague expressions.

[0052] In step 232, a pre-trained intent recognition model is invoked to perform multi-granular semantic parsing on the standardized set of operation predicates, extracting the logical dependencies and action triggering conditions between the operation predicates, and generating an initial intent label set. The pre-trained intent recognition model is trained on a large amount of text data and can understand the semantics of operation predicates and the relationships between them. For the standardized operation predicate "Check whether the temperature and pressure of the production equipment in workshop A are normal," the intent recognition model analyzes that "check" is the core action, "production equipment in workshop A" is the operation object, "temperature and pressure" are the specific parameters to be detected, and "whether it is normal" is the judgment standard. Simultaneously, the model identifies the logical dependencies between operation predicates, that is, first determining the operation object, then checking the specific parameters of that object, and finally making a judgment based on the standard. Action triggering conditions may involve checking at certain time intervals or triggering checks when certain abnormal signs appear. Based on these analyses, the model generates an initial intent label set, such as "equipment status detection" and "temperature and pressure detection." Deep learning models, such as Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs), can be used as intent recognition models, as these models have good performance in processing sequential data and semantic understanding.

[0053] In step 233, semantic verification is performed on the initial intent tag set based on a pre-set safety production domain knowledge base. Each tag in the initial intent tag set is compared with the standard operation scenarios in the domain knowledge base to verify its matching degree, generating a confidence score for the verified intent tag. The safety production domain knowledge base contains various standard operation scenarios and corresponding intent tags, allowing comparison of the initial intent tags with the tags in the knowledge base. For the initial intent tags "equipment status detection" and "temperature and pressure detection," matching standard scenarios can be found in the knowledge base. If the knowledge base explicitly defines temperature and pressure detection of production equipment in a workshop environment as a normal equipment status detection operation, then these two tags will have a high matching degree with the standard scenario. A confidence score is generated for each tag based on the matching degree; the higher the matching degree, the higher the confidence score. Methods such as cosine similarity can be used to calculate the matching degree between tags.

[0054] In step 234, the initial intent tag set is dynamically weighted based on the confidence level of the verified intent tags. Intent tags with a matching degree higher than a preset verification threshold for the standard operating scenario are assigned high confidence weights, generating a weighted intent tag set. The preset verification threshold is a pre-defined standard used to determine whether the matching degree between a tag and the standard scenario is high enough. If the confidence level of an intent tag is higher than this threshold, it means that its matching degree with the standard operating scenario is high, and it can be assigned a high confidence weight. For the tags "device status detection" and "temperature and pressure detection," if their confidence levels are both higher than the preset threshold, they will be assigned higher weights in the weighted intent tag set. This dynamic weight allocation highlights tags with a high matching degree with the standard operating scenario, providing a more accurate basis for subsequent priority determination.

[0055] In step 235, the weighted intent label set is input into a predefined task priority rule base. Based on the mapping relationship between safety response levels and operational urgency defined in the rule base, each intent label is mapped to a corresponding priority weight. The predefined task priority rule base is developed based on actual needs and experience in safe production, specifying the urgency of various operations under different safety response levels. For the labels "equipment status detection" and "temperature and pressure detection," the rule base may stipulate that under high safety response levels, equipment temperature and pressure detection are very urgent tasks that need to be executed immediately, thus assigning these two labels higher priority weights. Under low safety response levels, the urgency of these tasks may be relatively low, and the priority weights will decrease accordingly. By querying the rule base, each label in the weighted intent label set is mapped to its corresponding priority weight, thereby initially determining the priority of each task.

[0056] In step 236, the set of entity objects involved in the detection instruction and the timeliness constraints of the operation are extracted based on the contextual features of the detection instruction, and the priority weights are dynamically adjusted. The contextual features of the detection instruction include information such as the time and location of the instruction issuance and the current production status. For the instruction "Check whether the temperature and pressure of all production equipment in workshop A are normal," if workshop A is currently performing a critical production task, then the temperature and pressure detection of the equipment becomes more urgent, and the priority weight of the relevant task can be increased. At the same time, the timeliness constraints of the operation also affect the priority weights. For example, if the instruction requires the detection to be completed within 1 hour, then the priority of the task will be increased accordingly. Taking all these factors into account, the previously determined priority weights are dynamically adjusted to make the priority determination more in line with the actual situation.

[0057] In step 237, the weighted intent label set is sorted based on the corrected priority weights to generate a task type priority queue. The order of each element in the queue reflects the urgency of its corresponding operation predicate in safety production inspection, and the head element of the queue is marked as the core task type of the inspection instruction. The weighted intent label set is sorted in descending order according to the corrected priority weights, with labels having higher priority weights appearing earlier. For the labels "equipment status detection" and "temperature and pressure detection," if the corrected priority weight of "temperature and pressure detection" is higher, it will be placed at the front of the queue. The generated task type priority queue clearly shows the urgency of each task, allowing for the scheduling of subsequent data processing and analysis, prioritizing the core task types at the head of the queue.

[0058] As one implementation method, step 300 involves performing cross-modal fusion processing on the security detection data using a jointly trained multimodal feature extraction model to generate a security feature vector for each time-series node, including:

[0059] Step 310: Decompose the safety detection data into visual modal data, acoustic modal data, and equipment status modal data;

[0060] Step 320: Call the visual encoder to extract spatial features from the visual modal data to generate visual features that include regional thermal distribution and motion trajectory;

[0061] Step 330: Call the acoustic encoder to perform spectral decomposition and abnormal waveform detection on the acoustic modal data to generate acoustic energy distribution characteristics;

[0062] Step 340: Input the equipment status modal data into the status encoder for operating condition mode recognition and generate the equipment operating status vector;

[0063] Step 350: Use a cross-modal attention network to align and fuse visual features, acoustic energy distribution features, and device operating state vectors to generate a security feature vector for each time-series node.

[0064] In step 310, the safety detection data is broken down into visual modal data, acoustic modal data, and equipment status modal data. The data from different modalities reflects different aspects of the production environment and equipment operation. Data can be categorized according to its source and characteristics, classifying data collected by different sensors into their respective modalities.

[0065] In step 320, the visual encoder is a model specifically designed for processing image and video data. It can extract valuable spatial features from visual modal data. In the example of workshop A, for video footage captured by a camera, the visual encoder can analyze information such as color and brightness in different areas of the image to generate regional thermal distribution features. Simultaneously, the visual encoder can also track the motion trajectories of people and objects, determining their direction and speed by analyzing changes in object position across consecutive frames. For example, it can track the movement paths of workers within the workshop and the movement trajectories of materials on conveyor belts. A convolutional neural network (CNN) can be used as the visual encoder. CNNs have powerful feature extraction capabilities in image and video processing; through multi-layer convolution and pooling operations, they can extract spatial features at different levels.

[0066] In step 330, an acoustic encoder is invoked to perform spectral decomposition and abnormal waveform detection on the acoustic modal data, generating acoustic energy distribution characteristics. The acoustic encoder converts the acoustic signal into a frequency domain representation, analyzing the frequency components and energy distribution of the sound. In workshop A, the sound of equipment operation under normal conditions has specific spectral characteristics; when equipment malfunctions, the spectrum changes. The acoustic encoder performs a Fast Fourier Transform (FFT) on the collected acoustic data, converting the time-domain signal into a frequency-domain signal to obtain the sound's spectrum. By analyzing the spectrum, the energy distribution of different frequency components can be identified. Deep learning-based acoustic models, such as variants of Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs), can be used to implement spectral decomposition and abnormal waveform detection of the acoustic data.

[0067] In step 340, the equipment state modal data is input into a state encoder for operating condition pattern recognition, generating an equipment operating state vector. The state encoder can determine the current operating condition mode of the equipment based on various operating parameters. In workshop A, the equipment state modal data includes multiple parameters such as temperature, pressure, and current. The state encoder performs comprehensive analysis on these parameters. For example, for a motor, the state encoder can determine whether the motor is in a starting, normal operation, or overload state based on the magnitude and trend of the current. Simultaneously, by combining parameters such as temperature and pressure, it further determines the overall operating state of the equipment. The state encoder converts this information into an equipment operating state vector, where each element represents a specific operating condition feature. For example, the first element of the vector might represent the temperature state, the second element might represent the pressure state, and so on. Machine learning algorithms, such as Support Vector Machines (SVM) or decision trees, can be used to classify and model the equipment state modal data to achieve operating condition pattern recognition.

[0068] In step 350, a cross-modal attention network is used to align and fuse visual features, acoustic energy distribution features, and equipment operating state vectors to generate a safety feature vector for each time-series node. The cross-modal attention network can automatically adjust the weights of different modal features, allowing the system to focus more on important information. In the scenario of workshop A, features from different modalities may reflect the equipment's safety status to varying degrees under different circumstances. For example, in some cases, visual features may better reflect external damage to the equipment, while acoustic features may be better at detecting internal faults. The cross-modal attention network assigns appropriate weights to features from different modalities based on the current task and data conditions. For example, when detecting obvious external damage to the equipment, the weight of visual features can be increased; when detecting internal faults, the weight of acoustic features can be increased. Through feature alignment and fusion, features from different modalities are integrated into a unified safety feature vector, which can comprehensively and accurately reflect the equipment's safety status at each time-series node. Deep learning models based on attention mechanisms, such as variants of the Transformer architecture, can be used to achieve cross-modal feature alignment and fusion.

[0069] As one implementation method, step 400 involves selecting key data segments from the time-series data stream based on the dynamic matching results of the detection instruction features and the security feature vectors of each time-series node, including:

[0070] Step 410: Based on the dynamic matching results between the detection instruction feature vector and the security feature vector of each time node, calculate the similarity score between each time node and the detection instruction feature vector, and generate an initial matching degree sequence;

[0071] Step 420: Perform time window sliding analysis on the initial matching degree sequence to identify candidate data segments in the time series data stream where the similarity scores of multiple consecutive time series nodes all exceed the dynamic threshold, and generate a set of candidate data segments;

[0072] Step 430: Based on the task type priority in the detection instruction features, adjust the length weight of each candidate data segment in the candidate data segment set. The length weight of the candidate data segment corresponding to the high priority task is increased, and the length weight of the candidate data segment corresponding to the low priority task is decreased, thereby generating an optimized candidate data segment set.

[0073] Step 440: Input the optimized candidate data segment set into the pre-trained temporal correlation model, analyze the contextual dependency of each candidate data segment in the temporal data stream, remove isolated candidate data segments whose correlation with the preceding and following time nodes is less than the correlation threshold, and merge candidate data segments with time overlap to generate a set of correlated optimized data segments.

[0074] Step 450: Based on the distribution density of each data segment in the time-series data stream in the association optimization data segment set, perform density clustering on the association optimization data segment set, and select data segments with a distribution density higher than the preset clustering threshold as the final key data segment set. Each key data segment in the key data segment set contains a preset minimum number of consecutive time-series nodes and covers the set of entity objects specified in the detection instruction features.

[0075] In step 410, based on the dynamic matching results between the detection instruction feature vector and the safety feature vector of each time-series node, the similarity score between each time-series node and the detection instruction feature vector is calculated, generating an initial matching degree sequence. First, the detection instruction feature vector and the safety feature vector of each time-series node are normalized to ensure they are within the same numerical range, avoiding the impact of different numerical scales on the accuracy of similarity calculation. For example, the value of an element in the detection instruction feature vector might be between 0 and 100, while the corresponding element in the safety feature vector might be between 0 and 1; normalization eliminates this difference. Next, based on the semantic coverage of the entity distribution vector in the detection instruction feature vector and the dynamic distribution of the task weight matrix, the weights of the normalized detection instruction feature vector are dynamically adjusted to generate a dynamically weighted detection instruction feature vector. This is because different entities and tasks may have different importance in the detection instruction, requiring adjustment according to the actual situation. For example, in an instruction to check whether the temperature and pressure of production equipment in workshop A are normal, if the temperature detection task has a higher priority, then the weight of temperature-related elements in the dynamically weighted detection instruction feature vector will increase.

[0076] Then, a multi-dimensional similarity comparison is performed between the dynamically weighted detection instruction feature vector and the normalized security feature vector of each time-series node, including three independent calculation paths: semantic coverage overlap, entity distribution direction consistency, and task weight sensitivity. For the semantic coverage overlap calculation path, the entity association network topology features in the dynamically weighted detection instruction feature vector can be extracted and compared with the entity spatial distribution features in the normalized security feature vector to perform regional overlap analysis, generating a semantic coverage overlap score. For example, if the detection instruction focuses on equipment in several specific areas within workshop A, and the security feature vector of a certain time-series node shows that there is relevant equipment operation data in these areas, then the semantic coverage overlap score will be high. For the entity distribution direction consistency calculation path, the entity distribution vector in the dynamically weighted detection instruction feature vector and the entity distribution vector in the normalized security feature vector can be projected to generate an entity distribution direction consistency score. If the directions of the two vectors are relatively consistent, it indicates that the entity distribution is similar, and the score will be high. For the task weight sensitivity calculation path, the task weight matrix in the dynamically weighted detection instruction feature vector and the risk sensitivity matrix in the normalized security feature vector can be multiplied element-wise to generate a task weight sensitivity score. If a task has a large weight in the task weight matrix and the risk sensitivity of that task in the security feature vector is also high, then the task weight sensitivity score will be high.

[0077] The semantic coverage overlap score, entity distribution direction consistency score, and task weight sensitivity score are input into a pre-trained fusion decision model. The three scores are dynamically weighted and fused based on the task type priority in the detection instruction features to generate a comprehensive similarity score for each temporal node. For example, if the temperature detection task has a high priority in the detection instruction, the weight of the temperature-related part of the semantic coverage overlap score will be increased during fusion. Next, a temporal dependency analysis model is invoked to perform contextual correction on the comprehensive similarity score. This model smoothly corrects the comprehensive similarity score of the current temporal node based on the dynamic matching relationship between the change trend of the safety feature vectors of adjacent temporal nodes and the feature vectors of the detection instructions. Finally, the corrected comprehensive similarity scores are arranged in temporal order to generate an initial matching degree sequence. The value of each element in the sequence reflects the dynamic matching degree between the corresponding temporal node and the feature vector of the detection instruction, and the changes in the values ​​of adjacent elements follow the contextual constraints output by the temporal dependency analysis model.

[0078] In step 420, a time window sliding analysis is performed on the initial matching degree sequence to identify candidate data segments in the time-series data stream where the similarity scores of multiple consecutive time-series nodes all exceed a dynamic threshold, generating a set of candidate data segments. The time window is a fixed-length time period. This window can be slid across the time-series data stream to view the similarity scores of multiple consecutive time-series nodes within the window. For example, if the time window is set to 10 time-series nodes, and the similarity scores of all 10 time-series nodes within a certain window exceed the dynamic threshold, it indicates that the data in this time period is highly correlated with the detection command, and the data in this time period will be identified as a candidate data segment. The dynamic threshold is adjusted according to the overall distribution of the data and the requirements of the detection command to ensure that suitable candidate data segments are selected.

[0079] In step 430, based on the task type priority in the detection instruction features, the length weight of each candidate data segment in the candidate data segment set is adjusted. Specifically, the length weight of candidate data segments corresponding to high-priority tasks is increased, while the length weight of candidate data segments corresponding to low-priority tasks is decreased, generating an optimized candidate data segment set. In the example of detecting production equipment in workshop A, if temperature detection is a high-priority task in the detection instruction, the length weight of candidate data segments related to temperature detection will be increased to ensure that these critical data are not missed. Conversely, the length weight of candidate data segments corresponding to low-priority tasks will be decreased to avoid interference from too much irrelevant data. Through this length weight adjustment, data segments related to high-priority tasks can be more accurately selected.

[0080] In step 440, the optimized candidate data segment set is input into the pre-trained temporal correlation model. The model analyzes the contextual dependencies of each candidate data segment in the temporal data stream, removes isolated candidate data segments whose correlation with preceding and following time nodes is less than the correlation threshold, and merges candidate data segments with temporal overlap to generate an optimized data segment set. The temporal correlation model can learn the temporal order and interdependencies between candidate data segments.

[0081] In step 450, based on the distribution density of each data segment in the time-series data stream, density clustering is performed on the optimized data segment set. Data segments with a distribution density higher than a preset clustering threshold are selected as the final key data segment set. Each key data segment in the key data segment set contains a preset minimum number of consecutive time-series nodes and covers the set of entity objects specified in the detection instruction features. The density clustering algorithm can group adjacent and densely distributed data segments together based on their distribution in the time-series data stream. For example, if data segments in a certain region are densely distributed and their density exceeds the preset clustering threshold, these data segments will be clustered into one class. These high-density data segments are selected as the final key data segment set. Simultaneously, each key data segment needs to contain a preset minimum number of consecutive time-series nodes to ensure data continuity and integrity, and it must cover the set of entity objects specified in the detection instruction features, such as production equipment in workshop A, thereby ensuring the relevance of the key data segment to the detection instruction.

[0082] As one implementation method, step 410, based on the dynamic matching results between the detection instruction feature vector and the security feature vector of each time-series node, calculates the similarity score between each time-series node and the detection instruction feature vector, and generates an initial matching degree sequence, including:

[0083] Step 411: Normalize the detection instruction feature vector and the security feature vector of each time node respectively to generate normalized detection instruction feature vector and normalized time node security feature vector;

[0084] Step 412: Based on the semantic coverage of the entity distribution vector in the detection instruction feature vector and the dynamic distribution of the task weight matrix, dynamically adjust the weights of the normalized detection instruction feature vector to generate a dynamically weighted detection instruction feature vector.

[0085] Step 413: Perform a multi-dimensional similarity comparison between the dynamically weighted detection instruction feature vector and the normalized security feature vector of each time node. The multi-dimensional similarity comparison includes three independent calculation paths: semantic coverage overlap, entity distribution direction consistency, and task weight sensitivity.

[0086] Step 414: For the semantic coverage overlap calculation path, extract the entity association network topology features from the dynamically weighted detection instruction feature vector, and perform regional overlap analysis with the entity spatial distribution features from the normalized security feature vector to generate a semantic coverage overlap score.

[0087] Step 415: For the entity distribution orientation consistency calculation path, perform orientation projection calculation on the entity distribution vector in the dynamically weighted detection instruction feature vector and the entity distribution vector in the normalized security feature vector to generate the entity distribution orientation consistency score.

[0088] Step 416: For the task weight sensitivity calculation path, perform element-wise multiplication of the task weight matrix in the dynamically weighted detection instruction feature vector with the risk sensitivity matrix in the normalized security feature vector to generate the task weight sensitivity score.

[0089] Step 417: Input the semantic coverage overlap score, entity distribution direction consistency score, and task weight sensitivity score into the pre-trained fusion decision model, and dynamically weight and fuse the three scores according to the task type priority in the detection instruction features to generate a comprehensive similarity score for each time node.

[0090] Step 418: Call the temporal dependency analysis model to perform contextual correction on the comprehensive similarity score. The temporal dependency analysis model performs smooth correction on the comprehensive similarity score of the current temporal node based on the dynamic matching relationship between the change trend of the security feature vector of adjacent temporal nodes and the feature vector of the detection instruction.

[0091] Step 419: Arrange the corrected comprehensive similarity scores in chronological order to generate an initial matching degree sequence. The value of each element in the initial matching degree sequence reflects the dynamic matching degree between the corresponding chronological node and the detection instruction feature vector, and the value changes of adjacent elements follow the contextual association constraints output by the chronological dependency analysis model.

[0092] In step 411, the detection instruction feature vector and the security feature vector of each time node are normalized respectively to generate the normalized detection instruction feature vector and the normalized time node security feature vector.

[0093] In step 412, based on the semantic coverage of the entity distribution vector in the detection instruction feature vector and the dynamic distribution of the task weight matrix, the weights of the normalized detection instruction feature vector are dynamically adjusted to generate a dynamically weighted detection instruction feature vector.

[0094] In step 413, the dynamically weighted detection instruction feature vector is compared with the normalized security feature vector of each time-series node using a multi-dimensional similarity comparison. This multi-dimensional similarity comparison includes three independent calculation paths: semantic coverage overlap, entity distribution direction consistency, and task weight sensitivity. These three calculation paths measure the similarity between the detection instruction feature vector and the security feature vector from different perspectives, providing a more comprehensive reflection of their matching degree. Semantic coverage overlap focuses on the overlap between the entities and concepts involved in the detection instruction and those contained in the security feature vector; entity distribution direction consistency examines whether the orientation of entity distribution in the detection instruction feature vector and the security feature vector is consistent; and task weight sensitivity considers the degree of matching between the task priority in the detection instruction and the risk sensitivity in the security feature vector. Through these three independent calculation paths, similarity can be evaluated from multiple dimensions, avoiding the limitations of a single-dimensional evaluation.

[0095] In step 414, for the semantic coverage overlap calculation path, the entity association network topology features in the dynamically weighted detection instruction feature vector are extracted, and regional overlap analysis is performed with the entity spatial distribution features in the normalized safety feature vector to generate a semantic coverage overlap score. The entity association network topology features describe the relationships and structure between entities in the detection instruction, such as the connection relationships between devices and the distribution relationships of devices in different areas of the workshop. The entity spatial distribution features reflect the spatial distribution of entities in the safety feature vector. Regional overlap analysis is performed on these two features to determine the degree of overlap between the entity region involved in the detection instruction and the region where the entities in the safety feature vector are located.

[0096] In step 415, for the entity distribution direction consistency calculation path, the direction projection calculation is performed on the entity distribution vector in the dynamically weighted detection instruction feature vector and the entity distribution vector in the normalized security feature vector to generate an entity distribution direction consistency score. The entity distribution vector represents the distribution of entities in space or feature dimensions. By performing direction projection calculation on two entity distribution vectors, it can be determined whether their distribution directions are consistent. If the directions of the two vectors are relatively consistent, it indicates that the entity distribution is similar, and the entity distribution direction consistency score is high.

[0097] In step 416, for the task weight sensitivity calculation path, the task weight matrix in the dynamically weighted detection instruction feature vector and the risk sensitivity matrix in the normalized safety feature vector are multiplied element-wise to generate a task weight sensitivity score. In safety production inspection, the detection instructions clearly define the importance of different tasks, which is reflected in the task weight matrix. The risk sensitivity matrix in the safety feature vector reflects the risk sensitivity of different tasks at each time point. If a task has a large weight in the task weight matrix and a high value in the risk sensitivity matrix, then the product result for that task will be large, and the task weight sensitivity score will be high. This indicates that the task has a good match with the detection instructions and is relatively sensitive to risk.

[0098] In step 417, the semantic coverage overlap score, entity distribution direction consistency score, and task weight sensitivity score are input into the pre-trained fusion decision model. The three scores are dynamically weighted and fused according to the task type priority in the detection instruction features to generate a comprehensive similarity score for each time-series node. These three scores reflect the degree of matching between the detection instruction feature vector and the security feature vector from different perspectives, but their importance may vary depending on the task type priority of the detection instruction. For example, if high-temperature equipment temperature detection is a high-priority task in the detection instruction, then the weights of the semantic coverage overlap score, entity distribution direction consistency score, and task weight sensitivity score related to high-temperature equipment temperature will be relatively high during fusion. The pre-trained fusion decision model is trained based on a large amount of historical data and can assign appropriate weights to the three scores according to the task type priority in the detection instruction features, and then perform a weighted summation.

[0099] In step 418, the temporal dependency analysis model is invoked to perform context-based correction on the comprehensive similarity score. This model smooths the comprehensive similarity score of the current temporal node based on the dynamic matching relationship between the changing trends of the safety feature vectors of adjacent temporal nodes and the feature vectors of detection commands. In actual production processes, the data of each temporal node are not isolated; they exhibit temporal continuity and correlation. For example, when detecting the operating status of production equipment in workshop A, the equipment status of the current temporal node is usually correlated with the status of adjacent temporal nodes. If the safety feature vectors of adjacent temporal nodes show relatively stable equipment operating status, but the comprehensive similarity score of the current temporal node suddenly fluctuates significantly, there may be data errors or anomalies. The temporal dependency analysis model analyzes the changing trends of the safety feature vectors of adjacent temporal nodes and their matching relationship with the feature vectors of detection commands to adjust the comprehensive similarity score of the current temporal node. If the matching degree of adjacent nodes is high and the trend of change is stable, while the current node's score is too low, the model can appropriately increase its score; conversely, if the matching degree of adjacent nodes is low and the trend of deterioration is strong, while the current node's score is too high, the model can decrease its score. This makes the overall similarity score more consistent with actual production conditions and avoids misjudgments caused by individual abnormal data.

[0100] In step 419, the corrected comprehensive similarity scores are arranged in chronological order to generate an initial matching degree sequence. The value of each element in the initial matching degree sequence reflects the dynamic matching degree between the corresponding time-series node and the detection instruction feature vector, and the changes in the values ​​of adjacent elements follow the contextual constraints output by the time-series dependency analysis model. The comprehensive similarity scores, after contextual correction, are arranged sequentially in chronological order to form a sequence. This sequence visually demonstrates how the matching degree between each time-series node and the detection instruction changes over time. For example, in equipment inspection in workshop A, the initial matching degree sequence clearly shows the degree of compliance between the equipment's operating status and the requirements of the detection instruction at different time points. Because the changes in the values ​​of adjacent elements follow the contextual constraints output by the time-series dependency analysis model, this sequence not only reflects the independent matching of each time-series node but also demonstrates the continuity and correlation of the data in the time dimension. By analyzing this initial matching degree sequence, key data segments related to the detection instruction can be more accurately selected, providing strong data support for subsequent anomaly pattern recognition and risk assessment.

[0101] As one implementation method, step 500 involves calling a pre-trained security decision model to perform anomaly pattern recognition and risk assessment on critical data segments, generating a structured security report corresponding to the detection instructions, including:

[0102] Step 510: Perform multi-granularity slicing on the key data segments to generate hierarchical data blocks containing global state features and local anomaly features;

[0103] Step 520: Input the hierarchical data blocks into the convolutional recurrent network of the security decision model for spatiotemporal feature extraction;

[0104] Step 530: The extracted spatiotemporal features are compared with the preset abnormal pattern library through the pattern matching layer in the security decision model;

[0105] Step 540: Generate risk assessment indicators that include risk level, anomaly type, and scope of impact based on the comparison results;

[0106] Step 550: Generate a structured security report containing visualization charts and handling suggestions based on the task type in the risk assessment indicators and detection instruction characteristics.

[0107] In step 510, the key data segment is sliced ​​at multiple granularities to generate hierarchical data blocks containing both global state features and local anomaly features. The key data segment contains various data related to equipment operation over a period of time, and this data is analyzed at different granularities. For example, the key data segment for a production machine in workshop A might contain continuous data on multiple parameters such as temperature, pressure, and rotational speed. This data is sliced ​​according to different time scales. For instance, data over a longer period can be analyzed as global state features to understand the overall operating trend of the equipment; simultaneously, data over a shorter period can be mined as local anomaly features to identify potential sudden anomalies. This multi-granularity slicing generates data blocks at different levels, each containing equipment operating information at a specific granularity. A sliding window technique can be used to implement multi-granularity slicing, allowing data blocks at different time scales to be obtained by adjusting the window size and step size.

[0108] In step 520, the hierarchical data blocks are input into the convolutional recurrent network (CRN) of the safety decision model for spatiotemporal feature extraction. The CRN combines the advantages of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), effectively processing spatial features and temporal series information in the data. For global state features in the hierarchical data blocks, the convolutional layers can extract spatial features, such as the correlation between different parameters of the equipment. The recurrent layers, on the other hand, can process the temporal series information of the data, capturing the changing trends of the equipment's operating status over time. For local anomaly features, the CRN can more sensitively detect sudden changes and abnormal patterns in the data. For example, when analyzing equipment temperature data, the CRN can identify anomalies such as sudden increases or decreases in temperature. By inputting the hierarchical data blocks into the CRN, key spatiotemporal features related to abnormal patterns and risks can be extracted.

[0109] In step 530, the extracted spatiotemporal features are compared with a pre-defined anomaly pattern library using the pattern matching layer in the safety decision model. The pre-defined anomaly pattern library stores various known safety anomalies and their corresponding feature patterns. The extracted spatiotemporal features are compared one by one with the patterns in the anomaly pattern library, and the similarity between them is calculated. For example, the anomaly pattern library may contain feature patterns for anomalies such as equipment overheating and excessive pressure. If the extracted spatiotemporal features have a high similarity to a certain anomaly pattern, then the corresponding anomaly may exist. For example, methods such as cosine similarity can be used to calculate the similarity between the spatiotemporal features and the anomaly patterns.

[0110] In step 540, a risk assessment index is generated based on the comparison results, including risk level, anomaly type, and scope of impact. By comparing spatiotemporal characteristics with the anomaly pattern library, it can be determined whether an anomaly exists and its risk level can be further assessed. If the similarity exceeds a preset threshold, an anomaly is identified. The risk level is determined based on the severity of the anomaly and its potential impact, such as high, medium, or low. Simultaneously, the anomaly type is determined based on the matched anomaly pattern, such as equipment failure or operational error. Furthermore, the potential scope of the anomaly's impact is analyzed, such as whether it affects only a single piece of equipment or extends to surrounding equipment and personnel. By comprehensively considering these factors, a risk assessment index including risk level, anomaly type, and scope of impact is generated, providing a basis for subsequent decision-making.

[0111] In step 550, a structured safety report containing visual charts and remedial recommendations is generated based on the risk assessment indicators and the task type in the inspection instructions. The visual charts intuitively display the risk situation, helping managers quickly understand the safety status of the equipment. For example, a line graph can be used to show the trend of equipment parameters over time, and a bar chart can be used to compare the distribution of different risk levels. The remedial recommendations provide specific countermeasures based on the risk assessment results and the requirements of the inspection instructions. If the inspection instructions require a focus on the equipment's temperature and pressure, and the risk assessment indicators show that the equipment temperature is too high, then the remedial recommendations might include reducing the equipment load and checking the cooling system. By integrating the visual charts and remedial recommendations into the structured safety report, the report's format and content are customized according to the task type of the inspection instructions to meet the needs of different users.

[0112] As one implementation method, the aforementioned collaborative optimization method for the jointly trained multimodal feature extraction model and the security decision model includes the following steps 10-70:

[0113] Step 10: Input the spatial feature sequence output by the visual encoder, the spectral feature sequence output by the acoustic encoder, and the working condition feature sequence output by the state encoder into the cross-modal attention network to generate a time-aligned multimodal joint feature sequence.

[0114] Step 20: Match the multimodal joint feature sequence with the equipment failure type labels in the labeled safety events using timestamps to generate a joint training sample set with time-series annotations;

[0115] Step 30: Extract features from the joint training sample set through the spatiotemporal convolutional layer in the security decision model, and output the predicted risk level and probability distribution of anomaly type;

[0116] Step 40: Compare the predicted risk level with the environmental risk level label in the labeled safety event node by node to generate the first error gradient. At the same time, calculate the cross-entropy between the probability distribution of the anomaly type and the equipment failure type label to generate the second error gradient.

[0117] Step 50: Input the multimodal joint feature sequence into the discriminator network, calculate the distribution difference between it and the feature template extracted from real security events, and generate the third error gradient;

[0118] Step 60: Based on the weighted fusion results of the first error gradient, the second error gradient, and the third error gradient, generate the joint optimization direction of the multimodal feature extraction model and the security decision model;

[0119] Step 70: Synchronously adjust the weight parameters of the cross-modal attention network, the kernel parameters of the spatiotemporal convolutional layer, and the discrimination threshold of the discriminator network according to the joint optimization direction, until the fusion value of the first error gradient and the second error gradient reaches the preset convergence condition.

[0120] In step 10, the spatial feature sequence output by the visual encoder, the spectral feature sequence output by the acoustic encoder, and the condition feature sequence output by the state encoder are input into the cross-modal attention network to generate a time-aligned multimodal joint feature sequence. This method generates a time-aligned multimodal joint feature sequence, providing a more accurate data foundation for subsequent training and optimization.

[0121] In step 20, the multimodal joint feature sequence is timestamped with the equipment failure type labels in the labeled safety events to generate a joint training sample set with time-series annotations. The labeled safety events contain known equipment failure types and their corresponding time information. Each time point in the multimodal joint feature sequence is compared with the timestamp in the labeled safety event, and the multimodal joint features are associated with the corresponding equipment failure type label. For example, if the labeled safety event records a motor failure at a specific time point, the features corresponding to that time point in the multimodal joint feature sequence can be found and associated with the label "motor failure." Through this timestamp matching, a joint training sample set with time-series annotations is generated. This sample set contains multimodal features and corresponding equipment failure type labels, providing supervised learning data for model training.

[0122] In step 30, the spatiotemporal convolutional layer in the safety decision model extracts features from the joint training sample set, outputting the predicted risk level and probability distribution of anomaly types. The spatiotemporal convolutional layer effectively processes the spatiotemporal information in the joint training sample set, extracting features related to risk and anomalies. In the safety inspection of workshop A, the spatiotemporal convolutional layer can analyze the changes in multimodal joint feature sequences over time, as well as the spatial relationships between different modal features. For example, it can identify the changing trends of parameters such as equipment temperature and pressure over a period of time, and the correlation between these parameters and features such as equipment appearance and sound. Through the extraction and analysis of these features, the safety decision model outputs the predicted risk level, such as high, medium, and low levels, and the probability distribution of anomaly types, i.e., the likelihood of each anomaly type occurring. In this way, the current safety status of the equipment and possible anomalies can be preliminarily determined.

[0123] In step 40, the predicted risk level is compared node-by-node with the environmental risk level label in the labeled safety event to generate a first error gradient. Simultaneously, the anomaly type probability distribution and the equipment failure type label are cross-entropy calculated to generate a second error gradient. The predicted risk level is the output of the safety decision model, while the environmental risk level label in the labeled safety event represents the known actual situation. The two are compared node-by-node, and the error at each node is calculated. For example, if the predicted risk level is low, but the actual environmental risk level label is high, then an error exists. By statistically analyzing the errors at all nodes, the first error gradient is generated, reflecting the degree of difference between the predicted and actual risk levels. Simultaneously, the anomaly type probability distribution and the equipment failure type label are cross-entropy calculated to generate a second error gradient, reflecting the difference between the predicted anomaly type probability distribution and the actual situation.

[0124] In step 50, the multimodal joint feature sequence is input into the discriminator network, and the distribution difference between it and the feature templates extracted from real security events is calculated to generate a third error gradient. The discriminator network determines whether the input multimodal joint feature sequence originates from a real security event. The feature templates extracted from real security events represent typical feature distributions under normal and abnormal conditions. The multimodal joint feature sequence is compared with these feature templates, and the distribution difference between them is calculated. For example, methods such as maximum mean difference (MMD) can be used to measure the distribution difference. Through this comparison, a third error gradient is generated, which reflects the degree of difference between the multimodal joint feature sequence and the feature templates of real security events. This error gradient can help adjust the multimodal feature extraction model so that the generated features are closer to the features of real security events.

[0125] In step 60, based on the weighted fusion result of the first, second, and third error gradients, a joint optimization direction for the multimodal feature extraction model and the security decision model is generated. These three error gradients reflect the model's performance from different perspectives. The first error gradient focuses on the accuracy of predicting risk levels, the second on the accuracy of predicting anomaly types, and the third on the degree of matching between multimodal features and real security event features. Different weights are assigned to these three error gradients, and then they are weighted and fused. For example, if the current security detection task prioritizes the accuracy of predicting risk levels, the weight of the first error gradient might be larger. Through weighted fusion, a comprehensive error gradient is obtained. Based on the direction of this error gradient, the joint optimization direction of the multimodal feature extraction model and the security decision model is determined, i.e., the direction in which the model parameters need to be adjusted.

[0126] In step 70, the weight parameters of the cross-modal attention network, the kernel parameters of the spatiotemporal convolutional layer, and the discrimination threshold of the discriminator network are simultaneously adjusted according to the joint optimization direction until the fusion value of the first and second error gradients reaches the preset convergence condition. The cross-modal attention network is responsible for the fusion of multimodal features, the spatiotemporal convolutional layer is used for feature extraction in the safety decision model, and the discriminator network is used to determine the authenticity of features. The parameters of these networks are adjusted synchronously according to the joint optimization direction. For example, when the joint optimization direction indicates a need to improve the fusion effect of multimodal features, the weight parameters of the cross-modal attention network can be adjusted to better fuse features from different modalities. During the adjustment process, the fusion value of the first and second error gradients can be continuously calculated and compared with the preset convergence condition. The preset convergence condition can be an error threshold; when the fusion value is less than this threshold, it indicates that the model's performance has reached a certain requirement, and the training process can stop. Through this iterative adjustment, the performance of the multimodal feature extraction model and the safety decision model is continuously optimized, enabling them to work together better and improving the accuracy and reliability of safety production detection.

[0127] Throughout the collaborative optimization process, it is crucial to accurately process various data and error gradients, and to appropriately adjust model parameters. For calculating error gradients, suitable mathematical methods and algorithms must be employed to ensure the accuracy of the results. When adjusting model parameters, optimization algorithms, such as stochastic gradient descent (SGD) or its variants, should be used to improve the efficiency and stability of the adjustment. Simultaneously, the training process needs to be monitored and evaluated to promptly identify and address any issues.

[0128] To improve the effectiveness of collaborative optimization, several optimization strategies can be employed. For example, batch normalization can be used to accelerate model convergence and reduce training time. Additionally, model fusion can be used to combine multiple different models, thereby improving their generalization ability and performance. During training, data augmentation techniques can be used to expand the joint training sample set, increasing sample diversity and thus enhancing the model's robustness.

[0129] Steps 10-70 provide a complete method and process for the collaborative optimization of the multimodal feature extraction model and the safety decision model. Through operations such as time alignment, sample labeling, feature extraction, error calculation, gradient fusion, and parameter adjustment, the performance and collaborative working ability of these two models can be continuously improved, providing more accurate and reliable support for safety production detection.

[0130] As one implementation method, the method provided in this embodiment of the invention may further include:

[0131] Step 61: Divide the first error gradient into risk level intervals to generate high-risk interval error components and normal interval error components;

[0132] Step 62: Perform anomaly type clustering analysis on the second error gradient to generate core fault type error components and edge anomaly type error components;

[0133] Step 63: Based on the task type priority in the detection instruction features, assign enhanced weighting coefficients to the error components of the high-risk interval and the error components of the core fault type;

[0134] Step 64: Input the weighted first error gradient and the second error gradient into the dynamic fusion controller to generate the main optimization direction of the model parameters;

[0135] Step 65: Align the third error gradient input distribution with the controller to generate the sub-optimization direction for feature distribution optimization;

[0136] Step 66: Coordinate the adjustment order of the main optimization direction and the secondary optimization direction through a priority scheduling strategy. In the early stage of training, the secondary optimization direction is executed first to improve feature alignment accuracy, and in the later stage of training, the main optimization direction is switched to enhance the risk prediction capability.

[0137] Step 67: Dynamically control the parameter update frequency of the cross-modal attention network and the security decision model based on the output results of the scheduling strategy, so as to enable the co-evolution of multimodal feature extraction and risk prediction.

[0138] In step 61, the first error gradient is divided into risk level intervals, generating a high-risk interval error component and a normal interval error component. The first error gradient reflects the difference between the predicted risk level and the environmental risk level label in the safety event annotation. In the safety inspection of workshop A, the risk level can usually be divided into different levels such as high, medium, and low. According to the preset risk level interval standard, the first error gradient is divided into a high-risk interval error component and a normal interval error component. For example, when there is a difference between the predicted risk level and the actual environmental risk level label in the high-risk level interval, this part of the error will be classified as the high-risk interval error component; while the errors in the medium and low risk level intervals will be classified as the normal interval error component. This classification helps to deal with the errors under different risk levels more effectively, because the errors of the high-risk level may have a more serious impact on safe production and require more attention.

[0139] In step 62, anomaly type clustering analysis is performed on the second error gradient to generate core fault type error components and marginal anomaly type error components. The second error gradient is calculated by cross-entropy using the anomaly type probability distribution and equipment fault type labels; it reflects the difference between the predicted and actual anomaly types. In workshop A, equipment may exhibit various anomaly types, such as motor failures, sensor failures, and pipeline leaks. Clustering analysis is performed on these anomaly types, classifying common anomalies with significant production impact as core fault types and relatively rare anomalies with less impact as marginal anomalies. Then, based on the clustering results, the second error gradient is divided into core fault type error components and marginal anomaly type error components. For example, motor failure is a common and important anomaly type in workshop A, so errors related to motor failure are classified as core fault type error components; while errors from occasional minor component damage are classified as marginal anomaly type error components. This classification allows for greater focus on the identification and prediction of core fault types during the optimization process.

[0140] In step 63, based on the task type priority in the detection instruction features, enhanced weighting coefficients are assigned to the high-risk interval error component and the core fault type error component. The task type priority in the detection instruction features reflects the importance of different detection tasks in safe production. In the safety inspection of workshop A, if the detection instruction focuses on the operating status of high-risk equipment and the identification of core fault types, then higher weighting coefficients will be assigned to the high-risk interval error component and the core fault type error component.

[0141] In step 64, the weighted first and second error gradients are input into the dynamic fusion controller to generate the main optimization direction of the model parameters. The dynamic fusion controller is a module for comprehensively processing error gradients. It can fuse the weighted first and second error gradients according to the characteristics and weights of different error gradients. In the scenario of workshop A, the error components of the high-risk interval, the normal interval, the core fault type, and the marginal anomaly type, after weight adjustment, are comprehensively considered. The dynamic fusion controller generates a comprehensive error gradient, which represents the main optimization direction for adjusting the model parameters. The dynamic fusion controller can adopt a multi-layer neural network structure. The input layer receives the weighted first and second error gradients, which reflect the errors in predicting the risk level and the probability distribution of the anomaly type, respectively. Multiple hidden layers are set in the middle, using activation functions such as ReLU for nonlinear transformation to enhance the model's expressive power. The output layer outputs the fused error gradient as the main optimization direction of the model parameters. In implementation, the network weights are first initialized, and then the gradient descent method is used to iteratively update the weights based on the training data. The fusion method is continuously adjusted so that the main optimization direction of the output can effectively guide the adjustment of model parameters and improve model performance.

[0142] In step 65, the third error gradient is input to the distribution alignment controller to generate a secondary optimization direction for feature distribution optimization. The third error gradient reflects the distribution difference between the multimodal joint feature sequence and the feature template extracted from real security events. The role of the distribution alignment controller is to adjust the multimodal feature extraction model based on the third error gradient, making the distribution of the generated multimodal joint feature sequence closer to the feature template of real security events. In workshop A, different security events may correspond to different feature distributions. The distribution alignment controller determines the direction of feature distribution optimization, i.e., the secondary optimization direction, based on the third error gradient. For example, if the multimodal joint feature sequence differs significantly from the distribution of the real feature template in certain feature dimensions, the distribution alignment controller will instruct the model to adjust its parameters to make the distribution in these feature dimensions more consistent. The distribution alignment controller can be implemented using a neural network structure. Its input is the third error gradient, which reflects the distribution difference between the multimodal joint feature sequence and the feature template of real security events. The network can consist of an input layer, a hidden layer, and an output layer. The input layer receives the error gradient data, the hidden layer performs nonlinear transformations, and increases the model's expressive power through activation functions such as ReLU. The output layer outputs the direction and magnitude of the adjustment parameters. In implementation, gradient descent is used to update network weights to minimize the distribution difference between the joint multimodal features and the real template. Continuous iterative training gradually aligns the distribution of multimodal features with the real template, improving the model's feature extraction accuracy.

[0143] In step 66, a priority scheduling strategy coordinates the adjustment order of the primary and secondary optimization directions. In the early stages of training, the secondary optimization direction is prioritized to improve feature alignment accuracy, while in the later stages, the primary optimization direction is switched to enhance risk prediction capabilities. This priority scheduling strategy aims to rationally arrange the adjustment order of the primary and secondary optimization directions to achieve better optimization results. In the early stages of training, the features generated by the multimodal feature extraction model may differ significantly from the feature templates of real security events. In this case, the secondary optimization direction is prioritized, adjusting the multimodal feature extraction model to make the feature distribution closer to the real template, thus improving feature alignment accuracy. For example, at the beginning of security detection model training, the focus is on the matching degree between the multimodal joint feature sequence and the real feature template, adjusting model parameters through a distribution alignment controller. In the later stages of training, when the feature alignment accuracy reaches a certain level, the primary optimization direction can be switched to, enhancing the model's risk prediction capabilities based on the weighted first and second error gradients, enabling the model to more accurately predict risk levels and anomaly types.

[0144] In step 67, the parameter update frequency of the cross-modal attention network and the safety decision model is dynamically controlled based on the output of the scheduling strategy, enabling the co-evolution of multimodal feature extraction and risk prediction. The scheduling strategy determines the adjustment order of the primary and secondary optimization directions, allowing for dynamic control of the parameter update frequency of the cross-modal attention network and the safety decision model. In the early stages of training, prioritizing the secondary optimization direction allows for more frequent parameter updates to the cross-modal attention network, improving feature alignment accuracy. For example, increasing the number of parameter updates in the cross-modal attention network allows for faster adjustment of the fusion method of different modal features, making the multimodal joint feature sequence closer to the true feature template. Later in training, when switching to the primary optimization direction, more attention can be paid to updating the parameters of the safety decision model to enhance risk prediction capabilities. This dynamic control enables the co-evolution of multimodal feature extraction and risk prediction, ensuring effective optimization of both models at different training stages and ultimately improving the performance of the entire safety production detection system.

[0145] As one implementation method, the method provided in this embodiment of the invention may further include:

[0146] Step 71: After each parameter update, extract the updated multimodal joint feature sequence and risk prediction results;

[0147] Step 72: Input the updated multimodal joint feature sequence into the discriminator network to generate a new distribution dissimilarity index;

[0148] Step 73: Compare the updated risk prediction results with the labeled safety events to generate new risk level errors and anomaly type errors;

[0149] Step 74: Calculate the comprehensive optimization efficiency index for the current training phase based on the new distribution difference index, risk level error, and anomaly type error;

[0150] Step 75: When the overall optimization efficiency index fails to improve for N consecutive training cycles, trigger the weight coefficient reset mechanism of the dynamic fusion controller;

[0151] Step 76: During the weight coefficient reset process, the fusion weights of the first error gradient and the second error gradient are reallocated according to the error distribution pattern in the historical training data;

[0152] Step 77: Synchronize the reset fusion weights to the priority scheduling strategy and redefine the switching conditions for primary and secondary optimization directions;

[0153] Step 78: The optimized state information is fed back to the cross-modal attention network through the feedback channel of the discriminator network, forming a closed-loop optimization circuit from feature extraction to risk prediction and then to parameter adjustment.

[0154] In step 71, after each parameter update, the updated multimodal joint feature sequence and risk prediction results are extracted. The multimodal joint feature sequence is formed by fusing features from multiple modalities such as vision, acoustics, and equipment status, containing rich information about the equipment's operating status. The risk prediction results are the judgments about the equipment's risk level and anomaly type output by the safety decision model based on these feature sequences. After each update to the parameters of the cross-modal attention network, spatiotemporal convolutional layer, and discriminator network, these updated information can be recalculated and extracted. For example, after updating the weight parameters of the cross-modal attention network, the fusion method of different modal features changes, and the multimodal joint feature sequence will also change accordingly, allowing for timely extraction of new sequences. Simultaneously, the safety decision model performs risk prediction based on the new feature sequences, obtaining updated risk prediction results. This information will be used for subsequent evaluation and optimization.

[0155] In step 72, the updated multimodal joint feature sequence is input into the discriminator network to generate a new distribution dissimilarity index. The discriminator network determines whether the input multimodal joint feature sequence originates from a real security event. The feature templates extracted from real security events represent typical feature distributions under normal and abnormal conditions. The updated multimodal joint feature sequence is compared with these feature templates to calculate the distribution dissimilarity between them. For example, methods such as maximum mean difference (MMD) can be used to measure distribution dissimilarity, with the formula: Where P and Q are the distributions of the multimodal joint feature sequence and the true feature template, respectively. This is a mapping function that maps data to the Regenerative Kernel Hilbert Space (RKHS), where m and n are the number of samples in the two distributions, respectively. Through this comparison, a new distribution dissimilarity index is generated, reflecting the degree of difference between the updated multimodal joint feature sequence and the feature template of a real security event.

[0156] In step 73, the updated risk prediction results are compared with the labeled safety events to generate new risk level errors and anomaly type errors. The labeled safety events include known equipment failure types and environmental risk level labels, representing actual situations. The updated risk prediction results are compared node-by-node with the labeled safety events. For risk levels, the difference between the predicted risk level and the actual environmental risk level label is calculated to obtain a new risk level error. For anomaly types, the difference between the predicted anomaly type probability distribution and the equipment failure type label is calculated, and a new anomaly type error is obtained through cross-entropy calculation. The cross-entropy formula is: Here, p(i) is the probability distribution of the actual equipment failure type labels, and q(i) is the probability distribution of the predicted anomaly types. This comparison yields new risk level errors and anomaly type errors, which reflect the predictive accuracy of the updated safety decision-making model.

[0157] In step 74, the comprehensive optimization efficiency index for the current training phase is calculated based on the new distribution dissimilarity index, risk level error, and anomaly type error. The comprehensive optimization efficiency index is a metric that comprehensively measures the model's optimization performance, considering both the accuracy of multimodal feature extraction and the predictive accuracy of the safety decision-making model. Different weights can be assigned to the new distribution dissimilarity index, risk level error, and anomaly type error, and then a weighted sum is performed to obtain the comprehensive optimization efficiency index.

[0158] In step 75, when the overall optimization efficiency index fails to improve for N consecutive training epochs, the weight coefficient reset mechanism of the dynamic fusion controller is triggered. During training, the overall optimization efficiency index should gradually increase as training progresses. If the index fails to improve for N consecutive training epochs, it indicates that the current optimization strategy may have gotten stuck in a local optimum or encountered other problems. At this point, the weight coefficient reset mechanism of the dynamic fusion controller can be triggered. The dynamic fusion controller is responsible for fusing the weighted first error gradient with the second error gradient to generate the main optimization direction of the model parameters. The weight coefficient reset mechanism readjusts the weights of these error gradients to change the optimization direction, attempt to escape local optima, and allow the model to continue optimizing towards better performance.

[0159] In step 76, during the weight coefficient reset process, the fusion weights of the first and second error gradients are reallocated based on the error distribution patterns in the historical training data. Historical training data records the model's errors at different training stages, including risk level errors and anomaly type errors. The distribution patterns of these errors are analyzed, such as which types of errors occur more frequently in certain stages and which errors have a greater impact on model performance. Based on these analysis results, the fusion weights of the first and second error gradients are reallocated. For example, if historical data shows that risk level errors are large and have a significant impact on model performance at a certain stage, then during the weight coefficient reset, the weight of the first error gradient can be increased, making the model focus more on the accuracy of risk level prediction.

[0160] In step 77, the reset fusion weights are synchronized to the priority scheduling strategy, and the switching conditions for primary and secondary optimization directions are redefined. The priority scheduling strategy coordinates the adjustment order of primary and secondary optimization directions. In the early stages of training, secondary optimization directions are prioritized to improve feature alignment accuracy, while in the later stages, the focus shifts to the primary optimization direction to enhance risk prediction capabilities. The reset fusion weights change the emphasis of the primary optimization direction, therefore these weights need to be synchronized to the priority scheduling strategy. Based on the new fusion weights, the switching conditions for primary and secondary optimization directions are redefined. For example, if the weight of the first error gradient increases after the reset, it may switch to the primary optimization direction earlier, allowing the model to focus on improving risk prediction capabilities sooner.

[0161] In step 78, the optimized state information is fed back to the cross-modal attention network through the feedback channel of the discriminator network, forming a closed-loop optimization loop from feature extraction to risk prediction and then to parameter adjustment. During the calculation of the distribution difference index, the discriminator network accumulates information about the differences between the multimodal joint feature sequence and the true feature template. This optimized state information is fed back to the cross-modal attention network through the feedback channel. The cross-modal attention network is responsible for fusing multimodal features. Based on this feedback information, it can adjust the weight allocation of different modal features, making the generated multimodal joint feature sequence closer to the feature template of a real security event. In this way, a closed-loop optimization loop is formed, from feature extraction to risk prediction, then parameter adjustment based on the prediction results, and finally the adjustment information is fed back to the feature extraction stage, continuously optimizing and improving the overall performance of the model.

[0162] Steps 71-78 construct a closed-loop optimization circuit in the collaborative optimization of the multimodal feature extraction model and the safety decision model. Through operations such as extracting and updating information, calculating indicators, resetting weights, adjusting strategies, and feedback optimization, the performance of the model can be continuously optimized, improving the accuracy and reliability of safety production detection.

[0163] As one implementation method, the method provided in this embodiment of the invention may further include: pushing the generated structured security report to the target user terminal and recording the user operation log; extracting the user's modifications to the security report and the confirmation response time as feedback signals; converting the feedback signals into model optimization reward values ​​through a reinforcement learning framework; fine-tuning the risk assessment strategy in the security decision model online based on the reward values; and establishing a mapping relationship library between feedback data and original detection instructions for subsequent semantic parsing optimization of detection instructions.

[0164] This implementation constitutes a model optimization mechanism based on user feedback. By collecting and analyzing user feedback on structured security reports, the security decision model is fine-tuned online, and the semantic parsing of subsequent detection instructions is optimized.

[0165] Specifically, the generated structured security report is pushed to the target user terminals and user operation logs are recorded. In the safety inspection scenario of Workshop A, after completing the anomaly pattern identification and risk assessment of key data segments, a structured security report containing visual charts and handling suggestions is generated. This report presents detailed information such as the equipment's risk level, anomaly type, and scope of impact, as well as handling suggestions for different anomaly situations. This report is pushed to relevant management personnel, technical personnel, and other target user terminals, which can be devices such as computers and mobile phones. Simultaneously, user operation logs for the report are recorded, including when the user opened the report, what content they viewed, and whether they printed or forwarded it. These operation logs will provide data support for subsequent analysis of user behavior and feedback.

[0166] The generated structured security report is pushed to target user terminals and user operation logs are recorded. In the safety inspection scenario of Workshop A, after completing the anomaly pattern identification and risk assessment of key data segments, a structured security report containing visual charts and handling suggestions is generated. This report presents detailed information such as the equipment's risk level, anomaly type, and scope of impact, as well as handling suggestions for different anomaly situations. This report is pushed to relevant management personnel, technical personnel, and other target user terminals, which can be devices such as computers and mobile phones. Simultaneously, user operation logs for the report are recorded, including when the user opened the report, what content they viewed, and whether they printed or forwarded it. These operation logs will provide data support for subsequent analysis of user behavior and feedback.

[0167] Extract user modifications to security reports and their confirmation response times as feedback signals. After reviewing a structured security report, users may modify the report content based on their experience and the actual situation, or confirm the risk assessments and handling recommendations in the report. Extract these modifications; for example, users may revise their judgments on certain anomaly types or add additional handling recommendations. Simultaneously, record the user's confirmation response time—the time interval from report submission to user confirmation. This time reflects the user's level of acceptance of the report and the efficiency of handling emergencies. For example, if a user quickly confirms the report within a short period, it indicates that the report content is likely accurate; conversely, if the user does not confirm for a long time or makes extensive modifications, it may indicate inaccuracies or incompleteness in the report. These modifications and confirmation response times will serve as important feedback signals for subsequent model optimization.

[0168] This approach utilizes a reinforcement learning framework to transform feedback signals into model optimization reward values. Reinforcement learning is a machine learning method that uses an agent to interact with its environment and learn the optimal policy based on the reward signals received from that environment. In this scenario, user feedback signals are treated as environmental feedback and converted into model optimization reward values ​​according to predefined rules. For example, if the user makes few modifications to the report and the confirmation response time is short, it indicates that the model's risk assessment and handling suggestions are relatively accurate, and a higher reward value can be given; conversely, if the user makes many modifications and the confirmation response time is long, it indicates that the model's performance needs improvement, and a lower reward value can be given. In this way, user feedback is transformed into quantifiable reward signals to guide the optimization of the security decision-making model.

[0169] The risk assessment strategy in the security decision-making model is fine-tuned online based on reward values. The risk assessment strategy of the security decision-making model determines how to judge the risk level and anomaly type of a device based on key input data segments. These strategies are adjusted according to reward values. If the reward value is high, it indicates that the current risk assessment strategy is relatively effective and can be maintained to a certain extent; if the reward value is low, the risk assessment strategy can be adjusted. For example, the thresholds in the anomaly pattern library can be adjusted, or the weights of different features in the risk assessment can be changed. Through this online fine-tuning, the security decision-making model can continuously adapt to actual conditions and improve the accuracy of risk assessment.

[0170] A mapping library is established between feedback data and original detection commands for subsequent semantic parsing optimization of detection commands. Feedback data includes information such as user modifications, confirmation response time, and model optimization reward values. The original detection commands are the initial instructions issued to guide security detection. The mapping library associates feedback data with the original detection commands. For example, if a security report corresponding to a detection command receives a high reward value from the user, it indicates that the semantic parsing of that command is relatively accurate, and this experience can be referenced when processing similar commands in the future. If a report corresponding to a command receives a low reward value, it indicates that there may be a problem with the semantic parsing of the command, and adjustments need to be made in subsequent parsing. Through this mapping library, historical feedback information can be combined in the subsequent semantic parsing process of detection commands to optimize the parsing results and improve the accuracy of command understanding.

[0171] Steps 600-1000 provide a model optimization and instruction parsing optimization mechanism based on user feedback. Through operations such as pushing reports, collecting feedback, converting reward values, fine-tuning the model, and establishing a mapping relationship library, the performance of the safety decision model and the semantic parsing accuracy of detection instructions can be continuously improved, better serving safety production detection work.

[0172] As one implementation method, the method provided in this embodiment of the invention may further include: extracting keywords and semantic role labeling of the modified content to identify risk dimensions of concern to users; analyzing the correlation between user confirmation response time and report complexity to optimize the presentation of visualization charts; performing correlation analysis between the risk dimensions of concern to users and the features of the original detection instructions to generate an instruction understanding preference vector; adjusting the expansion range of the entity association network in conjunction with the instruction understanding preference vector during subsequent detection instruction parsing; and synchronizing the optimized parameters to the multimodal feature extraction model and the security decision model through an incremental learning algorithm.

[0173] Specifically, the first step is to extract keywords and perform semantic role labeling on the user's modifications to the structured safety report to identify the risk dimensions the user is concerned about. Keyword extraction involves finding the words that best represent the core content of the text. This can be achieved using the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm, whose formula is TF-IDF(t, d, D) = TF(t,d) × IDF(t, D), where TF(t, d) represents the frequency of word t in document d, and IDF(t, D) represents the inverse document frequency, used to measure the general importance of word t in the entire document set D. For example, in a structured safety report about chemical production safety, if the user modifies the content to "the temperature of the reactor has been too high recently, requiring close attention," the TF-IDF algorithm can be used to extract "reactor" and "temperature too high" as keywords. Semantic role labeling analyzes the semantic role played by each word in the sentence, such as agent, patient, time, and place, which can be achieved using a pre-trained semantic role labeling model. In the example above, the "reaction vessel" is the subject of the action, and "excessive temperature" describes the state. Through this analysis, we can identify that the user is concerned about the risk dimension related to the temperature of the reaction vessel.

[0174] Next, the correlation between user confirmation response time and report complexity is analyzed to optimize the presentation of visualizations. Report complexity can be measured by factors such as the number of charts, data dimensions, and text length. Confirmation response times for multiple users with reports of varying complexity can be recorded, and statistical methods such as regression analysis can be used to identify the relationship between the two. For example, if it is found that users take longer to confirm reports with many data dimensions and complex charts, data dimensionality reduction techniques, such as principal component analysis (PCA), can be used to reduce data dimensions and simplify charts. Alternatively, a hierarchical presentation approach can be adopted, first displaying key information and main conclusions, and then expanding on detailed data as needed. This improves user comprehension of the report and reduces confirmation response time.

[0175] Then, the risk dimensions that users focus on are correlated with the features of the original detection instructions to generate an instruction comprehension preference vector. The features of the original detection instructions include information such as entity distribution vectors and task weight matrices. The correlation between the risk dimensions that users focus on and the entity objects and task types in the detection instructions can be calculated, and the correlation values ​​are used as elements of the vector. For example, if the detection instruction is to inspect equipment safety and environmental safety in a chemical production workshop, and the risk dimension that users focus on is excessively high reactor temperature, analysis reveals that the reactor is an entity object in equipment safety inspection, and excessively high temperature is closely related to the equipment's operating status. Correlation calculation shows that this risk dimension has a correlation of 0.8 with the equipment safety inspection task and a correlation of 0.2 with the environmental safety inspection task. This is used to generate an instruction comprehension preference vector reflecting the user's preferences for different inspection tasks and entity objects.

[0176] In subsequent detection instruction parsing, the scope of the entity association network is adjusted based on the instruction understanding preference vector. The entity association network is generated by semantically expanding the set of entity objects input into a pre-trained domain knowledge graph during semantic parsing of the detection instructions. Based on the instruction understanding preference vector, for entities with high user interest, a broader semantic expansion is performed in the domain knowledge graph, increasing the number of associated entities and the types of relationships. For example, since the user previously focused on the temperature of a reactor, in subsequent detection instruction parsing related to chemical production safety, the entity object of the reactor is expanded in the domain knowledge graph to include more related entities such as heating systems, cooling systems, and temperature sensors, as well as their causal and dependency relationships. This allows the entity association network to more comprehensively cover the information the user might be interested in.

[0177] Finally, the optimized parameters are synchronized to the multimodal feature extraction model and the safety decision model using an incremental learning algorithm. Incremental learning refers to the ability of a model to learn and adapt to new data without forgetting its previous learning, building upon its existing knowledge of a subset of data. An online gradient descent algorithm can be used to update the weight parameters of both the multimodal feature extraction model and the safety decision model based on the optimized parameters. For example, in the multimodal feature extraction model, adjustments to the expansion range of the entity association network alter the feature distribution of the input data. Using an online gradient descent algorithm, the weight parameters of components such as the visual encoder, acoustic encoder, and state encoder are gradually adjusted based on the new feature distribution and the optimized parameters, enabling the model to better handle the new input data. Similarly, for the safety decision model, the parameters of components such as the convolutional recurrent network and pattern matching layer are updated based on the optimized parameters, improving the model's ability to identify abnormal patterns and assess risks. Through this incremental learning process, the co-evolution of multimodal feature extraction and risk prediction is achieved, enabling the model to better meet user needs and improve the accuracy and efficiency of safety production monitoring data processing.

[0178] By executing the above steps, we can fully utilize user feedback on structured safety reports, optimize the model and detection instruction parsing process, form a closed-loop optimization system, and continuously improve the performance of safety production detection data processing and user experience.

[0179] This invention provides a computer system, such as... Figure 2As shown, the computer system 100 includes a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, for example, via a bus 102. Optionally, the computer system 100 may also include a transceiver 104. It should be noted that in practical applications, the transceiver 104 is not limited to one type, and the structure of this computer system 100 does not constitute a limitation on the embodiments of the present invention.

[0180] This invention provides a computer system comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more programs are executed by the processors, they implement the method provided in this invention.

Claims

1. A method for processing safety production inspection data based on digitalization, characterized in that, include: Acquire security detection data in the target scenario and receive detection instructions for the security detection data, wherein the security detection data includes a continuous time-series data stream collected by multi-dimensional sensors; The detection instructions are semantically parsed to generate detection instruction features that include the relationship between task type and entity distribution; The security detection data is fused across modally using a jointly trained multimodal feature extraction model to generate a security feature vector for each time-series node. Specifically, this includes: splitting the security detection data into visual modal data, acoustic modal data, and device status modal data; using a visual encoder to extract spatial features from the visual modal data, generating visual features containing regional thermal distribution and motion trajectories; using an acoustic encoder to perform spectral decomposition and abnormal waveform detection on the acoustic modal data, generating acoustic energy distribution features; inputting the device status modal data into a status encoder for operating condition pattern recognition, generating a device operating status vector; and using a cross-modal attention network to align and fuse the visual features, acoustic energy distribution features, and device operating status vector to generate a security feature vector for each time-series node. Based on the dynamic matching results between the detection command features and the security feature vectors of each time-series node, key data segments are selected from the time-series data stream; The pre-trained security decision model is invoked to perform abnormal pattern recognition and risk assessment on the key data segment, and a structured security report corresponding to the detection instruction is generated. The collaborative optimization method for the jointly trained multimodal feature extraction model and the safety decision model includes: inputting the spatial feature sequence output by the visual encoder, the spectral feature sequence output by the acoustic encoder, and the working condition feature sequence output by the state encoder into a cross-modal attention network to generate a time-aligned multimodal joint feature sequence; matching the timestamps of the multimodal joint feature sequence with the equipment fault type labels in the labeled safety events to generate a joint training sample set with time-series annotations; extracting features from the joint training sample set through the spatiotemporal convolutional layer in the safety decision model to output the predicted risk level and anomaly type probability distribution; and performing node-by-node matching between the predicted risk level and the environmental risk level labels in the labeled safety events. The system compares the data to generate a first error gradient, and simultaneously calculates the cross-entropy of the anomaly type probability distribution and the equipment fault type label to generate a second error gradient. The multimodal joint feature sequence is input into the discriminator network, and the distribution difference between it and the feature template extracted from real security events is calculated to generate a third error gradient. Based on the weighted fusion result of the first, second, and third error gradients, a joint optimization direction for the multimodal feature extraction model and the security decision model is generated. According to the joint optimization direction, the weight parameters of the cross-modal attention network, the kernel parameters of the spatiotemporal convolutional layer, and the discrimination threshold of the discriminator network are adjusted synchronously until the fusion value of the first and second error gradients reaches a preset convergence condition.

2. The method as described in claim 1, characterized in that, The step of semantically parsing the detection command to generate detection command features containing the relationship between task type and entity distribution includes: The detection instruction is parsed using a syntax tree to extract the set of entity objects and the set of operation predicates from the detection instruction; The set of entity objects is input into a pre-trained domain knowledge graph for semantic expansion, generating an expanded entity association network. The intent is classified into the set of operation predicates to determine the task type priority of the detection instruction; The topological features of the entity association network and the task type priority are combined to generate multi-dimensional detection instruction features, wherein the detection instruction features include an entity distribution vector and a task weight matrix. The multi-dimensional detection command features are dynamically weighted using an attention mechanism to generate an optimized detection command feature vector.

3. The method as described in claim 2, characterized in that, The step of classifying the intent of the operation predicate set and determining the task type priority of the detection instruction includes: The set of operation predicates is standardized to generate a standardized set of operation predicates; The pre-trained intent recognition model is invoked to perform multi-granular semantic parsing on the standardized set of operation predicates, extract the logical dependencies and action triggering conditions between operation predicates, and generate an initial set of intent labels. Based on a preset safety production domain knowledge base, the initial intent tag set is semantically verified, and each tag in the initial intent tag set is matched with the standard operation scenario in the domain knowledge base to generate the verified intent tag confidence score. The initial intent tag set is dynamically weighted based on the confidence level of the verified intent tags, wherein intent tags with a matching degree higher than a preset verification threshold to the standard operation scenario are given high confidence weights, thereby generating a weighted intent tag set. The weighted intent tag set is input into a predefined task priority rule base. Based on the mapping relationship between security response level and operation urgency defined in the rule base, each intent tag is mapped to a corresponding priority weight. By combining the contextual features of the detection command, the set of entity objects involved in the detection command and the timeliness constraints of the operation are extracted, and the priority weight is dynamically corrected. The weighted intent label set is sorted based on the corrected priority weights to generate a task type priority queue. The order of each element in the queue reflects the urgency of the corresponding operation predicate in safety production detection, and the head element of the queue is marked as the core task type of the detection instruction.

4. The method as described in claim 1, characterized in that, The dynamic matching result based on the detection command features and the security feature vectors of each time-series node is used to filter out key data segments from the time-series data stream, including: Based on the dynamic matching results between the detection instruction feature vector and the security feature vector of each time node, the similarity score between each time node and the detection instruction feature vector is calculated to generate an initial matching degree sequence. A time window sliding analysis is performed on the initial matching degree sequence to identify candidate data segments in the time series data stream where the similarity scores of multiple consecutive time series nodes all exceed a dynamic threshold, thereby generating a set of candidate data segments. Based on the task type priority in the detection instruction features, the length weight of each candidate data segment in the candidate data segment set is adjusted to generate an optimized candidate data segment set. The optimized candidate data segment set is input into a pre-trained temporal correlation model to analyze the contextual dependency of each candidate data segment in the temporal data stream. Isolated candidate data segments with a correlation degree less than the correlation threshold with the preceding and following time nodes are removed, and candidate data segments with temporal overlap are merged to generate an optimized correlation data segment set. Based on the distribution density of each data segment in the associated optimized data segment set in the time-series data stream, density clustering is performed on the associated optimized data segment set, and data segments with a distribution density higher than a preset clustering threshold are selected as the final key data segment set. Each key data segment in the final key data segment set contains a preset minimum number of consecutive time-series nodes and covers the set of entity objects specified in the detection instruction features.

5. The method as described in claim 4, characterized in that, The process involves invoking a pre-trained security decision model to perform anomaly pattern recognition and risk assessment on the critical data segment, generating a structured security report corresponding to the detection command, including: The key data segments are sliced ​​at multiple granularities to generate hierarchical data blocks containing global state features and local anomaly features; The hierarchical data blocks are input into the convolutional recurrent network of the security decision model for spatiotemporal feature extraction; The extracted spatiotemporal features are compared with a preset abnormal pattern library through the pattern matching layer in the security decision model. Based on the comparison results, risk assessment indicators are generated that include risk level, anomaly type, and scope of impact. Based on the risk assessment indicators and the task type in the detection instruction characteristics, a structured security report containing visual charts and handling suggestions is generated.

6. The method as described in claim 4, characterized in that, The process of calculating the similarity score between each time-series node and the detection instruction feature vector based on the dynamic matching result between the detection instruction feature vector and the security feature vector of each time-series node, and generating an initial matching degree sequence, includes: The detection instruction feature vector and the security feature vector of each time node are normalized respectively to generate normalized detection instruction feature vector and normalized time node security feature vector. Based on the semantic coverage of the entity distribution vector in the detection instruction feature vector and the dynamic distribution of the task weight matrix, the weights of the normalized detection instruction feature vector are dynamically adjusted to generate a dynamically weighted detection instruction feature vector. The dynamically weighted detection instruction feature vector is compared with the normalized security feature vector of each time node in a multi-dimensional similarity comparison. The multi-dimensional similarity comparison includes three independent calculation paths: semantic coverage overlap, entity distribution direction consistency, and task weight sensitivity. For the semantic coverage overlap calculation path, the entity association network topology features in the dynamically weighted detection instruction feature vector are extracted, and regional overlap analysis is performed with the entity spatial distribution features in the normalized security feature vector to generate a semantic coverage overlap score. For the entity distribution direction consistency calculation path, the entity distribution vector in the dynamically weighted detection instruction feature vector and the entity distribution vector in the normalized security feature vector are subjected to directional projection calculation to generate an entity distribution direction consistency score. For the task weight sensitivity calculation path, the task weight matrix in the dynamically weighted detection instruction feature vector and the risk sensitivity matrix in the normalized security feature vector are multiplied element-wise to generate a task weight sensitivity score. The semantic coverage overlap score, entity distribution direction consistency score, and task weight sensitivity score are input into the pre-trained fusion decision model. The three scores are dynamically weighted and fused according to the task type priority in the detection instruction features to generate a comprehensive similarity score for each time node. The temporal dependency analysis model is invoked to perform context-related correction on the comprehensive similarity score. The temporal dependency analysis model performs smooth correction on the comprehensive similarity score of the current temporal node based on the dynamic matching relationship between the change trend of the security feature vector of adjacent temporal nodes and the feature vector of the detection instruction. The corrected comprehensive similarity scores are arranged in chronological order to generate an initial matching degree sequence. The value of each element in the initial matching degree sequence reflects the dynamic matching degree between the corresponding time node and the detection instruction feature vector, and the value changes of adjacent elements follow the contextual association constraints output by the time-series dependency analysis model.

7. The method as described in claim 1, characterized in that, The method further includes: The generated structured security report is pushed to the target user's terminal and user operation logs are recorded; Extract the user's modifications to the security report and the confirmation response time as feedback signals; The feedback signal is converted into a model optimization reward value using a reinforcement learning framework; The risk assessment strategy in the security decision-making model is fine-tuned online based on the reward value; Establish a mapping library between feedback data and original detection commands for semantic parsing and optimization of subsequent detection commands.

8. A computer system, characterized in that, include: One or more processors; Memory; One or more computer programs; The one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, they implement the method as described in any one of claims 1 to 7.

Images