Limited space operation violation judgment method and system based on multi-modal evidence representation

Abstract

The invention provides a limited space operation violation judgment method and system based on multi-mode evidence representation, and belongs to the field of operation safety monitoring. The method comprises the following steps: firstly, acquiring environment parameter data of limited space operation, behavior image data and health sign data of a worker and operation flow confirmation data, and generating a multi-modal original data set; secondly, preprocessing the multi-modal original data set to generate a standardized multi-modal data set; then carrying out multi-modal evidence representation on the set to generate three types of evidence representation results of environment, behavior and health; performing violation judgment through a preset multi-mode evidence fusion model based on the various evidence representation results, and generating violation judgment results; and finally, according to a violation judgment result, triggering an early warning signal and generating an emergency disposal suggestion. According to the method, violation judgment is carried out through multi-modal evidence representation, so that the concealment risk of multi-factor coupling is deeply mined, and the judgment accuracy of limited space operation violation is improved.

Description

Technical Field

[0001] This invention belongs to the field of operational safety monitoring technology, specifically relating to a method and system for determining violations in confined space operations based on multimodal evidence representation. Background Technology

[0002] Confined space work refers to work activities carried out in enclosed or partially enclosed, poorly ventilated spaces such as storage tanks, pipelines, and basements. Due to the special and complex nature of the environment, it is considered a high-risk type of work. To ensure the safety of workers, real-time monitoring and risk warning using information technology has become an important means. This type of technology typically involves the collection and processing of various types of data on site, particularly the analysis of image, sound, or sensor signals using pattern recognition technology to identify potential hazards or violations.

[0003] In existing technologies, safety monitoring methods for confined space operations often employ single-dimensional technical approaches. For example, some solutions utilize video surveillance equipment and image recognition algorithms to detect whether workers are wearing personal protective equipment such as safety helmets as required, or to identify specific dangerous actions such as falls. Other solutions focus on environmental monitoring, deploying gas sensor networks to monitor the concentration of oxygen and toxic gases in the space in real time, triggering alarms when measured values ​​exceed preset static thresholds.

[0004] However, the aforementioned existing technical solutions have significant technical shortcomings. Because various monitoring systems operate independently, their data processing and judgment logic is isolated, making it impossible to effectively integrate and correlate multi-source information. This leads to difficulties in identifying complex risks resulting from the combined effects of multiple factors such as environment and behavior. Furthermore, isolated judgment methods are prone to false alarms or missed alarms due to noise from single sensors or limitations of algorithms, and the lack of cross-validation mechanisms based on multi-source evidence results in insufficient reliability of the overall judgment. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for determining violations in confined space operations based on multimodal evidence representation. It aims to deeply explore the hidden risks of multi-factor coupling by performing collaborative pattern recognition and evidence fusion judgment on multimodal data such as environment, behavior and health, so as to achieve accurate and dynamic judgment of violations in confined space operations and improve the accuracy and reliability of safety monitoring of confined space operations.

[0006] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for determining violations in confined space operations based on multimodal evidence representation, comprising the following steps:

[0008] Acquire environmental parameter data, worker behavior image data and health sign data, and work process confirmation data for confined space operations to generate a multimodal raw data set;

[0009] The original multimodal dataset is preprocessed to generate a standardized multimodal dataset;

[0010] Multimodal evidence representation is performed on a standardized multimodal dataset to generate environmental evidence representation results, behavioral evidence representation results, and health evidence representation results. The multimodal evidence representation includes environmental evidence representation, behavioral evidence representation, and health evidence representation, and there is a dynamic triggering relationship between the three types of representation. The dynamic triggering relationship is that when any one type of evidence representation result meets the set conditions, the enhancement analysis, cross-validation, or collaborative analysis of the remaining evidence representations is triggered.

[0011] Based on the environmental evidence representation results, behavioral evidence representation results, and health evidence representation results, violation determination is performed through a pre-set multimodal evidence fusion model to generate violation determination results;

[0012] Based on the violation determination results, an early warning signal is triggered and emergency response suggestions are generated.

[0013] Furthermore, environmental parameter data, worker behavior image data, health sign data, and work process confirmation data are acquired for confined space operations to generate a multimodal raw dataset, including:

[0014] Environmental parameter data is acquired through pre-set sensors used for environmental monitoring;

[0015] Behavioral image data is acquired by a pre-set image acquisition device, and the behavioral image data is denoised and calibrated to generate calibrated behavioral image data.

[0016] Health data are obtained through pre-set devices used for health monitoring.

[0017] Work process confirmation data is obtained through a pre-set device for process recording;

[0018] By integrating environmental parameter data, calibrated behavioral image data, health sign data, and work process confirmation data, a multimodal raw data set is obtained.

[0019] Furthermore, the original multimodal dataset is preprocessed to generate a standardized multimodal dataset, including:

[0020] Environmental parameter data are normalized to generate standardized environmental data;

[0021] Low-light enhancement and feature extraction are performed on the calibrated behavioral image data to generate enhanced behavioral image data;

[0022] Sliding window smoothing is applied to health indicator data to generate smoothed health data.

[0023] Perform time-series alignment on the work process confirmation data to generate time-series aligned process data;

[0024] By integrating standardized environmental data, enhanced behavioral image data, smoothed health data, and time-aligned process data, a standardized multimodal data set is obtained.

[0025] Furthermore, multimodal evidence characterization is performed on the standardized multimodal dataset to generate environmental evidence characterization results, including:

[0026] Analyze environmental parameter data in a standardized multimodal dataset, identify environmental anomaly patterns, and generate environmental anomaly clustering results;

[0027] The Euclidean distance from the data points to the boundary of the normal operating condition cluster is calculated based on the results of the environmental anomaly clustering, the degree of deviation of environmental parameters is quantified, and the probability value of environmental violation is generated through a nonlinear mapping function.

[0028] When the probability of environmental violation exceeds a preset threshold, enhanced analysis of behavioral and health evidence representation is triggered. This integrates the probability of environmental violation, abnormal environmental parameter data, and the triggering status of dynamic triggering relationships to generate environmental evidence representation results.

[0029] Furthermore, multimodal evidence representation is performed on the standardized multimodal dataset to generate behavioral evidence representation results, including:

[0030] Target detection is performed on behavioral image data in a standardized multimodal dataset to identify the wearing status of safety equipment and generate equipment detection results;

[0031] Extract the spatiotemporal features of staff actions, identify dangerous behaviors, and generate behavioral anomaly scores;

[0032] By integrating equipment detection results and abnormal behavior scores, a probability value of behavioral violations is generated.

[0033] When the probability of a behavioral violation is higher than a preset threshold, the results of the associated health evidence representation are cross-validated, and the validation results are integrated to generate behavioral evidence representation results.

[0034] Furthermore, multimodal evidence characterization is performed on the standardized multimodal dataset to generate health evidence characterization results, including:

[0035] Analyze smoothed health data in a standardized multimodal dataset to generate health trend indicators;

[0036] The health trend indicators are compared with the personalized preset health baselines of the staff to generate a level of health abnormality.

[0037] The level of health abnormality is quantified into a probability value of health violation through a nonlinear mapping function;

[0038] When the probability of health violations increases, the environmental evidence representation and behavioral evidence representation from the same period are analyzed in a coordinated manner, and the results of the coordinated analysis are used to generate the health evidence representation result.

[0039] Furthermore, based on the environmental evidence representation results, behavioral evidence representation results, and health evidence representation results, a violation determination is performed using a pre-set multimodal evidence fusion model, generating violation determination results, including:

[0040] The results of environmental evidence characterization, behavioral evidence characterization, and health evidence characterization are mapped to the confidence levels of environmental evidence violations, behavioral evidence violations, and health evidence violations, respectively.

[0041] A dynamic weighted summation algorithm is used to fuse the violation confidence scores of the three types of evidence to generate an initial fused violation score;

[0042] The initial fusion violation score is adaptively adjusted in stages by combining the time sequence markers of the work process to generate violation judgment results; the stage adaptive adjustment is to dynamically adjust the weights of the three types of evidence in the dynamic weighted summation algorithm as the work process stages change.

[0043] Furthermore, based on the violation determination results, an early warning signal is triggered and emergency response suggestions are generated, including:

[0044] The violation determination results are sent to the preset monitoring platform in real time, triggering the audible and visual alarm signals of the corresponding work area;

[0045] Based on case-based reasoning technology, similar cases are matched in a historical accident case database, effective handling measures are extracted and adjusted according to the current scenario to generate emergency response suggestions;

[0046] Adjust the operating procedures according to emergency response recommendations.

[0047] Secondly, the present invention provides a system for determining violations in confined space operations based on multimodal evidence representation, comprising:

[0048] The data acquisition module is used to acquire environmental parameter data, worker behavior image data and health sign data, and work process confirmation data for confined space operations, and generate a multimodal raw data set;

[0049] The preprocessing module is used to preprocess the original multimodal dataset to generate a standardized multimodal dataset.

[0050] The evidence characterization module is used to perform multimodal evidence characterization on standardized multimodal datasets, generating environmental evidence characterization results, behavioral evidence characterization results, and health evidence characterization results. The multimodal evidence characterization includes environmental evidence characterization, behavioral evidence characterization, and health evidence characterization, and there is a dynamic triggering relationship between the three types of characterization. The dynamic triggering relationship is that when any one type of evidence characterization result meets the set conditions, the remaining evidence characterization is triggered to perform enhanced analysis, cross-validation, or collaborative analysis.

[0051] The violation determination module is used to determine violations based on environmental evidence representation results, behavioral evidence representation results, and health evidence representation results, and generates violation determination results through a preset multimodal evidence fusion model.

[0052] The early warning feedback module is used to trigger early warning signals and generate emergency response suggestions based on the violation determination results.

[0053] Thirdly, the present invention provides a computer device, the device including a processor and a memory:

[0054] The memory is used to store computer programs and send the instructions of the computer programs to the processor;

[0055] The processor executes, according to the instructions of the computer program, a method for determining violations in confined space operations based on multimodal evidence representation, as described in the first aspect.

[0056] In summary, this invention provides a method and system for determining violations in confined space operations based on multimodal evidence representation. First, this invention acquires environmental parameter data, worker behavior image data, health sign data, and operation process confirmation data for confined space operations, generating a multimodal raw data set. Second, it preprocesses the multimodal raw data set to generate a standardized multimodal data set. Then, it performs multimodal evidence representation on the standardized multimodal data set, generating environmental evidence representation results, behavioral evidence representation results, and health evidence representation results. The multimodal evidence representation includes environmental evidence representation, behavioral evidence representation, and health evidence representation, and there is a dynamic triggering relationship among the three types of representations. The dynamic triggering relationship is that when any one type of evidence representation result meets a set condition, it triggers enhanced analysis, cross-validation, or collaborative analysis of the remaining evidence representations. Based on the environmental evidence representation results, behavioral evidence representation results, and health evidence representation results, a violation determination is performed using a preset multimodal evidence fusion model, generating a violation determination result. Finally, based on the violation determination result, an early warning signal is triggered, and emergency response suggestions are generated. This invention uses multimodal evidence representation with dynamic triggering relationships and a multimodal evidence fusion model to determine violations, thereby deeply exploring the hidden risks of multi-factor coupling, achieving accurate and dynamic determination of violations in confined space operations, and improving the accuracy and reliability of safety monitoring. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart of a method for determining violations in confined space operations based on multimodal evidence representation provided in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the dynamic triggering relationship and violation probability curve of multimodal evidence representation provided in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the multimodal evidence fusion model and violation determination result curve provided in the embodiments of the present invention;

[0061] Figure 4 A block diagram illustrating the composition of a confined space operation violation determination device based on multimodal evidence representation, provided in an embodiment of the present invention;

[0062] Figure 5 This is a block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0064] Please see Figure 1 This embodiment provides a method for determining violations in confined space operations based on multimodal evidence representation, including the following steps:

[0065] S11: Acquire environmental parameter data, worker behavior image data and health sign data, and work process confirmation data for confined space operations, and generate a multimodal raw data set.

[0066] It should be noted that environmental parameter data refers to quantitative data reflecting the environmental safety status within a confined space, collected through preset environmental monitoring sensors, including but not limited to oxygen concentration, hydrogen sulfide concentration, carbon monoxide concentration, combustible gas concentration, temperature, and humidity.

[0067] Behavioral image data refers to image or video data captured by a pre-set image acquisition device (such as a camera) that records the actions, postures, and operational behaviors of workers in a confined space.

[0068] Health vital signs data refers to quantitative data that reflects the real-time physiological status of employees, including heart rate, blood oxygen saturation, and body temperature, collected through pre-set health monitoring devices (such as smart bracelets and wearable sensors).

[0069] Work process confirmation data refers to structured data that records the execution status of each step in confined space operations, including work permit approval information, step execution timestamps, and key operation confirmation records.

[0070] The multimodal raw data set is a unified data set formed by integrating the above-mentioned types of data according to a preset format.

[0071] S12: Preprocess the original multimodal dataset to generate a standardized multimodal dataset.

[0072] It should be noted that preprocessing refers to a series of data cleaning, correction, and transformation operations performed on the multimodal raw dataset, including but not limited to: removing outliers caused by sensor noise, deblurring / cropping behavioral images, filling missing values ​​in health sign data, and standardizing workflow confirmation data into a unified format, in order to improve data quality.

[0073] A standardized multimodal dataset refers to a multimodal dataset that, after preprocessing, is converted into a format that conforms to a preset standard (such as uniform data dimensions, quantization range, and encoding method).

[0074] S13: Perform multimodal evidence representation on the standardized multimodal dataset to generate environmental evidence representation results, behavioral evidence representation results, and health evidence representation results; the multimodal evidence representation includes environmental evidence representation, behavioral evidence representation, and health evidence representation, and there is a dynamic triggering relationship between the three types of representation; the dynamic triggering relationship is that when any one type of evidence representation result meets the set conditions, it triggers the enhancement analysis, cross-validation, or collaborative analysis of the remaining evidence representations.

[0075] It should be noted that multimodal evidence representation refers to the process of transforming various types of data in a standardized multimodal dataset into intermediate results with evidentiary attributes through a specific algorithm. The core is to extract key features and quantitative indicators related to the determination of operational violations from the original data.

[0076] Environmental evidence characterization refers to the process of extracting features of environmental safety status (such as whether it deviates from normal operating conditions and the degree of deviation) from standardized environmental parameter data through algorithms such as anomaly detection and cluster analysis, and transforming them into evidence related to the possibility of environmental violations.

[0077] Behavioral evidence representation refers to the process of extracting features of workers' work behavior (such as whether they are wearing safety equipment or whether there are dangerous actions) from standardized behavioral image data through algorithms such as target detection and action recognition, and transforming them into evidence related to the possibility of behavioral violations.

[0078] Health evidence characterization refers to the process of extracting the characteristics of an employee's physiological state (such as the presence or trend of abnormal physiological fluctuations) from standardized health data through algorithms such as trend analysis and threshold judgment, and transforming them into evidence related to the likelihood of violations associated with health risks.

[0079] Dynamic triggering relationships refer to the linkage mechanism among three types of evidence representations: environment, behavior, and health. Specifically, when the result of any one type of evidence representation meets preset conditions (such as the probability of environmental violation exceeding a threshold or the detection of dangerous actions in behavior), enhanced processing of the remaining two types of evidence representations is automatically triggered, including enhanced analysis, cross-validation, or collaborative analysis. Enhanced analysis refers to improving the resolution of data processing, increasing the dimensionality of feature extraction, or improving the detection sensitivity of the algorithm (such as performing more detailed action frame analysis on behavioral images) to obtain more accurate evidence results, depending on the type of evidence representation triggered. Cross-validation refers to using the result of one type of evidence representation to verify the reliability of the result of another type of evidence representation (such as using abnormal fluctuations in health sign data to verify whether dangerous actions in behavioral images are caused by physiological discomfort), avoiding misjudgments from a single source of evidence. Collaborative analysis refers to combining the results of two or three types of evidence representations for comprehensive analysis (such as simultaneously considering environmental hypoxia, the behavior of staff not wearing respirators, and health data of abnormally elevated heart rate) to identify complex violation risks involving multiple coupled factors.

[0080] The environmental evidence characterization results, behavioral evidence characterization results, and health evidence characterization results are used to quantify evidence related to environmental violations, behavioral violations, and health risk-related violations, respectively.

[0081] S14: Based on the environmental evidence representation results, behavioral evidence representation results, and health evidence representation results, violation determination is performed through a preset multimodal evidence fusion model to generate violation determination results.

[0082] It should be noted that the preset multimodal evidence fusion model refers to the warning information triggered by the violation judgment result, which is used to remind on-site personnel or remote monitoring platforms. It can be presented in the form of audible and visual alarms (such as on-site alarms) and message push (such as notifications from the monitoring APP).

[0083] The violation determination result is the output of the pre-set multimodal evidence fusion model. It is the final judgment on whether there is a violation in the confined space operation. It can include a binary judgment of "violation / non-violation", violation type (such as environmental exceedance violation, behavioral operation violation), violation severity level and other information.

[0084] S15: Based on the violation determination results, trigger an early warning signal and generate emergency response suggestions.

[0085] It should be noted that the warning signal refers to the warning information triggered based on the result of the violation judgment, which is used to remind the personnel at the work site or the remote monitoring platform. It can be presented in the form of audible and visual alarms (such as on-site alarms) and message push (such as notifications from the monitoring APP).

[0086] Emergency response recommendations are specific measures to be taken in response to the current violation scenario, generated based on the violation determination results (such as the type and severity of the violation). These recommendations include, but are not limited to, suspending operations, evacuating personnel, ventilating the environment, and repairing equipment.

[0087] This embodiment provides a method for determining violations in confined space operations based on multimodal evidence representation. This method integrates four heterogeneous data sources—environment, personnel behavior, physiological health, and work procedures—to form a comprehensive, time-synchronized multimodal data foundation. Based on this, standardized preprocessing transforms each modality of data into a quantifiable form of evidence. Crucially, the representation of environmental, behavioral, and health evidence is not independent but rather establishes a dynamically triggered, interconnected relationship. This allows risk signs in any dimension to trigger enhanced analysis of other dimensions, achieving cross-validation of information and in-depth risk assessment. Finally, a multimodal evidence fusion model comprehensively analyzes these interconnected pieces of evidence, generating a more accurate and reliable global violation determination conclusion that surpasses single information sources. This conclusion automatically triggers subsequent early warning and emergency response procedures, achieving comprehensive and in-depth perception of the safety status of confined space operations. By integrating data from multiple modalities, it overcomes the shortcomings of traditional single-monitoring methods, such as information bias and susceptibility to misjudgments and omissions, thus improving the breadth and depth of violation event identification. The dynamic triggering and evidence fusion mechanism can intelligently discover the inherent correlations between risks of different dimensions, effectively identifying complex and hidden risks caused by the combined effects of multiple factors that are difficult to detect by simple rules, thereby improving the accuracy and reliability of violation determination. Ultimately, the violation determination results are directly linked to early warning and emergency response recommendations, forming a complete management process from risk perception and intelligent analysis to closed-loop handling, enhancing the timeliness and relevance of emergency response, and fundamentally improving the safety level of confined space operations.

[0088] In one embodiment of the present invention, environmental parameter data, worker behavior image data, health sign data, and work process confirmation data for confined space operations are acquired to generate a multimodal raw data set, including:

[0089] S21: Obtain environmental parameter data through preset sensors used for environmental monitoring.

[0090] For example, in terms of acquiring environmental parameter data, an environmental monitoring sensor network is deployed within the confined space and at key locations such as entrances. This network includes at least oxygen concentration sensors, hydrogen sulfide sensors, carbon monoxide sensors, and combustible gas sensors, supplemented by temperature and humidity sensors. These sensors continuously measure and report their respective parameter values ​​at a preset sampling frequency, such as once per second. Each data point is accompanied by a precise timestamp, forming a continuous stream of environmental parameter data.

[0091] S22: Acquire behavioral image data through a preset image acquisition device, and perform noise reduction and calibration processing on the behavioral image data to generate calibrated behavioral image data.

[0092] For example, in acquiring worker behavior image data, an industrial-grade low-light wide-angle camera is installed at a location that fully covers the work area without blind spots as the image acquisition device. This device captures a high frame rate video stream as the raw worker behavior image data. To ensure the accuracy of subsequent behavior recognition, a Gaussian filtering algorithm is first used to denoise each frame of the acquired image data to eliminate interference caused by insufficient light or sensor noise. The image data is then calibrated using camera distortion correction. Using a pre-shot checkerboard calibration board, the camera's intrinsic parameter matrix and distortion coefficients are calculated. The intrinsic parameter matrix and distortion coefficients are then applied to perform an inverse geometric transformation on the real-time image to eliminate barrel or pincushion distortion caused by the wide-angle lens.

[0093] S23: Obtain health sign data through a pre-set device for health monitoring.

[0094] For example, in terms of acquiring health data, wearable health monitoring devices are provided to every worker entering a confined space, such as smart sensors integrated into a safety helmet or worn on the wrist. These devices monitor and collect key health data in real time, primarily including critical physiological indicators such as heart rate and blood oxygen saturation. The data is timestamped and transmitted wirelessly to a data aggregation node.

[0095] S24: Obtain work process confirmation data through a preset device for process recording.

[0096] For example, in acquiring work process confirmation data, a digital process recording device is used, such as an explosion-proof tablet computer held by the operator or a work site check-in system implemented through radio frequency identification (RFID) technology. When completing key process nodes such as ventilation, inspection, and approval, operators need to perform a confirmation operation on this device. The device automatically records the operator, operation content, and precise time of each confirmation action, forming structured work process confirmation data.

[0097] S25: Integrate environmental parameter data, calibrated behavioral image data, health sign data, and work process confirmation data to obtain a multimodal raw data set.

[0098] Environmental parameter data, processed calibrated behavioral image data, health sign data, and work process confirmation data are integrated. Based on the unified timestamps in each data stream, the four modalities of data within the same moment or a very small time window are associated to construct a time-seriesd multidimensional data record, ultimately generating a complete and comprehensive multimodal raw data set.

[0099] In one embodiment of the present invention, the original multimodal data set is preprocessed to generate a standardized multimodal data set, including:

[0100] S31: Normalize the environmental parameter data to generate standardized environmental data.

[0101] Because different environmental sensors, such as oxygen concentration sensors and hydrogen sulfide concentration sensors, have significantly different dimensions and numerical ranges in their measurements, a minimum-maximum normalization method can be used to map various environmental parameter data to a uniform range of 0 to 1, generating standardized environmental data. The calculation method is as follows:

[0102] ;

[0103] in, This represents the normalized, standardized environmental data. X is the real-time measurement value from the sensor. and These are the minimum and maximum values ​​of the normal fluctuation range of this parameter, obtained from safety regulations or historical data statistics.

[0104] S32: Perform low-light enhancement and feature extraction on the calibrated behavioral image data to generate enhanced behavioral image data.

[0105] Confined space operating environments often have poor lighting conditions, and directly acquired images may suffer from low contrast, high noise, and blurred details, severely affecting the accuracy of subsequent behavior recognition. A deep learning-based low-light enhancement network model is employed to process each frame of the image. This model can adaptively brighten dark areas, restore colors, and suppress noise, outputting enhanced behavior image data with improved visual quality.

[0106] S33: Perform sliding window smoothing on health indicator data to generate smooth health data.

[0107] Raw health data, such as heart rate, may contain high-frequency noise or spikes due to minor bodily movements or momentary equipment interference. To extract the true trend of these changes, a sliding window smoothing algorithm is used. A fixed-size time window is set and slides forward on the data stream. Each time the window moves one step, the average value of all data points within the window is calculated, and this average value is used as the smoothed health data for that time point to generate smoothed health data that truly reflects the changes in the physiological state of the staff.

[0108] S34: Perform time-series alignment on the work process confirmation data to generate time-series aligned process data.

[0109] The original workflow confirmation data consists of discrete event points. To integrate and analyze this data with continuous environmental, behavioral, and health data, it needs to be converted into a time-series state signal. The time interval between two consecutive workflow confirmation points is marked as the workflow stage represented by the previous confirmation point. For example, after a ventilation confirmation event occurs, a continuous time-aligned workflow data is generated until the next gas detection confirmation event occurs, with the state value always being the ventilation stage.

[0110] S35: Integrate standardized environmental data, enhanced behavioral image data, smoothed health data, and time-aligned process data to obtain a standardized multimodal data set.

[0111] By using a unified high-precision timestamp as an index, the standardized environmental data, enhanced behavioral image data, smoothed health data, and time-series aligned process data obtained through the above processing are integrated into a structured data record to generate a standardized multimodal data set.

[0112] In one embodiment of the present invention, multimodal evidence characterization is performed on a standardized multimodal dataset to generate environmental evidence characterization results, including:

[0113] S41: Analyze environmental parameter data in a standardized multimodal dataset, identify environmental anomaly patterns, and generate environmental anomaly clustering results.

[0114] This process can employ an unsupervised learning algorithm based on density clustering. This algorithm is pre-trained using a large amount of standardized environmental data from historical safe operation periods, thereby constructing high-density core regions, or normal operating condition clusters, representing safe or normal states in a multi-dimensional parameter space. The standardized environmental data vector at the current moment is then fed into the pre-trained model. If the current data point falls within the core region of any normal operating condition cluster, it is considered normal; conversely, if the model identifies it as an outlier or noise point, meaning it does not belong to any known safe state pattern, it is marked as an environmental anomaly, and an environmental anomaly clustering result is generated, indicating that the current environmental state deviates from the normal pattern.

[0115] S42: Calculate the Euclidean distance from the data points to the boundary of the normal operating condition cluster based on the results of environmental anomaly clustering, quantify the degree of deviation of environmental parameters, and generate the probability value of environmental violation through a nonlinear mapping function.

[0116] If the current environmental data is determined to be abnormal, its deviation is quantified as the Euclidean distance from the data point to the boundary of the nearest normal operating condition cluster. The larger this distance, the further the environmental state deviates from normal. This deviation is converted into an environmental violation probability value between 0 and 1 using a non-linear mapping function. This conversion can be implemented using the following function:

[0117] ;

[0118] in, This represents the final calculated probability value of environmental violations. This represents the degree of deviation of environmental parameters, specifically the normalized distance between the current abnormal environmental data vector and the centroid of the nearest normal operating condition cluster. This distance is directly output by the clustering analysis step. k is an adjustable gain coefficient used to control the steepness of the probability curve, set according to the risk sensitivity of the actual application scenario. It is a deviation from the baseline threshold, representing the maximum acceptable deviation. It is usually set as the statistical boundary of the normal operating condition cluster, such as a position calculated based on historical data at three times the standard deviation.

[0119] S43: When the probability value of environmental violation exceeds the preset threshold, the enhanced analysis of behavioral evidence representation and health evidence representation is triggered. The environmental violation probability value, abnormal environmental parameter data and the triggering status of dynamic triggering relationships are integrated to generate environmental evidence representation results.

[0120] A preset threshold is set, for example, 0.7. When the calculated environmental violation probability value exceeds this threshold, a high violation risk is determined in the current environment, triggering a dynamic linkage mechanism. The dynamic linkage mechanism sends an enhanced analysis command to the behavioral evidence representation module and the health evidence representation module. For example, the command might request the behavioral evidence representation module to use a more accurate attitude estimation algorithm, or request the health evidence representation module to shorten the data collection and analysis cycle. The current violation probability value, specific abnormal parameters, and trigger status are integrated to form a complete environmental evidence representation result.

[0121] In one embodiment of the present invention, multimodal evidence characterization is performed on a standardized multimodal dataset to generate behavioral evidence characterization results, including:

[0122] S51: Perform target detection on behavioral image data in a standardized multimodal dataset, identify the wearing status of safety equipment, and generate equipment detection results.

[0123] This step utilizes a pre-trained deep learning object detection model to analyze each frame of the video stream in real time. This model is specifically trained to identify critical targets in confined space work scenarios, particularly workers and their required safety equipment, such as helmets, respirators, and seatbelts. For each worker instance detected in the image, the model simultaneously checks whether the appropriate safety equipment is worn on their head, face, and torso. The output is a structured equipment detection result, including classification information such as not wearing, incorrectly wearing, or fully wearing.

[0124] S52: Extract the spatiotemporal features of staff actions, identify dangerous behaviors, and generate behavioral anomaly scores.

[0125] First, the positions of key skeletal points such as the shoulder, elbow, wrist, hip, knee, and ankle of the worker are accurately identified in each frame of the image. Then, the coordinates of these key points in consecutive time-series frames are combined to form a dynamic skeletal sequence describing the changes in human posture over time. Next, the dynamic skeletal sequence is input into a spatiotemporal graph convolutional network. This network, trained on a large number of labeled videos, learns and recognizes specific dangerous behavior patterns, such as sudden falls, prolonged immobility, and entering unauthorized dangerous areas. Finally, it outputs a real-time behavioral anomaly score; the higher the score, the greater the similarity between the detected action pattern and the defined dangerous behavior pattern.

[0126] S53: Integrate equipment detection results and behavior anomaly scores to generate a behavior violation probability value.

[0127] The equipment detection results and behavioral anomaly scores are combined to generate a comprehensive behavioral violation probability value:

[0128] ;

[0129] in, This represents the probability value of a violation of regulations. It is a quantitative score for equipment violations based on equipment inspection results. This is a discrete value. For example, not wearing a respirator is 0.9, not wearing a safety helmet is 0.6, and wearing all the necessary equipment is 0. It is the normalized value of the behavioral abnormality score, and its value range is between 0 and 1. and These are the weighting coefficients of the two, which sum to 1 and are dynamically adjusted according to the risk focus of different operational stages.

[0130] S54: When the probability value of behavioral violation is higher than the preset threshold, cross-validate the associated health evidence representation results and integrate the validation results to generate behavioral evidence representation results.

[0131] An adjustable threshold, such as 0.8, is set. When the probability of a behavioral violation exceeds this threshold, the employee's behavior is deemed highly risky, and a cross-validation mechanism with health evidence representation is immediately initiated. Using the timestamp of the current behavioral event as a key index, health evidence representation results corresponding exactly to the time of the high-risk behavior are obtained. If the returned health evidence representation results show an abnormal health status, the employee's physiological data is considered to increase the certainty of the violation event due to the current behavioral abnormality. If the returned health evidence representation results show a normal health status, the high probability of a behavioral violation is considered to be caused by algorithmic misjudgment or a non-dangerous, large-scale movement. The behavioral violation probability value and the cross-validation status marker information are structured and encapsulated to generate the behavioral evidence representation results.

[0132] In one embodiment of the present invention, multimodal evidence characterization is performed on a standardized multimodal dataset to generate health evidence characterization results, including:

[0133] S61: Analyze smoothed health data in a standardized multimodal dataset to generate health trend indicators.

[0134] The rate of change or short-term fluctuation of key health indicators, such as heart rate and blood oxygen saturation, within a preset time window is calculated as a core health trend indicator. For example, by calculating the first derivative of heart rate data over a short period of time, the speed of its rise or fall can be quantified, thereby capturing dangerous signals such as sudden increases in heart rate.

[0135] S62: Compare health trend indicators with staff’s personalized preset health baselines to generate a health abnormality level.

[0136] The real-time calculated health trend indicators are compared with a preset health baseline to generate a health abnormality level. The health baseline is a personalized physiological parameter model collected and established for each worker under normal conditions before entering confined space operations. It includes the worker's normal physiological indicator mean and fluctuation range. The comparison process involves comparing the current health trend indicator with the worker's personal baseline model. Based on the degree and rate of deviation from the baseline model, the health status is divided into multiple health abnormality levels, such as normal, mild abnormality, and severe abnormality. A health trend indicator that consistently exceeds the normal fluctuation range of its personal baseline will be judged as a higher level of abnormality.

[0137] S63: The level of health abnormality is quantified into a probability value of health violation through a non-linear mapping function.

[0138] Based on the assessed level of health abnormality, the health risk is quantified, generating a probability value for health violation between 0 and 1. This step is achieved through a predefined mapping function that maps discrete health abnormality levels to continuous probability values. The mapping relationship is as follows:

[0139] ;

[0140] in, This represents the final probability value of a health violation. It represents the health abnormality level generated in the previous step. M is a non-linear mapping function.

[0141] S64: When the probability of health violations increases, conduct a collaborative analysis of environmental evidence representation and behavioral evidence representation from the same period, and generate health evidence representation results by combining the results of the collaborative analysis.

[0142] When the calculated probability of a health violation increases significantly, collaborative analysis is performed. For example, if an abnormally rapid heart rate is detected in a worker, leading to an increased probability of a health violation, the health evidence characterization module will immediately send query requests to the other two modules to determine the potential cause of this physiological abnormality. This includes checking for excessive concentrations of toxic or harmful gases and analyzing whether the worker was engaged in strenuous physical labor. After integrating this information, the final health evidence characterization result is formed, which includes the violation probability value and preliminary causal inferences derived from the collaborative analysis. The multimodal evidence dynamic triggering logic is as follows: Figure 2 As shown, environmental anomalies first trigger behavioral / health enhancement analysis, then behavioral violations trigger health cross-validation, ultimately forming a collaborative judgment to accurately capture risks coupled with multiple factors. Specifically, Figure 2 Using work time as the horizontal axis and violation probability as the vertical axis, the study shows the changes and dynamic linkages of three types of violation probabilities in confined space operations: environment (hydrogen sulfide concentration), behavior (not wearing a respirator), and health (heart rate / blood oxygen). Initially, the probabilities of the three types of violations remained low. When the environmental violation probability increased (environmental abnormality), enhanced analysis of behavioral and health evidence was triggered. Subsequently, the behavioral violation probability increased (behavioral abnormality), triggering cross-validation of health evidence. When the health violation probability increased (health deterioration), it was linked with environmental and behavioral evidence for collaborative analysis. The three probabilities eventually approached the violation threshold (0.7), intuitively demonstrating the multi-dimensional risk linkage assessment mechanism.

[0143] In one embodiment of the present invention, based on environmental evidence characterization results, behavioral evidence characterization results, and health evidence characterization results, a violation determination is performed using a preset multimodal evidence fusion model to generate a violation determination result, including:

[0144] S71: Map the environmental evidence characterization results, behavioral evidence characterization results, and health evidence characterization results to environmental evidence violation confidence levels, behavioral evidence violation confidence levels, and health evidence violation confidence levels, respectively.

[0145] The results of environmental, behavioral, and health evidence characterization are transformed into a unified quantitative indicator: the violation confidence level. The environmental violation probability values ​​included in the environmental evidence characterization results are directly mapped to the environmental evidence violation confidence level. Similarly, the comprehensive adjusted violation confidence level generated after cross-validation in the behavioral evidence characterization results is extracted as the behavioral evidence violation confidence level. The health violation probability values ​​in the health evidence characterization results are mapped to the health evidence violation confidence level.

[0146] S72: A dynamic weighted summation algorithm is used to fuse the violation confidence scores of the three types of evidence to generate an initial fused violation score.

[0147] The confidence scores for violations based on environmental evidence, behavioral evidence, and health evidence are weighted and fused to generate a comprehensive fused violation score. The calculation uses a linear weighted summation model, as shown in the following formula:

[0148] ;

[0149] in, This is the final result of the violation determination. , and These represent the confidence levels of violation of environmental, behavioral, and health evidence calculated in the aforementioned steps, respectively. , and These are dynamic weighting coefficients that change with the stage t of the work process, and at any time t, the sum of the three is 1. For example, in the gas detection stage, the weight of environmental evidence will be increased; while in the personnel operation stage, the weights of behavioral evidence and health evidence will be increased accordingly.

[0150] S73: The initial fusion violation score is adaptively adjusted in stages by combining the time sequence markers of the work process to generate violation judgment results; the stage adaptive adjustment is to dynamically adjust the weights of the three types of evidence in the dynamic weighted summation algorithm as the work process stages change.

[0151] The weight coefficients of corresponding work process stages are adjusted based on the time sequence markers of the work process to generate a violation determination result. The final violation determination result is a quantitative violation risk score. When this score exceeds a preset determination threshold, a violation is formally confirmed. The multimodal evidence fusion logic is as follows: Figure 3 As shown, the weighting of behavior / health is dynamically increased during the work phase, and violations are determined after the combined score exceeds the threshold, accurately matching the core risk points at different stages. Specifically, Figure 3 (a) shows the changes in the confidence of violations of environment, behavior and health over the duration of the operation, and also marks the dynamic weights of different stages (environmental weight of 0.5 in the preparation stage, and behavior and health weights of 0.85 in total in the operation stage). Figure 3 (b) shows the violation score obtained based on dynamic weighted fusion. As the operation progresses, the fusion score gradually increases. When it exceeds the preset threshold (0.7), it enters the violation judgment range, realizing accurate violation judgment based on the risk of the operation stage.

[0152] In one embodiment of the present invention, triggering an early warning signal and generating emergency response suggestions based on the violation determination result includes:

[0153] S81: The violation determination result is sent to the preset monitoring platform in real time, triggering the audible and visual alarm signal of the corresponding work area.

[0154] When the violation determination result output by the multimodal evidence fusion model, i.e., the final violation determination score, exceeds the preset safety threshold, the early warning and handling process is immediately initiated. First, the violation determination result, along with a summary of the key evidence leading to this determination, is sent in real-time via a network interface to a preset central monitoring platform. This platform, deployed in a remote monitoring center or on-site command center, activates its built-in alarm module immediately upon receiving the violation determination result, triggering audible and visual alarm signals associated with the confined space work site. For example, a high-decibel alarm sounds on-site, a high-brightness warning light flashes, and simultaneously, an alarm window pops up prominently on the monitoring screen of the monitoring platform, clearly displaying the location, time, type, and severity level of the violation, ensuring that supervisory personnel are aware of the danger immediately.

[0155] S82: Based on case-based reasoning technology, match similar cases in the historical accident case database, extract effective handling measures, and adjust them according to the current scenario to generate emergency response suggestions.

[0156] Simultaneously with triggering the alarm, an emergency response suggestion generation program is initiated. This program is based on a historical accident case database, which stores detailed records of numerous confined space work accidents or violations throughout history. Each record is a case, containing multimodal data characteristics at the time of the accident, the type of violation, the development process, the emergency measures taken, and the final outcome. When a new violation occurs, the current violation determination result and its associated evidence features are used as the target case. The database is searched for the most similar historical case. The similarity calculation is a multidimensional feature matching process, comprehensively considering multiple aspects such as abnormal patterns of environmental parameters, categories of behavioral violations, and trends in health indicators. When the most similar historical accident case or set of cases is matched, the verified effective emergency response measures from these cases are extracted and adaptively adjusted based on the specific context of the current violation to generate emergency response suggestions. These suggestions may include specific operational instructions such as immediately evacuating all personnel, activating forced ventilation equipment, and assigning monitoring personnel with rescue equipment to enter the confined space.

[0157] S83: Adjust the operating procedures according to emergency response recommendations.

[0158] The generated emergency response recommendations are immediately pushed to the monitoring platform and the smart terminals of on-site personnel. These recommendations not only guide immediate responses, but their core content is also used to dynamically adjust current work processes. For example, if an abnormal ambient gas concentration is detected, the system may recommend suspending all hot work operations in the emergency response recommendations and update this instruction to the pending list of work process confirmation data. This mandates that personnel can only continue the process after the environment returns to normal and is retested and confirmed. This adjustment is real-time, directly embedding risk intervention into work process management and preventing the recurrence of similar risks. Simultaneously, the entire process of this violation, from data collection to the final handling result, will be fully recorded and stored as a new case in the historical accident case database, thereby continuously improving the accuracy and effectiveness of future emergency response recommendations.

[0159] Based on the same inventive concept, this application also provides a system for determining violations in confined spaces based on multimodal evidence representation, used to implement the aforementioned method for determining violations in confined spaces based on multimodal evidence representation. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations in the embodiments of the system for determining violations in confined spaces based on multimodal evidence representation provided below can be found in the limitations of the method for determining violations in confined spaces based on multimodal evidence representation described above, and will not be repeated here.

[0160] Please see Figure 4This invention also provides a confined space operation violation determination system based on multimodal evidence representation, comprising:

[0161] The data acquisition module is used to acquire environmental parameter data, worker behavior image data and health sign data, and work process confirmation data for confined space operations, and generate a multimodal raw data set;

[0162] The preprocessing module is used to preprocess the original multimodal dataset to generate a standardized multimodal dataset.

[0163] The evidence characterization module is used to perform multimodal evidence characterization on standardized multimodal datasets, generating environmental evidence characterization results, behavioral evidence characterization results, and health evidence characterization results. The multimodal evidence characterization includes environmental evidence characterization, behavioral evidence characterization, and health evidence characterization, and there is a dynamic triggering relationship between the three types of characterization. The dynamic triggering relationship is that when any one type of evidence characterization result meets the set conditions, the remaining evidence characterization is triggered to perform enhanced analysis, cross-validation, or collaborative analysis.

[0164] The violation determination module is used to determine violations based on environmental evidence representation results, behavioral evidence representation results, and health evidence representation results, and generates violation determination results through a preset multimodal evidence fusion model.

[0165] The early warning feedback module is used to trigger early warning signals and generate emergency response suggestions based on the violation determination results.

[0166] To verify the feasibility of this invention in practice, it was applied to a tower maintenance operation scenario at a large chemical plant. This plant requires regular internal cleaning and maintenance of reaction towers containing trace amounts of residual hydrogen sulfide, a typical high-risk confined space operation. Traditional monitoring methods rely on fixed gas detectors at the entrance and visual observation by external personnel, making it difficult to monitor the behavior and health status of internal workers in real time and comprehensively. This has led to near-miss accidents of poisoning and asphyxiation due to negligence in supervision and worker violations. The chemical plant decided to adopt the method of this invention for intelligent safety monitoring and violation determination throughout the entire tower maintenance operation process.

[0167] In this implementation, the chemical plant deployed an environmental monitoring device containing sensors for oxygen, hydrogen sulfide, and carbon monoxide concentrations inside the reaction tower; industrial-grade low-light cameras were installed in locations covering the main work areas; smart safety helmets with integrated heart rate and blood oxygen saturation monitoring functions were provided for the two workers entering the tower; and each step of the work process, such as ventilation, testing, and tower entry permits, required confirmation via a handheld explosion-proof tablet computer. The system collects this multimodal data and processes and analyzes it using the method of this invention.

[0168] To verify the beneficial effects of this invention, a typical operation process was recorded and analyzed. Approximately 30 minutes into the operation, a worker, feeling stuffy, removed their respirator and placed it aside without authorization. Simultaneously, the agitation of the sludge at the bottom of the tower caused the local hydrogen sulfide concentration to slowly rise from a safe 5 ppm to 12 ppm. The following is a detailed description of the handling process and effectiveness verification of the method of this invention when this incident occurred.

[0169] First, the environmental evidence characterization module detected that the hydrogen sulfide concentration data deviated from the safe operating condition cluster learned through the DBSCAN algorithm. The degree of deviation of the environmental parameters was calculated and converted into an environmental violation probability value of 0.78. This value exceeded the preset threshold of 0.7, indicating a high environmental risk and immediately triggering an instruction requiring the behavioral and health evidence characterization module to perform enhanced analysis.

[0170] Upon receiving the enhanced analysis command, the behavioral evidence representation module analyzes the behavioral image data of the workers. Using the YOLO target detection model, it identifies one worker as not wearing a respirator, generating an equipment violation score of 0.9. Simultaneously, analysis of the worker's spatiotemporal characteristics using the ST-GCN model reveals slight swaying and non-operational leaning behaviors, generating a behavioral anomaly score of 0.4. Based on the weighted fusion strategy, the generated comprehensive behavioral violation probability value is 0.82, also exceeding the preset threshold.

[0171] Meanwhile, analysis revealed that the worker's smoothed heart rate rapidly increased from 85 bpm to 110 bpm, and blood oxygen saturation decreased from 98% to 94%. Compared to their baseline health, the health trend indicators were significantly abnormal, classified as severely abnormal, resulting in a quantified health violation probability value of 0.91. Further analysis showed a strong correlation between the deterioration of health indicators and increased hydrogen sulfide concentration in environmental evidence and the lack of respirator use in behavioral evidence.

[0172] Subsequently, the multimodal evidence fusion model fused the environmental violation probability value (0.78), behavioral violation probability value (0.82), and health violation probability value (0.91). Since the current operation is within the tower, the model, according to a phase-adaptive adjustment mechanism, assigned higher weights (0.4 and 0.45) to behavioral and health evidence. The final calculated fused violation score was 0.85, far exceeding the violation determination threshold of 0.75.

[0173] An audible and visual alarm was immediately triggered, and a bright alarm popped up on the large screen in the central monitoring room, clearly indicating that worker A inside the tower was not wearing a respirator, had an abnormal heart rate, and had a slightly elevated hydrogen sulfide concentration. Simultaneously, based on case analysis and matching with similar historical poisoning and asphyxiation incidents, emergency response recommendations were generated: 1. Immediately order all personnel to evacuate via communication equipment; 2. Activate emergency forced ventilation; 3. Monitoring personnel should don positive-pressure air respirators and prepare for rescue, and these recommendations were sent to the on-site monitoring and central control room.

[0174] Data comparison shows that the method of this invention has advantages in the timeliness and accuracy of risk identification. In traditional methods, a hydrogen sulfide concentration of 12 ppm may not trigger the high-limit alarm of a fixed detector, and external monitoring personnel may not notice the removal of masks in time due to perspective limitations. In this embodiment, this invention, through the collaborative analysis and dynamic triggering of multimodal evidence, accurately identifies complex risks at the nascent stage of hazard occurrence, issuing an early warning at least 2-3 minutes earlier than traditional methods, thus gaining valuable time for safe evacuation and emergency response.

[0175] In summary, the method of this invention demonstrates good performance in practical applications. Regarding the accuracy of the judgment results, the fusion model based on multimodal evidence and dynamic weights can accurately quantify composite risks, making the judgment results more targeted and accurate, and effectively reducing the false alarm rate. Its significant advantages in response speed and decision support can improve the safety level of confined space operations and effectively prevent serious accidents.

[0176] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0177] Reference Figure 5The present invention also provides a computer device, including: a memory and a processor, and a computer program stored in the memory. When the computer program is executed on the processor, it implements the method for determining violations of confined space operations based on multimodal evidence representation as described in any of the above methods.

[0178] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. They may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, they may also include input / output devices, network access devices, etc.

[0179] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0180] In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the computer device. The memory is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0181] This invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it implements the method for determining violations of confined space operations based on multimodal evidence representation as described in any of the above methods.

[0182] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0183] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the method for determining violations of confined space operations based on multimodal evidence representation as described in any of the above methods.

[0184] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0185] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0186] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0187] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for limited space work violation determination based on multi-modal evidence representation, characterized in that, The method comprises the following steps: Obtaining environmental parameter data, behavior image data and health sign data of workers and operation process confirmation data of limited space operation, generating a multi-modal original data set; Preprocessing the multi-modal original data set to generate a standardized multi-modal data set; Multi-modal evidence characterization is performed on the standardized multi-modal data set to generate environmental evidence characterization results, behavior evidence characterization results and health evidence characterization results; The multi-modal evidence characterization includes environmental evidence characterization, behavior evidence characterization and health evidence characterization, and there is a dynamic triggering relationship between the three types of characterization; when any type of evidence characterization result meets the set condition, the remaining evidence characterization is triggered for enhanced analysis, cross-validation or collaborative analysis; Based on the environmental evidence characterization results, behavior evidence characterization results and health evidence characterization results, a pre-set multi-modal evidence fusion model is used for violation judgment to generate a violation judgment result; According to the violation judgment result, a warning signal is triggered and an emergency disposal suggestion is generated.

2. The method of claim 1, wherein the method further comprises: Obtaining environmental parameter data, behavior image data and health sign data of workers and operation process confirmation data of limited space operation, generating a multi-modal original data set, comprising: Obtaining the environmental parameter data through a pre-set sensor for environmental monitoring; Obtaining the behavior image data through a pre-set device for image acquisition, and performing denoising and calibration processing on the behavior image data to generate calibrated behavior image data; Obtaining health sign data through a pre-set device for health monitoring; Obtaining the operation process confirmation data through a pre-set device for process recording; Integrating the environmental parameter data, calibrated behavior image data, health sign data and operation process confirmation data to obtain the multi-modal original data set.

3. The method of claim 2, wherein the method further comprises: Preprocessing the multi-modal original data set to generate a standardized multi-modal data set, comprising: Performing normalization processing on the environmental parameter data to generate standardized environmental data; Performing low-illumination enhancement and feature extraction on the calibrated behavior image data to generate enhanced behavior image data; Performing sliding window smoothing processing on the health sign data to generate smoothed health data; Performing time sequence alignment on the operation process confirmation data to generate time sequence aligned process data; Fusing the standardized environmental data, enhanced behavior image data, smoothed health data and time sequence aligned process data to obtain the standardized multi-modal data set.

4. The method of claim 1, wherein the method further comprises: Multi-modal evidence characterization is performed on the standardized multi-modal data set to generate environmental evidence characterization results, comprising: Analyzing the environmental parameter data in the standardized multi-modal data set to identify environmental abnormal patterns and generate environmental abnormal clustering results; Calculating the Euclidean distance of data points to the boundary of the normal working condition cluster based on the environmental abnormal clustering results to quantify the degree of deviation of environmental parameters, and generating environmental violation probability values through a nonlinear mapping function; When the environmental violation probability value exceeds a preset threshold, triggering an enhanced analysis of the behavior evidence characterization and the health evidence characterization, integrating the environmental violation probability value, the abnormal environmental parameter data, and the triggering state of the dynamic triggering relationship, and generating an environmental evidence characterization result.

5. The method of claim 1, wherein the method further comprises: The standardized multi-modal data set is subjected to multi-modal evidence characterization to generate a behavior evidence characterization result, including: Target detection is performed on the behavior image data in the standardized multi-modal data set to identify the safety equipment wearing state, and an equipment detection result is generated; The spatiotemporal features of the worker's actions are extracted to determine dangerous behaviors and generate a behavior anomaly score; The equipment detection result and the behavior anomaly score are fused to generate a behavior violation probability value; When the behavior violation probability value is higher than a preset threshold, cross-validation is performed in association with the health evidence characterization result to generate a behavior evidence characterization result.

6. The method of limited space work violation determination based on multi-modal evidence characterization of claim 1, wherein, The standardized multi-modal data set is subjected to multi-modal evidence characterization to generate a health evidence characterization result, including: The health trend indicators are compared with the worker's individualized preset health baseline to generate a health anomaly level; The health anomaly level is quantified into a health violation probability value through a nonlinear mapping function; When the health violation probability value rises, a collaborative analysis is performed on the environmental evidence characterization and the behavior evidence characterization of the same period, and a health evidence characterization result is generated in combination with the collaborative analysis result. Based on the environmental evidence characterization result, the behavior evidence characterization result, and the health evidence characterization result, a violation determination result is generated through a preset multi-modal evidence fusion model, including:

7. The method of claim 1, wherein the method further comprises: The environmental evidence characterization result, the behavior evidence characterization result, and the health evidence characterization result are respectively mapped into environmental evidence violation confidence, behavior evidence violation confidence, and health evidence violation confidence; A dynamic weighted summation algorithm is used to fuse the violation confidence of the three types of evidence to generate an initial fusion violation score; The initial fusion violation score is subjected to stage-adaptive adjustment in combination with the timing marks of the work process to generate a violation determination result; the stage-adaptive adjustment is a dynamic adjustment of the weights of the three types of evidence in the dynamic weighted summation algorithm with the change of the work process stage. The triggering of the warning signal and the generation of the emergency disposal suggestion according to the violation determination result include:

8. The method of limited space work violation determination based on multi-modal evidence characterization of claim 1, wherein, The violation determination result is sent to a preset monitoring platform in real time to trigger an audible and visual alarm signal in the corresponding work area; Based on case reasoning technology, similar cases are matched in a historical accident case library, effective disposal measures are extracted and adjusted in combination with the current scene to generate an emergency disposal suggestion; The work process is adjusted according to the emergency disposal suggestion. It includes:

9. A limited space work violation determination system based on multi-modal evidence characterization, comprising: A data acquisition module for acquiring environmental parameter data, behavior image data and health sign data of workers, and work process confirmation data of confined space work to generate a multi-modal raw data set; A preprocessing module for preprocessing the multi-modal raw data set to generate a standardized multi-modal data set; ​ The evidence characterization module is configured to perform multi-modal evidence characterization on the standardized multi-modal data set to generate environment evidence characterization results, behavior evidence characterization results, and health evidence characterization results. The multi-modal evidence characterization includes environment evidence characterization, behavior evidence characterization, and health evidence characterization, and the three types of characterization have a dynamic triggering relationship. When any type of evidence characterization result meets a set condition, the dynamic triggering relationship triggers enhanced analysis, cross-validation, or collaborative analysis of the remaining evidence characterization. The violation determination module is configured to perform violation determination based on the environment evidence characterization results, the behavior evidence characterization results, and the health evidence characterization results through a pre-set multi-modal evidence fusion model to generate violation determination results. The early warning feedback module is configured to trigger an early warning signal and generate an emergency disposal suggestion according to the violation determination results.

10. A computer device, comprising: The device includes a processor and a memory: The memory is configured to store a computer program and send instructions of the computer program to the processor; The processor executes the method according to the instructions of the computer program. The device includes a processor and a memory: The memory is configured to store a computer program and send instructions of the computer program to the processor; The processor executes the method according to the instructions of the computer program.

Images