Workshop intelligent safety system based on multi-modal data analysis

Abstract

This invention provides a smart safety system for workshops based on multimodal data analysis, relating to the field of industrial safety technology. It includes a perception layer, an edge layer, a platform layer, and an application layer. The perception layer includes camera equipment installed in the workshop for real-time monitoring of operator behavior and equipment operation; current and vibration sensors installed on various devices for collecting equipment current and vibration characteristics; and temperature and humidity sensors and gas detection sensors for collecting workshop temperature and humidity data, as well as gas detection sensors for collecting harmful gas concentrations. The video data collected by the cameras and data collected by the sensors in the perception layer are transmitted to an edge computing control terminal in the edge layer for preliminary processing. The pre-processed data is then uploaded to the platform layer for comprehensive analysis using a multimodal data fusion algorithm. The analysis results are transmitted to the application layer for early warning and to the display layer for graphical display.

Description

Technical Field

[0001] This invention relates to the field of industrial safety technology, specifically to a smart workshop safety system based on multimodal data analysis. Background Technology

[0002] Currently, safety management in industrial workshops primarily relies on traditional video surveillance systems. These systems collect real-time video footage by installing cameras within the workshop, which is then observed and managed by staff in a monitoring room via displays. While some systems have begun to incorporate basic video analytics algorithms, significant technical limitations remain. Furthermore, traditional video analytics algorithms are poorly adapted to complex scenarios, failing to effectively identify and predict potential safety hazards, resulting in high false alarm and false negative rates. More importantly, the response mechanisms of these systems are overly reactive, relying mainly on manual intervention to handle anomalies, failing to provide timely warnings and automatic responses, thus resulting in poor accident prevention effectiveness.

[0003] Another common type of safety management solution is the single-sensor-based safety early warning system. These systems deploy single-type sensors (such as those for temperature, smoke, or gas) in the workshop and set fixed thresholds for alarms. However, due to the limited data dimensions, the system can only monitor a single environmental parameter, failing to achieve multi-dimensional safety assessments and resulting in biased safety evaluations. Furthermore, the early warning mechanism of these systems is overly simplistic, relying solely on fixed thresholds and lacking data correlation analysis capabilities, making them unsuitable for complex industrial environments. A more prominent problem is the system's isolation; various sensors operate independently, lacking coordination and data sharing, severely impacting the overall system effectiveness.

[0004] In some industrial workshops, safety management still relies on traditional manual inspections. This model requires dedicated personnel to regularly patrol the workshop, manually record safety hazards, and judge handling solutions based on experience. This approach not only requires a significant investment of human resources, resulting in high personnel training costs and management difficulties, but is also extremely inefficient. Due to limited inspection frequency, 24 / 7 monitoring is impossible, and emergency response is slow. More importantly, safety assessments depend excessively on individual experience, lack standardized criteria, and fail to establish a scientific management system. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a smart workshop safety system based on multimodal data analysis. By combining data collected from different sensors and performing comprehensive analysis, it provides all-round data support for workshop safety, enabling the system to more accurately and quickly identify and issue early warnings of risks.

[0006] This invention is achieved through the following technical solution:

[0007] A smart safety system for workshops based on multimodal data analysis is characterized by comprising a perception layer, an edge layer, a platform layer, and an application layer. The perception layer includes camera equipment installed in the workshop for real-time monitoring of the behavior of workshop operators and workshop equipment; current sensors and vibration sensors installed on various workshop equipment for collecting current and vibration data; and temperature and humidity sensors for collecting workshop temperature and humidity data, as well as smoke detection sensors for collecting combustible gas concentration values ​​in the smoke. The video data collected by the camera equipment and the sensor data collected by each sensor in the perception layer are transmitted to the edge computing control terminal in the edge layer for preliminary processing. The pre-processed data is then uploaded to the platform layer for comprehensive analysis of the workshop safety status using a multimodal data fusion algorithm. The analysis results are transmitted to the application layer for early warning and to the display layer for graphical display.

[0008] Furthermore, the video data collected by the camera devices in the perception layer and the sensor data collected by each sensor are transmitted to the edge computing control terminal in the edge layer for preliminary processing, specifically including:

[0009] The edge computing control terminal decodes video data from the camera device into image frames in real time, and feeds the image frames into the trained YOLO v5 target detection model for specific target detection and takes screenshots. At the same time, the data transmitted by the current sensor, vibration sensor, temperature and humidity sensor and smoke detection sensor are filtered to remove noise.

[0010] Furthermore, the noise elimination process for the data transmitted by the current sensor, vibration sensor, temperature and humidity sensor, and smoke detection sensor through a data filtering algorithm specifically refers to:

[0011] To address potential pulse noise in current and vibration sensors, a median filtering method is used to select the median value within a given window as the output. To ensure system response, the median filtering window is set to 5. The calculation formula is as follows:

[0012]

[0013] in The filtered output value at time t represents the signal value obtained after median filtering; x(t) represents the original sensor data at time t; median{*} represents the operation of taking the median in the window, that is, sorting the 5 data points in the window in ascending order and selecting the median value as the filtered output value;

[0014] For temperature and humidity sensors and smoke detection sensors, the mean filtering method is used to smooth the signal by calculating the average value of data within a fixed window, thereby reducing high-frequency noise and random errors. The mean filtering method sets the window size to 4, and the calculation formula is as follows:

[0015]

[0016] in This represents the filtered output value at time t.

[0017] Furthermore, the pre-processed data is uploaded to the platform layer for comprehensive analysis using a multimodal data fusion algorithm, specifically including:

[0018] The platform layer performs dimensionality reduction and noise reduction on the processed sensor data using principal component analysis (PCA) to obtain dimensionality-reduced data. Then, it uses Bayesian estimation to fuse the dimensionality-reduced data for analysis to determine the normal or abnormal trend of the workshop.

[0019] The platform layer applies a trained target tracking algorithm to the processed video data to further analyze the behavioral characteristics in the scene, identify unsafe behaviors, and take screenshots and mark the unsafe behaviors as high-risk or low-risk.

[0020] Furthermore, unsafe behaviors include workers entering hazardous areas of the workshop, workers climbing heights not wearing safety belts, workers not wearing proper clothing, workers incorrectly or not wearing safety helmets, workers smoking, or workers not placing safety work signs properly.

[0021] Furthermore, the platform layer uses Principal Component Analysis (PCA) to reduce the dimensionality and remove noise from the sensor data collected by each sensor. The specific steps are as follows:

[0022] First, the sensor data collected by each sensor is standardized.

[0023]

[0024] Where x i Z represents data collected by different sensors. i Representing standardized data, a matrix σ is the sample mean, and σ is the sample standard deviation;

[0025] Next, calculate the covariance matrix ∑ of the standardized data. The calculation formula is as follows:

[0026]

[0027] Z i This represents the standardized data matrix, where n is the number of samples. It is the mean vector of the standardized samples;

[0028] Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues ​​λ. i and the corresponding eigenvector W i Choose the one with the largest eigenvalue λ. max The eigenvector W, whose standardized data is projected onto the principal components, is represented as:

[0029] Y i =Z i ·W, i = 1, 2, 3, 4;

[0030] Obtain the dimensionality-reduced data Y i The fusion is estimated using Bayesian methods, and the formula is as follows:

[0031]

[0032] Where P(A|Y1,Y2) is the posterior probability of event A occurring given the dimensionality-reduced data Y1 and Y2; P(Y1|A) and P(Y2|A) are the probabilities of observing Y1 and Y2 given that event A has occurred, i.e., the conditional probabilities of each sensor data; P(A) is the prior probability of event A, representing the probability of event A without other information; P(Y1,Y2) is the total probability of observing data Y1 and Y2, serving as a normalization factor.

[0033] Furthermore, the platform layer performs dimensionality reduction on the current data Xc collected by the current sensor and the vibration data Xv collected by the vibration sensor using principal component analysis (PCA) to obtain current data Yc and vibration data Yv. Prior probabilities P(normal) and P(abnormal) for normal and abnormal operating conditions are defined in the historical operating data of the device. Based on the current data Yc and vibration data Yv, the conditional probability distributions P(Yc|normal), P(Yv|normal), P(Yc|abnormal), and P(Yv|abnormal) for the current data Yc and vibration data Yv under normal and abnormal operating conditions are obtained.

[0034] Joint probability:

[0035] P(Yc, Yv) = P(Yc|Normal)·P(Yv|Normal)·P(Normal) + P(Yc|Abnormal)·P(Yv|Abnormal)·P(Abnormal)

[0036] Calculate the posterior probability of abnormal operating conditions using Bayesian estimation:

[0037]

[0038] By calculating the joint probability and posterior probability, it is found that if P(abnormal|Yc,Yv) exceeds the set threshold, where the threshold is set to 0.6.

[0039] Furthermore, the platform layer performs dimensionality reduction on the temperature and humidity data Nc collected by the temperature and humidity sensor and the combustible gas concentration data Mv collected by the smoke detection sensor, obtaining the temperature and humidity data Sc and the combustible gas concentration data Sv. Prior probabilities P(normal) and P(fire risk) for normal and fire risk conditions are defined in the historical temperature and humidity data and combustible gas concentration data recorded in the workshop. Based on the temperature and humidity data Sc and the combustible gas concentration data Sv, the conditional probability distributions P(Sc|normal), P(Sv|normal), P(Sc|fire risk), and P(Sv|fire risk) of the temperature and humidity data and smoke concentration data under normal and fire risk conditions are obtained.

[0040] Joint probability:

[0041] P(Sc, Sv) = P(Sc|Normal)·P(Sv|Normal)·P(Normal) + P(Sc|Fire Risk)·P(Sv|Fire Risk)·P(Fire Risk);

[0042] Calculate the posterior probability of fire risk conditions using Bayesian estimation:

[0043]

[0044] By calculating the joint probability and the posterior probability, it is concluded that if the posterior probability P(fire risk|Sc,Sv) exceeds the set threshold, then the workshop is considered to have a fire risk, where the threshold is set to 0.55.

[0045] Furthermore, the application layer includes a comprehensive security hazard management module, a risk warning module, and an alarm module;

[0046] The comprehensive safety hazard management module is used to record and analyze real-time data uploaded from the platform layer, providing suggestions for workshop safety management.

[0047] The risk warning module receives data and analysis results uploaded from the platform layer, categorizing the data into three types: equipment anomalies, fire risks, and unsafe behaviors. Based on the analysis results, if the workshop shows a fire risk trend, it is marked as high priority; if the equipment shows an anomaly trend and the uploaded current and vibration data exceed the set alarm thresholds, it is marked as a major equipment anomaly (medium priority); if the equipment shows an anomaly trend and the uploaded current and vibration data do not exceed the set alarm thresholds, it is marked as a minor equipment anomaly (low priority). The risk warning module also marks high-risk unsafe behaviors as high priority and low-risk unsafe behaviors as low priority based on the analysis results. After setting the alarm priorities, the risk warning module sends alarm signals to the alarm module.

[0048] The alarm module is used to receive alarm signals from the risk warning module and issue an alarm.

[0049] Furthermore, the current sensor, vibration sensor, temperature and humidity sensor, and smoke detection sensor are all equipped with fixed alarm thresholds. Specifically, the current sensor has an alarm threshold of 10A, the vibration sensor has an alarm threshold of 1000μm, the temperature and humidity sensor has an alarm threshold of 50℃, the humidity sensor has an alarm threshold of 60%RH, and the smoke detection sensor has an alarm threshold of 10ppm for combustible gases. If the data monitored by a single sensor exceeds the alarm threshold, a threshold alarm signal is sent to the alarm module, which then triggers the alarm module to sound an alarm.

[0050] When a threshold alarm signal and an analysis result trigger an alarm simultaneously, the system prioritizes the threshold alarm signal. If only a single sensor issues a threshold alarm signal, but the analysis result does not show any abnormality, the system will record the alarm and wait for other sensor signals or multi-source information fusion analysis in the next cycle. If the analysis result shows an abnormal trend, but does not reach the alarm threshold of a single sensor, the system will mark it as a low-priority alarm.

[0051] Furthermore, the display layer includes a large display screen, a PC terminal, and a mobile terminal.

[0052] The beneficial effects of this invention are as follows:

[0053] 1. This invention integrates multiple heterogeneous data sources such as current, vibration, video streams, temperature and humidity, and combustible gas concentration, and performs correlation analysis on the data using a fusion algorithm based on Bayesian estimation and principal component analysis (PCA). This allows for the extraction of complementary information from different data sources, effectively improving the accuracy and response speed in identifying abnormal situations, providing comprehensive data support for workshop safety, and enabling the system to more accurately and quickly identify and warn of risks.

[0054] 2. The edge computing control terminal in this invention has the ability to perform hardware H264 decoding of video streams and run the YOLO v5 target detection model efficiently. It can analyze video data and detect specific target objects in real time near the device, reducing data transmission bandwidth, reducing the computing load on the cloud, and improving the system's response speed. It can realize the instant identification and response to key events in the workshop, thereby better meeting the high requirements for real-time performance and stability in the workshop environment.

[0055] 3. This invention adopts a modular architecture, which makes the entire system highly scalable and easy to maintain. The system is divided into a presentation layer, an application layer, a platform layer, an edge layer, and a perception layer. Each layer performs its own function and cooperates with each other. From data collection in the perception layer, computing in the edge layer, analysis in the platform layer to application display and early warning, a well-defined and clearly defined architecture is formed. The modular design allows each part of the system to be upgraded, maintained, or replaced independently without affecting the functions of other modules. Attached Figure Description

[0056] Figure 1 This is a system architecture diagram of the present invention. Detailed Implementation

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, a smart safety system for workshops based on multimodal data analysis includes a perception layer, an edge layer, a platform layer, and an application layer. The perception layer includes camera equipment installed in the workshop for real-time monitoring of the behavior of workshop operators and workshop equipment; current sensors and vibration sensors installed on various workshop equipment for collecting current and vibration data; and temperature and humidity sensors for collecting workshop temperature and humidity data, as well as smoke detection sensors for collecting combustible gas concentration values ​​in the smoke. The video data collected by the camera equipment and the sensor data collected by each sensor in the perception layer are transmitted to the edge computing control terminal in the edge layer for preliminary processing. The pre-processed data is then uploaded to the platform layer for comprehensive analysis of the workshop safety status using a multimodal data fusion algorithm. The analysis results are transmitted to the application layer for early warning and to the display layer for graphical display.

[0060] The workshop equipment includes overhead cranes, forklifts, lathes, drilling machines, milling machines, and boring machines; current sensors, vibration sensors, temperature and humidity sensors, and gas detection sensors convert the collected data into electrical signals, which are then converted from analog to digital signals and transmitted to the edge computing terminal via Ethernet; the camera equipment transmits video streams to the edge computing control terminal via the RTSP protocol.

[0061] Example 2

[0062] This embodiment further elaborates and supplements the implementation of the present invention based on Embodiment 1.

[0063] The video data collected by the camera devices in the perception layer and the sensor data collected by various sensors are transmitted to the edge computing control terminal in the edge layer for preliminary processing, specifically including:

[0064] The edge computing control terminal decodes video data from the camera device into image frames in real time, and feeds the image frames into the trained YOLO v5 target detection model for specific target detection and takes screenshots. At the same time, the data transmitted by the current sensor, vibration sensor, temperature and humidity sensor and smoke detection sensor are filtered to remove noise.

[0065] The data transmitted by current sensors, vibration sensors, temperature and humidity sensors, and smoke detection sensors undergo noise elimination through data filtering algorithms. Specifically, this means:

[0066] To address potential pulse noise in current and vibration sensors, a median filtering method is used to select the median value within a given window as the output. To ensure system response, the median filtering window is set to 5. The calculation formula is as follows:

[0067]

[0068] in The filtered output value at time t represents the signal value obtained after median filtering; x(t) represents the original sensor data at time t; median{*} represents the operation of taking the median in the window, that is, sorting the 5 data points in the window in ascending order and selecting the median value as the filtered output value;

[0069] For temperature and humidity sensors and smoke detection sensors, the mean filtering method is used to smooth the signal by calculating the average value of data within a fixed window, thereby reducing high-frequency noise and random errors. The mean filtering method sets the window size to 4, and the calculation formula is as follows:

[0070]

[0071] in This represents the filtered output value at time t.

[0072] Example 3

[0073] This embodiment further elaborates and supplements the implementation of the present invention based on Embodiment 1 or Embodiment 2.

[0074] The pre-processed data is uploaded to the platform layer for comprehensive analysis using a multimodal data fusion algorithm, specifically including:

[0075] The platform layer performs dimensionality reduction and noise reduction on the processed sensor data using principal component analysis (PCA) to obtain dimensionality-reduced data. Then, it uses Bayesian estimation to fuse the dimensionality-reduced data for analysis to determine the normal or abnormal trend of the workshop.

[0076] The platform layer applies a trained target tracking algorithm to the processed video data to further analyze the behavioral characteristics in the scene, identify unsafe behaviors, and take screenshots and mark the unsafe behaviors as high-risk or low-risk.

[0077] Unsafe behaviors include workers entering hazardous areas of the workshop, workers climbing heights not wearing safety belts, workers not wearing proper clothing, workers not wearing safety helmets correctly or not wearing them at all, workers smoking, or workers not placing safety work signs properly.

[0078] The platform layer uses Principal Component Analysis (PCA) to reduce the dimensionality and remove noise from the sensor data collected by each sensor. The specific steps are as follows:

[0079] First, the sensor data collected by each sensor is standardized.

[0080]

[0081] Where x i Z represents data collected by different sensors. i Representing standardized data, a matrix σ is the sample mean, and σ is the sample standard deviation;

[0082] Next, calculate the covariance matrix ∑ of the standardized data. The calculation formula is as follows:

[0083]

[0084] Z i This represents the standardized data matrix, where n is the number of samples. It is the mean vector of the standardized samples;

[0085] Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues ​​λ. i and the corresponding eigenvector W i Choose the one with the largest eigenvalue λ. max The eigenvector W, whose standardized data is projected onto the principal components, is represented as:

[0086] Y i =Z i ·W, i = 1, 2, 3, 4;

[0087] Obtain the dimensionality-reduced data Y i The fusion is estimated using Bayesian methods, and the formula is as follows:

[0088]

[0089] Where P(A|Y1,Y2) is the posterior probability of event A occurring given the dimensionality-reduced data Y1 and Y2; P(Y1|A) and P(Y2|A) are the probabilities of observing Y1 and Y2 given that event A has occurred, i.e., the conditional probabilities of each sensor data; P(A) is the prior probability of event A, representing the probability of event A without other information; P(Y1,Y2) is the total probability of observing data Y1 and Y2, serving as a normalization factor.

[0090] Furthermore, the platform layer performs dimensionality reduction on the current data Xc collected by the current sensor and the vibration data Xv collected by the vibration sensor using principal component analysis (PCA) to obtain the current data Yc and vibration data Yv. Prior probabilities P(normal) and P(abnormal) for normal and abnormal operating conditions are defined in the historical operating data of the equipment. Based on the current data Yc and vibration data Yv, the conditional probability distributions P(Yc|normal), P(Yv|normal), P(Yc|abnormal), and P(Yv|abnormal) for the current data Yc and vibration data Yv under normal and abnormal operating conditions are obtained.

[0091] Joint probability:

[0092] P(Yc,Yv)=P(Yc|Normal)·P(Yv|Normal)·P(Normal)+P(Yc|Abnormal)·P(Yv|Abnormal)·P(Abnormal)

[0093] Calculate the posterior probability of abnormal operating conditions using Bayesian estimation:

[0094]

[0095] By calculating the joint probability and posterior probability, it is found that if P(abnormal|Yc,Yv) exceeds the set threshold, where the threshold is set to 0.6.

[0096] As an example, the platform layer obtains the current data (Yc = [2.3, 2.5, 2.6, 2.7]) and vibration data (Yv = [0.5, 0.6, 0.55, 0.65]) of a piece of equipment in the workshop, uploaded by the edge layer over a period of time. Here, mean(Yc) = 2.525 is the sample mean of the current data, σ(Yc) = 0.1708 is the sample standard deviation of the current data, mean(Yv) = 0.575 is the sample mean of the vibration data, and σ(Yv) = 0.0707 is the sample standard deviation of the vibration data; standardized data:

[0097]

[0098] Similarly, we can obtain Zv = [-1.06, 0.35, -0.35, 1.06], then the standardized data matrix is ​​Z:

[0099]

[0100] Covariance matrix ∑:

[0101]

[0102] The eigenvalues ​​are calculated as λ1 = 2.5 and λ2 = 0.1667. The corresponding eigenvectors are... Select the eigenvector v1 with the largest eigenvalue, and project its standardized data onto the principal components: Further analysis reveals that under normal operating conditions, P(-1.68|normal)≈0.148, P(0.14|normal)≈0.396, P(0.064|normal)≈0.398, and P(1.68|normal)≈0.148; under abnormal operating conditions, P(-1.68|abnormal)≈0.0006, P(0.14|abnormal)≈0.121, P(0.064|abnormal)≈0.104, and P(1.68|abnormal)≈0.391. Prior probabilities are set as P(normal) = 0.9 and P(abnormal) = 0.1.

[0103] The joint probability under normal operating conditions is:

[0104] P(Y|Normal)=0.148×0.396×0.398×0.148≈0.0035;

[0105] The joint probability under abnormal operating conditions is:

[0106] P(Y|abnormal)=0.0006×0.121×0.104×0.391≈0.000003;

[0107] Total probability:

[0108] P(Y)=0.0035×0.9+0.000003×0.1≈0.00315;

[0109] Posterior probability under normal operating conditions:

[0110]

[0111] Posterior probability of abnormal operating conditions:

[0112]

[0113] The joint probability is a weighted average of the total probabilities of the observed data under all possible operating conditions. It reflects the likelihood of observing a specific combination of data under all known operating conditions. The posterior probability is the probability that the equipment is in an abnormal operating condition given the observed data, obtained by calculating the joint probability and the posterior probability. P(normal|Y)≈0.999 is greater than P(abnormal|Y)≈0.0001, therefore it is determined that the equipment is trending towards normal operating conditions.

[0114] The platform layer performs dimensionality reduction on the temperature and humidity data Nc collected by the temperature and humidity sensor and the combustible gas concentration data Mv collected by the smoke detection sensor in the workshop using principal component analysis (PCA) to obtain the temperature and humidity data Sc and the combustible gas concentration data Sv. Prior probabilities P(normal) and P(fire risk) are defined for normal operating conditions and fire risk conditions in the historical temperature and humidity data and combustible gas concentration data recorded in the workshop. Based on the temperature and humidity data Sc and the combustible gas concentration data Sv, the conditional probability distributions P(Sc|normal), P(Sv|normal), P(Sc|fire risk), and P(Sv|fire risk) of the temperature and humidity data and smoke concentration data under normal and fire risk conditions are obtained.

[0115] Joint probability:

[0116] P(Sc,Sv)=P(Sc|Normal)·P(Sv|Normal)·P(Normal)+P(Sc|Fire Risk)·P(Sv|Fire Risk)·P(Fire Risk);

[0117] Calculate the posterior probability of fire risk conditions using Bayesian estimation:

[0118]

[0119] By calculating the joint probability and the posterior probability, it is concluded that if the posterior probability P(fire risk|Sc,Sv) exceeds the set threshold, then the workshop is considered to have a fire risk, where the threshold is set to 0.55.

[0120] After receiving data from multiple sensors, the platform layer first performs dimensionality reduction and noise reduction using PCA to simplify the multidimensional data into its main components and reduce noise interference. The dimensionality-reduced data is then subjected to multi-source information fusion analysis through Bayesian estimation to more accurately analyze the overall status of the workshop. Bayesian estimation can integrate the correlation between various sensors, improve the ability to identify abnormal trends, and thus provide a more global anomaly judgment. This analysis is more suitable for detecting potential trend risks or equipment status anomalies and has strong predictive power.

[0121] Example 4

[0122] This embodiment further elaborates and supplements the implementation of the present invention based on Embodiment 1, Embodiment 2 or Embodiment 3.

[0123] The application layer includes a comprehensive security hazard management module, a risk warning module, and an alarm module;

[0124] The comprehensive safety hazard management module is used to record and analyze real-time data uploaded from the platform layer, providing suggestions for workshop safety management.

[0125] The risk warning module receives data and analysis results uploaded from the platform layer, categorizing the data into three types: equipment anomalies, fire risks, and unsafe behaviors. Based on the analysis results, if the workshop shows a fire risk trend, it is marked as high priority; if the equipment shows an anomaly trend and the uploaded current and vibration data exceed the set alarm thresholds, it is marked as a major equipment anomaly (medium priority); if the equipment shows an anomaly trend and the uploaded current and vibration data do not exceed the set alarm thresholds, it is marked as a minor equipment anomaly (low priority). The risk warning module also marks high-risk unsafe behaviors as high priority and low-risk unsafe behaviors as low priority based on the analysis results. After setting the alarm priorities, the risk warning module sends alarm signals to the alarm module.

[0126] The alarm module is used to receive alarm signals from the risk warning module and issue an alarm.

[0127] Furthermore, the current sensor, vibration sensor, temperature and humidity sensor, and smoke detection sensor are all equipped with fixed alarm thresholds. The current sensor has a threshold of 10A, the vibration sensor has a threshold of 1000μm, the temperature and humidity sensor has a temperature threshold of 50℃, the humidity threshold is 60%RH, and the smoke detection sensor has a combustible gas threshold of 10ppm. If the data monitored by a single sensor exceeds the threshold, the alarm module is triggered to sound an alarm.

[0128] In implementation, a fixed alarm threshold is set for each sensor. Once the parameter monitored by the sensor exceeds the fixed alarm threshold, that is, exceeds the safe range, a threshold alarm signal is sent to the alarm module to immediately trigger the alarm. This type of alarm response is rapid and suitable for high-risk scenarios, such as fire risk detection. When the temperature, humidity, or flammable gas concentration exceeds the alarm threshold set by the sensor, the alarm is triggered immediately, facilitating rapid intervention by operators.

[0129] When a threshold alarm signal and an analysis result trigger an alarm simultaneously, the system prioritizes the threshold alarm signal. If only a single sensor issues a threshold alarm signal, but the analysis result does not show any abnormality, the system records the alarm and waits for other sensor signals or multi-source information fusion analysis in the next cycle to reduce the false alarm rate. If the analysis result shows an abnormal trend but does not reach the alarm threshold of a single sensor, the system marks it as a low-priority alarm to alert operators and prevent the accumulation of potential risks. This collaborative mode combines real-time performance with accuracy, minimizing false alarms and missed alarms while ensuring workshop safety, and enabling more flexible responses to safety risks in complex environments. Through the above judgment steps, the risk warning module can effectively screen and prioritize high-risk factors and ensure that different types of risks receive appropriate responses.

[0130] Furthermore, the presentation layer includes a large display screen, PC terminals, and mobile devices. The presentation layer serves as the terminal for data display and information presentation within the system. It displays data graphically on the workshop's large screen, PC terminals, and mobile devices. The large screen displays real-time monitoring footage, equipment status, and safety warnings. Operators can perform portable monitoring and management via PC and mobile terminals. The presentation layer uses a WebSocket interface to achieve real-time data synchronization with the application layer, ensuring the accuracy and timeliness of information and providing operators with a comprehensive safety monitoring view.

Claims

1. A workshop intelligent safety system based on multimodal data analysis, characterized in that: It includes a perception layer, an edge layer, a platform layer, and an application layer. The perception layer includes camera equipment installed in the workshop for real-time monitoring of the behavior of workshop operators and workshop equipment; current sensors and vibration sensors installed on various workshop equipment for collecting current and vibration data; and temperature and humidity sensors for collecting workshop temperature and humidity data, as well as smoke detection sensors for collecting combustible gas concentration values ​​in the smoke. The video data collected by the camera equipment in the perception layer and the sensor data collected by each sensor are transmitted to the edge computing control terminal in the edge layer for preliminary processing. The pre-processed data is then uploaded to the platform layer for comprehensive analysis of the workshop safety status using a multimodal data fusion algorithm. The analysis results are transmitted to the application layer for early warning, and finally to the display layer for graphical display.

2. The workshop intelligent safety system based on multimodal data analysis as described in claim 1, characterized in that: The video data collected by the camera devices in the perception layer and the sensor data collected by various sensors are transmitted to the edge computing control terminal in the edge layer for preliminary processing, specifically including: The edge computing control terminal decodes video data from the camera device into image frames in real time, and feeds the image frames into the trained YOLO v5 target detection model for specific target detection and takes screenshots. At the same time, the data transmitted by the current sensor, vibration sensor, temperature and humidity sensor and smoke detection sensor are filtered to remove noise.

3. The workshop intelligent safety system based on multimodal data analysis as described in claim 2, characterized in that: The noise removal process for the data transmitted by the current sensor, vibration sensor, temperature and humidity sensor, and smoke detection sensor through a data filtering algorithm specifically refers to: To address potential pulse noise in current and vibration sensors, a median filtering method is used to select the median value within a given window as the output. To ensure system response, the median filtering window is set to 5. The calculation formula is as follows: in The filtered output value at time t represents the signal value obtained after median filtering; x(t) represents the original sensor data at time t; median{*} represents the operation of taking the median in the window, that is, sorting the 5 data points in the window in ascending order and selecting the median value as the filtered output value; For temperature and humidity sensors and smoke detection sensors, the mean filtering method is used to smooth the signal by calculating the average value of data within a fixed window, thereby reducing high-frequency noise and random errors. The mean filtering method sets the window size to 4, and the calculation formula is as follows: in This represents the filtered output value at time t.

4. The workshop intelligent safety system based on multimodal data analysis as described in claim 1, characterized in that: The pre-processed data is uploaded to the platform layer for comprehensive analysis using a multimodal data fusion algorithm, specifically including: The platform layer performs dimensionality reduction and noise reduction on the processed sensor data using principal component analysis (PCA) to obtain dimensionality-reduced data. Then, it uses Bayesian estimation to fuse the dimensionality-reduced data and performs multi-source information fusion analysis to determine the normal or abnormal trend of the workshop. The platform layer applies a trained target tracking algorithm to the processed video data to further analyze the behavioral characteristics in the scene, identify unsafe behaviors, and take screenshots and mark the unsafe behaviors as high-risk or low-risk.

5. A workshop intelligent safety system based on multimodal data analysis as described in claim 4, characterized in that: Unsafe behaviors include workers entering hazardous areas of the workshop, workers climbing heights not wearing safety belts, workers not wearing proper clothing, workers incorrectly or not wearing safety helmets, workers smoking, or safety work signs not being placed properly.

6. The workshop intelligent safety system based on multimodal data analysis as described in claim 4, characterized in that: The platform layer uses Principal Component Analysis (PCA) to reduce the dimensionality and remove noise from the sensor data collected by each sensor. The specific steps are as follows: First, the sensor data collected by each sensor is standardized. Where x i Z represents data collected by different sensors. i Representing standardized data, a matrix σ is the sample mean, and σ is the sample standard deviation; Next, calculate the covariance matrix ∑ of the standardized data. The calculation formula is as follows: Z i This represents the standardized data matrix, where n is the number of samples. It is the mean vector of the standardized samples; Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues ​​λ. i and the corresponding eigenvector W i Choose the one with the largest eigenvalue λ. max The eigenvector W, whose standardized data is projected onto the principal components, is represented as: Y i =Z i ·W,i=1,2,3,4; Obtain the dimensionality-reduced data Y i The fusion is estimated using Bayesian methods, and the formula is as follows: Where P(A|Y1,Y2) is the posterior probability of event A occurring given the dimensionality-reduced data Y1 and Y2; P(Y1|A) and P(Y2|A) are the probabilities of observing Y1 and Y2 given that event A has occurred, i.e., the conditional probabilities of each sensor data; P(A) is the prior probability of event A, representing the probability of event A without other information; P(Y1,Y2) is the total probability of observing data Y1 and Y2, serving as a normalization factor.

7. A workshop intelligent safety system based on multimodal data analysis as described in claim 6, characterized in that: The platform layer performs dimensionality reduction on the current data Xc collected by the current sensor and the vibration data Xv collected by the vibration sensor using principal component analysis (PCA) to obtain the current data Yc and vibration data Yv. Prior probabilities P(normal) and P(abnormal) for normal and abnormal operating conditions are defined in the historical operating data of the equipment. Based on the current data Yc and vibration data Yv, the conditional probability distributions P(Yc|normal), P(Yv|normal), P(Yc|abnormal), and P(Yv|abnormal) for the current data Yc and vibration data Yv under normal and abnormal operating conditions are obtained, with the joint probability: P(Yc, Yv) = P(Yc|Normal)·P(Yv|Normal)·P(Normal) + P(Yc|Abnormal)·P(Yv|Abnormal)·P(Abnormal) Calculate the posterior probability of abnormal operating conditions using Bayesian estimation: By calculating the joint probability and posterior probability, it is found that if P(abnormal|Yc,Yv) exceeds the set threshold, where the threshold is set to 0.

6.

8. A workshop intelligent safety system based on multimodal data analysis as described in claim 6, characterized in that: The platform layer performs dimensionality reduction on the temperature and humidity data Nc collected by the temperature and humidity sensor and the combustible gas concentration data Mv collected by the smoke detection sensor in the workshop using principal component analysis (PCA) to obtain the temperature and humidity data Sc and the combustible gas concentration data Sv. Prior probabilities P(normal) and P(fire risk) are defined for normal and fire risk conditions in the historical temperature and humidity data and combustible gas concentration data recorded in the workshop. Based on the temperature and humidity data Sc and the combustible gas concentration data Sv, the conditional probability distributions P(Sc|normal), P(Sv|normal), P(Sc|fire risk), and P(Sv|fire risk) of the temperature and humidity data and smoke concentration data under normal and fire risk conditions are obtained. Joint probability: P(Sc,Sv)=P(Sc|Normal)·P(Sv|Normal)·P(Normal)+P(Sc|Fire Risk)·P(Sv|Fire Risk)·P(Fire Risk); Calculate the posterior probability of fire risk conditions using Bayesian estimation: By calculating the joint probability and the posterior probability, it is concluded that if the posterior probability P(fire risk|Sc,Sv) exceeds the set threshold, then the workshop is considered to have a fire risk, where the threshold is set to 0.

55.

9. A workshop intelligent safety system based on multimodal data analysis as described in claim 4, characterized in that: The application layer includes a comprehensive security hazard management module, a risk warning module, and an alarm module; The comprehensive safety hazard management module is used to record and analyze real-time data uploaded from the platform layer, providing suggestions for workshop safety management. The risk warning module is used to receive data and analysis results uploaded from the platform layer, and divides the data into three categories, including equipment abnormalities, fire risks, and unsafe behaviors. Then, based on the analysis results, if the workshop has a fire risk trend, it is marked as high priority; if the equipment has an abnormal trend and the current and vibration data in the uploaded data exceed the set alarm threshold, it is marked as major equipment abnormality, medium priority; if the equipment has an abnormal trend and the uploaded current and vibration data do not exceed the set alarm threshold, it is marked as minor equipment abnormality, low priority. Based on the analysis results, the risk warning module marks high-risk unsafe behaviors as high priority and low-risk unsafe behaviors as low priority; after setting the alarm priority, the risk warning module sends an alarm signal to the alarm module. The alarm module is used to receive alarm signals from the risk warning module and issue an alarm.

10. A workshop intelligent safety system based on multimodal data analysis as described in claim 9, characterized in that: The current sensor, vibration sensor, temperature and humidity sensor, and smoke detection sensor are all equipped with fixed alarm thresholds. The current sensor has an alarm threshold of 10A, the vibration sensor has an alarm threshold of 1000μm, the temperature and humidity sensor has an alarm threshold of 50℃, the humidity sensor has an alarm threshold of 60%RH, and the smoke detection sensor has an alarm threshold of 10ppm for combustible gases. If the data monitored by a single sensor exceeds the set alarm threshold, a threshold alarm signal is sent to the alarm module, which then triggers the alarm module to sound an alarm.

11. A workshop intelligent safety system based on multimodal data analysis as described in claim 10, characterized in that: When the threshold alarm signal and the analysis result trigger an alarm simultaneously, the system prioritizes processing the threshold alarm signal. If only a single sensor issues a threshold alarm signal, and the analysis results do not show any abnormality, the system will record the alarm and wait for other sensor signals or multi-source information fusion analysis in the next cycle; if the analysis results show an abnormal trend, but do not reach the alarm threshold of a single sensor, the system will mark it as a low-priority alarm.

12. The workshop intelligent safety system based on multimodal data analysis as described in claim 1, characterized in that: The display layer includes a large display screen, a PC terminal, and a mobile terminal.

Images