Industrial workshop safety production behavior identification method based on multi-mode deep learning

Abstract

The invention provides an industrial workshop safety production behavior identification method based on multi-modal deep learning. The method comprises the following steps: acquiring multi-modal data of an industrial workshop; performing feature extraction and weighted fusion on the multi-modal data to generate a fusion feature vector; performing feature enhancement on the fused feature vector to obtain enhanced features; performing time sequence feature extraction on the enhanced features by using a time sequence difference convolutional network, and generating a worker behavior classification result and a risk prediction result; encrypting the result of the worker behavior classification and the result of the risk prediction by adopting symmetric encryption and asymmetric encryption; constructing a federal learning framework according to the encrypted result of worker behavior classification and the result of risk prediction; and performing compliance detection on the result of real-time worker behavior classification and the result of real-time risk prediction by using the optimized global model, judging whether the behaviors of the workers conform to the safety production specifications of the workshop, and generating a corresponding risk level and an alarm response.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal deep learning, and in particular relates to a method for identifying safe production behaviors in industrial workshops based on multimodal deep learning. Background Art

[0002] With the rapid development of industrial automation, the issue of safe production management in industrial workshops has received increasing attention. In recent years, production activities in industrial workshops have become increasingly dependent on intelligent equipment and automated production lines. However, industrial workshops still face many safety management challenges. Traditional safety management methods, such as manual inspections and single-sensor monitoring, are no longer able to meet the requirements for efficient, real-time, and comprehensive safety monitoring in modern industrial environments. Especially with the increasing complexity of production processes and the diversification of workers' working environments, how to accurately identify workers' safety behaviors and monitor equipment status in real time have become urgent challenges in the field of safety management.

[0003] Existing safety behavior recognition technologies rely primarily on two approaches: behavioral analysis based on video surveillance, and environmental monitoring based on sensor data. Video surveillance-based safety behavior recognition technologies typically use image processing and computer vision to analyze whether workers are wearing safety equipment or operating according to regulations. While these technologies can provide relatively intuitive behavior analysis results, their recognition accuracy is limited by factors such as lighting, angle, and occlusion, and they lack sufficient reliability in complex environments. Furthermore, video surveillance technology suffers from poor real-time performance, high data storage pressure, and insufficient detailed information in the surveillance footage, all of which limit its application in industrial environments.

[0004] Another common technology is behavioral monitoring based on sensor data, such as temperature sensors, vibration sensors, and environmental monitoring equipment. These devices can provide real-time data feedback on the factory environment. However, sensor technology itself has issues with data accuracy and reliability, especially when analyzing data from multiple dimensions. Sensor data alone is often unable to determine whether workers have violated safety regulations, and it is also difficult to detect potential safety hazards. Furthermore, monitoring methods based on a single data source often overlook the interaction of diverse information, resulting in safety hazards not being discovered in the early stages.

[0005] Although some comprehensive technologies have attempted to combine video and sensor data for multimodal monitoring in recent years, many problems remain. First, the multimodal data fusion techniques used in existing methods are insufficient when dealing with large amounts of heterogeneous data, resulting in low processing efficiency and difficulty in achieving cross-modal information sharing and analysis. Second, many methods for identifying safety behaviors rely on the analysis of static data, which makes the system lack sufficient flexibility and adaptability when facing dynamic environments and real-time changes, and is unable to accurately predict future safety hazards. Summary of the Invention

[0006] The purpose of this invention is to propose a method for identifying safe production behaviors in industrial workshops based on multimodal deep learning. By combining multimodal data fusion, time series dynamic prediction and privacy protection technology, the present invention can provide a comprehensive, multi-level safety behavior recognition system, which can effectively improve the overall safety management level of industrial workshops while ensuring high efficiency and real-time performance.

[0007] An embodiment of the present invention provides a method for identifying safe production behaviors in industrial workshops based on multimodal deep learning, the method comprising:

[0008] Acquire multimodal data from industrial workshops, including video data, sensor data, and audio data;

[0009] Extracting features from the multimodal data and weighted fusing them to generate a fused feature vector;

[0010] Perform feature enhancement on the fused feature vector to obtain enhanced features;

[0011] A temporal difference convolutional network is used to extract temporal features from the enhanced features, which are input into a fully connected layer. The results of worker behavior classification and risk prediction are obtained by combining behavioral smoothing regularization with multi-task learning.

[0012] Symmetric encryption and asymmetric encryption are used to encrypt the results of the worker behavior classification and the results of the risk prediction, generating encrypted results of the worker behavior classification and the results of the risk prediction, and performing hash verification to verify data integrity; wherein, the symmetric encryption encrypts the results of the worker behavior classification and the results of the risk prediction, generating symmetrically encrypted data and a key used for symmetrical encryption; the asymmetric encryption encrypts the key used for encryption, generating an encrypted key used for symmetrical encryption;

[0013] A federated learning framework is constructed based on the encrypted worker behavior classification results and risk prediction results to generate an optimized global model; wherein local models of the federated learning framework are trained based on the generated encrypted worker behavior classification results and risk prediction results, a global model is obtained by weighted averaging the local models, and the global model is distributed to various device nodes in the industrial workshop to perform multiple rounds of federated learning training to generate an optimized global model;

[0014] Real-time worker behavior data and real-time safety risk prediction data are obtained, and the results of real-time worker behavior classification are obtained based on the real-time worker behavior data, and the results of real-time risk prediction are obtained based on the real-time safety risk prediction data. The optimized global model is used to perform compliance testing on the results of real-time worker behavior classification and real-time risk prediction, to determine whether the workers' behavior complies with the safety production regulations of the workshop, and to generate corresponding risk levels and alarm responses.

[0015] Furthermore, the shape of the fused feature vector is (N, T, D fused ), where N is the number of samples, T is the number of time steps, and D fused is the characteristic dimension;

[0016] The step of performing feature enhancement on the fused feature vector to obtain enhanced features specifically includes:

[0017] Dynamically generate feature enhancement items through local temporal difference;

[0018] The fused feature vector is enhanced based on the feature enhancement item to generate an enhanced feature.

[0019] Furthermore, the behavior smoothing regularization is used to smooth the results of worker behavior classification and risk prediction to avoid overfitting; and the multi-task learning is used to simultaneously process the two tasks of worker behavior classification and risk prediction.

[0020] Furthermore, when performing symmetric encryption and asymmetric encryption to encrypt the results of the worker behavior classification and the risk prediction, the k-anonymization method is used to desensitize the worker's identity information to ensure that the worker's identity information will not be leaked.

[0021] Furthermore, the federated learning framework further includes:

[0022] On each device node, the local model is trained through back-propagation and optimization algorithms.

[0023] Furthermore, in the federated learning framework, differential privacy technology is used to process updates of the federated learning framework to ensure that updates of a single node do not leak its local data.

[0024] Furthermore, the optimized global model is used to perform compliance testing on the results of the real-time worker behavior classification and the real-time risk prediction to determine whether the worker's behavior complies with the workshop's safety production regulations and generate corresponding risk levels and alarm responses, specifically including:

[0025] Build a compliance detection model based on the optimized global model;

[0026] Based on the compliance detection model, the real-time worker behavior classification results are judged and a compliance indicator is output, that is, whether the worker's current behavior complies with safety regulations;

[0027] Determine in real time whether an alarm is required based on the results of the real-time risk prediction and the identification of compliance;

[0028] Once an alarm is triggered, management personnel will be notified, and different levels of alarm information will be generated based on the risk level and non-compliant identification of workers' behavior, so that management personnel can respond quickly.

[0029] Furthermore, the construction of a compliance detection model based on the optimized global model specifically includes:

[0030] The output of the compliance detection model is a binary output, where output 1 indicates that the behavior complies with safety regulations, and output 0 indicates that the behavior does not comply with safety regulations; the input of the compliance detection model is the optimized global model and the results of real-time worker behavior classification.

[0031] Furthermore, the real-time determination of whether an alarm is required based on the real-time risk prediction result and the compliance indicator specifically includes:

[0032] An alarm threshold is set based on the results of real-time risk prediction. When the results of real-time risk prediction exceed the alarm threshold, an alarm signal is triggered. The alarm threshold is adjusted according to the actual safety standards and historical data of the workshop to ensure that the system responds to high-risk events in a timely manner.

[0033] Furthermore, the different levels of alarm information include:

[0034] Alarm level: Determine the alarm level based on the value of the real-time risk prediction result;

[0035] Violation details: The system will display the worker's real-time worker behavior classification;

[0036] Real-time video monitoring: By accessing real-time video monitoring data, the real-time images of the alarm area are automatically pushed to the management personnel;

[0037] Alarm response: Each alarm signal will notify relevant personnel via SMS, email, and APP.

[0038] The beneficial technical effects of the present invention are at least as follows:

[0039] In response to these shortcomings in the prior art, the present invention proposes a new method for identifying safe production behaviors in industrial workshops, aiming to overcome the shortcomings of the prior art in data fusion, real-time performance, and safety prediction. By integrating information from multimodal data sources (such as video, sensors, audio, etc.), the present invention can improve recognition accuracy while reducing the reliance of traditional methods on a single data source. More importantly, the present invention can effectively integrate temporal dynamic features and combine modern deep learning technology to achieve real-time monitoring and dynamic prediction of worker behavior and equipment status in the workshop. This method can not only accurately identify safe behaviors, but also promptly warn of potential risks. Especially in a complex and changeable environment such as an industrial workshop, it can adapt to different operating scenarios and achieve intelligent safety management.

[0040] Furthermore, the present invention proposes a highly effective data and privacy protection solution, ensuring the privacy and security of sensitive information such as worker behavior and equipment status during real-time monitoring and data analysis. This innovation ensures that the present invention not only ensures safety but also meets the stringent data privacy requirements of modern industrial workshops. This is particularly true in scenarios involving both personal worker data and production data, ensuring that the safety behavior recognition system in industrial workshops meets safety compliance requirements without infringing on employee privacy.

[0041] By combining multimodal data fusion, time series dynamic prediction and privacy protection technology, the present invention can provide a comprehensive, multi-level safety behavior recognition system, which can effectively improve the overall safety management level of industrial workshops while ensuring high efficiency and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0043] Figure 1 This is a flow chart of the method for identifying safe production behaviors in industrial workshops based on multimodal deep learning of the present invention. DETAILED DESCRIPTION

[0044] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0045] In one embodiment, Figure 1 As shown, a method for identifying safe production behaviors in industrial workshops based on multimodal deep learning is provided; the method includes the following steps:

[0046] S1. Acquire multimodal data of an industrial workshop, including video data, sensor data, and audio data; perform feature extraction and weighted fusion on the multimodal data to generate a fused feature vector.

[0047] Specifically, data collection:

[0048] Video data:

[0049] Collection equipment: High-resolution cameras installed in the workshop, with a video capture resolution of 1920×1080 and a sampling frequency of 30 frames per second.

[0050] Collection method: Each surveillance camera collects real-time video streams of workers' work areas, including whether workers are wearing safety protection equipment and whether they are operating in accordance with regulations.

[0051] Sensor data:

[0052] Collection equipment: Environmental monitoring sensors (such as temperature and humidity sensors, vibration sensors, and pressure sensors) are installed in key equipment and work areas in the workshop.

[0053] Collection method: These sensors collect real-time environmental data within the workshop, such as temperature, humidity, and pressure. Each sensor data is sampled at a 1Hz frequency, ensuring real-time monitoring of equipment and environmental changes.

[0054] Audio data:

[0055] Collection equipment: Audio sensors installed in the workshop are mainly used to collect audio information during workers' operations (such as the working sounds of machines, workers' operating voices, etc.).

[0056] Collection method: The audio sensor has a sampling frequency of 44.1kHz and is used to capture background noise and employee voices in the workshop to help identify safety incidents (such as worker communication and equipment failure sounds).

[0057] Further, data cleaning and denoising:

[0058] Video data cleaning:

[0059] Denoising: Use Gaussian blur and median filtering to denoise the video to reduce the impact of changes in the workshop environment (such as lighting changes, camera shake, etc.).

[0060] Data repair: If video frames are lost or certain areas are blocked (such as a worker with his back to the camera), video frame interpolation and image filling technology are used to repair missing frames and partial images to maintain the continuity and integrity of the video.

[0061] Sensor data cleaning:

[0062] Outlier detection and removal: Sensor data is detected based on the Z-score to remove abnormal data caused by equipment failure or external environmental influences. For example, if the value of the temperature and humidity sensor is outside the reasonable range (such as the temperature exceeds 100°C), the data is considered abnormal.

[0063] Missing value handling: Missing data points are interpolated using linear interpolation or sliding window interpolation based on adjacent data. Long-term missing data segments are marked as invalid data.

[0064] Audio data cleaning:

[0065] Noise Removal: Use spectral subtraction and Wiener filtering to reduce the noise of audio signals and remove background noise (such as device operation sounds and external interference sounds).

[0066] Audio segmentation: Cut the continuous audio signal into segments of fixed time periods (for example, one segment every 5 seconds) to facilitate subsequent feature extraction.

[0067] Furthermore, to ensure that multimodal data can work together in subsequent deep learning models, we standardize and normalize different data sources.

[0068] Video data normalization:

[0069] Map the pixel value range in the video from [0,255] to [0,1], that is, perform normalization:

[0070]

[0071] Among them, X video Represents the normalized video pixel value, X raw is the original video pixel value.

[0072] Sensor data normalization:

[0073] The sensor data is processed using the Z-score normalization method to ensure that the mean of each sensor data is 0 and the standard deviation is 1:

[0074]

[0075] Among them, X sensor represents the raw sensor data, μ sensor and σ sensor are the mean and standard deviation of the sensor data, respectively.

[0076] Audio data normalization:

[0077] Perform time-frequency transformation on the audio data, convert it from the time domain to the frequency domain, extract audio features through Mel-frequency cepstral coefficients (MFCC), and then normalize it:

[0078]

[0079] Among them, X MFCC It is the audio feature extracted by Mel frequency cepstral coefficients. is the normalized audio feature.

[0080] Furthermore, in order to adapt to subsequent time series model processing, video, sensor and audio data need to be cut and segmented.

[0081] Specifically, video data segmentation involves cutting the video data into smaller segments based on time periods, such as 10-second segments. During segmentation, ensure that each segment contains continuous worker actions to facilitate subsequent behavioral analysis.

[0082] Sensor data segmentation: Sensor data is also segmented by time windows with a window size of 5 seconds. The sensor data in each window is used to describe the status of the device or environment within that time period.

[0083] Audio data segmentation: The audio data will be segmented into 5-second segments, and the corresponding Mel-Frequency Cepstral Coefficient (MFCC) features will be extracted for each segment.

[0084] By precisely cleaning, standardizing, and segmenting video, sensor, and audio data, this step provides a high-quality data foundation for subsequent deep learning models. Innovation lies in the use of specific image enhancement, sensor data interpolation, and audio noise reduction methods, ensuring that the data is not only clean but also meets model training requirements, thereby ensuring efficient learning and accurate predictions for subsequent models.

[0085] Furthermore, feature extraction and fusion strategies are performed:

[0086] Video data feature extraction:

[0087] Video data has strong spatiotemporal information. We use convolutional neural networks (CNNs) to extract spatial features from videos. For example, we use the pre-trained ResNet50 model to extract spatial feature representations of video frames. The ResNet50 model consists of 50 convolutional layers, which effectively capture details in images, such as workers' posture and movements, as well as the operating status of equipment.

[0088] For each video frame, the extracted spatial feature representation is a D video -dimensional feature vector.

[0089] Feature extraction formula:

[0090]

[0091] in, is the feature of each frame of video extracted from ResNet50, X video It is the normalized raw video data.

[0092] Sensor data feature extraction:

[0093] Sensor data is typically continuous time series data. Long Short-Term Memory (LSTM) networks are used to process this data and capture temporal dependencies. LSTMs are well-suited for capturing long-term dependencies in time series data and can effectively handle temporal variations in sensor data.

[0094] The temporal features of sensor data are extracted through the LSTM model, and the output is D sensor -dimensional feature vector.

[0095] Feature extraction formula:

[0096]

[0097] in, represents the sensor time series features extracted from LSTM, is the normalized sensor data.

[0098] Audio data feature extraction:

[0099] In audio data processing, we combine convolutional neural networks (CNNs) and LSTM networks. CNN first extracts the time-frequency features of the audio, and then passes the extracted features to LSTM to capture the temporal information of the audio.

[0100] We use Mel-Frequency Cepstral Coefficients (MFCCs) to extract audio features. These features effectively describe the spectral characteristics of audio signals, especially highlighting key audio signals (such as worker voices or equipment malfunction sounds) amidst background noise in a workshop.

[0101] The output audio feature is a D audio dimensional vector.

[0102] Feature extraction formula:

[0103]

[0104] in, represents the features extracted from the audio data, It is the normalized audio data.

[0105] Furthermore, since the features of video, sensor, and audio data have different spatial and temporal properties, we adopt a weighted fusion strategy when fusing these features:

[0106] Characteristic vectors for each data type Fusion is performed by weighted averaging. The weight of each data source (w video ,w sensor ,w audio ) can be automatically optimized through the training process.

[0107] The feature vector after weighted fusion is represented as X fused ,This fused feature vector will serve as the input for subsequent steps (behavior recognition and risk prediction).

[0108] Weighted fusion formula:

[0109]

[0110] Among them, X fused is the fused feature, w video ,w sensor , and w audio are the weight coefficients for video, sensor, and audio data, and are the data features of each modality.

[0111] Furthermore, to prevent feature overfitting, we introduced a regularization term. By adding an additional regularization term, we penalize the weight of each modality feature, thereby reducing the model's over-reliance on a single modality. This regularization term ensures that the model learns information from multiple data sources in a balanced manner.

[0112] Regularization formula:

[0113]

[0114] in, is the regularization loss function, and λ is a regularization hyperparameter that controls the weight of the regularization term. This regularization term penalizes excessive weights to ensure that each modality feature is not overly biased during fusion.

[0115] By combining weighted fusion, LSTM, and CNN, we are able to extract complementary feature information from video, sensor, and audio data, thereby enhancing the model's ability to identify safety behaviors in workshops.

[0116] Finally, the fused feature vector X is output fused It will serve as input for subsequent steps (time series behavior recognition and security risk prediction) and be provided to subsequent deep learning models for training and prediction.

[0117] As can be seen, the multimodal feature extraction method, combining weighted fusion, LSTM, and CNN, ensures that video, sensor, and audio data are fully integrated, providing efficient and high-quality feature input for subsequent behavior recognition and risk prediction. Through regularization mechanisms, we further optimize the feature fusion process, reduce the risk of overfitting, and enhance the model's generalization capabilities. This solution effectively improves the accuracy of safe behavior recognition in industrial workshop environments, meeting the core requirements of the patent.

[0118] S2. Enhance the fused feature vector to obtain enhanced features; use a temporal difference convolutional network to extract temporal features from the enhanced features, input the temporal features into a fully connected layer, and combine behavioral smoothing regularization with multi-task learning to obtain worker behavior classification results and risk prediction results.

[0119] Furthermore, the video, sensor and audio features are obtained through weighted fusion, with the shape of (N, T, D fused ), where N is the number of samples, T is the number of time steps, and D fused is the characteristic dimension.

[0120] Specifically, at this stage, the goal is to identify worker behaviors and predict potential safety risks. To capture dynamically changing behavioral patterns, we will use a time series network model to effectively extract and predict features from time series data.

[0121] Furthermore, to address sudden behavioral changes in industrial workshops, we designed a dynamic feature generation (DFA) approach. This innovative approach enhances features at each time step, better capturing short-term variations in time series data. For example, workers may experience sudden behavioral changes during urgent operations, changes that are often overlooked in traditional time series modeling.

[0122] Through local temporal differencing, feature enhancement terms are dynamically generated, which can capture sudden changes in worker behavior and incorporate them into the original features to help the model better understand these changes.

[0123] This innovation enhances the model's sensitivity to local temporal changes, and is particularly suitable for frequent and sudden operational changes in workshop environments (such as sudden worker behavior, equipment abnormalities, etc.).

[0124] Enhanced feature formula:

[0125] X aug =X fused +γ·(X fused -X fused,t-1 ) (9)

[0126] Among them, X aug is the enhancement feature, γ is the dynamic enhancement regulation factor, X fused,t-1 is the fused feature vector of the previous time step, X fused is the fused feature vector of the current time step.

[0127] This approach enables the model to adjust input features based on workers' real-time behavior, avoiding sudden behavioral patterns that traditional methods may ignore and improving the model's responsiveness to short-term behavioral changes.

[0128] Furthermore, in the temporal feature extraction stage, we introduced a temporal difference convolutional network (TDCN), which combines a convolutional neural network (CNN) with a temporal difference operation and is specifically designed to process multimodal data with temporal dependencies. Although traditional LSTM networks can capture long-term temporal dependencies, they may not be as flexible as convolutional networks when dealing with rapid changes in shorter time frames.

[0129] The innovation of TDCN lies in its ability to extract local differential features from time series data through differential convolution. These features are highly effective in describing subtle changes and rapid reactions in worker behavior. Compared to traditional LSTMs, TDCN is better at capturing short-term, sudden behavioral patterns, such as abnormal worker operations or rapid equipment reactions.

[0130] TDCN processing formula:

[0131] X TDCN =Conv(ΔX fused ) (10)

[0132] Where ΔX fused =X fused -X fused,t-1 is the feature after temporal difference, Conv represents the convolution operation, X TDCNIt is the temporal feature extracted by TDCN.

[0133] TDCN uses differential convolution operations to effectively capture small differences and instantaneous changes in time series data, which is particularly suitable for dynamic environments such as industrial workshops with sudden changes.

[0134] After extracting the time series features, we input them into the fully connected layer (FC) for worker behavior classification and safety risk prediction. Through the FC layer, we map the time series features generated by LSTM or TDCN to specific behavior categories and risk predictions.

[0135] Worker behavior classification: We use the Softmax function to classify worker behavior, such as determining whether workers are wearing safety equipment and whether they are operating in accordance with regulations.

[0136] Safety Risk Prediction: We use regression analysis to map time-series features into continuous risk level predictions. This allows the model to output the potential safety risk posed by worker behavior (e.g., “low risk,” “medium risk,” “high risk”).

[0137] Worker behavior classification formula:

[0138] Y class =Softmax(FC(h T )) (11)

[0139] Among them, Y class is the result of worker behavior classification, h T is the temporal feature after TDCN processing.

[0140] Risk prediction formula:

[0141] Y risk =FC regression (h T ) (12)

[0142] Among them, Y risk Represents the result of risk prediction, FC regression is the regression layer.

[0143] Worker behavior classification and safety risk prediction are carried out simultaneously. The model can classify worker behavior based on real-time collected data, predict potential safety risks, and realize intelligent safety management.

[0144] Furthermore, to optimize model performance, especially in dynamic and complex shop floor environments, we introduced behavioral smoothing regularization (BSR) and multi-task learning (MTL) strategies. The BSR regularization term helps smooth the outputs of the LSTM and TDCN, preventing overfitting. MTL enables the model to simultaneously handle both worker behavior classification and risk prediction, making the learning process more comprehensive.

[0145] BSR regularization formula:

[0146]

[0147] in, is the behavioral smoothing regularization term, X TDCN,t represents the TDCN output at the t-th time step.

[0148] Multi-task learning formula:

[0149]

[0150] in, is the total loss, Classify losses for worker behavior, Forecasting losses for risk, is the behavioral smoothing regularization term.

[0151] Ultimately, the output from all model processing will provide the following information:

[0152] Worker behavior classification results: Softmax is used to output the category of worker behavior, such as "wearing safety equipment" and "not wearing safety equipment."

[0153] Security risk prediction results: Output potential security risk levels through regression, such as "low risk", "medium risk" or "high risk".

[0154] These outputs will be used to generate real-time safety warnings, helping workshop managers identify potential safety hazards in a timely manner.

[0155] As can be seen, the introduction of innovative technologies such as dynamic feature generation, temporal difference convolutional networks, and multi-task learning has made temporal behavior recognition and safety risk prediction more accurate and efficient. The model not only identifies worker behavior in real time but also predicts potential safety risks, providing real-time, safe decision-making support for workshop managers, significantly improving workshop safety management.

[0156] S3. Use symmetric encryption and asymmetric encryption to encrypt the results of the worker behavior classification and the results of the risk prediction, generate encrypted results of the worker behavior classification and the results of the risk prediction, and perform hash verification to verify data integrity; wherein, the symmetric encryption encrypts the results of the worker behavior classification and the results of the risk prediction, generates symmetrically encrypted data and a key used for symmetrical encryption; the asymmetric encryption encrypts the key used for encryption, and generates an encrypted key used for symmetric encryption.

[0157] Specifically, this solution uses a combination of symmetric encryption (AES) and asymmetric encryption (RSA):

[0158] Symmetric encryption (AES): Symmetric encryption methods (such as AES) are suitable for encrypting large amounts of data. We will use AES to encrypt the worker behavior classification results and security risk prediction results. The encryption process uses a shared key K AES , which can effectively protect the privacy of data.

[0159] AES encryption process: Encrypt each output data: Y class and Y risk Encrypted into C class and C risk The specific encryption formula is as follows:

[0160] C class =AES(Y class ,K AES ) (15)

[0161] C risk =AES(Y risk ,K AES ) (16)

[0162] Among them, C class and C risk are the encrypted data, K AES The key used for encryption.

[0163] Note: The main purpose of the AES encryption process is to ensure that worker behavior data and security risk data are not leaked during storage and transmission.

[0164] Asymmetric encryption (RSA): Asymmetric encryption algorithms (such as RSA) are used to protect the security of key transmission. We will use the RSA algorithm to encrypt the AES key K AES Encryption is performed to ensure the security of the key during transmission.

[0165] RSA encryption process: Encrypt K using the recipient's public key AES :

[0166] C AES =RSA(K AES ,Public Key) (17)

[0167] Among them, C AES It is the AES key after RSA encryption, and Public Key is the public key of the recipient.

[0168] Description: The main purpose of RSA encryption is to ensure that the key K AES It will not be stolen or tampered with during the transmission process, ensuring the security of the data encryption process.

[0169] As you can see, combining AES and RSA encryption ensures both efficient data encryption and secure key transmission. This encryption method has high practical value in industrial workshop environments where efficient data transmission and protection are required.

[0170] Furthermore, to further protect worker privacy, we employ data desensitization technology, particularly when processing worker behavior classification and safety risk prediction results, to ensure that worker identity information is not leaked. We use k-anonymization to remove any information that could reveal an individual's identity.

[0171] k-anonymization: To protect worker identities, we desensitize sensitive data using k-anonymization. This method removes worker identities, retaining only data related to worker behavior and risk profiles. This ensures that even if the data is leaked, no specific individuals can be identified.

[0172] Desensitized data: After k-anonymization, Y class and Y risk It still provides enough information for subsequent analysis, but cannot be linked to an individual's identity.

[0173] By combining encryption and data desensitization technologies, this solution effectively avoids the risk of worker personal information leakage. Especially when processing worker behavioral data, it ensures data privacy while retaining sufficient information for behavioral analysis and risk prediction.

[0174] Furthermore, to ensure that the encrypted data has not been tampered with during transmission, we introduced a hash check mechanism. Hash check ensures that each data packet is integrity checked during transmission. The receiver can compare the hash value with the received data to determine whether the data has been tampered with.

[0175] Hash check: After encryption, the worker behavior classification data C class and security risk prediction data Crisk In this example, we will use a hash algorithm such as SHA-256 to calculate its hash value H. class and H risk , and then transmit these hashes along with the encrypted data.

[0176] Hash check formula:

[0177] H class =SHA-256(C class ) (18)

[0178] H risk =SHA-256(C risk ) (19)

[0179] Among them, H class and H risk This is the hash value of the worker behavior classification results and safety risk prediction results. After decryption, the recipient calculates the hash value and compares it with the hash value during transmission to ensure that the data has not been tampered with during transmission.

[0180] Combining hashing and encryption technology, data not only maintains privacy and security during transmission, but also ensures data integrity. Even in insecure transmission environments, data can be effectively protected and tamper-proof.

[0181] Furthermore, the encrypted worker behavior classification results (C class ): Worker behavior classification data is encrypted with AES to ensure the privacy and security of the data during storage and transmission.

[0182] The encrypted security risk prediction results (C risk ): Security risk prediction data is encrypted by AES to ensure the privacy and security of data during storage and transmission.

[0183] This encrypted data will be stored in the workshop's security system and made available to management personnel through a decryption process when needed. In the workshop management system, this encrypted data will be decrypted and used to generate real-time safety warnings, helping to promptly identify and prevent safety risks.

[0184] By combining innovative methods such as AES and RSA encryption, data desensitization (k-anonymization), and hashing, we can immediately and comprehensively safeguard the privacy, security, and integrity of industrial shop floor data. This solution, in particular, provides effective privacy protection and data encryption measures when processing worker behavior and equipment data, and ensures data security during transmission through integrity verification. This innovative combination of technologies enables our patented solution to meet the stringent data privacy and security requirements of industrial shops, while ensuring efficient and secure model application.

[0185] S4. Construct a federated learning framework based on the encrypted worker behavior classification results and risk prediction results to generate an optimized global model; wherein the local models of the federated learning framework are obtained by performing local model training based on the generated encrypted worker behavior classification results and risk prediction results, and a global model is obtained by performing weighted averaging on the local models, and the global model is distributed to various equipment nodes in the industrial workshop to perform multiple rounds of federated learning training to generate an optimized global model.

[0186] Specifically, federated learning is a distributed training framework that allows different device nodes to independently train models on local data and periodically upload updated model parameters (rather than the original data) to a central server. The central server aggregates the models received from multiple nodes to improve the accuracy of the global model.

[0187] Local model training: Each device node (such as each sensor or surveillance camera in the workshop) will be based on the encrypted and desensitized data C class and C risk Perform local training to generate a local model M local This data is encrypted locally to ensure privacy protection.

[0188] Training process: On each device node, the local model is trained through back propagation and optimization algorithms (such as Adam optimizer). Assume M local is the local model of the node, and the training process is:

[0189]

[0190] in, is the model parameter of the tth round, η is the learning rate, is the local loss function, C class and C risk It is encrypted worker behavior classification and safety risk prediction data.

[0191] Example: Assume that node 1 has C class and C risk The worker's behavior and risk data. Node 1 will use the Adam optimizer to train a local model M using local data. local , and continuously adjust the model parameters during the training process to minimize the loss function

[0192] Furthermore, after each round of local training, each device node uploads its updated model parameters to the central server for aggregation. To ensure that the aggregated model effectively represents the data distribution of all nodes, we use a weighted average method to aggregate the model updates of each node.

[0193] Weighted model aggregation: The model contribution weight of each node is proportional to its local data volume. The central server allocates the weight of the model to the node based on the data volume w. i Perform weighted averaging on the model parameters to obtain the global model θ global :

[0194]

[0195] Among them, w i is the weight coefficient of the i-th node, N is the total number of nodes participating in the training, θ local,i is the local model parameter of the ith node.

[0196] Note: This aggregation process ensures that the contribution of each node is adjusted according to its data volume, so that the global model can better fit the characteristics of different nodes and data distribution.

[0197] Furthermore, once the global model is obtained through aggregation, the central server will send the new global model θ global Distribute to each device node for use in the next round of training. Through multiple rounds of federated learning training, the performance of the model will gradually improve.

[0198] Model Update and Security: To ensure data privacy, we use differential privacy technology to process model updates, ensuring that updates to a single node do not leak its local data.

[0199] Differential Privacy Protection:

[0200] Add noise to the model gradient of each node to ensure privacy. The formula is:

[0201]

[0202] in, is the gradient with noise added, is the noise added to the gradient, σ 2 is the variance of the noise.

[0203] Innovation: By using differential privacy technology, we ensure that the gradient update of each node does not expose the local data of the node, thereby protecting data privacy.

[0204] It is understandable that after several rounds of federated learning training, the central server obtains the optimized global model θ globalThe model can be used to predict workers’ behaviors and potential risks. The final global model will be applied to the shop floor system to provide managers with real-time safety warnings.

[0205] Output: The optimized global model (θ global ): A global model that contains the data features of all nodes, obtained through federated learning training and optimization, and is used to classify workers' behaviors and predict risks in real time.

[0206] Example: In the final optimized model θ global The accuracy of whether workers are wearing safety equipment and safety risk prediction has been greatly improved, providing workshop managers with more effective safety warnings.

[0207] As can be seen, the introduction of a federated learning framework and the use of innovative methods such as weighted average aggregation and differential privacy effectively improve model performance while ensuring data privacy. Through federated learning, the model can learn data characteristics from different workshops and sensor nodes while ensuring privacy, ultimately generating an efficient and secure global model that provides more accurate behavior recognition and risk prediction. This step provides workshop management with more powerful intelligent safety analysis capabilities, effectively improving the level of workshop safety management.

[0208] S5. Acquire real-time worker behavior data and real-time safety risk prediction data, obtain the results of real-time worker behavior classification based on the real-time worker behavior data, obtain the results of real-time risk prediction based on the real-time safety risk prediction data, and use the optimized global model to perform compliance testing on the results of real-time worker behavior classification and real-time risk prediction to determine whether the workers' behavior complies with the workshop's safety production regulations, and generate corresponding risk levels and alarm responses.

[0209] Specifically, the input data of this step comes directly from the output of step 5, including the optimized global model θ global and real-time worker behavior data and security risk prediction data This data will be used as the basis for compliance checks and risk assessments.

[0210] Input data: Global optimization model (θ global ): A global model optimized through a federated learning framework that can accurately predict real-time behavioral data.

[0211] Real-time worker behavior data Worker behavior data obtained based on real-time sensors and video surveillance indicates the worker's current behavior category (such as wearing safety equipment, operating specifications, etc.).

[0212] Real-time security risk prediction data Safety risk assessment based on real-time behavioral data reflects the potential safety risks brought about by workers' behavior.

[0213] Furthermore, we first use the optimized global model θ global Behavioral data at every moment Conduct compliance testing to determine whether the workers’ behavior complies with the safety production regulations of the workshop. This step is done through the compliance testing model (M compliance ) to execute.

[0214] Compliance Detection Model (M compliance ): This model is based on the global model θ global , based on the real-time data input Judge workers' behavior and output whether it is compliant That is, whether the workers' current behavior complies with safety regulations.

[0215] Compliance judgment formula:

[0216]

[0217] in, is a binary output, Indicates that the behavior complies with safety regulations. Indicates that the behavior does not comply with safety regulations.

[0218] It is understandable that using the global model θ global Conducting behavioral compliance assessments to ensure real-time matching of worker behavior with production safety regulations helps accurately detect potential safety hazards.

[0219] Furthermore, based on real-time security risk prediction data and compliance judgment results We further established an alarm triggering mechanism. The core goal of this mechanism is to determine in real time whether an alarm is needed based on the risk level and the compliance of the worker's behavior.

[0220] Based on the predicted security risk data Set an alarm threshold θ risk_threshold ,when When the threshold is exceeded, an alarm signal is triggered. This threshold is adjusted based on the actual safety standards of the workshop and historical data to ensure that the system responds promptly to high-risk events.

[0221] Alarm trigger formula: If and Then trigger the alarm:

[0222]

[0223] in, It is an indicator function. When the condition is met, it returns 1, indicating that the alarm is triggered; otherwise, it returns 0.

[0224] As you can see, this mechanism combines worker behavior compliance with predicted safety risks to provide a dynamic, intelligent alarm triggering mechanism. This is more accurate than the traditional approach that relies solely on risk prediction and can more effectively prevent safety incidents.

[0225] Furthermore, once an alarm is triggered, the system will not only notify the management personnel, but also and violations of workers' behavior Generate different levels of alarm information to enable managers to respond quickly.

[0226] Alarm content:

[0227] Alarm level: According to The value of determines the level of the alarm. For example, if Very high, indicating an emergency has occurred, the system will issue a "high risk" alarm and require immediate processing; if If the risk is medium, the system will issue a "medium risk" alarm and recommend further monitoring measures.

[0228] Violation details: The system will display the worker's behavior classification information For example, workers did not wear safety helmets or operated in an irregular manner.

[0229] Real-time video surveillance: The system can access real-time video surveillance data and automatically push real-time images of the alarm area to management personnel, helping them make judgments more quickly.

[0230] Alarm response: Each alarm signal will notify relevant personnel via SMS, email, APP notification, etc. The system can also reduce accidents through automated response measures (such as automatically locking equipment, suspending production lines, etc.).

[0231] Furthermore, the output data:

[0232] Alarm signal: Based on real-time judgment of behavioral compliance and risk assessment, the system will issue an alarm signal, including the type and level of the alarm and the specific behavior of the worker's violation.

[0233] Real-time monitoring data: When an alarm is triggered, the system will provide real-time video surveillance of the current security risk area so that managers can evaluate and handle it in a timely manner.

[0234] As can be seen, by designing a comprehensive compliance monitoring and alerting mechanism that combines global model optimization results, real-time behavioral data, and safety risk assessments, an efficient and accurate shop floor safety monitoring solution is provided. This mechanism enables the system to issue real-time alerts and provide multi-level responses based on worker compliance and potential safety risks. This innovative alerting mechanism significantly enhances shop floor safety management capabilities, enabling timely detection and response to safety hazards, effectively reducing the occurrence of safety incidents.

[0235] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0236] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the system described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0237] In the description of this application, it should be noted that the terms "upper" and "lower" etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or the orientations or positional relationships in which the invented product is usually placed when in use. These are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on this application.

[0238] It should also be noted that, in the description of this application, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying safe production behaviors in industrial workshops based on multimodal deep learning, characterized by: The method comprises: Acquire multimodal data from industrial workshops, including video data, sensor data, and audio data; Extracting features from the multimodal data and weighted fusing them to generate a fused feature vector; Perform feature enhancement on the fused feature vector to obtain enhanced features; A temporal difference convolutional network is used to extract temporal features from the enhanced features, which are input into a fully connected layer. The results of worker behavior classification and risk prediction are obtained by combining behavioral smoothing regularization with multi-task learning. Symmetric encryption and asymmetric encryption are used to encrypt the results of the worker behavior classification and the results of the risk prediction, generating encrypted results of the worker behavior classification and the results of the risk prediction, and performing hash verification to verify data integrity; wherein, the symmetric encryption encrypts the results of the worker behavior classification and the results of the risk prediction, generating symmetrically encrypted data and a key used for symmetrical encryption; the asymmetric encryption encrypts the key used for encryption, generating an encrypted key used for symmetrical encryption; A federated learning framework is constructed based on the encrypted worker behavior classification results and risk prediction results to generate an optimized global model; wherein local models of the federated learning framework are trained based on the generated encrypted worker behavior classification results and risk prediction results, a global model is obtained by weighted averaging the local models, and the global model is distributed to various device nodes in the industrial workshop to perform multiple rounds of federated learning training to generate an optimized global model; Real-time worker behavior data and real-time safety risk prediction data are obtained, and the results of real-time worker behavior classification are obtained based on the real-time worker behavior data, and the results of real-time risk prediction are obtained based on the real-time safety risk prediction data. The optimized global model is used to perform compliance testing on the results of real-time worker behavior classification and real-time risk prediction, to determine whether the workers' behavior complies with the safety production regulations of the workshop, and to generate corresponding risk levels and alarm responses.

2. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 1 is characterized in that: The shape of the fused feature vector is (N, T, D fused ), where N is the number of samples, T is the number of time steps, and D fused is the characteristic dimension; The step of performing feature enhancement on the fused feature vector to obtain enhanced features specifically includes: Dynamically generate feature enhancement items through local temporal difference; The fused feature vector is enhanced based on the feature enhancement item to generate an enhanced feature.

3. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 1 is characterized in that: The behavior smoothing regularization is used to smooth the results of worker behavior classification and risk prediction to avoid overfitting; the multi-task learning is used to simultaneously process the two tasks of worker behavior classification and risk prediction.

4. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 1 is characterized in that: When performing symmetric encryption and asymmetric encryption to encrypt the results of the worker behavior classification and the risk prediction, the k-anonymization method is used to desensitize the worker's identity information to ensure that the worker's identity information will not be leaked.

5. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 1 is characterized in that: The federated learning framework further includes: On each device node, the local model is trained through back-propagation and optimization algorithms.

6. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 5 is characterized in that: In the federated learning framework, differential privacy technology is used to process updates of the federated learning framework to ensure that updates of a single node do not leak its local data.

7. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 1 is characterized in that: The optimized global model is used to perform compliance testing on the results of the real-time worker behavior classification and the real-time risk prediction, to determine whether the worker's behavior complies with the workshop's safety production regulations, and to generate corresponding risk levels and alarm responses, specifically including: Build a compliance detection model based on the optimized global model; Based on the compliance detection model, the real-time worker behavior classification results are judged and a compliance indicator is output, that is, whether the worker's current behavior complies with safety regulations; Determine in real time whether an alarm is required based on the results of the real-time risk prediction and the identification of compliance; Once an alarm is triggered, management personnel will be notified, and different levels of alarm information will be generated based on the risk level and non-compliant identification of workers' behavior, so that management personnel can respond quickly.

8. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 7 is characterized in that: The construction of a compliance detection model based on the optimized global model specifically includes: The output of the compliance detection model is a binary output, where output 1 indicates that the behavior complies with safety regulations, and output 0 indicates that the behavior does not comply with safety regulations; the input of the compliance detection model is the optimized global model and the results of real-time worker behavior classification.

9. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 7 is characterized in that: The real-time determination of whether an alarm is required based on the real-time risk prediction result and the compliance indicator specifically includes: An alarm threshold is set based on the results of real-time risk prediction. When the results of real-time risk prediction exceed the alarm threshold, an alarm signal is triggered. The alarm threshold is adjusted according to the actual safety standards and historical data of the workshop to ensure that the system responds to high-risk events in a timely manner.

10. The method for identifying safe production behaviors in industrial workshops based on multimodal deep learning according to claim 7, characterized in that: The different levels of alarm information include: Alarm level: Determine the alarm level based on the value of the real-time risk prediction result; Violation details: The system will display the worker's real-time worker behavior classification; Real-time video monitoring: By accessing real-time video monitoring data, the real-time images of the alarm area are automatically pushed to the management personnel; Alarm response: Each alarm signal will notify relevant personnel via SMS, email, and APP.

Images