Real-time industrial injury risk assessment and early warning method based on multi-modal data fusion

Abstract

The invention discloses a real-time industrial injury risk assessment and early warning method based on multi-modal data fusion. The method comprises the following steps: firstly, respectively extracting static numerical type, static category type and multi-modal time series data, such as features of text and image data, then aligning and fusing the data to generate efficient representation of the multi-modal data, then training a prediction model off line, deploying the model on line for real-time risk assessment, and finally carrying out real-time risk assessment. In order to adapt to continuously updated data streams, model parameters are optimized through an online incremental learning technology, and the accuracy of the model prediction amount is ensured. And finally, determining whether an early warning is given or not through prediction data of the prediction model within a period of time. According to the method, multi-modal data features are fused, model parameters are updated through online incremental learning, and the precision of model prediction is ensured. The industrial injury risk is effectively evaluated, and a corresponding early warning mechanism is triggered according to the evaluation result, so that an enterprise is helped to take precautionary measures in time, and the industrial injury risk is reduced.

Description

Technical Field

[0001] The present application relates to the field of real-time work injury risk assessment and early warning, and specifically to a real-time work injury risk assessment and early warning method based on multimodal data fusion. Background Art

[0002] With the increasingly stringent industrial safety standards and the increasing attention to work-related injury risk management, real-time assessment and early warning of work-related injury risks have become an important task in enterprise safety management.

[0003] The existing patent application number is 202310744832.3, which discloses an analysis and processing platform for work-related injury prevention, specifically involving the field of big data management technology, including the step of evaluating the user's work-related injury risk index γ1 based on the user's industry and contact equipment. The process of obtaining the user's work-related injury risk index γ1 is as follows: obtain the frequency of work-related injuries and the degree of work-related injury hazards in each industry, the number of employees in the industry, the number of people who have suffered work-related injuries, and calculate the industry risk coefficient; obtain the frequency of work-related injury accidents in each type of equipment, the degree of work-related injury hazards caused by each type of equipment, the number of people using the equipment, the number of people who have suffered work-related injuries using the equipment, and calculate the risk coefficient of the equipment; obtain the user's industry and contact equipment information, and obtain the probability of work-related injuries for each type of equipment based on work-related injury big data. Suppose the user contacts m types of equipment and the contact frequency of each type of equipment, and calculate the equipment risk actually borne by the user; calculate the user's work-related injury risk index.

[0004] However, the above methods and traditional work-related injury risk assessment have certain defects. They often rely on historical data and static analysis, and ignore the comprehensive use of multi-dimensional data, resulting in delayed response and insufficient accuracy of the early warning system.

[0005] Especially in complex working environments, factors such as employees' personal information, accident type, injury severity, injured part, physical environment of the workplace, and equipment status can significantly affect the probability of work-related injuries. At the same time, streaming data such as video surveillance of the accident site, chemical substance monitoring, current and voltage detection, environmental condition monitoring, and other real-time information can provide key context to better identify potential risks.

[0006] In order to help enterprises more effectively protect employee safety and reduce the accident rate, the method based on multimodal data fusion can not only integrate these heterogeneous data, but also designs a real-time work injury risk assessment and early warning method based on multimodal data fusion. Summary of the invention

[0007] The present invention proposes a real-time work injury risk assessment and early warning method based on multimodal data fusion.

[0008] The input of the model is heterogeneous data consisting of static numerical data, static categorical data and dynamic time series data. The dynamic time series data includes numerical data recorded by sensors, categorical data, log text data and image stream data from video surveillance. It is necessary to extract features from multimodal data and align and fuse them, and then input the fused features into the neural network for learning to obtain the probability of each type of work-related injury. Finally, these probabilities are combined with the LSTM neural network to give early warnings.

[0009] The present invention discloses a real-time work injury risk assessment and early warning method based on multimodal data fusion, including the following method: Multimodal data input: Multimodal data is heterogeneous data consisting of static numerical data, static categorical data, and dynamic time series data. Dynamic time series data includes numerical data and categorical data recorded by sensors, log text data, and image stream data from video surveillance. Data processing: Extract features from the input multimodal data and align and fuse them. Then input the fused features into the neural network model to obtain the probability of each type of work-related injury. Result output: The probability of each type of work-related injury is combined with the LSTM neural network to output the probability of each type of work-related injury and its warning information.

[0010] As an embodiment of the present invention, the multimodal data input includes the following steps: Step 1: For input size Numerical number First, a fully connected layer is used to extract features, and then an activation function is used to enhance the nonlinear representation. The final feature is , the dimension is , as input for subsequent tasks; Step 2: For input size Categorical data , first convert it into a binary vector representation through one-hot encoding, and the resulting dimension is ,in is the total number of all categories, and then the one-hot encoded data is input into the fully connected layer to obtain the final feature , the dimension is , as input for subsequent tasks; Step 3: For input size , by numerical data and categorical data Dynamic time series data composed of , process them separately, for dimensions Numeric data , through a fully connected layer, the dimension of the generated feature vector is ,in is the set numerical feature output dimension. For dimensions Categorical Data , first convert it into a numerical representation through one-hot encoding, and the resulting dimension is , then, a fully connected layer is used to map it to The characteristic representation of is the representation dimension of categorical features, and the above dimensions are , Concatenate to get the final output feature representation , whose dimensions are ,in , used for subsequent tasks; Step 4: For input size Text data After that, each log text is preprocessed, including removing stop words, punctuation marks and restoring the word form. Then, BERT is used to convert each text into a feature representation, and the resulting dimension is feature vector, where is the dimension of word embedding, which is then mapped to The feature representation of , used for subsequent tasks; Step 5: Enter the dimensions Video recording information After that, firstly, for each frame of image After preprocessing, including scaling, normalization, and enhancement, the dimension is images of the same size, then, using the pre-trained Extract the features of the above image, and the feature representation dimension obtained is , then, a fully connected layer is used to map the extracted features to The feature representation of is the final output feature representation , for subsequent tasks.

[0011] As an embodiment of the present invention, the data processing includes the following steps: Step 6: After completing the multimodal data feature extraction from steps 1 to 5, these feature vectors , , , and To perform fusion, first, align , and The frequency of for , , The lowest common multiple, and Expand to Dimensions; By copying, and Two eigenvectors are consistent with the other eigenvectors in the frequency dimension; all eigenvectors , , , and The dimensions of ,in is the output dimension of the feature; these feature vectors are concatenated to form the final fused feature representation , used for subsequent model training and reasoning tasks.

[0012] As an embodiment of the present invention, the following steps are also included: Step 7: After feature extraction, the final feature vector Input to the neural network model The risk of work-related injuries is assessed in is the input feature representation, No model parameters.

[0013] As an embodiment of the present invention, the neural network model The network structure includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer. The output layer is set to six neurons. Activation function to output the probability distribution of various types of work-related injury risks.

[0014] As an embodiment of the present invention, the neural network model The training is based on historically collected data and is completed in an offline environment. Then the offline trained neural network model is Deployed online to conduct real-time work injury risk assessment.

[0015] As an embodiment of the present invention, the work injury risk assessment method comprises the following steps: Step 8: The input layer receives the dimension The characteristic data of is the batch size, is the length of the time series, is the feature dimension of each sampling; The convolutional layer uses The specific calculation formula for the convolution kernel size is: ; in, is the weight matrix of the convolution kernel (dimension ), is the bias term (dimension ), the output dimension is ; The pooling layer uses the maximum pooling operation to further reduce the time dimension, and its calculation formula is: ,in, is the pooling window size, and the output dimension is ; The output features of the pooling layer enter the fully connected layer after flattening, and the dimension after flattening is ; The feature extraction formula of the fully connected layer is: ,in, is the weight matrix of the fully connected layer (dimension ), is the bias term (dimension ), the output dimension is ; The output layer is set to six neurons, and its softmax activation function is defined as: ; in, Indicates The predicted probability of the risk of a work-related injury is trained using the cross entropy loss function. The loss formula is: ; in, is the actual label, To predict the probability, in offline training on historical data, use The optimizer iteratively updates the parameters In order to minimize the losses, the probability distribution of each type of work-related injury risk is finally generated.

[0016] As an embodiment of the present invention, the parameters The update method includes the following steps: Step 9: Continuously update the model during real-time risk assessment Parameters To adapt to the current environment and ensure the accuracy of the neural network model, the parameters are updated through the input data stream using elastic weight fixing technology and small batch gradient descent method. , so that the neural network model Ability to dynamically adapt to new data characteristics to maintain the accuracy of risk assessment.

[0017] As an implementation mode of the present invention, the elastic weight fixing technology includes: use The information matrix is ​​used to quantify the influence of model parameters on the loss function. Each element of the information matrix Representation parameters and The covariance between the gradients of is given by: in, is the loss function Parameters By sampling the training data, we can estimate the above expected value and form Information Matrix The complete structure: ; In got After the information matrix is ​​constructed The loss function can be expressed as: ; in, is the loss of the current task, is calculated in the previous task The information matrix elements, is the parameter value learned in the previous task, is a weight hyperparameter used to adjust the importance of new and old task knowledge; The mini-batch gradient descent method includes: model Ability to flexibly adjust parameters as new data is received , achieving dynamic adaptation to new features; By dividing the training data into several small batches for iterative updates, model training can be accelerated and convergence performance can be improved. In the formula, the parameter update is: ,in, is the learning rate, is the loss function For model parameters gradient.

[0018] By By combining this technology with the mini-batch gradient descent method, the model can maintain efficient and accurate performance in the face of a continuous influx of new data, thereby improving the accuracy of risk assessment while ensuring the effective retention of old knowledge.

[0019] As an embodiment of the present invention, the result output method comprises the following steps: Step 10: Use a sliding window to provide real-time evaluation by selecting the most recent predicted probability data to determine whether an early warning is needed. The input data is ,in For the selected time window size, use The neural network processes the data and outputs a binary classification result with 6 dimensions, where each dimension corresponds to a warning indicator of a work-related injury (0 or 1); Said The basic structure of the network includes input gate, forget gate and output gate. These gating mechanisms work together to control the inflow, retention and output of information. The input of the model is , which means that at the time step For each time step, the model receives a six-dimensional vector, which constitutes a three-dimensional tensor input data with a dimension of ; Core variables , , , , , ,in yes The number of hidden units, which is usually a hyperparameter of the model; The specific process is as follows: First, the previous state is controlled by the forget gate The information is retained, and the calculation process is, ,in is the weight matrix of the forget gate, is the bias term; then the current input is determined by the input gate Amount of information that needs to be updated: The calculation process is ,in, is the weight matrix of the input gate, is the bias term. Next, improve the candidate memory unit to generate a new memory unit: The calculation process is ,in, is the weight matrix of the candidate states, is the bias term. At the same time, the current input and the previous state are combined to update the memory unit: The calculation process is ,in, is the current cell state, represents element-wise multiplication, Control the effect of the current unit state on the output: The calculation process is ,in, is the weight matrix of the output gate, is the bias term, Finally, the current hidden state is calculated by the current memory unit and the output gate: The calculation process is ,go through The iteration of time steps, Will output a hidden state sequence , , , , where each Generate the final risk warning indication and convert the hidden state of the last time step Input into a fully connected layer and use the sigmoid activation function to get the predicted probability of each work-related injury: ,in, represents the risk warning probability of each type of work-related injury, where is the weight matrix of the output layer, is the bias term of the output layer; Binary classification results: whether to issue an early warning, and perform threshold processing on the output value: if , then it is considered that the corresponding work-related injury risk needs to issue an early warning, and the output is 1; otherwise, the output is 0.

[0020] The beneficial effects of the present invention are as follows: This paper proposes a new neural network model that effectively integrates multimodal data, trains a risk assessment model, and deploys the model online. The model is updated using incremental learning technology to ensure the accuracy of the model prediction. Then, according to the probability predicted by the model, The model determines whether an early warning is needed, helping companies to more effectively protect employee safety and reduce accident rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Overall structure diagram.

[0022] Figure 2 Diagram of the risk assessment model.

[0023] Figure 3 Risk warning model diagram. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and examples.

[0025] This method integrates multidimensional data from multiple sources to build a deep learning model that can assess the risk of work-related injuries in real time and provide early warnings. The data processed include static data (such as numerical age and categorical gender), as well as dynamic data (such as numerical equipment status parameters and chemical monitoring data, text-based log records, and accident site video surveillance data in the form of image streams). Through a specific fusion algorithm, these data are effectively processed and analyzed to achieve accurate work-related injury risk assessment. After data fusion, the model will comprehensively consider various factors, assess the probability of work-related injuries, and trigger the corresponding early warning mechanism based on the assessment results, thereby helping companies take preventive measures in a timely manner to reduce the risk of work-related injuries.

[0026] This method obtains the vector representation of sample information by efficiently extracting, aligning and fusing the features of multimodal data. Then, it selects the feature information within a certain period of time and predicts it through a neural network to obtain the probability of each type of work-related injury. Finally, based on these probabilities, Neural network for early warning.

[0027] The present invention is a deep learning neural network model, whose input is multimodal data, such as numerical, textual, image data, etc., and whose output is the probability and warning information of each type of work-related injury.

[0028] The overall structure of the method is as follows Figure 1 As shown in the figure, the input heterogeneous data is fused and feature extracted to obtain features for different tasks, and the features of these tasks are contributed through the cross-attention mechanism to obtain the final features of each task, which are input into the predicted neural network of each task for prediction and loss calculation. Finally, each loss is weighted and optimized to better learn from different tasks.

[0029] In order to achieve the above object, the technical solution of the present invention comprises the following steps: Here are the detailed steps Step 1: For input size Numeric data First, features are extracted through a fully connected layer, which uses a weight matrix (Dimensions are ) and the bias vector (Dimensions are ) is linearly transformed to get the output , the dimension is Then apply The activation function performs nonlinear mapping to obtain the final feature representation , whose dimensions are .

[0030] Step 2: For input size Categorical data , and convert it into a binary vector representation through one-hot encoding: Assume that each categorical feature has possible categories, among which Indicates categorical features, one-hot encoding will expand each categorical feature into one dimensional binary vector. The output dimension after one-hot encoding is ,in is the total number of all categories.

[0031] Then the one-hot encoded data is input into the fully connected layer to extract features. Given the weight matrix is (Dimensions are ), the bias vector is (Dimensions are ). After the linear transformation of the fully connected layer, we get , and then Nonlinear mapping through ReLU activation function , and obtain the final output features , the dimension is .

[0032] Step 3: For input size Dynamic time series data When , numerical data and categorical data are processed separately. Numeric data , feature extraction is performed through a fully connected layer. Given the weight matrix of the fully connected layer is , whose dimensions are , the bias term is , whose dimensions are The linear transformation process can be expressed as: , where the input data The dimension is , after being processed by the fully connected layer, the dimension of the output feature is .

[0033] Then the dimension is Categorical data To process, first perform one-hot encoding in the same way as step 2 to convert it into a numerical representation. The dimensions generated by this process are ,in Represents the number of categories after one-hot encoding. The one-hot encoded data is input into the fully connected layer for feature mapping. The weight matrix of this layer is set as , the dimension is , the bias term is , whose dimensions are The linear transformation process is , the dimension of the output feature is of The feature representation of .

[0034] Finally, the numerical features are represented and categorical feature representation Splice to get the final output feature representation , whose dimensions are ,in .

[0035] Step 4: For input size Text data First, we remove stop words and punctuation marks from each log text, and perform preprocessing such as word form restoration to reduce the noise and redundant information of the text data. After preprocessing, we use the pre-trained BER model to encode each text to generate context-related feature representations. The feature representation after BERT processing is recorded as , whose dimensions are ,in is the dimension of word embedding.

[0036] Then, to further optimize the feature representation, Input to the fully connected layer for mapping. The weight matrix of the fully connected layer is set as , whose dimensions are , the bias term is , whose dimensions are . The linear transformation can be expressed as: , input data After processing through the fully connected layer, the final feature representation is obtained , the dimension is .

[0037] Step 5: For input size Video recording information First, each frame of the image is scaled, normalized, and data enhanced. Image scaling adjusts each frame of the image to a uniform size. Next, the pre-trained ResNet-19 model is used to extract features from the processed images. The dimension of the feature representation obtained is ,in It is the feature dimension of ResNet-19 output. After extracting effective features, these features are mapped through a fully connected layer. The weight matrix of this fully connected layer The dimension is , bias term The dimension is The feature mapping is achieved through the following linear transformation: , and obtain the final output features , the dimension is .

[0038] Step 6: After completing the data feature extraction from steps 1 to 5, all feature vectors need to be , , , and First, we need to align , and These three eigenvectors have different sampling frequencies, namely , , In order to align different frequencies, the time dimensions of these three features need to be interpolated to the same frequency , this frequency is , which is the least common multiple. The alignment process uses linear interpolation to generate new time steps. The specific interpolation formula is as follows: ; In the above formula, Represents an interpolation function that expands or compresses the time dimension of the original data into a new frequency by resampling the original data , thus achieving , and Time alignment.

[0039] After aligning these three eigenvectors, we also need to consider and The frequency expansion of and It does not contain the time dimension itself (that is, the initial dimension is ), which needs to be expanded by copying Specifically, the expansion process can be expressed as: ; Through the above method, and Replicate it in the frequency dimension, so that its time dimension is the same as , and Be consistent.

[0040] Finally, after interpolation and replication, the dimensions of all feature vectors are adjusted to ,in is the sum of the output dimensions of each feature vector. Specifically: After all features are aligned, these feature vectors are combined together using concatenation to generate the final fused feature representation. . The concatenation operation is performed on the feature dimension and can be expressed as: Through this fusion method, multimodal features can be effectively integrated to form a feature vector with comprehensive expression capabilities. , providing a comprehensive and consistent input basis for subsequent model training and reasoning tasks. The overall data fusion is Figure 1 shown.

[0041] Step 7: Prediction model Figure 2 As shown, after the feature extraction is completed, the final fused feature vector Input to Convolutional Neural Network , in order to classify and predict the risk of work-related injuries. The network structure includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer receives dimensions of The characteristic data of is the batch size, is the length of the time series, is the feature dimension of each sampling. The convolution layer uses The specific calculation formula for the convolution kernel size is: ; in, is the weight matrix of the convolution kernel (dimension ), is the bias term (dimension ), the output dimension is Subsequently, the pooling layer uses the Max Pooling operation to further reduce the time dimension, and its calculation formula is: ,in, is the pooling window size, and the output dimension is .

[0042] The output features of the pooling layer enter the fully connected layer after flattening, and the dimension after flattening is The feature extraction formula of the fully connected layer is: ,in, is the weight matrix of the fully connected layer (dimension ), is the bias term (dimension ), the output dimension is The output layer is set to six neurons, and its softmax activation function is defined as: ; in, Indicates The predicted probability of the risk of a class of work-related injuries. The model is trained using the cross entropy loss function, and the loss formula is: ; in, is the actual label, is the predicted probability. Offline training is performed using historical data, and the Adam optimizer is used to iteratively update the parameters In order to minimize the losses, the probability distribution of each type of work-related injury risk is finally generated.

[0043] Step 8: Use the offline trained model Deploy it online to conduct real-time work injury risk assessment. After receiving the new data stream, first perform feature extraction, alignment and fusion on the multimodal data according to steps 1 to 6 to obtain the fused feature vector . Then, The data is input into the model for prediction and the probability distribution of various types of work-related injury risks is calculated. Through this process, the system can assess risks in real time and provide timely decision support for safety management.

[0044] Step 9: In order to effectively update the model during the real-time risk assessment process Parameters , an incremental learning strategy is combined with data flow to cope with the dynamic changes of new data features. Specifically, the elastic weight fixation (EWC) technology and mini-batch gradient descent method are used to ensure that the model maintains the memory and protection of previous learning experience while continuously optimizing the accuracy of risk assessment.

[0045] The core of EWC technology is to limit the change of model parameters by introducing additional loss terms to protect the previously learned knowledge. The construction of this loss term is based on the importance of the parameters, and the Fisher information matrix is ​​used to quantify the influence of model parameters on the loss function. The Fisher information matrix is ​​an important statistical tool that can reflect the sensitivity and importance of each parameter of the model in a specific task. Specifically, each element of the Fisher information matrix Representation parameters and The covariance between the gradients of is defined as: ,in, is the loss function Parameters By sampling the training data, we can estimate the above expected value and form the complete structure of the Fisher information matrix F: ; After obtaining the Fisher information matrix, the loss function of EWC is constructed, which can be expressed as: ; in, is the loss of the current task, is the Fisher information matrix calculated in the previous task. elements, is the parameter value learned in the previous task, is a weight hyperparameter that adjusts the importance of new tasks and old task knowledge. In this way, EWC prioritizes those parameters with high importance in previous tasks when updating model parameters, thereby effectively avoiding the destruction of old knowledge when learning new tasks.

[0046] Combined with mini-batch gradient descent, the model Ability to flexibly adjust parameters as new data is received , to achieve dynamic adaptation to new features. The mini-batch gradient descent method accelerates model training and improves convergence performance by dividing the training data into several mini-batches for iterative updates. In each mini-batch B, the formula for updating parameters is: ,in, is the learning rate, is the loss function L on the model parameters In this way, the model can not only use the latest data features for optimization, but also reduce the risk of overfitting that may occur during model training due to the randomness of small batches.

[0047] In summary, by combining EWC technology with the mini-batch gradient descent method, the model can maintain efficient and accurate performance in the face of a continuous influx of new data, improve the accuracy of risk assessment, and at the same time ensure the effective retention of old knowledge, providing strong support for real-time risk assessment.

[0048] Step 10: Warning neural network such as Figure 3 As shown in the figure, in the early warning system for work-related injury risk, in order to accurately determine when to issue an early warning, a sliding window method is used. By selecting the most recent predicted probability data to provide real-time evaluation, it ensures that it can dynamically adapt to changes in the risk of work-related injuries. The input data dimensions are ,in

[0049] Indicates the size of the selected time window. To process this sequence data, we chose the long short-term memory (LSTM) network. The basic structure of the LSTM network includes input gate, forget gate and output gate. These gating mechanisms work together to control the inflow, retention and output of information.

[0050] In the current setting, the input to the LSTM model is , which means that at the time step For each time step, the model receives a six-dimensional vector, which constitutes a three-dimensional tensor input data with a dimension of .

[0051] LSTM core variables , , , , , , where m is the number of LSTM hidden units, which is usually a hyperparameter of the model.

[0052] The specific process is as follows: First, the previous state is controlled by the forget gate. The information is retained, and the calculation process is, ,in is the weight matrix of the forget gate, is the bias term. Then the current input is determined by the input gate The amount of information that needs to be updated is calculated as follows: ,in, is the weight matrix of the input gate, is the bias term. Next, we improve the candidate memory cells to generate new memory cells. The calculation process is ,in is the weight matrix of the candidate states, is the bias term. At the same time, the current input and the previous state are combined to update the memory unit. The calculation process is ,in is the current cell state, represents element-by-element multiplication. It is also necessary to control the influence of the current cell state on the output. The calculation process is ,in, is the weight matrix of the output gate, is the bias term. Finally, the current hidden state is calculated by the current memory unit and the output gate. The calculation process is ,go through For each iteration of time steps, LSTM will output a hidden state sequence , , , , where each .

[0053] In order to generate the final risk warning indication, the hidden state of the last time step is Input into a fully connected layer and use the sigmoid activation function to get the predicted probability of each work-related injury: here, represents the risk warning probability of each type of work-related injury, where is the weight matrix of the output layer, is the bias term of the output layer.

[0054] To achieve a binary classification result (i.e., whether to issue an early warning), the output value is thresholded. , then it is considered that the corresponding work-related injury risk needs to issue an early warning, and the output is 1; otherwise, the output is 0.

[0055] In summary, the sliding window method combined with the LSTM network can effectively use recent historical data to make real-time early warning decisions for work-related injury risks. This method not only improves the accuracy of early warnings, but also enhances the model's ability to learn time series features, providing reliable support for work-related injury risk management.

Claims

1. A real-time work injury risk assessment and early warning method based on multimodal data fusion, characterized in that: The following methods are included: Multimodal data input: Multimodal data is heterogeneous data consisting of static numerical data, static categorical data, and dynamic time series data. Dynamic time series data includes numerical data and categorical data recorded by sensors, log text data, and image stream data from video surveillance. Data processing: extract features from the input multimodal data, align and fuse them, and then input the fused features into the neural network model to obtain the probability of each type of work-related injury; Result output: The probability of each type of work-related injury is combined with the LSTM neural network to output the probability of each type of work-related injury and its warning information.

2. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 1 is characterized in that: The multimodal data input comprises the following steps: Step 1: For input size Numerical number First, a fully connected layer is used to extract features, and then an activation function is used to enhance the nonlinear representation. The final feature is , the dimension is , as input for subsequent tasks; Step 2: For input size Categorical data , first convert it into a binary vector representation through one-hot encoding, and the resulting dimension is ,in is the total number of all categories, and then the one-hot encoded data is input into the fully connected layer to obtain the final feature , the dimension is , as input for subsequent tasks; Step 3: For input size , by numerical data and categorical data Dynamic time series data composed of , process them separately, for dimensions Numeric data , through a fully connected layer, the dimension of the generated feature vector is ,in is the set numerical feature output dimension. For dimensions Categorical data , first convert it into a numerical representation through one-hot encoding, and the resulting dimension is , then, a fully connected layer is used to map it to The characteristic representation of is the representation dimension of categorical features, and the above dimensions are , Concatenate to get the final output feature representation , whose dimensions are ,in , used for subsequent tasks; Step 4: For input size Text data After that, each log text is preprocessed, including removing stop words, punctuation marks and restoring the word form. Then, BERT is used to convert each text into a feature representation, and the resulting dimension is feature vector, where is the dimension of word embedding, which is then mapped to The feature representation of , used for subsequent tasks; Step 5: Enter the dimensions Video recording information After that, firstly, for each frame of image After preprocessing, including scaling, normalization, and enhancement, the dimension is images of the same size, then, using the pre-trained Extract the features of the above image, and the feature representation dimension obtained is , then, a fully connected layer is used to map the extracted features to The feature representation of is the final output feature representation , for subsequent tasks.

3. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 2 is characterized in that: The data processing comprises the following steps: Step 6: After completing the multimodal data feature extraction from steps 1 to 5, these feature vectors , , , and To perform fusion, first, align , and The frequency of , The lowest common multiple, and Expand to Dimensions; By copying, and Two eigenvectors are consistent with the other eigenvectors in the frequency dimension; All eigenvectors , , and The dimensions of ,in is the output dimension of the feature; these feature vectors are concatenated to form the final fused feature representation , used for subsequent model training and reasoning tasks.

4. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 3 is characterized in that: The following steps are also included: Step 7: After feature extraction, the final feature vector Input to the neural network model The risk of work-related injuries is evaluated in , where h is the input feature representation, is the model parameter.

5. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 4 is characterized in that: The neural network model The network structure includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer. The output layer is set to six neurons. Activation function to output the probability distribution of various types of work-related injury risks.

6. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 5 is characterized in that: The neural network model The training is based on historically collected data and is completed in an offline environment. Then the offline trained neural network model is Deployed online to conduct real-time work injury risk assessment.

7. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 6 is characterized in that: The work injury risk assessment method comprises the following steps: Step 8: The input layer receives the dimension The characteristic data of is the batch size, is the length of the time series, is the feature dimension of each sampling; The convolutional layer uses The specific calculation formula for the convolution kernel size is: ； in, is the weight matrix of the convolution kernel, with dimension , is the bias term, and its dimension is , the output dimension is ; The pooling layer uses the maximum pooling operation to further reduce the time dimension, and its calculation formula is: ； in, is the pooling window size, and the output dimension is ; The output features of the pooling layer enter the fully connected layer after flattening, and the dimension after flattening is ; The feature extraction formula of the fully connected layer is: ,in, is the weight matrix of the fully connected layer (dimension ), is the bias term, and its dimension is , the output dimension is ; The output layer is set to six neurons, The activation function is defined as: ； in, Indicates The predicted probability of the risk of a work-related injury is trained using the cross entropy loss function. The loss formula is: ； in, is the actual label, To predict the probability, in offline training on historical data, use The optimizer iteratively updates the parameters In order to minimize the losses, the probability distribution of each type of work-related injury risk is finally generated.

8. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 7 is characterized in that: The parameters The update method includes the following steps: Step 9: Continuously update the model during real-time risk assessment Parameters To adapt to the current environment and ensure the accuracy of the neural network model, the parameters are updated through the input data stream using elastic weight fixing technology and small batch gradient descent method. , so that the neural network model Ability to dynamically adapt to new data characteristics to maintain the accuracy of risk assessment.

9. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 8 is characterized in that: The elastic weight fixing technology includes: use The information matrix is ​​used to quantify the influence of model parameters on the loss function. Each element of the information matrix Representation parameters and The covariance between the gradients of is given by: ； in, is the loss function Parameters By sampling the training data, we can estimate the expected value and form The complete structure of the information matrix F is: ； In got After the information matrix is ​​obtained, the loss function of EWC is constructed, which can be expressed as: ； in, is the loss of the current task, is calculated in the previous task The information matrix elements, is the parameter value learned in the previous task, is a weight hyperparameter used to adjust the importance of new and old task knowledge; The mini-batch gradient descent method includes: model Ability to flexibly adjust parameters as new data is received , achieving dynamic adaptation to new features; By dividing the training data into several small batches for iterative updates, model training can be accelerated and convergence performance can be improved. In the formula, the parameter update is: ,in, is the learning rate, is the loss function For model parameters The gradient of By combining this technology with the mini-batch gradient descent method, the model can maintain efficient and accurate performance in the face of a continuous influx of new data, thereby improving the accuracy of risk assessment while ensuring the effective retention of old knowledge.

10. The real-time work injury risk assessment and early warning method based on multimodal data fusion according to claim 1 is characterized in that: The result output method comprises the following steps: Step 10: Use a sliding window to provide real-time evaluation by selecting the most recent predicted probability data to determine whether an early warning is needed. The input data is ,in For the selected time window size, use The neural network processes the data and outputs a binary classification result with 6 dimensions, where each dimension corresponds to a warning indication of a work-related injury, 0 or 1; Said The basic structure of the network includes input gate, forget gate and output gate. These gating mechanisms work together to control the inflow, retention and output of information. The input of the model is , which means that at the time step For each time step, the model receives a six-dimensional vector, which constitutes a three-dimensional tensor input data with dimensions of n, t, 6; Core variables , , , , , ,in yes The number of hidden units, usually a hyperparameter of the model; The specific process is as follows: First, the previous state is controlled by the forget gate The information is retained, and the calculation process is, ; in is the weight matrix of the forget gate, is the bias term; then the current input is determined by the input gate Amount of information that needs to be updated: The calculation process is ,in, is the weight matrix of the input gate, Is the bias term, and then improve the candidate memory unit to generate a new memory unit: The calculation process is ,in is the weight matrix of the candidate states, is the bias term, which combines the current input and the previous state to update the memory unit: The calculation process is ,in is the current cell state, Represents element-by-element multiplication, controlling the effect of the current cell state on the output: The calculation process is ,in, is the weight matrix of the output gate, is the bias term, and the current hidden state is finally calculated by the current memory unit and the output gate: The calculation process is ,go through The iteration of time steps, Will output a hidden state sequence , , , , where each , generate the final risk warning indication, and convert the hidden state of the last time step Input into a fully connected layer and use the sigmoid activation function to get the predicted probability of each work-related injury: ,in, represents the risk warning probability of each type of work-related injury, where is the weight matrix of the output layer, is the bias term of the output layer; Binary classification results: whether to issue an early warning and perform threshold processing on the output value: if , then it is considered that the corresponding work-related injury risk needs to issue an early warning, and the output is 1; otherwise, the output is 0.

Images