An ergonomic risk assessment method and device based on posture information and scene information
By combining deep learning methods with human posture and scene information, and using graph convolutional and temporal convolutional networks for feature and decision-level fusion, the accuracy and reliability issues of ergonomic risk assessment in existing technologies are solved, achieving a more efficient risk assessment effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA NORMAL UNIV
- Filing Date
- 2023-06-13
- Publication Date
- 2026-07-14
Smart Images
Figure CN116704415B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of ergonomics, and in particular to ergonomic risk assessment. Background Technology
[0002] With societal development, the pace of life is accelerating and work tasks are becoming increasingly demanding. Inappropriate posture during work or rest can easily lead to musculoskeletal disorders. Ergonomic risk assessment can help people improve their sitting posture during daily office or factory work, evaluating and scoring the postures of workers across various industries to correct poor posture. Traditional ergonomic risk assessment methods rely on manual observation, questionnaires, and interviews to collect data and assign scores. These methods are inefficient and inaccurate, limiting the application of ergonomic risk assessment.
[0003] To address the aforementioned issues, a better solution is to utilize deep learning methods for ergonomic risk assessment. Currently, there are two main deep learning methods: the first processes video scene information to perform ergonomic risk assessment, and the second processes human posture data from videos to obtain ergonomic risk scores. However, both existing deep learning methods exhibit low accuracy and reliability in ergonomic risk assessment for multimodal videos and complex scenes and movements. Summary of the Invention
[0004] Therefore, the purpose of this invention is to provide an ergonomic risk assessment method based on posture information and scene information, which combines human posture features and scene features. Human posture features contain information about human posture, while scene features contain information about the interaction between the human body and the environment and information about human movements. Combining the two features to complement each other can improve the accuracy of movement assessment.
[0005] An ergonomic risk assessment method based on posture and scene information, characterized by comprising:
[0006] Step S1 preprocesses the human posture video of the risk to be assessed to obtain a unified image sequence and a spatiotemporal human skeleton map.
[0007] Step S2 extracts scene spatial features and human posture spatial features from the unified image sequence and spatiotemporal human skeleton map respectively, and performs feature-level fusion on the scene spatial features and human posture spatial features to obtain feature-level fusion results.
[0008] Step S3 predicts the scene spatial features and human posture spatial features respectively to obtain the first action category prediction result and the second action category prediction result. The first action category prediction result and the second action category prediction result are then fused at the decision level to obtain the decision level fusion result.
[0009] Step S4 connects the decision-level fusion result with the feature-level fusion result to perform action evaluation and obtain an ergonomic risk assessment score.
[0010] Further, step S2 includes:
[0011] S2A extracts the scene spatial features from the unified image sequence;
[0012] S2B extracts the human posture spatial features from the spatiotemporal human skeleton map;
[0013] S2C performs feature-level fusion of the scene space features and human posture space features to obtain the feature-level fusion result.
[0014] Further, step S3 includes:
[0015] S31A performs action segmentation on the scene spatial features extracted from the unified image sequence to obtain scene spatiotemporal features;
[0016] S32A performs classification and action recognition on the spatiotemporal features of the scene to obtain the prediction result of the first action category;
[0017] S31B performs motion segmentation on the human body posture spatial features to obtain human body posture spatiotemporal features.
[0018] S32B performs classification and action recognition on the spatiotemporal features of the human posture to obtain the prediction result of the second action category;
[0019] S3C performs decision-level fusion of the first action category prediction result and the second action category prediction result to obtain the decision-level fusion result.
[0020] Further, step S4 includes:
[0021] S41 connects the decision-level fusion result as a supervision factor with the feature-level fusion result to obtain a cascaded fusion feature;
[0022] S42 performs motion evaluation on the serial fusion features to obtain an ergonomic evaluation score.
[0023] Further, step S1 includes:
[0024] Extract the 2D joint coordinates (x, y) of the human body from the image sequence of the human posture video of the risk to be assessed;
[0025] For each frame of image, the coordinates of the human body 2D joint points are extracted as nodes of the graph, and the natural connections of the human skeleton are used as edges between nodes, thereby constructing the human skeleton graph of each frame.
[0026] Connect the corresponding nodes of each node in the time frame with edges to construct a spatiotemporal human skeleton diagram.
[0027] Further, step S2B includes: performing graph convolution operation on the spatiotemporal human skeleton map to obtain the human posture spatial features; before performing graph convolution operation on the spatiotemporal human skeleton map, the spatiotemporal human skeleton map is first preprocessed by partitioning.
[0028] Further, in step S2B, the graph convolution operation is performed using a CTR-GCNs network, and the graph convolution operation formula is:
[0029]
[0030] Where Λ represents the adjacency matrix of a joint in a single frame, I represents the self-matrix of the joint connection, and W represents the weight matrix formed by superimposing the weight vectors of multiple output channels.
[0031] Further, in step S2A, VGG16 is used as an image feature extraction network, and the N images of the unified image sequence are used as the input data of the VGG16 to extract the scene spatial features.
[0032] The present invention also provides an ergonomic risk assessment device based on posture information and scene information, characterized in that it includes:
[0033] The extraction module is used to extract images from each frame of a human pose video to be assessed, thereby obtaining an image sequence;
[0034] The size unification module is used to resize each image in the image sequence to obtain a unified image sequence;
[0035] The human skeleton diagram construction module is used to extract the coordinates of human 2D joint points from each image in the image sequence, and construct a spatiotemporal human skeleton diagram based on the human 2D joint point coordinates.
[0036] A scene space feature extraction module is used to extract scene space features from the unified image sequence;
[0037] The human posture spatial feature extraction module is used to extract human posture spatial features from the spatiotemporal human skeleton map;
[0038] The feature-level fusion module is used to perform feature-level fusion of the scene space features and the human posture space features to obtain the feature-level fusion result.
[0039] The scene spatiotemporal feature extraction module is used to perform action segmentation on all scene spatial features extracted from the unified image sequence to obtain scene spatiotemporal features;
[0040] The scene feature action recognition module classifies and recognizes the spatiotemporal features of the scene to obtain a first action category prediction result.
[0041] The human posture spatiotemporal feature extraction module is used to perform action segmentation on the human posture spatial features to obtain human posture spatiotemporal features.
[0042] The human body feature action recognition module is used to classify and recognize the spatiotemporal features of the human body posture to obtain a second action category prediction result.
[0043] The decision-level fusion module is used to perform decision-level fusion of the first action category prediction result and the second action category prediction result to obtain a decision-level fusion result;
[0044] A serial fusion module is used to connect the decision-level fusion result as a supervision factor with the feature-level fusion result to obtain a serial fusion feature.
[0045] The motion assessment module performs motion assessment on the serial fusion features to obtain an ergonomic assessment score.
[0046] Furthermore, the parameters in the ergonomic assessment device can be updated via update step S5, which specifically includes the following steps:
[0047] S51 calculates the cross-entropy loss value based on the decision-level fusion result;
[0048] S52 calculates the logistic regression value based on the ergonomics assessment score;
[0049] S53 sums the cross-entropy loss value and the logistic regression loss value to obtain the final loss value;
[0050] S54 updates the parameters based on the final loss value.
[0051] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description
[0052] Figure 1 This is a flowchart of an ergonomic risk assessment method based on posture information and scene information according to this application;
[0053] Figure 2 To execute Figure 1 The diagram shows a structural block diagram of an ergonomic risk assessment device based on posture and scene information.
[0054] Figure 3 This is an example diagram of the spatiotemporal human skeleton in this application;
[0055] Figure 4 Diagram of CTR-GCNs network structure;
[0056] Figure 5 A flowchart illustrating the update steps;
[0057] Figure 6 This is a structural diagram of the update steps. Detailed Implementation
[0058] This application studies and analyzes ergonomic risk assessment for multimodal videos and complex scenes and movements. It argues that current ergonomic risk assessment algorithms using deep learning methods only consider one aspect of human activity or scene information, resulting in a singular assessment perspective and impacting accuracy and reliability. Therefore, this application incorporates the spatial information of the scene into the scoring of human behavior and postures in a given environment, integrating it into the overall posture assessment process to improve the accuracy of action recognition, thereby further enhancing the accuracy and reliability of action assessment.
[0059] Please see Figure 1 and Figure 2 , Figure 1 A flowchart of the ergonomic risk assessment method based on posture and scene information provided in this application. Figure 2 To execute Figure 1 The diagram shows the structural block diagram of the ergonomic risk assessment device based on posture and scene information.
[0060] The ergonomic risk assessment method based on posture and scene information in this application includes:
[0061] S1: Preprocess the human posture videos to be assessed for risk to obtain a unified image sequence and a spatiotemporal human skeleton map. Specifically, step S1 includes the following steps:
[0062] S10: Extract images from each frame of the human pose video to be assessed to obtain an image sequence. This step S10 is performed by the image extraction module 10.
[0063] S1A: Resize each image in the image sequence to obtain a unified image sequence. In some embodiments, the images are uniformly resized to 224*224 pixels. This step S1A is performed by the size unification module 1A.
[0064] S1B: Extract the 2D human body joint coordinates (x, y) from each image in the image sequence, and then construct a spatiotemporal human skeleton map based on all the extracted 2D human body joint coordinates (x, y). This step S1B is performed by the human skeleton map construction module 1B.
[0065] Specifically, firstly, the LCR-Net pose estimation network is used to extract the 2D joint coordinates (x, y) of the human body in the image sequence, with 25 joint coordinates per frame. Then, for each frame, the 2D joint coordinates of the human body are extracted, i.e., the skeleton nodes of the human body are used as nodes in the graph, and the natural connections of the skeleton are used as edges between nodes, thus constructing the human skeleton graph for each frame. Then, the corresponding nodes of each node in the time frame are also connected with edges, which constitutes a graph containing temporal and spatial information, i.e., a spatiotemporal human skeleton graph.
[0066] Please see Figure 3 , Figure 3 This is an example diagram of the spatiotemporal human skeleton diagram of this application. The spatiotemporal human skeleton diagram is represented by G=(V,E), where V represents a node, and V={v ti The set of edges |t=1,...,T,i=1,...,N} includes all skeletal joints in the image sequence; the edge set E consists of two subsets. The first subset describes the intra-skeleton connections of each frame, representing a set of naturally connected human joints in a single frame, denoted as E. s ={v ti v tj| (i,j)∈H}; the second subset describes inter-frame edges, which connect the same joints in consecutive frames. All edges on a joint in time represent its trajectory over time, denoted as E. s ={v ti v (t+1)i};Eigenvector F(v ti It consists of a coordinate vector and the estimated confidence level of the i-th joint in the frame.
[0067] S2: Extract scene information and pose information from the unified image sequence and the spatiotemporal human skeleton image respectively to obtain scene spatial features and human pose spatial features. Then, perform feature-level fusion on the scene spatial features and human pose spatial features to obtain the feature-level fusion result. Step S2 specifically includes the following steps.
[0068] S2A: Extracting scene spatial features from a unified image sequence.
[0069] Specifically, VGG16 is used as the image feature extraction network. N images from a uniform image sequence constitute the input data for VGG16, and scene spatial features are extracted from the images. This step S2A is performed by the scene spatial feature extraction module 2A.
[0070] S2B: Perform graph convolution operation on the spatiotemporal human skeleton map to obtain human pose spatial features. This step S2B is executed by the human pose spatial feature extraction module 2B.
[0071] Before performing graph convolution operations on the spatiotemporal human skeleton image, the spatiotemporal human skeleton image is first preprocessed by partitioning.
[0072] Specifically, the spatiotemporal human skeleton diagram specifies which nodes near the root node will be convolved during the convolution process within a single frame, using two partitioning methods: distance partitioning and spatial configuration partitioning. Since the motion of body parts can be roughly divided into concentric motion and eccentric motion, the average coordinates of all joints in the skeleton in a single frame are considered its centroid, i.e., the root node.
[0073] Distance partitioning: based on the distance from the node to the root node V ti The distance d(·,V) ti The neighbor set is partitioned. For example, if D=1, the neighbor set will be divided into two subsets, where d=0 represents the root node itself, and the remaining neighbor nodes are located in the subset d=1. Therefore, there will be two different weight vectors that can model local differential properties, such as the relative translation between joints.
[0074] Spatial configuration partitioning: Divide the neighbor set into three subsets: ① the root node itself; ② the centripetal group, which consists of adjacent nodes that are closer to the center of gravity of the skeleton than the root node; ③ the centrifugal group.
[0075] Then, graph convolution operation is performed on the partitioned spatiotemporal human skeleton map to obtain the spatial features of human posture.
[0076] Specifically, channel-based topology-refined graph convolutional networks (CTR-GCNs) are used to process the human spatiotemporal skeleton map, thereby obtaining the human pose space features. The CTR-GCNs network structure is as follows: Figure 4 As shown, the CTR-GCN model network consists of ten basic modules, followed by a global average pooling and a softmax classifier for predicting action labels. Each module comprises a spatial modeling module, a temporal modeling module, and residual connections.
[0077] The graph convolution formula for the root node itself is:
[0078]
[0079] Among them, Λ ii =∑j (Λ ij +Λ ij ), where matrices Λ and I are the adjacency matrix of a joint in a single frame and the self matrix representing the connection of the joint, respectively, and W is the weight matrix formed by superimposing the weight vectors of multiple output channels.
[0080] For a partitioning strategy with multiple subsets, namely distance partitioning and spatial configuration partitioning (including the root node itself), the graph convolution formula for the human skeleton in a single frame is:
[0081]
[0082] The adjacency matrix is decomposed into several matrices, where Λ+I=∑ j Λ j .
[0083] CTR-GCNs networks can dynamically learn human skeleton maps from different channels of different samples and effectively aggregate joint features from different channels for skeleton-based action recognition. The processing of spatiotemporal skeleton maps by CTR-GCNs networks makes them more effective in learning spatiotemporal features, more robust to pose estimation noise, and has stronger representational capabilities than other graph convolutional networks.
[0084] S2C: This step involves fusing scene spatial features with human pose spatial features at the feature level to obtain the feature-level fusion result. This step S2C is executed by the feature-level fusion module 2C.
[0085] Specifically, scene spatial features and human pose spatial features are fused together in the third dimension (channel dimension) to obtain feature-level fusion results; feature-level fusion allows for early interaction between the scene and human pose extraction networks, enabling early fusion.
[0086] S3: Predict the scene spatial features and human pose spatial features respectively to obtain the first action category prediction result and the second action category prediction result. Then, perform decision-level fusion on the first action category prediction result and the second action category prediction result to obtain the decision-level fusion result. S3 specifically includes the following steps.
[0087] S31A: Perform action segmentation on the scene spatial features extracted from the unified image sequence to obtain scene spatiotemporal features. This step is performed by the scene spatiotemporal feature extraction module 31A.
[0088] Specifically, the scene spatial features are fed into an encoder-decoder temporal convolutional (ED-TCN) network for action segmentation to obtain the scene's spatiotemporal features. The ED-TCN network consists of two parts: an encoder and a decoder. The encoder has l layers, and each layer uses... Indicates; F lT represents the number of convolutional kernels in the l-th layer. l This represents the number of time steps. Each layer includes convolution, a non-linear activation function, and max pooling. The formula for calculating E(l) in the l-th layer is as follows:
[0089] E (l) =max_poiling(f(W*E) (l-1) +))
[0090] Where W is the encoder convolution parameter and b is the encoder convolution network bias parameter.
[0091] The decoder also has l layers, each layer using Among them, F l T represents the number of convolutional kernels in the l-th layer. l This indicates the number of corresponding time steps. Each layer includes upsampling, convolution, and a non-linear activation function. The formula for calculating the current action at time t is as follows:
[0092]
[0093] Where U is the decoder convolution parameter and c is the decoder convolution network bias parameter.
[0094] The ED-TCN network can model the global temporal information of the spatial features of the entire sequence, thereby obtaining the spatiotemporal features.
[0095] S32A: Classify and identify actions based on the spatiotemporal features of the scene to obtain the prediction result of the first action category. This step is performed by the scene feature action recognition module 32A.
[0096] Specifically, the spatiotemporal features of the scene are fed into a fully connected layer for classification and action recognition, leading to the prediction result of the first action category.
[0097] S31B: Perform motion segmentation on the spatial features of human posture to obtain the spatiotemporal features of human posture. This step is performed by the spatiotemporal feature extraction module 31B.
[0098] Specifically, the spatial features of human pose are fed into an encoder-decoder temporal convolutional (ED-TCN) network for action segmentation to obtain the spatiotemporal features of human pose.
[0099] S32B: Classifies and identifies the spatiotemporal features of human posture to obtain the prediction result of the second action category. This step is performed by the human feature action recognition module 32B.
[0100] Specifically, the spatiotemporal features of human posture are fed into a fully connected layer for action classification and recognition, leading to the prediction result of the second action category.
[0101] S3C: The prediction results of the first action category and the second action category are fused at the decision level to obtain the decision-level fusion result. This step is performed by the decision-level fusion module 3C.
[0102] Specifically, the prediction results of the first action category and the prediction results of the second action category are added together at the corresponding elements; the decision-level fusion performs a weighted average of the predictions from the scene and human pose streams and then performs post-fusion.
[0103] S4: Connect the decision-level fusion result with the feature-level fusion result to perform motion evaluation and obtain the ergonomic risk assessment score. Step S4 specifically includes the following steps.
[0104] S41: The decision-level fusion result is used as a supervision factor and concatenated with the feature-level fusion result to obtain the serial fusion feature. This step is performed by the serial fusion module 41.
[0105] Specifically, the decision-level fusion results and feature-level fusion results are concatenated along the third dimension (channel dimension) to obtain concatenated fusion features. The aim is to use the decision-level fusion results as a supervisory factor to improve the performance of the action evaluation task.
[0106] S42: Perform motion evaluation on the serial fusion features to obtain an ergonomic evaluation score. This step is performed by motion evaluation module 42.
[0107] Specifically, the fused features are fed into a Long Short-Term Memory (LSTM) network for action evaluation to obtain an ergonomic evaluation score.
[0108] The ergonomic risk assessment method of this invention combines a soft-sharing approach from multi-task learning. Multi-task learning refers to simultaneously learning multiple related tasks, allowing these tasks to share knowledge during the learning process. This shared knowledge can be representations (features), model parameters, or learning algorithms, etc., utilizing the correlation between multiple tasks to improve the model's performance and generalization ability in each task. Multi-task learning can be viewed as a form of inductive transfer learning, that is, improving generalization ability by using information contained in related tasks as inductive biases. The ergonomic risk assessment method of this invention connects the decision-level fusion result and the feature-level fusion result of the multimodal action recognition task in the third dimension (channel dimension) through multi-task fusion. The aim is to use the decision-level fusion result as a supervisory factor to improve the performance of the action assessment task, allowing the action assessment task to "steal" some information from the action segmentation task to improve its own capabilities.
[0109] Please see Figure 5 and Figure 6 In some embodiments, the parameters of each module of the ergonomic risk assessment device based on posture information and scene information of the present invention are updated through an update step S5, which specifically includes the following steps.
[0110] S51: Calculate the cross-entropy loss value based on the decision-level fusion result. This step is performed by the cross-entropy loss calculation module 51.
[0111] S52: Calculate the logistic regression value based on the ergonomics assessment score. This step is performed by the logistic regression calculation module 52.
[0112] S53: Sum the cross-entropy loss value and the logistic regression loss value to obtain the final loss value. This step is performed by the final loss calculation module 53.
[0113] S54: Update the parameters of each module in steps S1-S4 based on the final loss value. This step is performed by update module 54.
[0114] Based on the same inventive concept, this application also provides an electronic device, which can be a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.) or other terminal device. The device includes one or more processors and a memory, wherein the processor is used to execute an image processing method according to an embodiment of the method; and the memory is used to store a computer program executable by the processor.
[0115] Based on the same inventive concept, this application also provides a computer-readable storage medium corresponding to the aforementioned embodiments of the image processing method. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the image processing method described in any of the above embodiments.
[0116] This application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0117] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.
Claims
1. A method for ergonomic risk assessment based on posture information and scene information, characterized in that, include: Step S1: Preprocess the human posture video of the risk to be assessed to obtain a unified image sequence and a spatiotemporal human skeleton map. Step S2: Extract scene spatial features and human posture spatial features from the unified image sequence and the spatiotemporal human skeleton map respectively. Then, perform feature-level fusion on the scene spatial features and human posture spatial features to obtain the feature-level fusion result. Step S3: Predict the scene spatial features and human posture spatial features respectively to obtain the first action category prediction result and the second action category prediction result. Then, perform decision-level fusion on the first action category prediction result and the second action category prediction result to obtain the decision-level fusion result. Step S4: Connect the decision-level fusion result with the feature-level fusion result to perform action evaluation and obtain an ergonomic risk assessment score; S2A extracts the scene spatial features from the unified image sequence; S2B extracts the spatial features of human posture from the spatiotemporal human skeleton diagram; S2C performs feature-level fusion of the scene space features and the human posture space features to obtain the feature-level fusion result. S31A performs action segmentation on the scene spatial features extracted from the unified image sequence to obtain scene spatiotemporal features; S32A performs action recognition on the spatiotemporal features of the scene to obtain the prediction result of the first action category; S31B performs motion segmentation on the human body posture spatial features to obtain human body posture spatiotemporal features; S32B performs classification and action recognition on the spatiotemporal features of the human posture to obtain the prediction result of the second action category; S3C performs decision-level fusion of the first action category prediction result and the second action category prediction result to obtain the decision-level fusion result; S41 The decision-level fusion result is used as a supervision factor and connected with the feature-level fusion result to obtain a cascaded fusion feature; S42 performs motion evaluation on the serial fusion features to obtain an ergonomic evaluation score.
2. The ergonomic risk assessment method based on posture information and scene information according to claim 1, characterized in that, Step S1 includes: Extract the 2D joint coordinates (x, y) of the human body from the image sequence of the human posture video of the risk to be assessed; For each frame of image, the coordinates of the human body 2D joint points are extracted as nodes of the graph, and the natural connections of the human skeleton are used as edges between nodes, thereby constructing the human skeleton graph of each frame. Connect the corresponding nodes of each node in the time frame with edges to construct a spatiotemporal human skeleton diagram.
3. The ergonomic risk assessment method based on posture information and scene information according to claim 2, characterized in that, Step S2B includes: performing graph convolution operation on the spatiotemporal human skeleton map to obtain the human posture spatial features; before performing graph convolution operation on the spatiotemporal human skeleton map, the spatiotemporal human skeleton map is first preprocessed by partitioning.
4. The ergonomic risk assessment method based on posture information and scene information according to claim 3, characterized in that: In step S2B, the graph convolution operation is performed using a CTR-GCNs network. The graph convolution operation formula is: in This represents the adjacency matrix of the joints in a single frame. denoted by , W represents the matrix of the joint connection itself, and W represents the weight matrix formed by the superposition of weight vectors of multiple output channels.
5. The ergonomic risk assessment method based on posture information and scene information according to claim 4, characterized in that: In step S2A, VGG16 is used as the image feature extraction network. The N images of the unified image sequence are used as the input data of the VGG16 to extract the scene spatial features.
6. An ergonomic risk assessment device based on posture information and scene information, characterized in that, include: The extraction module is used to extract images from each frame of a human pose video to be assessed, thereby obtaining an image sequence; The size unification module is used to resize each image in the image sequence to obtain a unified image sequence; The human skeleton diagram construction module is used to extract the coordinates of human 2D joint points from each image in the image sequence, and construct a spatiotemporal human skeleton diagram based on the human 2D joint point coordinates. A scene space feature extraction module is used to extract scene space features from the unified image sequence; The human posture spatial feature extraction module is used to extract human posture spatial features from the spatiotemporal human skeleton map; The feature-level fusion module is used to perform feature-level fusion of the scene space features and the human posture space features to obtain the feature-level fusion result. The scene spatiotemporal feature extraction module is used to perform action segmentation on all scene spatial features extracted from the unified image sequence to obtain scene spatiotemporal features; The scene feature action recognition module classifies and recognizes the spatiotemporal features of the scene to obtain a first action category prediction result. The human posture spatiotemporal feature extraction module is used to perform action segmentation on the human posture spatial features to obtain human posture spatiotemporal features. The human body feature action recognition module is used to classify and recognize the spatiotemporal features of the human body posture to obtain a second action category prediction result. The decision-level fusion module is used to perform decision-level fusion of the first action category prediction result and the second action category prediction result to obtain a decision-level fusion result; A serial fusion module is used to connect the decision-level fusion result as a supervision factor with the feature-level fusion result to obtain a serial fusion feature. The motion assessment module performs motion assessment on the serial fusion features to obtain an ergonomic assessment score.
7. The ergonomic risk assessment device based on posture information and scene information according to claim 6, characterized in that, The parameters in the ergonomic assessment device can be updated through update step S5, which specifically includes the following steps: S51 Calculates the cross-entropy loss value based on the decision-level fusion result; S52 Calculate the logistic regression value based on the ergonomics assessment score; S53 sums the cross-entropy loss value and the logistic regression loss value to obtain the final loss value; S54 updates the parameters based on the final loss value.