Ergonomic risk assessment method based on multi-modal input and early fusion GCN

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for ergonomic risk assessment based on multimodal input and early fusion of GCN. The method involves acquiring 3D joint point data of the human body to be assessed; preprocessing the joint point data to obtain corresponding joint angle data and motion data; and inputting the joint point data, joint angle data, and motion data into a pre-trained ergonomic risk assessment model to obtain a corresponding ergonomic risk assessment score. The ergonomic risk assessment model is a neural network structure with GCN as the backbone and simultaneously fused with TCN. This solution reduces the cost of acquiring the data to be assessed, increases the dimensionality of the data, and extracts spatial and temporal features through early fusion of GCN and multi-scale TCN neural networks, respectively, reducing the number of model parameters and improving assessment accuracy.

Description

Technical Field

[0001] This application relates to the field of human muscle risk assessment, and in particular to an ergonomic risk assessment method, device, electronic device, and storage medium based on multimodal input and early fusion GCN. Background Technology

[0002] Musculoskeletal disorders (MSDs) have become a global occupational health concern. With the development of computer vision technology, human posture assessment tasks, combined with relevant ergonomic knowledge, can provide corresponding ergonomic scores for human movement postures, thereby standardizing movements and reducing the risk of MSDs.

[0003] However, existing technologies still suffer from problems such as high cost of acquiring motion data, relatively limited dimensions of motion data, and inaccurate ergonomics score evaluation. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for ergonomic risk assessment based on multimodal input and early fusion GCN.

[0005] In a first aspect, embodiments of this application provide an ergonomic risk assessment method based on multimodal input and early fusion GCN, the method comprising the following steps:

[0006] Obtain 3D joint point data of the human body to be evaluated;

[0007] The joint point data is preprocessed to obtain the corresponding joint angle data and motion data;

[0008] The joint point data, joint angle data, and motion data are input into a pre-trained ergonomic risk assessment model to obtain the corresponding ergonomic risk assessment score. The ergonomic risk assessment model is a neural network structure with GCN neural network as the backbone and TCN neural network as the fusion.

[0009] Furthermore, spatial feature data is extracted using the GCN neural network;

[0010] The spatial feature data is input into an attention mechanism, which includes a temporal attention mechanism, a spatial attention mechanism, and a channel attention mechanism. The attention mechanism has a preset dimensionality compression value.

[0011] Spatial feature data containing branch weights is obtained through the attention mechanism;

[0012] The spatial feature data containing branch weights is input into the pooling layer to obtain the pooled spatial feature data.

[0013] The pooled spatial feature data is input into the TCN neural network to extract temporal features and obtain temporal feature information.

[0014] The temporal feature information and the pooled spatial feature data are input into the fully connected layer, and the ergonomic risk assessment score is output according to the preset weight parameters and labels of the pre-trained ergonomic risk assessment model.

[0015] Furthermore, the training process of the ergonomic risk assessment model includes the following steps:

[0016] Obtain a dataset, the training dataset including joint point data and corresponding annotation data for the joint points, the annotation data indicating the ergonomic evaluation score for that joint point;

[0017] The dataset is divided into a training set and a validation set, and the training set is input into the ergonomic risk assessment model. The model is trained by setting the batch_size and learning_rate.

[0018] The number of blocks in the GCN neural network is adjusted based on the training results, and then the validation set is input into the adjusted ergonomic risk assessment model for validation.

[0019] Furthermore, after inputting the validation set into the adjusted ergonomic risk assessment model for validation, the method further includes the following steps:

[0020] Record the loss value in each validation process and preset the threshold for early stopping;

[0021] If the loss value output by the trained model is greater than the currently recorded minimum loss value, and the number of consecutive losses exceeds the set early stopping threshold, then the training process ends.

[0022] Further, the 3D joint point data of the human body to be evaluated is obtained, specifically including the following methods and steps:

[0023] Acquire an RGB video stream, wherein the RGB video stream contains human modality information;

[0024] Based on the RGB video stream, the 3D joint point data of the human body is extracted using the video_3d_pose model.

[0025] Furthermore, after extracting the 3D joint point data of the human body through the video_3d_pose model, the following method steps are also included:

[0026] Filter out zero-valued or rapidly changing 3D human joint point data and retain low-frequency components;

[0027] The missing 3D joint point data of the human body are filled in using linear interpolation.

[0028] Perform skeletal recognition on the 3D human joint point data to obtain a skeletal data sequence;

[0029] The midpoint between the left and right hips in the first skeleton of the skeletal sequence is determined as a reference point, and the distance from the neck to the reference point is calculated and defined as the standardized length.

[0030] Based on the standardized length, transform the coordinates of the human body 3D joint point data;

[0031] The center point of the skeletal data is preset, and the coordinates of the center point are subtracted from the coordinates of the human body 3D joint point data.

[0032] Furthermore, after extracting the 3D joint point data of the human body through the video_3d_pose model, the following method steps are also included:

[0033] Let the dataset of the human body 3D joint data be as follows:

[0034] Based on the joint dataset X, obtain the Motion dataset.

[0035] S={s i |i = 1, 2, 3, ..., V in}

[0036] s t =x[:,t+1,:]-x[:,t,:]

[0037] Based on the joint dataset X and the motion dataset S, obtain the bone length dataset L.

[0038] L={l i |i = 1, 2, ..., V in}

[0039] l i =x[:,:i]-x[:,:,i adj ]

[0040] i adjIndicates the node adjacent to i, and points to the center point;

[0041] Based on the joint dataset X, the bone length dataset L, and the motion dataset S, obtain the Angle angle dataset.

[0042] A={a i |i = 1, 2, ..., V in}

[0043]

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Attached Figure Description

[0045] Figure 1 A flowchart illustrating the steps of an ergonomic risk assessment method based on multimodal input and early fusion GCN in an exemplary embodiment;

[0046] Figure 2 This is an RGB video frame illustration of an ergonomic risk assessment method based on multimodal input and early fusion GCN provided in an exemplary embodiment.

[0047] Figure 3 A schematic diagram of 3D human joint points corresponding to RGB video frames, provided as an exemplary embodiment of an ergonomic risk assessment method based on multimodal input and early fusion GCN.

[0048] Figure 4 This is a schematic diagram of the structure of an ergonomic risk assessment model based on multimodal input and early fusion GCN provided in an exemplary embodiment;

[0049] Figure 5 This is a data illustration of the dataset used in an ergonomic risk assessment method based on multimodal input and early fusion GCN provided in an exemplary embodiment;

[0050] Figure 6 This is a schematic diagram illustrating the acquisition of the Motion dataset for an ergonomic risk assessment method based on multimodal input and early fusion GCN, provided in an exemplary embodiment.

[0051] Figure 7 This is a schematic diagram illustrating the acquisition of the Angle angle dataset for an ergonomic risk assessment method based on multimodal input and early fusion GCN, provided in an exemplary embodiment.

[0052] Figure 8 A flowchart illustrating the implementation of the attention mechanism in an ergonomic risk assessment method based on multimodal input and early fusion GCN, provided in an exemplary embodiment;

[0053] Figure 9 This is a schematic diagram illustrating the internal implementation of the attention mechanism in an ergonomic risk assessment method based on multimodal input and early fusion GCN, provided in an exemplary embodiment.

[0054] Figure 10 The network architecture diagram of MSTCN is provided in an exemplary embodiment of an ergonomic risk assessment method based on multimodal input and early fusion GCN.

[0055] Figure 11 This is a schematic diagram of a module for an ergonomic risk assessment device based on multimodal input and early fusion GCN, provided in an exemplary embodiment.

[0056] Figure 12 This is a schematic diagram of an electronic device provided in one exemplary embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0058] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0059] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0060] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0061] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0062] Musculoskeletal disorders (MSDs) are the most serious consequences of work-related musculoskeletal inadequacies, with symptoms defined as pain in one or more areas of the body. The accumulation of minor injuries from repetitive, prolonged work-related loads is considered a major cause of MSDs. MSDs are prevalent in labor-intensive industries such as construction and handicrafts. MSDs are chronic diseases, and symptoms are often subtle in their early stages; therefore, prevention and treatment should be the primary focus.

[0063] Currently, the evaluation methods for MSDs include:

[0064] (1) Self-report-based assessment methods, which collect relevant data from participants by analyzing staff logs and conducting interviews and questionnaires, are used to achieve the assessment effect. This method is simple to use, but due to the large gap in participants' cognitive and cultural levels, participants have a high degree of subjectivity in questionnaires and interviews, so the test results have a certain degree of subjectivity and unreliability.

[0065] (2) Evaluation method based on direct measurement. This method mainly evaluates MSDs by collecting bioelectrical signals or work posture data of those involved in the work and analyzing the trends and patterns of the data.

[0066] (3) Job observation-based assessment methods. Studies have confirmed a strong correlation between work posture, workload, repetition frequency, and duration and MSDs (Meningococcal Disorders). Standardizing these influencing factors can effectively reduce the risk of MSDs. This involves acquiring the main work postures and workload durations during daily work and assessing the associated disease risk through observational assessment methods. These methods typically use paper-and-pencil and tables to record relevant factors, with typical methods including the Rapid Entire Body Assessment (REBA) and Rapid Upperlimb Assessment (RULA). While observational assessment methods can assess MSD risk for individuals and provide ergonomic corrections and interventions for specific risk areas or postures, the process is cumbersome and inefficient. Furthermore, when the sample size is large, assessors are prone to errors.

[0067] (4) HPA-based assessment methods. With the development of computer vision technology, Human Activity Evaluation (HAE) has gained attention. This type of task captures changes in human movement and measures the quality of human movement according to predefined indicators, thus being applied to fields such as motor activity scoring and skill assessment. Human Postural Assessment (HPA) is a branch of HAE. HPA is widely used in fields such as medicine and biomechanics, and its key research direction lies in the application of ergonomics. Applying HPA tasks in conjunction with relevant ergonomic knowledge can provide corresponding ergonomic scores for human movement postures, thereby standardizing movements and reducing the risk of MSDs (Meningococcal Disorders).

[0068] In existing technologies, there are ergonomic analyses and assessments of musculoskeletal injury (MSD) risks for typical VDT office workers using questionnaire surveys combined with RULA; there are also methods based on Rapid Upper Limb Assessment (RULA) to assess ergonomic risks in mask production; there are also ergonomic scoring worksheets to implement classification algorithms for actions and postures; and there are also related methods that use a multi-task framework (MLT) based on 3D skeleton points to achieve action recognition and ergonomic assessment.

[0069] This application studies and analyzes the current research status of existing ergonomic risk assessment methods, and finds that there are still some problems to be solved. For example, manual methods such as questionnaires and observation assessments are subject to subjectivity, are labor-intensive, and have large errors. Most existing ergonomic risk assessment algorithms are based on 3D skeletons, which requires the use of relatively expensive Kinect cameras, resulting in high costs and inconvenience for promotion. The MLT method only considers the joint feature information, and its skeleton modal information is relatively simple, without comprehensive processing of various dimensions. MLT combined with LSTM to process the extracted and processed skeleton information, although LSTM can extract some information in the time dimension, has a large computational cost and poor performance.

[0070] Based on the above research and analysis results, and the existing technical problems in the background art, this application provides an ergonomic risk assessment method based on multimodal input and early fusion GCN, such as... Figure 1 As shown, the specific methods and steps include the following:

[0071] S201: Obtain 3D joint location data of the human body to be evaluated.

[0072] like Figure 2 and Figure 3 As shown, traditional human 3D skeleton data requires the use of a Kinect depth camera to extract the main joint points of the human body. In this embodiment, a video_pose_3d neural network is used to process the 2D RGB image information of the video stream to obtain the corresponding human 3D joint points. Specifically, the 2D RGB image information is obtained by segmenting the video stream containing human motion information frame by frame.

[0073] S202: Preprocess the joint position data to obtain the corresponding joint angle data and motion data.

[0074] In existing technologies, Kinect often encounters noise and missing joint points during the extraction of 3D skeleton data due to occlusion by obstacles and the position of the subject relative to the camera. Obtaining high-quality 3D skeleton data is crucial for assessing ergonomic risks. Therefore, in this embodiment, the skeleton data, i.e., the joint point data, is considered time-series data. Filtering and linear interpolation methods are used to process the 3D skeleton data to improve its quality. These data processing methods include noise filtering, normalization, and alignment.

[0075] Noise filtering is typically performed on the raw joint coordinates in most research works. Traditional image filtering techniques, such as Laplacian smoothing, Gaussian filtering, Discrete Cosine Transform (DCT), and Discrete Fourier Transform (DFT), have been used to transform the discrete coordinates of joint trajectories. Therefore, embodiments of this application filter out zero-valued or abruptly changing coordinates and retain low-frequency components as a reliable detection of human position. Linear interpolation can be used to fill in missing joint point data.

[0076] The normalization process is necessary because people's height and shooting distance vary greatly, resulting in potentially different human scales in different videos. To eliminate the influence of body shape and physique on muscle risk assessment, the skeletal data is normalized. In this embodiment, the midpoint between the left and right hips in the first skeleton of the sequence is selected as a reference point. The distance from the neck to this reference point is then calculated and defined as the normalized length. Finally, the joint position coordinates are transformed by scaling using the normalized length.

[0077] Alignment is necessary because the subject may not be in the same position relative to the camera during the action, and the action video is usually untrimmed, including the preparation and ending positions of the action instance. This embodiment translates the skeleton points to a uniform position, eliminating the influence of positional information. In this embodiment, since the offset from the camera center will be different, the system uses the center of the hip as the origin of the 3D coordinate system. All coordinates are successively subtracted from the coordinates of the hip center, and the skeleton data is translated to a position with the hip as the origin of the 3D coordinate axis (0, 0, 0), thereby achieving alignment.

[0078] S203: Input the joint point data, the joint angle data and the motion data into the pre-trained ergonomic risk assessment model to obtain the corresponding ergonomic risk assessment score. The ergonomic risk assessment model is a neural network structure with GCN neural network as the backbone and TCN neural network as the fusion.

[0079] The following section will describe the structural composition of the ergonomic risk assessment model in conjunction with the model training process:

[0080] like Figure 4 As shown, Figure 4This is a schematic diagram of the ergonomic risk assessment model. In this embodiment, ctr-gcn-baseline is used as the backbone. ctr-gcn (Channel-wise Topology RefinementGraph Convolution Network) proposes that different channels represent the correlation between motion features of different motion types at various joints. ctr-gcn dynamically and adaptively models the adjacency matrix based on the channels, capturing the relationships between joints. Only the structure and size of the adjacency matrix need to be defined based on the input joint data. ctr-gcn-baseline is a basic version of ctr-gn, with fewer parameters and training performance not significantly different from the full version of ctr-gcn. Therefore, this embodiment uses the baseline version as the backbone.

[0081] The three modal data—joint position data, joint angle data, and motion data—are used as input. According to existing medical ergonomic risk assessment methods, ergonomic risk is closely related to the joint positions, joint angles, and motion states between adjacent time intervals in the human posture at a given moment. Joint positions contain the most information and are therefore the most common data for assessing human muscle risk. Joint angles reflect the force and load between joints; for example, the occurrence of reverse joint angles can lead to dislocations and other significant muscle risks. Data between adjacent time intervals reflects the motion data of that posture; for example, a sudden and violent displacement of a key point within a short period can easily cause high-intensity joint loads, leading to problems such as strains. The labels in this project are calculated using the REBA method, which provides clear measurements of joint angles and motion speed. Angles and speeds are significant ergonomic indicators; therefore, we explicitly extract these features, which helps improve the accuracy of the model.

[0082] In this embodiment of the application, the dataset is a UW-IOM Dataset, such as... Figure 5 As shown, this dataset consists of 20 videos, primarily featuring lifting actions in industrial settings. The dataset provides the 3D coordinates of the human joints, action category, and ergonomic risk score for each video frame. The ergonomic score is labeled using the REBA method, with scores ranging from 1 to 15, where below 3 indicates low risk, 3-7 indicates medium risk, and 8-15 indicates high risk.

[0083] Specifically, the batch size was set to 8, and the learning rate was set to [5e-2, 1e-2, 5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5] for multiple rounds of training, and the results were observed. The dataset contains 20 videos; 15 were selected as the training set, and 5 as the validation set. A parallel GCN structure was used in the ergonomic risk assessment model. During training, numerous comparative experiments were conducted to adjust the number of GCN blocks and select the parallel structure with the best performance.

[0084] The data consists of the x, y, and z coordinates of 15 key points in each frame of the video, thus having three dimensions: C, T, and V. C represents the dimensions of the x, y, and z coordinates, T represents the description of the video frame, and V represents the coordinates of the key points.

[0085] Let the dataset of the human body 3D joint data be as follows:

[0086] Based on the joint dataset X, obtain the Motion dataset, such as... Figure 6 As shown,

[0087] S={s i |i = 1, 2, 3, ..., Vi n}

[0088] s t =x[:,t+1,:]-x[:,t,:]

[0089] Based on the joint dataset X and the motion dataset S, obtain the bone length dataset L.

[0090] L={l i |i = 1, 2, ..., V in}

[0091] l i =x[:,:i]-x[:,:,i adj ]

[0092] i adj Indicates the node adjacent to i, and points to the center point;

[0093] Based on the joint dataset X, the bone length dataset L, and the motion dataset S, obtain the angle dataset, such as... Figure 7 As shown,

[0094] A={a i |i = 1, 2, ..., Vin}

[0095]

[0096] The motion dataset uses the differences between keypoints in adjacent time frames. Because the subtraction is pairwise, a null value may occur. To maintain dimensional consistency, the null value is filled with 0. ":" is a representation in Python's NumPy library, representing all values ​​for that dimension. Therefore, x[:, t+1, :] - x[:, t, :] represents the subtraction of the x, y, and z coordinates of corresponding keypoints in adjacent time frames.

[0097] The bone length dataset is obtained by subtracting connected joints within the same time frame. It is predefined that the outer points are subtracted from the points closer to the center. Then, based on the bone length, the angles a(i, w) in the x, y, and z directions are calculated. Finally, the cosine formula is used to calculate the bone length dataset.

[0098] To prevent overfitting during training, we need to incorporate an early stopping mechanism. Early stopping involves stopping the training process earlier by setting a threshold, called "patience." If the number of consecutive times the model's loss on the validation set exceeds the minimum recorded validation loss exceeds this threshold, training is stopped early. This mechanism effectively prevents overfitting.

[0099] The output data of the parallel-structured GCN neural network is used as the input to the attention mechanism. For example... Figure 8 As shown, Figure 8 The flowchart shows the implementation of the attention mechanism.

[0100] Spatial attention focuses on data in a spatial dimension, learning the importance of joints and prioritizing those with a greater impact on muscle risk. This mechanism compresses other spatial dimensions using a mean function, then quantizes that dimension into a decimal using a conv1d and sigmoid function. The result is multiplied by the original input and then added back to the original input, thus achieving spatial attention.

[0101] Temporal attention focuses on data in the temporal dimension, extracting temporal information and resolving relationships between key points in different frames. The underlying concept of temporal attention is similar to that of spatial attention.

[0102] Channel attention focuses on data along the channel dimension. Since this model has many layers, the attention mechanism helps it learn more useful channel information, improving model accuracy. Channel attention compresses the temporal and spatial dimensions, then processes the result through fully connected layers and activation functions. Finally, the result is multiplied by the original input and added back to the original input, thus implementing the channel attention mechanism.

[0103] like Figure 9 As shown, Figure 9 This is a schematic diagram of the internal implementation of the attention mechanism. The implementation process of the attention mechanism is to multiply the compressed data in this dimension with the original data and add it back to the original data, thereby realizing the weights of different regions of this dimension.

[0104] The outputs of the three attention mechanisms are fused early on. The three-modal data trained on a certain block are concatenated and input into the main stream, thereby reducing the training load of the model and performing fusion training on the three-modal data, so that the data processed by the GCN network contains richer and more useful information.

[0105] like Figure 10 As shown, Figure 10 The diagram shows the network architecture of MSTCN, which uses multi-scale TCN to process data obtained from the GCN backbone network. Because CTR-GCN can adaptively learn spatial information between nodes, multi-scale TCN can effectively extract temporal information, allowing the network to learn richer information and thus improving model accuracy. The multi-scale flowchart uses a parallel structure and employs multi-scale kernel size and dilation to extract even richer information.

[0106] This application's embodiments are based on a 3D human estimation model to collect skeletal information, which has a wider range of applications and stronger universality, and can be more adaptable to industrial scenarios. By extracting multimodal information from the robust 3D skeleton data, the accuracy of the model is improved. Early fusion is used to significantly reduce the number of model parameters, and channel attention mechanism and branch weight are used to significantly improve algorithm performance. Multi-scale TCN is used to better process temporal information, extract richer temporal information, improve algorithm performance, and reduce the number of parameters.

[0107] like Figure 11 As shown, Figure 11 An ergonomic risk assessment device 400 based on multimodal input and early fusion GCN provided in this application embodiment includes:

[0108] The data acquisition module 401 is used to acquire 3D joint point data of the human body to be evaluated;

[0109] The data preprocessing module 402 is used to preprocess the joint position data to obtain the corresponding joint angle data and motion data;

[0110] The evaluation score acquisition module 403 is used to input the joint position data, the joint angle data and the motion data into a pre-trained ergonomic risk assessment model to obtain the corresponding ergonomic risk assessment score. The ergonomic risk assessment model is a neural network structure with GCN neural network as the backbone and TCN neural network as the fusion.

[0111] It should be noted that the ergonomic risk assessment device based on multimodal input and early fusion GCN provided in the above embodiments is only illustrated by the division of the above functional modules when executing the ergonomic risk assessment method based on multimodal input and early fusion GCN. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the ergonomic risk assessment device based on multimodal input and early fusion GCN and the ergonomic risk assessment method based on multimodal input and early fusion GCN provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0112] The electronic device includes a processor 910 and a memory 920. The main control chip may contain one or more processors 910. Figure 12 Taking a processor 910 as an example, the main control chip can contain one or more memory modules 920. Figure 12 Take a memory chip 920 as an example.

[0113] The memory 920, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the ergonomic risk assessment method program based on multimodal input and early fusion GCN described in any embodiment of this application, and the corresponding program instructions / modules of the ergonomic risk assessment method based on multimodal input and early fusion GCN described in any embodiment of this application. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on device usage, etc. Furthermore, the memory 920 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 920 may further include memory remotely located relative to the processor 910, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The processor 910 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 920, thereby implementing the ergonomic risk assessment method based on multimodal input and early fusion GCN described in any of the above embodiments.

[0115] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the ergonomic risk assessment method based on multimodal input and early fusion GCN described in any of the above embodiments.

[0116] This invention can take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0117] It should be understood that the embodiments of this application are not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from their scope. The scope of the embodiments of this application is limited only by the appended claims.

[0118] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the embodiments of this application, and these all fall within the protection scope of the embodiments of this application.

Claims

1. An ergonomic risk assessment method based on multi-modal input and early fusion GCN, characterized in that, Includes the following steps: Obtain 3D joint point data of the human body to be evaluated; The joint point data is preprocessed to obtain the corresponding joint angle data and motion data; The joint point data, joint angle data, and motion data are input into a pre-trained ergonomic risk assessment model to obtain the corresponding ergonomic risk assessment score. The ergonomic risk assessment model is a neural network structure with GCN neural network as the backbone and TCN neural network integrated. After inputting the joint position data, joint angle data, and motion data into the pre-trained ergonomic risk assessment model, the following steps are included: Spatial feature data is extracted using the GCN neural network. The spatial feature data is input into an attention mechanism, which includes a temporal attention mechanism, a spatial attention mechanism, and a channel attention mechanism. The attention mechanism has a preset dimensionality compression value. Spatial feature data containing branch weights is obtained through the attention mechanism; The spatial feature data containing branch weights is input into the pooling layer to obtain the pooled spatial feature data. The pooled spatial feature data is input into the TCN neural network to extract temporal features and obtain temporal feature information. The temporal feature information and the pooled spatial feature data are input into the fully connected layer, and the ergonomic risk assessment score is output according to the preset weight parameters and labels of the pre-trained ergonomic risk assessment model. Obtaining 3D joint point data of the human body to be evaluated includes the following steps: Acquire an RGB video stream, wherein the RGB video stream contains human modality information; Based on the RGB video stream, the 3D joint point data of the human body is extracted using the video_3d_pose model; After extracting the 3D joint point data of the human body using the video_3d_pose model, the following method steps are also included: Filter out zero-valued or rapidly changing 3D human joint point data and retain low-frequency components; The missing 3D joint point data of the human body are filled in using linear interpolation. Perform skeletal recognition on the 3D human joint point data to obtain a skeletal data sequence; The midpoint between the left and right hips in the first skeleton of the skeletal data sequence is determined as the reference point. The distance from the neck to the reference point is then calculated and defined as the standardized length. Based on the standardized length, transform the coordinates of the human body 3D joint point data; The center point of the skeletal data is preset, and the coordinates of the center point are subtracted from the coordinates of the human body 3D joint point data.

2. The ergonomic risk assessment method based on multi-modal input and early fusion GCN according to claim 1, characterized in that, The training process of the ergonomic risk assessment model includes the following steps: Obtain the dataset. The training dataset includes joint point data and corresponding annotation data for the joint points. The annotation data indicates the ergonomic evaluation score for the joint point. The dataset is divided into a training set and a validation set, and the training set is input into the ergonomic risk assessment model. The model is trained by setting the batch_size and learning_rate. The number of blocks in the GCN neural network is adjusted based on the training results, and then the validation set is input into the adjusted ergonomic risk assessment model for validation.

3. The ergonomic risk assessment method based on multimodal input and early fusion GCN as described in claim 2, characterized in that, After inputting the validation set into the adjusted ergonomic risk assessment model for validation, the method further includes the following steps: Record the loss value in each validation process and preset the threshold for early stopping; If the loss value output by the trained model is greater than the currently recorded minimum loss value, and the number of consecutive losses exceeds the set early stopping threshold, then the training process ends.

4. The ergonomic risk assessment method based on multimodal input and early fusion GCN as described in claim 1, characterized in that, After extracting the 3D joint point data of the human body using the video_3d_pose model, the following method steps are also included: Let the dataset of the human body 3D joint data joint be X= ; Based on the joint dataset X, obtain the Motion dataset. Based on the joint dataset X and the motion dataset S, obtain the bone length dataset L. Indicates and The adjacent nodes, and point to the center point; Based on the joint dataset X, the bone length dataset L, and the motion dataset S, obtain the angle dataset. 。 5. An ergonomic risk assessment device based on multimodal input and early fusion GCN for performing the ergonomic risk assessment method based on multimodal input and early fusion GCN as described in claim 1, characterized in that, include: The data acquisition module is used to acquire 3D joint point data of the human body to be evaluated; The data preprocessing module is used to preprocess the joint point data to obtain the corresponding joint angle data and motion data; The evaluation score acquisition module is used to input the joint position data, the joint angle data and the motion data into a pre-trained ergonomic risk assessment model to obtain the corresponding ergonomic risk assessment score. The ergonomic risk assessment model is a neural network structure with GCN neural network as the backbone and TCN neural network integrated.

6. An electronic device, characterized in that, include: At least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor performs the steps of the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images