Engineering machinery strike risk training system and method based on multi-modal data acquisition

Abstract

The invention discloses an engineering machinery strike risk training system and method based on multi-modal data acquisition. The system comprises a multi-modal data acquisition module, a data fusion processing module and a risk identification and evaluation module. The multi-modal data acquisition module is used for acquiring data in an engineering machinery operation environment in real time, wherein the data comprise action data, image data, sound data and inertial data. The data fusion processing module is used for carrying out standardization and normalization processing on the collected data of various modes and eliminating the dimensional influence, so that the data of different modes have comparability, meaningful features are extracted from the data of each mode, the features can reflect the correlation between essential information of the data and the modes, and the data fusion processing module is used for carrying out data fusion processing on the data of different modes. According to the method, features of different modes are fused by using a multi-mode fusion algorithm, the importance, correlation and complementarity of data of each mode can be considered, and a comprehensive feature vector is generated, so that effective fusion of the data is realized, and the effect of simulation training is ensured.

Description

Technical Field

[0001] The present invention relates to a training system and method, specifically a training system and method for construction machinery impact risks with multi-modal data acquisition, belonging to the technical field of simulation training. Background Technique

[0002] In the process of learning and mastering construction machinery operation skills, carrying out risk simulation training is a crucial teaching link. In the specific implementation of the simulation training, the system will highly simulate and collect various action data of the user when operating the construction machinery, and at the same time accurately simulate the impact injury scenarios that the construction machinery may cause under different working conditions. For example, simulating the impact injury caused by the falling of a heavy object due to improper operation when a crane hoists a heavy object, or the collision injury caused to surrounding personnel due to misoperation during the excavation operation of an excavator, etc.

[0003] Through such simulation training, it is possible to extremely effectively avoid the serious risks brought by improper operation during the actual use of construction machinery. Improper operation may stem from various factors such as the operator's unfamiliarity with the mechanical performance, the rustiness of the operation skills, or the judgment error in a complex environment. And risk simulation training allows the operator to experience the consequences of improper operation in advance in a safe and controllable virtual environment, so as to deeply realize the importance of standardized operation.

[0004] The importance of carrying out risk simulation training is self-evident. On the one hand, it greatly improves the operator's safety awareness and emergency handling ability. In the simulated scenario, when the operator faces sudden dangerous situations, they can learn how to quickly make correct responses and reduce the probability of accidents. On the other hand, simulation training can save a large amount of actual training costs. The actual operation training of construction machinery is often restricted by various factors such as site, equipment, and safety, while simulation training is not restricted by these conditions and can be carried out anytime and anywhere, and can be practiced repeatedly until the operator proficiently masters the correct operation skills. In addition, risk simulation training also helps enterprises reduce the economic losses and legal liabilities caused by operation accidents and ensure the normal operation and development of the enterprises. Therefore, when learning to use construction machinery, risk simulation training is an indispensable important link.

[0005] Under the existing technical system, when carrying out this key teaching link of risk simulation training, it is usually necessary to comprehensively and multi-dimensionally capture various relevant data of the user, mainly including action data, image data, and inertial data. These three different modal data have completely different description forms. Due to the complexity of the description forms and coupling relationships of the three modal data, it is difficult to unify and integrate them for representation.

[0006] Existing data processing methods often can only process data of a single modality, making it difficult to fully exploit the correlation information between different modalities of data. When these data cannot be effectively fused, the simulation training system cannot accurately restore the real operation scenario and mechanical motion state, which will significantly affect the effect of simulation training. Therefore, a construction machinery impact risk training system and method for multi-modal data acquisition are proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a construction machinery impact risk training system and method for multi-modal data acquisition to solve one of the problems mentioned in the above background technology.

[0008] The present invention is implemented by the following technical solutions: A construction machinery impact risk training system and method for multi-modal data acquisition, including a multi-modal data acquisition module, a data fusion processing module, and a risk identification and assessment module; The multi-modal data acquisition module is used to collect data in the operation environment of construction machinery in real time, including action data, image data, sound data, and inertial data; The multi-modal data acquisition module includes an action capture and analysis module, a depth image understanding module, a sound recognition and detection module, and an inertial data state analysis module; The action capture and analysis module is used to capture the actions and posture information of the operator using an optical motion capture system, and perform three-dimensional reconstruction and analysis; The depth image understanding module is used to collect the depth image of the operation environment using a depth camera, and use computer vision technology for scene understanding and obstacle detection to identify obstacles and potential dangerous areas in the operation environment in real time; The sound recognition and detection module is used to collect sound data during the operation of construction machinery, and use audio processing technology for sound recognition and anomaly detection to timely detect mechanical failures or abnormal sounds; The inertial data state analysis module is used to collect the acceleration and angular velocity of construction machinery and the operator through an IMU (inertial measurement unit) device, analyze the motion state and stability, evaluate the motion smoothness of construction machinery and the motion coordination of the operator, and identify potential motion risks; The data fusion processing module is used to fuse and process the collected multi-modal data using a multi-modal fusion algorithm, extract feature information, and achieve the organic fusion of multi-modal data; The risk identification and assessment module is used to analyze and process the fused data using a risk assessment algorithm, identify potential risk factors, and evaluate the risk level.

[0009] As a further preference of this technical solution: The multi-modal fusion algorithm includes the following steps: Data preprocessing: Standardize the collected motion data, image data, sound data, and inertial data to eliminate the influence of dimensions, and interpolate or remove missing or abnormal data; Feature extraction: Extract features from each modality data respectively, and use the principal component analysis method to reduce the dimension of high-dimensional features; Adaptive weight calculation: Calculate the adaptive weight based on mutual information according to the importance of each modality data in the current task; Data fusion: Use the calculated adaptive weights to perform weighted fusion on the features of each modality. The fused feature vector will contain information from all modalities, and the contribution of each modality is adjusted according to its importance; Post-processing and output: Further process the fused feature vector, perform normalization and denoising, and output the processed feature vector for use by the subsequent risk identification and assessment module.

[0010] As a further preferred embodiment of this technical solution: The steps of calculating the adaptive weight include: Calculate mutual information: For each modality data, calculate the mutual information between it and the task objective. Mutual information is used to measure the correlation between modality data and the task objective. The greater the mutual information, the stronger the correlation between the modality data and the task objective, and the greater the contribution to the task; Normalize mutual information: Normalize the mutual information of all modalities so that their sum is equal to 1. The normalized mutual information is used as the weight of each modality.

[0011] As a further preferred embodiment of this technical solution: The features extracted by the data fusion processing module include motion features, image features, sound features, and inertial features; The motion features include joint angles, speeds, and acceleration features, which are used to reflect the operation standardization and stability of the operator, and also include the patterns and frequencies of motion sequences, which are used to identify specific operation behaviors or habits; The image features include edges, textures, and colors, which are used to identify obstacles and equipment states in the operation environment, and also include the positions and postures of target objects, specifically the positions of components of construction machinery and the body postures of operators; The sound features include frequencies, energies, and timbres, which are used to identify mechanical failures and abnormal sounds, and also include the patterns and frequencies of sound events, which are used to judge the operating state of the machinery or the behavior of the operator; The inertial features include accelerations, angular velocities, and attitude angles, which are used to reflect the motion states of construction machinery and operators, and also include motion trajectories and patterns, which are used to analyze the motion smoothness of the machinery and the motion coordination of the operator.

[0012] As a further preference of this technical solution: The risk assessment algorithm includes the following steps: Feature selection: Select actions, images, sounds, and inertial features related to risk assessment; Model training: Build a risk assessment model, use historical data as the training set, train the model, and optimize the model parameters through cross-validation and grid search methods during the training process; Risk identification: Input the real-time collected and fused data into the trained model to obtain the risk prediction result, where the prediction result is the probability of risk occurrence and the risk category; Risk level assessment: According to the risk prediction result, combined with the preset risk level classification standard, assess the risk level; Output and feedback: Output the risk level assessment result to provide real-time risk warnings for operators.

[0013] As a further preference of this technical solution: The classification of the risk level includes: Low risk: The probability of risk occurrence is relatively low, and the consequences caused after occurrence are relatively light. The operator can continue normal operation, but needs to pay attention and remain vigilant; Medium risk: The probability of risk occurrence is moderate, and the consequences caused after occurrence are relatively serious. The operator needs to take certain preventive measures; High risk: The probability of risk occurrence is relatively high, and the consequences caused after occurrence are very serious. The operator should immediately stop the operation, take emergency measures to avoid the occurrence of risks, and report to the superior management personnel.

[0014] As a further preference of this technical solution: It further includes a virtual simulation scenario construction module, which is used to construct a virtual construction machinery operation environment, simulate real operation scenarios and risk factors, and provide an immersive training experience for operators.

[0015] As a further preference of this technical solution: It further includes a personalized training customization module, which is used to formulate a personalized training plan according to the actual situation and training needs of the operator.

[0016] As a further preference of this technical solution: It further includes a data storage and management module and an interaction module, which are used to store and manage the collected multi-modal data, training records, and evaluation results, and the interaction module is used to implement the human-computer interaction function.

[0017] A construction machinery strike risk training method for multi-modal data collection includes the following steps: Step 1: Construct a virtual construction machinery operation environment, simulate real operation scenarios and risk factors, and provide an immersive training experience for operators; Step 2: Use an optical motion capture system to capture the actions and posture information of the operator, and perform 3D reconstruction and analysis; Step 3: Use a depth camera to collect the depth images of the operating environment, and utilize computer vision technology for scene understanding and obstacle detection to real-time identify the obstacles and potential dangerous areas in the operating environment; Step 4: Collect the sound data during the operation of the construction machinery, and utilize audio processing technology for sound recognition and anomaly detection; Step 6: Collect the acceleration and angular velocity of the construction machinery and the operator through an IMU device, and analyze the motion state and stability; Step 8: Adopt a multi-modal fusion algorithm to fuse and process the collected multi-modal data, extract feature information, and achieve the organic fusion of multi-modal data; Step 7: Adopt a risk assessment algorithm to analyze and process the fused data, identify potential risk factors, and evaluate the risk level.

[0018] Advantages of the present invention: The present invention standardizes and normalizes the collected multi-modal data through a data fusion processing module to eliminate the influence of dimension, making the data of different modalities comparable, extracting meaningful features from the data of each modality, and these features can reflect the essential information of the data and the correlation between modalities. By fusing different modal features through a multi-modal fusion algorithm, comprehensively considering the importance, relevance, and complementarity of the data, a comprehensive feature vector is generated. By performing dimensionality reduction processing on the fused feature vector, the computational complexity is reduced, while noise and redundant information are removed, improving the purity and accuracy of the data, and thus achieving the effective fusion of data and ensuring the effect of simulation training. Brief Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a structural schematic diagram of a construction machinery strike risk training system for multi-modal data acquisition of the present invention; Figure 2 It is a step flow chart of the multi-modal fusion algorithm of the present invention; Figure 3 It is a step flow chart of the risk assessment algorithm of the present invention; Figure 4 It is a step flow chart of a construction machinery strike risk training method for multi-modal data acquisition of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Example See also Figures 1 - 4 ,The present invention provides a technical solution: a multi-modal data acquisition engineering machinery strike risk training system, including a multi-modal data acquisition module, a data fusion processing module and a risk identification and evaluation module; The multimodal data acquisition module is used to collect data in the operation environment of engineering machinery in real time, including motion data, image data, sound data and inertial data; The multimodal data acquisition module includes a motion capture analysis module, a deep image understanding module, a sound recognition detection module, and an inertial data state analysis module; The motion capture analysis module is used to capture the operator's motion and posture information using an optical motion capture system, and perform three-dimensional reconstruction and analysis; The depth image understanding module is used to use a depth camera to collect depth images of the operating environment, use computer vision technology to perform scene understanding and obstacle detection, and identify obstacles and potential danger areas in the operating environment in real time; The sound recognition and detection module is used to collect sound data during the operation of construction machinery, and uses audio processing technology to perform sound recognition and anomaly detection to promptly detect mechanical failures or abnormal sounds; The inertial data state analysis module is used to collect the acceleration and angular velocity of the construction machinery and operators through the IMU device, analyze the motion state and stability, evaluate the motion stability of the construction machinery and the motion coordination of the operators, and identify potential motion risks; The data fusion processing module is used to fuse and process the collected multimodal data using a multimodal fusion algorithm, extract feature information, and realize the organic fusion of multimodal data; The multimodal fusion algorithm includes the following steps: Data preprocessing: standardize the collected motion data, image data, sound data and inertial data, eliminate the impact of dimension, and interpolate or eliminate missing or abnormal data; Feature extraction: Extract features from each modality data separately, and use principal component analysis (PCA) to reduce the dimensionality of high-dimensional features; Adaptive weight calculation: Calculate the adaptive weight based on mutual information according to the importance of each modality data in the current task; Data fusion: Use the calculated adaptive weight to perform weighted fusion on the features of each modality. The fused feature vector will contain information from all modalities, and the contribution of each modality is adjusted according to its importance; Post - processing and output: Further process the fused feature vector, perform normalization and denoising, and output the processed feature vector for use by the subsequent risk identification and assessment module; The steps for calculating the adaptive weight include: Calculate mutual information: For each modality data, calculate the mutual information between it and the task objective. Mutual information is used to measure the correlation between modality data and the task objective. The larger the mutual information, the stronger the correlation between the modality data and the task objective, and the greater the contribution to the task; Normalize mutual information: Normalize the mutual information of all modalities so that their sum equals 1. The normalized mutual information is used as the weight for each modality.

[0023] The risk identification and assessment module is used to analyze and process the fused data using a risk assessment algorithm, identify potential risk factors, and assess the risk level; The risk assessment algorithm includes the following steps: Feature selection: Select actions, images, sounds, and inertial features relevant to risk assessment; Model training: Build a risk assessment model, use historical data as the training set to train the model. During the training process, optimize the model parameters through cross - validation and grid search methods; Risk identification: Input the real - time collected and fused data into the trained model to obtain the risk prediction result. The prediction result is the probability of risk occurrence and the risk category; Risk level assessment: According to the risk prediction result, combined with the preset risk level classification criteria, assess the risk level; Output and feedback: Output the risk level assessment result to provide real - time risk warning for the operator; The classification of risk levels includes: Low risk: The probability of risk occurrence is relatively low, and the consequences after occurrence are relatively minor. The operator can continue normal operations but needs to be vigilant; Medium risk: The probability of risk occurrence is moderate, and the consequences after occurrence are relatively serious. The operator should take certain preventive measures; High risk: The probability of risk occurrence is relatively high, and the consequences after occurrence are very serious. The operator should immediately stop operations, take emergency measures to avoid the occurrence of risks, and report to the superior management.

[0024] The assessment method of the risk level can be based on one or more of the following dimensions: Probability of risk occurrence: According to the probability of risk occurrence predicted by the model, risks are classified into different levels. For example, a risk with a probability higher than a certain threshold (such as 80%) is a high risk, and a risk with a probability lower than a certain threshold (such as 20%) is a low risk; Severity of risk impact: Evaluate the severity of the consequences that may occur after the risk occurs, such as casualties, equipment damage, production interruption, etc. Risks are classified into different levels according to the severity of the consequences; Comprehensive assessment: Combine the probability of risk occurrence and the severity of risk impact for comprehensive assessment. For example, using the risk matrix method, risks are classified into three levels or more levels, namely high, medium, and low.

[0025] In this embodiment, specifically: The features extracted by the data fusion processing module include motion features, image features, sound features, and inertial features; Motion features include joint angles, speeds, and acceleration features, which are used to reflect the standardization and stability of the operator's movements. They also include the patterns and frequencies of motion sequences, which are used to identify specific operation behaviors or habits; Image features include edges, textures, and colors, which are used to identify obstacles and equipment states in the operation environment. They also include the positions and postures of target objects, specifically the positions of components of construction machinery and the body postures of operators; Sound features include frequencies, energies, and timbres, which are used to identify mechanical failures and abnormal sounds. They also include the patterns and frequencies of sound events, which are used to judge the operating state of machinery or the behaviors of operators; Inertial features include accelerations, angular velocities, and attitude angles, which are used to reflect the motion states of construction machinery and operators. They also include motion trajectories and patterns, which are used to analyze the motion smoothness of machinery and the motion coordination of operators.

[0026] In this embodiment, specifically: It further includes a virtual simulation scenario construction module, which is used to construct a virtual operation environment for construction machinery, simulate real operation scenarios and risk factors, and provide an immersive training experience for operators.

[0027] In this embodiment, specifically: It further includes a personalized training customization module, which is used to formulate a personalized training plan according to the actual situation and training needs of the operator.

[0028] In this embodiment, specifically: It further includes a data storage and management module and an interaction module, which are used to store and manage the collected multi-modal data, training records, and evaluation results. The interaction module is used to implement the human-machine interaction function.

[0029] In this embodiment, specifically: there is also a training effect evaluation module, which is used to evaluate the training effect of the operator by using an evaluation algorithm; The evaluation algorithm includes the following steps: Collect data: Collect the training data of the operator from the training system, including motion data, image data (such as operation scenarios, equipment status), sound data (such as operation prompt sound feedback, abnormal sound records), inertial data (such as operation stability indicators), and interaction records during the training process (such as the operator's feedback, guidance records); Preprocess data: Clean, standardize, and normalize the collected data, remove noise and outliers, and ensure the consistency and availability of the data; Extract features: Extract features related to the training effect from the preprocessed data, such as motion accuracy, operation speed, equipment status recognition rate, sound feedback response time, operation stability indicators; Select features: Use the recursive feature elimination algorithm to screen out the features most relevant to the training effect evaluation; Select a model: Build a machine model according to the evaluation requirements and data characteristics; Train the model: Use historical training data (data containing known training effect labels) to train the model and optimize the model parameters; Input real-time data: Input the real-time data collected during the current training process into the trained evaluation model; Calculate evaluation indicators: According to the model output, calculate various evaluation indicators, including skill improvement degree, risk awareness improvement rate, operation stability improvement situation; Comprehensive evaluation: Weight and comprehensively combine various evaluation indicators to obtain the overall training effect evaluation result of the operator; Analyze the results: Conduct in-depth analysis of the evaluation results to identify the advantages and disadvantages of the operator during the training process; Provide feedback: According to the analysis results, provide personalized feedback and suggestions for the operator to help them improve the training method and training effect; Adjust the training plan: According to the evaluation results, adjust the training plan in a timely manner to ensure the achievement of the training objectives; A construction machinery impact risk training method for multi-modal data acquisition includes the following steps: Step 1: Build a virtual construction machinery operation environment to simulate real operation scenarios and risk factors, and provide an immersive training experience for the operator; Step 2: Use an optical motion capture system to capture the motion and posture information of the operator, and perform three-dimensional reconstruction and analysis; Step 3: Use a depth camera to collect the depth image of the operating environment, and utilize computer vision technology for scene understanding and obstacle detection to real-time identify obstacles and potential danger areas in the operating environment; Step 4: Collect the sound data during the operation of the construction machinery, and utilize audio processing technology for sound recognition and anomaly detection; Step 5: Collect the acceleration and angular velocity of the construction machinery and the operator through the IMU device, and analyze the motion state and stability; Step 6: Adopt a multi-modal fusion algorithm to fuse and process the collected multi-modal data, extract feature information, and achieve the organic fusion of multi-modal data; Step 7: Adopt a risk assessment algorithm to analyze and process the fused data, identify potential risk factors, and evaluate the risk level.

[0030] Working principle or structural principle: When in use, construct a virtual operating environment of the construction machinery through the virtual simulation scene construction module, simulate the real operation scene and risk factors, provide an immersive training experience for the operator, capture the motion and posture information of the operator through the optical motion capture system, and conduct three-dimensional reconstruction and analysis. Use a depth camera to collect the depth image of the operating environment, utilize computer vision technology for scene understanding and obstacle detection to real-time identify obstacles and potential danger areas in the operating environment, collect the sound data during the operation of the construction machinery, utilize audio processing technology for sound recognition and anomaly detection, collect the acceleration and angular velocity of the construction machinery and the operator through the IMU device, analyze the motion state and stability, adopt a multi-modal fusion algorithm to fuse and process the collected multi-modal data, extract feature information, and achieve the organic fusion of multi-modal data. Adopt a risk assessment algorithm to analyze and process the fused data, identify potential risk factors, and evaluate the risk level. Finally, evaluate the training effect of the operator through the training effect evaluation module.

[0031] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An engineering machinery impact risk training system for multi-modal data acquisition, characterized in that, It includes a multi-modal data acquisition module, a data fusion processing module, and a risk identification and assessment module; The multi-modal data acquisition module is used to collect data in the operating environment of construction machinery in real time, including motion data, image data, sound data, and inertial data; The multi-modal data acquisition module includes a motion capture and analysis module, a depth image understanding module, a sound recognition and detection module, and an inertial data status analysis module; The motion capture and analysis module is used to capture the motion and posture information of the operator using an optical motion capture system, and perform three-dimensional reconstruction and analysis; The depth image understanding module is used to collect the depth image of the operating environment using a depth camera, and use computer vision technology to perform scene understanding and obstacle detection, and real-time identify obstacles and potential dangerous areas in the operating environment; The sound recognition and detection module is used to collect sound data during the operation of construction machinery, and use audio processing technology to perform sound recognition and anomaly detection, and timely discover mechanical failures or abnormal sounds; The inertial data status analysis module is used to collect the acceleration and angular velocity of construction machinery and the operator through an IMU device, analyze the motion state and stability, evaluate the motion smoothness of construction machinery and the motion coordination of the operator, and identify potential motion risks; The data fusion processing module is used to fuse and process the collected multi-modal data using a multi-modal fusion algorithm, extract feature information, and achieve the organic fusion of multi-modal data; The risk identification and assessment module is used to analyze and process the fused data using a risk assessment algorithm, identify potential risk factors, and evaluate the risk level.

2. The construction machinery impact risk training system for multi-modal data acquisition according to claim 1, characterized in that The multi-modal fusion algorithm includes the following steps: Data preprocessing: Perform standardization processing on the collected motion data, image data, sound data, and inertial data to eliminate the influence of dimensions, and perform interpolation or elimination processing on missing or abnormal data; Feature extraction: Extract features from each modality data respectively, and use the principal component analysis method to reduce the dimension of high-dimensional features; Adaptive weight calculation: Calculate the adaptive weight based on mutual information according to the importance of each modality data in the current task; Data fusion: Use the calculated adaptive weight to perform weighted fusion on the features of each modality. The fused feature vector will contain the information of all modalities, and the contribution of each modality is adjusted according to its importance; Post-processing and output: Further process the fused feature vector, perform normalization and denoising processing, and output the processed feature vector for use by the subsequent risk identification and assessment module.

3. The construction machinery impact risk training system for multimodal data acquisition according to claim 2, wherein The steps of calculating the adaptive weight include: Calculate mutual information: For each modality data, calculate the mutual information between it and the task objective. Mutual information is used to measure the correlation between modality data and the task objective. The greater the mutual information, the stronger the correlation between the modality data and the task objective, and the greater the contribution to the task; Normalize mutual information: Normalize the mutual information of all modalities so that their sum is equal to 1. The normalized mutual information is used as the weight of each modality.

4. The construction machinery impact risk training system for multi-modal data collection according to claim 1, characterized in that, The features extracted by the data fusion processing module include motion features, image features, sound features, and inertial features; The described motion features include joint angles, speed, and acceleration features, which are used to reflect the operation standardization and stability of the operator, and also include the pattern and frequency of the motion sequence, which are used to identify specific operation behaviors or habits; The described image features include edges, textures, and colors, which are used to identify obstacles and equipment status in the operation environment, and also include the position and pose of the target object, specifically the positions of components of construction machinery and the body poses of the operator; The described sound features include frequency, energy, and timbre, which are used to identify mechanical failures and abnormal sounds, and also include the pattern and frequency of sound events, which are used to judge the operating status of the machinery or the behavior of the operator; The described inertial features include acceleration, angular velocity, and attitude angle, which are used to reflect the motion states of construction machinery and the operator, and also include the motion trajectory and pattern, which are used to analyze the motion smoothness of the machinery and the motion coordination of the operator.

5. The construction machinery impact risk training system for multi-modal data acquisition according to claim 1, characterized in that, The described risk assessment algorithm includes the following steps: Feature selection: Select motion, image, sound, and inertial features relevant to risk assessment; Model training: Construct a risk assessment model, use historical data as the training set, and train the model. During the training process, optimize the model parameters through cross-validation and grid search methods; Risk identification: Input the real-time collected and fused data into the trained model to obtain the risk prediction results, and the prediction results are the probability of risk occurrence and the risk category; Risk level assessment: According to the risk prediction results, combined with the preset risk level classification criteria, assess the risk level; Output and feedback: Output the risk level assessment results to provide real-time risk warnings for the operator.

6. The construction machinery impact risk training system for multimodal data acquisition according to claim 5, characterized in that, The classification of the described risk levels includes: Low risk: The probability of risk occurrence is relatively low, and the consequences caused after occurrence are relatively light. The operator can continue normal operation but needs to pay attention to remaining vigilant; Medium risk: The probability of risk occurrence is moderate, and the consequences caused after occurrence are relatively serious. The operator needs to take certain preventive measures; High risk: The probability of risk occurrence is relatively high, and the consequences caused after occurrence are very serious. The operator should immediately stop the operation, take emergency measures to avoid the occurrence of risks, and report to the superior management personnel.

7. A construction machinery impact risk training system for multi-modal data acquisition according to claim 1, characterized in that, It also includes a virtual simulation scenario construction module, which is used to construct a virtual operation environment for construction machinery, simulate real operation scenarios and risk factors, and provide an immersive training experience for the operator.

8. A construction machinery impact risk training system for multimodal data acquisition according to claim 1, characterized in that, It also includes a personalized training customization module, which is used to formulate a personalized training plan according to the actual situation and training needs of the operator.

9. The construction machinery impact risk training system for multi-modal data collection according to claim 1, characterized in that, It also includes a data storage management module and an interaction module, which are used to store and manage the collected multi-modal data, training records, and evaluation results, and the interaction module is used to implement the human-computer interaction function.

10. A training method for construction machinery impact risk with multi-modal data acquisition, which is applied to a training system for construction machinery impact risk with multi-modal data acquisition according to any one of claims 1-9, characterized in that, It includes the following steps: Step 1: Construct a virtual operation environment for construction machinery, simulate real operation scenarios and risk factors, and provide an immersive training experience for the operator; Step 2: Use an optical motion capture system to capture the motion and pose information of the operator, and perform three-dimensional reconstruction and analysis; Step 3: Use a depth camera to collect the depth image of the operating environment, and use computer vision technology for scene understanding and obstacle detection to real-time identify obstacles and potential dangerous areas in the operating environment; Step 4: Collect the sound data during the operation of construction machinery, and use audio processing technology for sound recognition and anomaly detection; Step 5: Collect the acceleration and angular velocity of construction machinery and operators through an IMU device, and analyze the motion state and stability; Step 6: Adopt a multi-modal fusion algorithm to fuse and process the collected multi-modal data, extract feature information, and achieve the organic fusion of multi-modal data; Step 7: Adopt a risk assessment algorithm to analyze and process the fused data, identify potential risk factors, and evaluate the risk level.

Images