A method, electronic equipment, and storage medium for identifying dangerous postures of personnel in boom trucks based on multi-source sensor fusion.

Abstract

This invention discloses a method, electronic device, and storage medium for identifying dangerous postures of personnel in a bucket truck based on multi-source sensor fusion. The method includes: acquiring multimodal data, including RGB video, depth image, IMU signal collected during personnel's work on the bucket truck, and human posture key point sequences extracted from the RGB video; preprocessing and feature extraction of the multimodal data to obtain multimodal features, the preprocessing including temporal alignment, spatial alignment, and data denoising; using the RGB video features as the primary modality, performing Cross-Attention interaction with the features of the depth image, IMU signal, and human posture key point sequence to enhance the RGB video features; calculating the adaptive weight of each modality feature and fusing the features of the four modalities; and identifying the posture category of the personnel on the bucket truck based on the fused features. This invention can achieve high-precision identification and real-time early warning of dangerous behaviors of personnel in bucket trucks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power distribution network non-power-off operation safety monitoring and personnel posture intelligent identification, and particularly relates to a boom truck personnel dangerous posture identification method based on multi-source sensor fusion, an electronic device and a storage medium. BACKGROUND

[0002] In the daily maintenance of the power industry, especially in the maintenance operation of transmission lines, the operating personnel generally need to perform high-altitude operation on the conductor or the tower. At present, the climbing tool that can be used for high-altitude live maintenance operation is mainly an insulating boom truck.

[0003] The high-altitude operation of the power grid, especially the live operation based on the insulating boom truck, is one of the work with the highest technical content and the largest risk coefficient in the power operation and maintenance. The traditional safety supervision highly depends on the naked-eye patrol and supervision of safety officers, and has problems such as strong subjectivity, inability to quantify the safety distance, and reaction lag. How to intelligently identify and warn the dangerous behavior of the boom truck high-altitude operating personnel has become the focus of the academic and industrial circles.

[0004] The statements herein only provide background technology related to the present application, and do not necessarily constitute the prior art. SUMMARY

[0005] The purpose of the present application is to provide a boom truck personnel dangerous posture identification method based on multi-source sensor fusion, an electronic device and a storage medium, to realize high-precision identification and real-time warning of the dangerous behavior of the boom truck high-altitude operating personnel, so as to effectively improve the safety guarantee capability in the high-altitude operation scene of the power grid.

[0006] In order to achieve the above purpose, the present application realizes the following technical scheme:

[0007] One aspect of the present application provides a boom truck personnel dangerous posture identification method based on multi-source sensor fusion, comprising:

[0008] Obtaining multi-modal data, including RGB video, depth image, IMU signal collected during personnel work on the boom truck, and human posture key point sequence extracted from the RGB video;

[0009] Respectively pre-processing and feature extracting the multi-modal data to obtain multi-modal features, the pre-processing including time series alignment, spatial alignment and data denoising;

[0010] Taking the RGB video feature as the main mode, respectively performing Cross-Attention interaction with the features of the depth image, the IMU signal and the human posture key point sequence to enhance the RGB video feature;

[0011] Adaptive weights of each modality feature are calculated, and the features of the four modalities are fused;

[0012] Based on the fused features, the posture category of the personnel on the arm truck is recognized.

[0013] Optionally, the multi-modal data is time-aligned in the following manner:

[0014] Taking the modality data with the highest sampling frequency as the reference, for the modality data with low sampling frequency, the missing timestamp values are supplemented by time interpolation;

[0015] The four kinds of multi-modal data are synchronously sliced in the same time period by means of sliding window.

[0016] Optionally, the multi-modal data is spatially aligned in the following manner:

[0017] For each modality data, the data in different coordinate systems is converted to a unified reference system by a corresponding homogeneous transformation matrix;

[0018] The posture estimation aligns all modality data to the body reference system with the target posture key point as the origin.

[0019] Optionally, the multi-modal data is respectively subjected to feature extraction, including:

[0020] The video spatio-temporal modeling network model is used to extract features from the RGB video;

[0021] The three-dimensional convolution expansion network model is used to extract features from the depth image;

[0022] The bidirectional recurrent network model is used to extract features from the IMU signal;

[0023] The spatio-temporal graph convolution network model is used to extract features from the human posture key point sequence.

[0024] Optionally, before enhancing the features of the RGB video, it further includes:

[0025] All modalities of features are uniformly mapped to the same dimension using a fully connected layer.

[0026] Optionally, the adaptive weights of each modality feature are calculated, including:

[0027] The adaptive weights of each modality feature are calculated by a fully connected layer, and the adaptive weight values are in the range of [0, 1] by means of Sigmoid activation function.

[0028] Optionally, the features of the four modalities are fused, including:

[0029] The fused features are calculated by the following fusion formula:

[0030]

[0031] wherein, is the fused feature, and is the element-wise product, is the enhanced feature of the RGB image, is the feature of the depth image, is the feature of the IMU signal, is the feature of the human pose keypoint sequence, , is the adaptive weight tensor of the corresponding modal feature.

[0032] Optionally, the method further comprises:

[0033] flattening or pooling the fused feature along the sequence dimension;

[0034] inputting the flattened feature into a fully connected network to determine the probability of the feature mapping to each pose category, thereby identifying the pose category of the person on the arm truck, wherein the fully connected network comprises two fully connected layers, the activation function of the first layer is ReLU, and the activation function of the second layer is Softmax.

[0035] In a second aspect, the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method described above.

[0036] In a third aspect, the present application further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0037] The present application has at least the following technical effects:

[0038] In summary, the method for identifying dangerous pose of person on arm truck based on multi-source sensor fusion provided by the present application combines RGB video, depth image, IMU signal and human pose keypoint sequence, which can significantly improve the accuracy and real-time performance of dangerous behavior recognition. Through cross-modal spatio-temporal alignment processing, the spatio-temporal heterogeneous problem of multi-source vision and sensor data is solved, effective fusion and joint modeling of cross-modal features are realized, multi-modal complementation is achieved through Cross-Attention interaction method, stable recognition performance can be maintained by relying on other modalities in the case of video occlusion, IMU signal loss or noise interference, thereby effectively reducing the false positive rate and the false negative rate. Through the feature fusion method of adaptive weight, the dynamic change trend of dangerous behavior can be captured, and dynamic perception and early warning of high-altitude operation dangerous behavior can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the description will be briefly introduced as follows. Obviously, the drawings in the following description are one embodiment of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings:

[0040] Figure 1 The flowchart of the personnel dangerous posture recognition method based on multi-source sensor fusion provided by an embodiment of the present application is shown in the figure.

[0041] Figure 2 The overall architecture of the present application is shown in the figure. DETAILED DESCRIPTION

[0042] The scheme proposed by the present application will be further described in detail below in combination with the drawings and specific embodiments. The advantages and features of the present application will be more apparent according to the following description. It should be noted that the drawings are greatly simplified and all use non-precise proportions, only to facilitate and clarify the purpose of assisting in the description of the embodiments of the present application. In order to make the purpose, features and advantages of the present application more apparent and easy to understand, please refer to the drawings. It should be noted that the structure, proportion, size, etc. shown in the drawings attached to the present specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the conditions for implementing the present application, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.

[0043] In the prior art, for the bucket arm vehicle operation in the power grid overhead operation, the dangerous behavior recognition method mainly includes the following types: traditional machine learning based method, single computer vision based method, deep learning based method and multi-source sensor fusion based method. However, the existing recognition methods have the following problems: 1) the multi-source vision and sensor data have space-time heterogeneity, the sampling rate, delay and coordinate system difference are not effectively aligned, which limits the cross-modal feature joint modeling capability; 2) dangerous posture recognition is mainly dependent on single frame image or local time sequence segment, which lacks modeling of continuous evolution of action, resulting in insufficient early warning capability; 3) in complex operation environment, factors such as light change, jitter, electromagnetic interference easily cause noise or loss of video and sensor signals, and the existing method lacks robust data compensation and redundancy mechanism.

[0044] In view of the above problems existing in the prior art, the present application provides a dangerous posture recognition method for personnel on a boom truck based on multi-source sensor fusion, which provides efficient and accurate dangerous posture detection by combining RGB video, depth image, inertial measurement unit (IMU) signal and posture key point sequence.

[0045] In combination Figure 1 , Figure 2 As shown in the figure, the embodiment provides a dangerous posture recognition method for personnel on a boom truck based on multi-source sensor fusion, which comprises the following steps:

[0046] Step S1, acquiring multi-modal data, the multi-modal data comprising: RGB video, depth image, IMU signal collected during personnel working on the boom truck, and human posture key point sequence extracted from the RGB video.

[0047] Generally, a camera and a depth sensor are installed on the fence of the boom truck, and the high-altitude worker needs to wear an IMU (inertial measurement unit) device on the key parts of the body before work. During the high-altitude work, the color video of the human body and the environment (RGB video) is captured by the camera, usually 30 frames of image data are collected per second, and the image data is stored in RGB format. At the same time, the depth sensor (such as Kinect, etc.) generates a depth image by measuring the distance with an infrared light beam to provide three-dimensional space information, each pixel in the depth image represents the depth value of a certain point in the scene, the unit is meter, and the depth image is collected synchronously with the RGB video. The IMU device measures acceleration and angular velocity in real time through the accelerometer and gyroscope, and the sampling frequency is usually 100 Hz or higher, capturing the acceleration and angular velocity information of X, Y and Z axes.

[0048] In order to ensure the original accuracy of the skeleton data, the two-dimensional or three-dimensional coordinates of each joint of the human body are extracted from the RGB video before subsequent processing, and the human posture key point sequence is obtained. The OpenPose algorithm can be used for extraction, 30 frames of data are collected per second, so as to provide the skeleton model information of the human body.

[0049] The above four kinds of multi-modal data can provide comprehensive support for subsequent dangerous posture recognition, and improve the recognition accuracy and system robustness.

[0050] Step S2, pre-processing and feature extraction are performed on the multi-modal data respectively to obtain multi-modal features, the pre-processing including time sequence alignment, space alignment and data denoising.

[0051] It can be understood that the data collected by the multi-source sensor often has significant differences in time and space, and such data heterogeneity brings great challenges to subsequent feature modeling and system identification. Therefore, in order to solve the heterogeneity problem between multi-source data, the embodiment performs time and space alignment operation when pre-processing multi-source data, as follows:

[0052] First, the following method is used for time series alignment of multi-modal data: taking the modality data with the highest sampling frequency as the reference, for the modality data with low sampling frequency, the missing timestamp value is supplemented by time interpolation; the four kinds of multi-modal data are synchronously sliced in the same time period by sliding window method.

[0053] Assume that the sampling points of the modality are , where is not necessarily continuous, and the goal of time interpolation is to predict the value at the new time point according to the existing data points. Linear interpolation can be used, which predicts the intermediate time point by the change rate of the previous and next data points.

[0054] For time series data, the four kinds of modality data are sliced by sliding window method to ensure that the data in each window can be processed as a time series. Set the window size as w , and the step size as s , then the data set D of each time period can be represented as: , where is the start time of the window, is all data in the i th window. The four kinds of modality data contained in each window will be synchronously sliced in the same time period for subsequent feature extraction and fusion. By sliding window slicing, long time series data can be effectively segmented, so that the data in each time period can be processed independently, thereby improving the recognition accuracy and system efficiency.

[0055] Then, the following method is used for spatial alignment of multi-modal data: for each modality data, the data in different coordinate systems is converted to a unified reference system by the corresponding homogeneous transformation matrix; all modality data is aligned to the body reference system with the target attitude key point as the origin by attitude estimation.

[0056] Assume that the data point of a modality in the original coordinate system is , and the new coordinate point after conversion to the target coordinate system can be converted by the following homogeneous transformation matrix : , where TThe homogeneous transformation matrix is 4x4, which contains translation, rotation, scaling and other transformation information, and can map data from the original coordinate system to the unified reference system. The specific value of the homogeneous transformation matrix can be calculated by the calibration process or the known rotation and translation relationship between coordinate systems.

[0057] The pose estimation aligns all data to the body reference system with the specific pose key point (e.g. pelvis) as the origin, ensuring spatial consistency between different modalities. Thus, it helps to make accurate spatial comparison and feature fusion between different modalities. Although the pelvis is usually used as the reference point, in specific applications, key points such as shoulders, spine or head can also be selected.

[0058] In addition, the data preprocessing also includes denoising the data. It can be understood that in the sensor acquisition process, the data is often affected by noise, interference and other problems, especially video images may be affected by occlusion, blur and other problems, IMU signals may be affected by vibration or other external factors, and depth images may also produce random noise and holes. In order to improve the quality of the data, the above three kinds of data are denoised and enhanced.

[0059] Specifically, for the RGB video, a blur denoising method can be used to remove noise and details in the image and smooth the video data. Gaussian filtering or mean filtering is usually used to blur the image.

[0060] For depth image denoising, a bilateral filtering and interpolation repair method can be used to deal with the random noise, holes and edge burrs that may appear in the depth image. This method can remove noise while preserving object edge features and filling in missing areas, thereby improving the integrity and reliability of depth data.

[0061] For IMU signal denoising, low-pass filtering and signal normalization methods can be used to process the IMU signal. Low-pass filtering can effectively remove high-frequency noise and retain the effective information in the IMU signal.

[0062] After preprocessing the multi-modal data, feature extraction is performed. In this embodiment, different deep learning networks are used to independently extract effective features of multi-modal data to fully utilize the advantages of each model on different data types.

[0063] Specifically, RGB video is an important modality in dangerous posture recognition, and its spatio-temporal characteristics contain dynamic information of human motion and scene changes. In order to efficiently extract the spatio-temporal features in the RGB video, the video spatio-temporal modeling network model (such as Video Swin Transformer or TimeSformer model) is adopted to extract the RGB video features. Taking the Video Swin Transformer model as an example, the input data format is (batch_size, time_steps, height, width, channels), and the input data is the video frame sequence of the RGB video . In the feature extraction process, the video frame is divided into local image windows after being input into the Video Swin Transformer model, the spatial features are extracted through local window self-attention, and the inter-frame time sequence relationship is modeled through cross-window self-attention, and finally the output data format is (batch_size, time_steps, 768), that is , as the RGB video feature output.

[0064] The depth image provides complementary spatial geometric information to the RGB image, which can effectively assist in recognizing dangerous postures. In order to extract the spatial geometric features of the depth image, the three-dimensional convolution expansion network model (such as I3D or Depth-Transformer model) is adopted to extract the features of the depth image. Taking the I3D model as an example, the input data format is (batch_size, time_steps, height, width, channels), and the input data is the video frame sequence of the depth image . In the feature extraction process, the video frame is input into the I3D model, and the spatial geometric information is modeled through three-dimensional convolution. The I3D network can capture the dynamic changes in the depth image and extract the spatial structure features related to the human posture, and finally the output data format is (batch_size, time_steps, 1024), that is , as the depth image feature output.

[0065] The IMU signal contains time series data such as acceleration and angular velocity of human motion, and can provide dynamic information of motion for posture recognition. In order to extract the time series features in the IMU signal, the embodiment adopts a bidirectional recurrent network model (such as BiLSTM or TCN) to extract features from the IMU signal. Taking the BiLSTM model as an example, the input data format is (batch_size, time_steps, num_features), the input data is the IMU signal sequence (acceleration, angular velocity, etc.), and in the feature extraction process, after the IMU signal is input into the BiLSTM network, the BiLSTM can capture the dynamic change rule according to the information before and after the time series, and extract the time series features related to the dangerous posture. Finally, the output data format is (batch_size, time_steps, 64), that is , as the IMU signal output.

[0066] The human posture key point sequence provides two-dimensional coordinate information of each joint position of the human body, and is an important structural feature in dangerous posture recognition. In order to effectively capture the spatial structure features of the human skeleton, the embodiment adopts a spatio-temporal graph convolution network model (such as the ST-GCN model) to extract features from the human posture key point sequence. Taking the ST-GCN model as an example, the input data format is (batch_size, time_steps, num_keypoints * 2), and the input data is the human posture key point sequence. In the feature extraction process, the ST-GCN extracts the structural features of the human skeleton and models the spatio-temporal relationship between the key points. Finally, the output data format is (batch_size, time_steps, 256), that is , as the key point feature output, which is expressed by the formula as follows: , wherein is the feature obtained by spatial convolution (spatial feature at time t), and are weight matrices of time convolution operation, used to process features in time dimension, is the final spatio-temporal feature, combining time and space information.

[0067] In step S3, the RGB video feature is taken as the main modality, and is respectively interacted with the features of the depth image, the IMU signal and the human posture key point sequence through Cross-Attention to enhance the RGB video feature.

[0068] It should be noted that, due to the different feature dimensions extracted by different modal data, the feature dimensions need to be aligned first to ensure that the subsequent fusion operation can be carried out under the same dimension. Specifically, all modal features can be uniformly mapped to the same dimension (for example, 728 dimensions) using a fully connected layer (MLP), as shown in the following formula:

[0069]

[0070]

[0071]

[0072]

[0073] wherein, , , respectively represent the RGB video features, depth image features, IMU signal features and human pose key point sequence features mapped to the same dimension.

[0074] The Cross-Attention mechanism enhances the interaction information between modalities, ensuring that the data of different modalities can be efficiently fused in the feature dimension. To reduce the complexity of multi-modal interaction, the embodiment adopts a main modal driven interaction mechanism. Since the RGB video contains the most rich spatio-temporal information among the four modalities, and is often used as the time reference for multi-modal acquisition, the RGB video features are selected as the main modal, and are respectively interacted with the features of the depth image, the IMU signal and the human pose key point sequence to enhance the RGB video features.

[0075] Specifically, the following formula can be used to achieve this:

[0076]

[0077]

[0078]

[0079] wherein, is a learnable projection matrix, respectively used to generate query vector , key vector and value vector , is a vector dimension, used to normalize the dot product result. represents the interaction result of the RGB video and the modal , and the modal includes the depth image , the IMU signal and human pose keypoint sequence , By fusing the interaction results (such as splicing or weighted sum) of the remaining three modalities, the final enhanced RGB feature is obtained. It should be noted that the four modal features involved in the formula here refer to the aforementioned features mapped to the same dimension.

[0080] Step S4, calculate the adaptive weight of each modal feature, and fuse the features of the four modalities.

[0081] The embodiment can make full use of the respective advantages of the four modal features (enhanced RGB video, depth image, IMU signal and pose keypoint) by fusing them: the RGB video provides dynamic information of human motion and scene changes, the depth image supplements spatial geometric features, the IMU signal provides acceleration and angular velocity data about motion, and the pose keypoint accurately describes the positions of human joints. A traditional single model cannot effectively combine these information.

[0082] The embodiment first calculates the adaptive weight of each modal feature, thereby dynamically adjusting the importance of each modal feature in the fusion process, so as to flexibly control the contribution of different modalities to the final fused feature, thereby significantly improving the accuracy and stability of recognition. Specifically, the adaptive weight of each modal feature is calculated through a fully connected layer, and the adaptive weight value is in the range of [0, 1] through a Sigmoid activation function, and the specific formula is as follows:

[0083]

[0084]

[0085]

[0086]

[0087] wherein, , are the adaptive weight tensors of the RGB video, the depth image, the IMU signal and the human pose keypoint sequence respectively, and the output dimensions thereof are consistent with the corresponding modal features to ensure that point-by-point weighting operation can be performed at the element level.

[0088] Then the four features are fused through the adaptive weight, and the fusion formula is:

[0089]

[0090] wherein, is the element-wise product, The fused feature has a dimension of (batch_size, time_steps, 728), , is an adaptive weight tensor corresponding to the modal feature.

[0091] In step S5, the posture category of the person on the arm truck is identified based on the fused feature.

[0092] In this embodiment, the fused feature is classified, and finally the posture category of the person is output. The posture category can include three categories of “normal”, “danger warning” and “failure”. The danger warning indicates that the posture has a certain danger, but has not yet failed. The failure indicates that the posture has entered a dangerous area and needs to be handled urgently. This classification method directly corresponds to the actual application of the dangerous posture recognition system, which can judge whether the posture is abnormal or dangerous in real time, and provides a warning function for the system to improve safety.

[0093] In the identification, the fused feature is first flattened or pooled according to the sequence dimension. Specifically, the time dimension is first pooled to compress the time sequence information to obtain (batch_size, 728), and then the dimension is reduced to (batch_size, 128) through full connection mapping as the input for subsequent classification. Then, the flattened feature is input into a full connection network to determine the probability of feature mapping to each posture category, so as to identify the posture category of the person on the arm truck. The full connection network includes two full connection layers. The activation function of the first layer is ReLU, and the activation function of the second layer is Softmax.

[0094] Specifically, the flattened feature is input into the first full connection layer FC1. The first layer has 256 neurons, and the activation function adopts ReLU, which can increase the nonlinear ability of the model. The output of the second full connection layer FC2 is mapped to the number of categories C (such as normal, dangerous, failure, etc.), and the activation function adopts Softmax to convert the result into a category probability distribution.

[0095]

[0096] ​In summary, the personnel dangerous posture recognition method based on multi-source sensor fusion provided by the application can significantly improve the accuracy and real-time performance of dangerous behavior recognition by combining RGB video, depth image, IMU signal and human body posture key point sequence, solve the spatio-temporal heterogeneity problem of multi-source vision and sensor data through cross-modal spatio-temporal alignment processing, realize effective fusion and joint modeling of cross-modal features, realize multi-modal complementation through the Cross-Attention interaction method, and can maintain stable recognition performance in the case of video occlusion, IMU signal loss or noise interference, thereby effectively reducing the false positive rate and the false negative rate, and the adaptive weight feature fusion method can capture the dynamic change trend of dangerous behavior, and realize dynamic perception and early warning of high-altitude operation dangerous behavior.

[0097] The training and optimization of the model in the embodiment are described below.

[0098] The data source of the data in the application relates to actual multi-source sensor actual capture data and public multi-source sensor related dangerous recognition data set. The data content includes: RGB video collected by an RGB camera (frame rate 30fps), depth image collected by a Kinect depth sensor (frame rate 30fps, synchronized with the RGB camera), IMU signal (acceleration / angular velocity) collected by a wearable IMU device (sampling frequency 100Hz), human body posture key point sequence (frame rate 30fps) extracted from the RGB video. The data label is divided into “normal”, “abnormal warning” (instability, about to fall, etc.), “fault” (abnormal illegal posture) three categories, which are labeled by experts combined with operation records. The sample distribution satisfies: about 50,000 segments; the proportion of “abnormal warning” and “fault” is about 15-25%; at least 5000 segments of each type to reduce extreme imbalance.

[0099] The loss function adopts a multi-class cross-entropy loss, as follows:

[0100]

[0101] Wherein, N is the total number of samples, C is the number of categories, is the true label, is the predicted probability.

[0102] The Adam optimizer is used for model training, and the initial learning rate is set to 0.001. Every 10 epochs, the learning rate is decayed to 0.95 times of the original.

[0103] The data set is divided into a training set, a validation set and a test set in a ratio of 7:1:2. The training set is used for model training, the validation set is used for performance evaluation and parameter adjustment during training, and the test set is used for final model evaluation after training. Training is stopped when the validation set loss no longer decreases, even if the training accuracy continues to improve. This method can ensure that the model does not overfit the training set.

[0104] The present application adopts a plurality of mainstream classification performance indicators for evaluation, including accuracy (Accuracy), precision (Precision), recall (Recall), and F1 score (F1-score). Specifically as follows:

[0105]

[0106]

[0107] Recall (Recall) is used to measure how many real samples of each category are successfully identified by the model, reflecting the model's ability to identify abnormal and minority categories. The formula is as follows:

[0108]

[0109] F1 score (F1-score) is used to comprehensively reflect the accuracy and recall ability of the model, and is suitable for class-imbalance data sets. The formula is as follows:

[0110]

[0111] The method of the present application is compared with other methods, and the comparison results are shown in Table 1, wherein RGB-only is a method of using only the Video Swin Transformer model to recognize dangerous postures based on RGB video, Depth-only represents a method of using only the I3D model to recognize dangerous postures based on depth image, IMU-only represents a method of using only the BiLSTM model to recognize dangerous postures based on IMU signal, Pose-only represents a method of using only the ST-GCN model to recognize dangerous postures based on human posture key point sequence, Early-Concat+MLP represents a method of aligning four kinds of modal data, then averaging, and then classifying through an mlp layer, and Late-AvgLogits represents a method of equally averaging the logits of four single-mode classifiers.

[0112] Table 1

[0113]

[0114] From the above table, the accuracy, precision, recall and F1 value of the method of the present application are significantly better than those of the prior art.

[0115] In some other embodiments, the embodiments of the present application also provide an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method as described above.

[0116] In some other embodiments, the embodiments of the present application also provide a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described above. All algorithms in the method of the present embodiment can be completed in the software of the electronic device or the readable storage medium, and are suitable for different application environments.

[0117] In summary, compared with the prior art, the present application has the following technical effects:

[0118] Traditional multi-modal fusion methods often face the problems of time alignment and spatial reference unification, especially in the multi-source data acquisition process, the data sampling rate, coordinate system difference and time synchronization of different modalities lead to difficulty in fusion and low recognition accuracy. The present application proposes a time alignment method based on interpolation and sliding window, and combines the homogeneous transformation matrix and the skeleton key point as an anchor point to realize spatial normalization, so as to ensure that the RGB, depth, IMU and attitude data are processed under a unified space-time reference, which has a significant improvement in accuracy and robustness compared with the traditional alignment method.

[0119] Existing dangerous posture recognition often relies on single-time-point state detection, which is difficult to capture the dynamic evolution process of dangerous behavior. The present application introduces Video Swin Transformer, BiLSTM and ST-GCN and other time series modeling methods to realize continuous modeling of multi-modal sequences, combines sliding window and dynamic weight mechanism to perceive the evolution trend of dangerous actions, and sets a "danger warning" intermediate state at the output end, so as to realize forward-looking identification and warning before the danger occurs completely.

[0120] Existing dangerous posture recognition systems often have some missing or disturbed modal data (such as video occlusion, IMU signal loss or noise interference) when processing multi-modal data. The present application takes RGB video features as the main modality, introduces depth image, IMU signal and attitude key point features through Cross-Attention to enhance and correct the RGB features, and combines an adaptive weight mechanism to dynamically adjust the contribution of each modality, so that the system can maintain stable recognition performance even in the case of video occlusion, significantly improving robustness and reliability.

[0121] It is to be appreciated that the term "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0122] While the application has been described in detail by reference to preferred embodiments thereof, it is to be understood that the detailed description is not intended to limit the application to the embodiments described herein. Many modifications and variations of the present application will be apparent to those of ordinary skill in the art upon reading this description. It is therefore contemplated to be within the scope of the present application to set forth and claim such modifications and variations of the application that come within the scope of the appended claims.

Claims

1. A method for recognizing dangerous postures of personnel on a boom truck based on multi-source sensor fusion, characterized in that, The method comprises the following steps: acquiring multi-modal data, including: RGB video, depth image, IMU signal collected during personnel work on the arm truck, and human posture key point sequence extracted from the RGB video; respectively pre-processing and extracting features of the multi-modal data to obtain multi-modal features, the pre-processing including time alignment, spatial alignment and data denoising; taking the RGB video features as the main mode, respectively performing Cross-Attention interaction with the features of the depth image, the IMU signal and the human posture key point sequence to enhance the RGB video features; calculating adaptive weights of each modal feature and fusing the features of the four modes; identifying the posture category of the personnel on the arm truck based on the fused features; wherein the fusing the features of the four modes comprises: calculating the fused features by using the following fusion formula: ， wherein, is the fused feature, is the element-wise product, is the enhanced RGB image feature, is the depth image feature, is the IMU signal feature, is the human pose keypoint sequence feature, is the adaptive weight tensor for the corresponding modal features; the identifying the posture category of the personnel on the arm truck based on the fused features comprises: flattening or pooling the fused features in sequence dimension; inputting the flattened features into a fully connected network to determine the probability of feature mapping to each posture category, thereby identifying the posture category of the personnel on the arm truck, wherein the fully connected network comprises two fully connected layers, the activation function of the first layer is ReLU, and the activation function of the second layer is Softmax.

2. The multi-source sensor fusion based personnel dangerous posture recognition method for a bucket arm vehicle according to claim 1, characterized in that, The following method is used for time alignment of multi-modal data: Taking the modal data with the highest sampling frequency as the reference, for the modal data with low sampling frequency, the missing timestamp values are supplemented by time interpolation; synchronously slicing the four kinds of multi-modal data in the same time period by sliding window.

3. The multi-source sensor fusion based personnel dangerous posture recognition method for a bucket arm vehicle according to claim 1, characterized in that, The following method is used for spatial alignment of multi-modal data: For each modal data, the data in different coordinate systems is converted to a unified reference system through the corresponding homogeneous transformation matrix; aligning all modal data to the body reference system with the target posture key point as the origin by posture estimation.

4. The multi-source sensor fusion based personnel dangerous posture recognition method of a bucket arm vehicle according to claim 1, characterized in that, Respectively extracting features of the multi-modal data, including: using a video spatio-temporal modeling network model to extract features of the RGB video; using a three-dimensional convolution expansion network model to extract features of the depth image; using a bidirectional recurrent network model to extract features of the IMU signal; using a spatio-temporal graph convolution network model to extract features of the human posture key point sequence.

5. The multi-source sensor fusion based personnel dangerous posture recognition method of a loader arm vehicle according to claim 1, characterized in that, Before enhancing the RGB video features, it further comprises: using a fully connected layer to uniformly map the features of all modes to the same dimension.

6. The multi-source sensor fusion based personnel dangerous posture recognition method of a loader arm vehicle according to claim 1, characterized in that, The calculation of adaptive weights of each modal feature comprises: respectively calculating adaptive weights of each modal feature through a fully connected layer, and making the adaptive weight values within the range of [0, 1] through a Sigmoid activation function.

7. An electronic device, comprising: The method comprises a processor and a memory, the memory stores a computer program, and the computer program is executed by the processor to realize the method of any one of claims 1-6.

8. A readable storage medium, characterized by, The computer program stored in the readable storage medium is executed by the processor to realize the method of any one of claims 1-6.

Images