Underground dangerous behavior identification method based on multi-modal joint training network
By using a multimodal joint training network under the mine, combining RGB video and depth map data, extracting human skeleton coordinates and hand details, and fusing multimodal data, the problem of behavior recognition in dim light environments under the mine is solved, and the problem of improving the Identification accuracy and robustness enhances safety and efficiency.
Patent Information
- Application Number
- CN202410979883.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-15
AI Technical Summary
In complex underground environments and under harsh conditions such as dim light, it is difficult to accurately identify the dangerous behavior of staff. The existing single-modal behavior recognition methods are difficult to cope with complex and changeable underground operation scenarios.
Using a multimodal joint training network method, combined with a downhole fixed camera to capture RGB streaming video and depth map data, HRNet extracts human skeleton coordinates, uses GCN model to identify bone data behavior, and performs multimodal through MHP graph and GOIS graph. State data fusion, trained using EfficientNet-B7 and ResNet-18, and combined training using transfer learning and multi-tuple loss functions.
It improves the accuracy and robustness of underground hazardous behaviors, reduces the rate of misjudgment, can effectively identify workers' behaviors in harsh environments, and enhances the safety and efficiency of mine operations.
Smart Images

Figure CN118968614A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of underground mine video processing and safety monitoring, and in particular relates to an underground dangerous behavior recognition method based on a multi-modal joint training network. Background Art
[0002] Human behavior recognition has extremely high research and application value in modern intelligent systems. It is widely used in human-computer interaction, virtual reality, underground warning and other fields. With the development of society, daily life and enterprise production have put forward higher requirements for the intelligence and automation of behavior recognition related products. In order to cope with diverse application scenarios, behavior recognition technology has put forward higher requirements in terms of accuracy and robustness. Although the current advanced single-modal behavior recognition methods have achieved certain achievements, they still face huge challenges in analyzing human behavior in complex and special scenarios.
[0003] In the working environment underground in mines, the task of identifying dangerous behaviors is particularly important. Accurately identifying and monitoring the behaviors of underground workers can help to timely discover and prevent potential dangers, reduce accidents, and ensure personnel safety. In recent years, research on the identification of dangerous behaviors underground has made significant progress. For example, a study based on LSTM neural networks proposed a method to predict potential dangers by analyzing the sequence of workers' movements, which significantly improved the timeliness and accuracy of behavior identification. Another study used a deep learning model to identify the posture and behavior of underground workers, and combined it with environmental perception information to judge potential dangers.
[0004] This complex and ever-changing mining environment has problems such as insufficient light, narrow space, and multiple potential sources of danger. Single-modal behavior recognition methods often find it difficult to cope with complex and ever-changing underground operation scenarios, so more advanced multimodal joint training models are needed to improve the accuracy and robustness of recognition. Some studies have proposed a method based on multi-sensor data fusion, which combines video, depth images, and sensor data to achieve comprehensive monitoring of underground workers' behavior. Researchers have also explored methods that combine infrared thermal imaging technology with conventional video data to effectively identify worker behavior in low-light environments. In addition, some studies have combined data from wearable devices with video surveillance data, and used multimodal data fusion technology to monitor worker behavior in real time, effectively reducing the misjudgment rate.
[0005] This research on behavior recognition algorithms based on multimodal data is of great significance for the identification of dangerous behaviors underground, and can significantly improve the safety and efficiency of underground mine operations. By identifying the behavior of underground workers, potential dangers can be discovered and prevented in a timely manner, accidents can be reduced, and personnel safety can be guaranteed. The multimodal behavior recognition model can utilize the complementary features between different modal data to provide a more comprehensive and accurate solution for the identification of human behavior in complex underground environments, and at the same time have a great impact on the prevention of dangerous behaviors underground.
[0006] To this end, we propose a method for underground dangerous behavior recognition based on a multimodal joint training network to solve the above problems. Summary of the invention
[0007] The technical problem to be solved by the present invention is to provide a method for identifying dangerous underground behaviors based on a multimodal joint training network, aiming to solve the problem of difficulty in accurately identifying the behavior of workers in mines due to harsh environmental conditions such as complex environment and dim light.
[0008] The objective of the present invention is achieved through the following technical solutions:
[0009] A method for identifying dangerous behaviors underground based on a multimodal joint training network first uses a fixed underground camera to capture RGB stream video and generate corresponding depth map data, which is pushed to the back-end device. The coordinates of key points of the human body are obtained using an advanced posture estimation algorithm and saved as readable data. On the one hand, the skeleton data can be used for skeleton data behavior recognition using the GCN model, and on the other hand, the corresponding complementary data needs to be generated using the skeleton coordinate points. The construction of the MHP map is generated by the depth map assisted by the RGB map, and the GOIS map is generated by the trained joint point weights. They are trained by EfficientNet-B7 and ResNet-18 respectively, and the specific steps are as follows:
[0010] S1. Video acquisition of underground personnel behavior and generation of skeleton data: RGB videos of personnel activities are acquired through fixed cameras underground. The whole process includes video acquisition, encoding transmission, video decoding and data storage. After capturing the original underground RGB video, the HRNet algorithm is used to extract human skeleton coordinate data;
[0011] S2, acquisition of behavioral characteristics of underground workers’ skeleton data;
[0012] S3, obtaining detailed feature information of underground workers’ hands;
[0013] S4. Global missing information extraction.
[0014] Compared with the prior art, the present invention has the following beneficial effects:
[0015] The present invention solves the problem of difficulty in accurately identifying the behavior of workers in mines due to harsh environmental conditions such as complex environment and dim light. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is the overall flow chart of the present invention;
[0017] Figure 2 It is a schematic diagram of underground video transmission in the present invention;
[0018] Figure 3 Schematic diagram of the HRNet model in the present invention;
[0019] Figure 4 Schematic diagram of MAS-GCN in the present invention;
[0020] Figure 5 is a flow chart for constructing the MHP diagram in the present invention;
[0021] Figure 6 This is a graph showing the results of constructing the GOIS graph in the present invention. DETAILED DESCRIPTION
[0022] The following will describe the implementation methods of the present application in detail with the help of accompanying drawings and examples, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.
[0023] The invention provides a method for identifying underground dangerous behaviors based on a multi-modal joint training network, and its overall structure is shown in the figure Figure 1 As shown, firstly, a fixed camera downhole is used to capture RGB stream video and generate corresponding depth map data, which is pushed to the back-end device, and the coordinates of key points of the human body are obtained using an advanced posture estimation algorithm and saved as readable data. On the one hand, the skeleton data can use the GCN model to perform skeleton data behavior recognition, and on the other hand, the corresponding complementary data needs to be generated using skeleton coordinate points. The construction of the MHP graph is generated by the depth map with the assistance of the RGB graph, and the GOIS graph is generated with the assistance of the trained joint point weights, which are trained by EfficientNet-B7 and ResNet-18 respectively. Finally, the present invention also provides another method for joint training of multimodal data, in which transfer learning and multi-group loss function are used.
[0024] 1. Underground personnel behavior video acquisition and skeleton data generation
[0025] First, it is necessary to obtain RGB video of the staff's activities through a fixed camera underground. The whole process includes key steps such as video acquisition, encoding transmission, video decoding and data storage. The specific process is shown as follows: Figure 2 ;
[0026] After capturing the original underground RGB video, we use the HRNet algorithm to extract the human skeleton coordinate data. The specific process is as follows:
[0027] (1) Use the OpenCV tool library to extract the original RGB video frame by frame at a frequency of 30fps;
[0028] (2) Preprocess each frame, including image scaling and normalization, to meet the input requirements of the HRNet model. At the same time, save the obtained RGB frame images separately and annotate their corresponding behavior categories as the RGB data source to be trained. All RGB videos pushed to the backend device are divided into training sets and test sets in a ratio of 7:3. The depth map data is processed in the same way.
[0029] (3) Then use the PyTorch framework to load the pre-trained HRNet model, input the pre-processed image into the HRNet model, and obtain the heatmap representation of the key points of the human body;
[0030] (4) Obtain the specific coordinates of each key point by parsing the generated heat map, and then post-process the obtained key points, that is, convert them back to the coordinate system of the original image;
[0031] (5) Finally, we save the coordinates of the human bone joints output by the HRNet model as .skeleton files, and package all .skeleton files into .pth files for subsequent underground human skeleton data behavior recognition tasks. After obtaining the .pth file, the data is divided into training set and test set according to the above 2. standards, and the training set includes as many dangerous behavior categories of underground personnel as possible.
[0032] The schematic diagram of the HRNet model for human posture estimation is as follows Figure 3 shown.
[0033] 2. Obtaining behavioral characteristics of underground workers’ skeleton data
[0034] Due to the dim and uneven light in the underground environment, the RGB video captured by the camera inevitably has different degrees of distortion, which will cause the spatial and temporal offset of the human joint coordinates output by HRNet. In addition, since the GCN model used for skeletal data behavior recognition often has the problem of excessive spatial and temporal smoothing, the final skeletal data behavior recognition effect is poor. In order to effectively solve this problem, the present invention proposes a multi-level spatial aggregation graph convolution (MAS-GCN) network to reduce spatial and temporal excessive smoothing using spatial and temporal enhancement and non-shared adjacency matrix.
[0035] The network structure of MAS-GCN is as follows Figure 4As shown in Figure 2, MAS-GCN performs spatial aggregation and temporal aggregation on the adjacency matrix of the spatial graph convolution, and calculates its unique non-shared graph representation for each input bone sequence. The temporal enhancement operation and the spatial enhancement operation are respectively and
[0036]
[0037] Among them, Reshape is the reshaping operation, C mean For integer operations, S pool and T pool They are spatial pooling operations and temporal pooling operations respectively. The calculation of the entire non-shared graph representation is divided into four branches. The top branch calculates the unique K-hop spatial aggregation of each graph, so that the model can capture the relationship between nodes under different synchronization numbers during the training process, thereby improving the model's ability to represent complex relationships between nodes. The calculation result of the K-hop spatial aggregation is A K :
[0038]
[0039] The two middle branches have similar structures, and their purpose is to obtain higher-dimensional features and perform temporal feature enhancement. First, the input is embedded into a high dimension through 1*1 convolution, then temporal aggregation is performed, and then the dimensions are converted, multiplied, and normalized before output.
[0040]
[0041] The bottom A ske It is the inherent physical connection of the human body, including the connection of the joints themselves. After parameterizing and initializing it, it is aggregated in space to obtain A p :
[0042] A p =S agg (para(A ske ))(4)
[0043] Then add the result to the result after intermediate time aggregation and perform spatial aggregation again to obtain f ST :
[0044]
[0045] Finally, we get the output f through residual connection and normalization operations. out :
[0046] f out =RES[(Relu(BN(f ST ))),fST ](6)
[0047] Among them, K represents the number of spatial layers that need to be aggregated during spatial convolution. In this paper, it is set to 3 following ST-GCN. agg represents spatial aggregation, para represents parameterization, T agg represents time aggregation, W a , W b With W c represents the corresponding weight matrix, A ske It represents the inherent connection of the human body.
[0048] The MAS-GCN proposed in the present invention starts from two different angles: the non-sharing of the adjacency matrix and the anti-smoothing of spatiotemporal features, which alleviates the defects of the GCN model to a certain extent, and further improves the performance and generalization of the model for skeletal data behavior recognition in harsh underground environments. We establish a strong connection between the data-driven features of the input skeletal sequence and the inherent graph representation of the human body; then perform feature aggregation in the spatial and temporal dimensions to alleviate the problem of feature smoothing. The MAS-GCN we proposed does not represent the inherent connection of the human body itself separately, but parameterizes it as the basic matrix for the construction of our new matrix, that is, integrates it into the non-shared matrix. For each skeleton subsequence, we calculate the adjacency matrix separately according to the relationship between it and the inherent connection of the human body, and perform feature aggregation in time and space accordingly, striving to highlight the features and thus mitigate the negative impact of feature smoothing and coordinate point drift.
[0049] 3. Acquisition of detailed feature information of underground workers’ hands
[0050] In underground human-object interaction (human and operation object), changes in the type of operation object will lead to changes in the action category, such as smoking and drinking water, which may bring completely different results. This is impossible to judge with pure skeleton sequences, because the skeleton is extremely similar throughout the action. At the same time, most human-object interaction actions occur mainly in the human hand joints (left hand and right hand), and the current mainstream posture estimation algorithm is difficult to obtain the extremely subtle joint point representation of both hands. In addition, the uneven light in the mine also makes the description of human body details in the corresponding RGB image blurred. If you want to explore the details of hand joints through skeleton data, it is not only costly but also difficult, because the hand joints are numerous and dense, and even if the subtle hand joint information is obtained using skeleton data, it is still difficult to provide human-object interaction information. On the contrary, the acquisition of hand RGB video is simple and feasible.
[0051] To address this problem, the present invention proposes an RGB modality behavior recognition information stream, which is added to the skeleton stream representation in the form of a patch. It is not only integrated at the final score level, but also can be combined with the trained skeleton stream information during the training process to re-correct the recognition results of the skeleton information stream. We call this information patch the Migrated Hand Patch (MHP), which adds hand detail information to the skeleton sequence while keeping the amount of computational data small. The MHP construction process is shown below: Figure 5 .
[0052] It is worth noting that in actual situations, due to the fixed camera angle and uneven light in the mine, it is difficult to obtain high-quality patch images by using only RGB video to construct the MHP map. This is because due to the above reasons, the corresponding hand joint information may be lost in some RGB frames, causing the final constructed MHP map to be distorted, blurred, and other problems. Based on this, the present invention uses the depth map captured simultaneously underground to synchronously optimize the construction of the MHP map. The specific construction process of the MHP map is shown in Algorithm 1:
[0053]
[0054]
[0055]
[0056] After obtaining the MHP graph, we transfer the knowledge points of the skeleton modality. The purpose of this operation is to use the knowledge learned from the skeleton modality to enhance the robustness of RGB modality learning, reduce irrelevant pixels, and focus on the hand area information. First, calculate the joint point weights of the GCN model:
[0057]
[0058] in, is the input feature of the j-th joint point of the k-th sample, is the relationship between joints i and j in the first channel of the kth sample, and w is the corresponding weight matrix. The calculation is:
[0059]
[0060] Where C represents the total number of channels, T represents the number of time frames, T is set to 15 according to the experiment, and i represents the joint point to be calculated. The specific representation of the mask matrix is W masked :
[0061]
[0062] and represents the weights of the left hand and the right hand respectively, and g represents the matrix conversion function. The skeleton knowledge transfer process obtained by the final GCN model training can be expressed as f masked :
[0063] f masked =f in ⊙W masked (11)
[0064] It is worth noting that in actual situations, due to the fixed camera angle and uneven light in the mine, it is difficult to obtain high-quality patch images by using only RGB video to construct the MHP map. This is because due to the above reasons, the corresponding hand joint information may be lost in some RGB frames, causing the final constructed MHP map to be distorted, blurred, and other problems. Based on this, the present invention uses the depth map captured simultaneously underground to synchronously optimize the construction of the MHP map. The specific construction process of the MHP map is shown in Algorithm 1:
[0065]
[0066]
[0067] After obtaining the MHP graph, we transfer the knowledge points of the skeleton modality. The purpose of this operation is to use the knowledge learned from the skeleton modality to enhance the robustness of RGB modality learning, reduce irrelevant pixels, and focus on the hand area information. First, calculate the joint point weights of the GCN model:
[0068]
[0069] in, is the input feature of the j-th joint point of the k-th sample, is the relationship between joints i and j in the first channel of the kth sample, and w is the corresponding weight matrix. The calculation is:
[0070]
[0071] Where C represents the total number of channels, T represents the number of time frames, T is set to 15 according to the experiment, and i represents the joint point to be calculated. The specific representation of the mask matrix is W masked :
[0072]
[0073] and represents the weights of the left hand and the right hand respectively, and g represents the matrix conversion function. The skeleton knowledge transfer process obtained by the final GCN model training can be expressed as f masked :
[0074] f masked =f in ⊙W masked (11)
[0075] To complete the construction of GOIS, we first search for key frames in RGB video. The importance of RGB frames is determined by using the weights of bone training. The dimension of weights is N*C*T*V. The weights are globally averaged in channels and spaces and reshaped to N*T. Then all frames N are traversed and the serial number of the frame with the largest score in [0,T] is saved as the index number of the key frame. Then a frame is randomly selected from the previous and next stages as the start and end frames of the action. A total of three frames are extracted and then cropped. The position of the human body in the RGB image is located according to the coordinates of the bone joints, and the upper left and lower right coordinates of the joint coordinates are obtained. Then, based on these two points, 196 pixels are expanded to the left and right, and 98 pixels are expanded to the top and bottom to reduce redundant background information. The configuration parameters are obtained through experiments. The construction of GOIS is shown in Algorithm 2.
[0076]
[0077]
[0078] The GOIS graph constructed by Algorithm 2 is shown in Figure 6.
[0079] The prediction results of different modal data training are:
[0080]
[0081] Among them, σ is the activation function, F i To train the network, W i For the corresponding training weights, the loss function of the entire joint training model is designed as a binary joint loss function, and the adaptive control factors α and β are used to regulate the skeleton modality data and RGN modality data respectively. α and β are updated with the gradient back propagation:
[0082]
[0083] Among them, L J , L B , L M and L G They represent the training loss functions of joints, bones, MHP and GOIS respectively, and they are all cross entropy loss functions:
[0084]
[0085] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention fall within the protection scope of the present invention.
[0086] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are within the scope of the technical solution of the present invention.
Claims
1. A method for identifying underground dangerous behaviors based on a multimodal joint training network, characterized in that: The steps include: S1. Video acquisition of underground personnel behavior and generation of skeleton data: RGB videos of personnel activities are acquired through fixed cameras underground. The whole process includes video acquisition, encoding transmission, video decoding and data storage. After capturing the original underground RGB video, the HRNet algorithm is used to extract human skeleton coordinate data; S2, acquisition of behavioral characteristics of underground workers’ skeleton data; S3, obtaining detailed feature information of underground workers’ hands; S4. Global missing information extraction.
2. The underground dangerous behavior identification method based on multi-modal joint training network according to claim 1 is characterized by: In step S1, the specific steps of extracting human skeleton coordinate data using the HRNet algorithm are as follows: S1.
1. Use the OpenCV tool library to extract the original RGB video frame by frame at a frequency of 30fps; S1.
2. Preprocess each frame, including image scaling and normalization, to meet the input requirements of the HRNet model. At the same time, save the obtained RGB frame images separately and annotate their corresponding behavior categories as the RGB data source to be trained. All RGB videos pushed to the backend device are divided into training sets and test sets in a ratio of 7:
3. The depth map data is processed in the same way. S1.
3. Then use the PyTorch framework to load the pre-trained HRNet model, input the pre-processed image into the HRNet model, and obtain the heatmap representation of the key points of the human body; S1.4, obtaining the specific coordinates of each key point by parsing the generated heat map, and then post-processing the obtained key points, that is, converting them back to the coordinate system of the original image; S1.
5. Finally, we save the coordinates of the human bone joints output by the HRNet model as .skeleton files, and package all .skeleton files into .pth files for subsequent underground human skeleton data behavior recognition tasks. After obtaining the .pth file, the data is also divided into training set and test set according to the above S1.2 standards, and the training set includes as many dangerous behavior categories of underground personnel as possible.
3. The underground dangerous behavior identification method based on multi-modal joint training network according to claim 2 is characterized by: In step S2, when acquiring the behavioral features of the underground worker's skeletal data, a multi-level spatial aggregation graph convolution (MAS-GCN) network is used to utilize spatiotemporal enhancement and a non-shared adjacency matrix to alleviate spatiotemporal over-smoothing. MAS-GCN performs spatial aggregation and temporal aggregation on the adjacency matrix of the spatial graph convolution, and calculates its unique non-shared graph representation for each input skeleton sequence. The temporal enhancement operation and the spatial enhancement operation are respectively and Among them, Reshape is the reshaping operation, C mean For integer operations, S pool and T pool They are spatial pooling operation and temporal pooling operation respectively. The calculation of the entire non-shared graph representation is divided into four branches. The top branch calculates the unique K-hop spatial aggregation of each graph, so that the model can capture the relationship between nodes under different synchronization numbers during the training process, thereby improving the model's ability to represent complex relationships between nodes. The calculation result of the K-hop spatial aggregation is A K : The two middle branches have similar structures. Their purpose is to obtain higher-dimensional features and perform temporal feature enhancement. First, the input is embedded into a high dimension through 1*1 convolution, then time aggregation is performed, and then the dimensions are converted, multiplied, and normalized before output. The bottom A ske It is the inherent physical connection of the human body, including the connection of the joints themselves, which is parameterized and initialized, and then aggregated in space to obtain A p : THE p =S agg (for(The ske )) (4) Then add the result to the result after intermediate time aggregation and perform spatial aggregation again to obtain f ST : Finally, after residual connection and normalization operations, the output f is obtained out : f out =RES[(Relu(BN(f ST ))),f ST ] (6) Among them, K represents the number of spatial layers that need to be aggregated during spatial convolution. In this paper, it is set to 3 following ST-GCN. agg represents spatial aggregation, para represents parameterization, T agg represents time aggregation, W a , W b With W c represents the corresponding weight matrix, A ske It represents the inherent connection of the human body.
4. The underground dangerous behavior identification method based on multi-modal joint training network according to claim 1 is characterized by: In step S3, the behavior recognition information flow of RGB modality is added to the skeleton flow representation in the form of a patch, the MHP graph is constructed using the RGB video, and the construction of the MHP graph is synchronously optimized using the depth map captured simultaneously underground. After the MHP graph is obtained, the skeleton modality knowledge point migration is performed on it. The purpose of this operation is to use the knowledge learned from the skeleton modality to enhance the robustness of RGB modality learning, reduce irrelevant pixels, and focus on the hand area information. First, the joint point weights of the GCN model are calculated: in, is the input feature of the j-th joint point of the k-th sample, is the relationship between joints i and j in the first channel of the kth sample, w is the corresponding weight matrix, and the mask map The calculation is: Where C represents the total number of channels, T represents the number of time frames, T is set to 15 according to the experiment, i represents the joint point to be calculated, and the specific representation of the mask matrix is W masked : and They represent the weights of the left hand and the right hand respectively, and g represents the matrix conversion function. The skeleton knowledge transfer process obtained by the final GCN model training can be expressed as f masked : f masked =f in ⊙W masked (11)。 5. The underground dangerous behavior identification method based on multi-modal joint training network according to claim 4 is characterized by: The algorithm used to construct the MHP graph is as follows: Input: depth map of underground personnel, RGB map of underground personnel, coordinates of human joints obtained by HRNet Output: MHP graph 1. Reconstruct the depth map file name 2. Adjust the depth map size 3.For each skeleton key point do 4.For each pixel around the key point do 5. Calculate the average depth of surrounding pixels 6.end for 7. Determine the minimum distance (MD) of the human body in the depth map 8. Update MD and the corresponding human index value (1 or 2) 9.if current distance>MD then 10. Not adopted (too far away) 11.else 12.if current distance <MD then 13. Use (appropriate distance) 14.end if 15.end if 16.end for 17. Randomly select N frames of RGB images and locate the coordinates of the hand joints 18. for t in T do 19.for v in V do 20. Cropping MHP (128×128) 21.end for 22.end for 23. Connect in time and space 24. Obtain the MHP diagram.
6. The underground dangerous behavior identification method based on multi-modal joint training network according to claim 1 is characterized by: In step S4, global missing information extraction adopts global missing information patching stream (GOIS), and the construction of GOIS is as follows: key frame search is performed on the RGB video, and the importance of the RGB frame is judged by using the skeleton training weight, the dimension of the weight is N*C*T*V, and the global average operation is performed on the channel and space and integerized to N*T, and then all frames N are traversed, and the serial number of the frame with the largest score in the [0, T] frame is saved as the index number of the key frame, and then a frame is randomly taken from the previous and next stages as the start frame and the end frame of the action, and a total of three frames are extracted, and then they are cropped, and the position of the human body in the RGB picture is located according to the coordinates of the bone joint points, and the upper left and lower right coordinate points of the joint point coordinates are obtained, and then these two points are used as references to expand 196 pixels on the left and right, and 98 pixels on the top and bottom to reduce redundant background information.
7. The underground dangerous behavior identification method based on multi-modal joint training network according to claim 6 is characterized by: In step S4, the GOIS construction algorithm is as follows: Input: underground personnel bone joint training weights, underground personnel bone joint point coordinates, underground personnel RGB image Output: GOIS map 1. Obtain the weights (N*C*T*V) after skeletal modal data training 2. Channel global average (N*T*V) 3. Spatial global average (N*T) 4.for n in range(N)do 5.for t in range(T)do 6. Select the frame with the largest weight and save the index number K 7.end do 8.end for 9. Randomly select 1 frame before K and 1 frame after K, a total of 3 frames 10. Determine the positions of the upper left joint and the lower right joint by the joint point coordinates 11. Pixel expansion and cropping 12. Stitch the processed RGB images in chronological order 13. Obtain the GOIS map.
Citation Information
Cited By
Human motion capture method based on mask perception graph convolution and skeleton prior
CN120088331A
A Human Motion Capture Method Based on Mask-Aware Graph Convolution and Skeletal Priors
CN120088331B
