A risk prediction method based on video key frame flood personnel action recognition

Abstract

The application discloses a risk prediction method based on video key frame flood personnel action recognition, including the following steps: extracting the key frame of the flood monitoring video; constructing a key frame image fast action recognition model; and performing risk level quantization based on disaster feature recognition. Through the integration of key frame extraction, three-dimensional pose estimation, action recognition and risk quantization and other technologies, efficient and accurate personnel behavior monitoring and risk assessment in a complex flood environment are realized; the method combines single-width flow information and multi-category behavior characteristics, significantly improves the accuracy and reliability of risk prediction, and ensures the timeliness and effectiveness of emergency response; the optimized deep learning model and efficient data processing process are adopted, the real-time performance and high performance in processing large-scale video data are ensured, and the monitoring efficiency and resource utilization in disaster management are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to a risk prediction method based on video key frame flood personnel action recognition. BACKGROUND

[0002] With the rapid development and wide application of computer vision and deep learning technology, flood personnel action recognition has become a new field of research, and the problems to be solved in disaster emergency response are becoming more and more complex. However, due to the diverse behavior patterns, complex dynamic environmental characteristics and high-dimensional image information of video data in flood scenes, the processing algorithm faces high computational complexity and real-time requirements, making it difficult to efficiently cope with the challenges of large-scale data and complex algorithms. In addition, the diversity of human behavior and the variability of the environment increase the difficulty of action recognition. Although the existing technology has made certain progress in real-time monitoring and multi-person body detection, there are still significant deficiencies in three-dimensional pose estimation, environmental adaptability and comprehensive risk quantification, and an integrated, efficient and accurate method is needed to improve the personnel monitoring and risk assessment capability in flood disasters. SUMMARY

[0003] At present, the flood personnel action recognition and risk prediction technology based on video key frames faces many challenges in practical application. First, the video data in flood scenes is huge and contains rich dynamic information and complex environmental characteristics, such as water flow speed, obstacle distribution, etc. This makes the existing action recognition algorithm have significantly increased computational complexity and real-time requirements when dealing with high-dimensional and diversified data. Second, the flood environment has high uncertainty and dynamic variability, and traditional two-dimensional pose estimation methods are difficult to accurately capture the three-dimensional pose and action details of personnel, resulting in insufficient accuracy and robustness of behavior recognition. In addition, existing technologies focus on single tasks, such as only behavior recognition or only 3D pose estimation, lack of systematic solutions that organically combine multiple tasks, and are difficult to achieve comprehensive risk quantification. Furthermore, the existing methods for risk level quantification combined with environmental factors such as water depth are limited, which cannot fully utilize the information advantage of multi-source data, limiting the accuracy and practicality of risk prediction. Finally, developing and deploying an efficient multi-task model requires deep knowledge of computer vision and deep learning, which poses a high technical threshold for experts in emergency management and geosciences. Therefore, how to provide an integrated, efficient and accurate flood personnel action recognition and risk prediction method based on video key frames, which can realize real-time and high-accuracy behavior recognition in complex and dynamic flood environments, and conduct comprehensive risk quantification combined with environmental factors, is a technical problem to be solved.

[0004] In view of the above technical problems in the related art, the present application provides a risk prediction method based on video key frame action recognition of personnel in flood, which can solve the above problems.

[0005] To achieve the above technical purposes, the technical scheme of the present application is as follows:

[0006] A risk prediction method based on video key frame action recognition of personnel in flood, comprising the following steps:

[0007] S100, extracting key frames of flood monitoring video: using a key frame extraction algorithm based on content change, by analyzing the inter-frame difference between consecutive video frames, extracting frames with significant changes as key frames;

[0008] S200, constructing a key frame image fast action recognition model;

[0009] S210, using OpenPose key point detection technology to perform high-precision human key point detection on the extracted key frames, and obtaining 3D posture information of the personnel;

[0010] S220, based on the detected key point information, using a deep learning model Transformer to classify the actions of the personnel into multiple categories, to accurately identify multiple personnel behaviors in the flood environment;

[0011] S300, risk level quantification based on disaster feature recognition;

[0012] S310, single-width flow information acquisition and processing: combining multi-source data fusion technology, real-time acquisition of single-width flow information of the flood area, and pre-processing of the data;

[0013] S320, risk score model design: designing a comprehensive risk score model, combining single-width flow information with identified personnel action features, to realize multi-dimensional risk level quantification;

[0014] S330, risk level division and early warning mechanism: according to the comprehensive risk score, dividing the personnel into multiple risk levels, and displaying the risk distribution through a dynamic risk map to ensure timely emergency response.

[0015] Further, step S100 specifically comprises the following steps:

[0016] S110, video frame reading: reading image data from the flood monitoring video frame by frame, processing in time sequence, and training the first N frames of the video to establish a background model;

[0017] S120, background difference: applying background difference method to each frame of image, calculating the difference between the current frame and the background model, and generating a foreground mask;

[0018] S130, feature extraction: extract the features of the foreground region, including color features, texture features, shape features;

[0019] S140, content transformation analysis: combine the color, texture, shape features extracted from each frame to form a feature vector, calculate the change of the feature vector between adjacent frames, and identify the frames with significant feature changes;

[0020] S150, cosine similarity calculation: use cosine similarity to measure the similarity between adjacent frames;

[0021] S160, K-means clustering: take all frames as input and apply K-means clustering algorithm for grouping;

[0022] S170, key frame selection: select the frames near the cluster center as key frames;

[0023] S180, key frame storage and management: store the extracted key frames in chronological order and establish an index.

[0024] Further, the extracted key frames are preprocessed, including denoising, image enhancement, cropping and scaling.

[0025] Further, step S210 specifically includes the following steps:

[0026] S211, key point detection: use the OpenPose model to detect human key points in the preprocessed key frame, and obtain the three-dimensional coordinates of the human key joints;

[0027] S212, pose optimization: through the normalization flow technology, the distribution modeling of the detected 3D joint position is carried out, and the accuracy and robustness of the pose estimation are improved.

[0028] Further, step S220 specifically includes the following steps:

[0029] S221, time sequence feature extraction: input the key point coordinate sequence in the continuous key frame into the Transformer model to extract the time sequence dynamic features of the action;

[0030] S222, action classification: classify the extracted features through a multi-class classifier, identify the current behavior type of the personnel, and output the corresponding behavior category probability distribution.

[0031] Further, the single-width flow information acquisition in step S310 is specifically: monitoring the flow rate and water depth of the flood by using the deployed flow rate sensor and water depth sensor, and supplementing the measurement results of the sensors by using the flow rate and water depth of the flood obtained from the media video data, integrating the water depth and flow rate information through multi-source data, and calculating the single-width flow Q = h x u, h is the real-time water depth, the unit is meter; u is the water flow rate, the unit is meter / second.

[0032] Further, the pre-processing of the single-width flow related data includes denoising and correction, time series smoothing, synchronization and calibration of the data.

[0033] Further, step S320 specifically includes the following steps:

[0034] S321, according to the size of the single-width flow, the single-width flow is divided into four dynamic levels of low flow, medium flow, high flow and extremely high flow, and the flow risk score is set according to the dynamic level of the single-width flow;

[0035] S322, according to the identified behavior type and frequency, the behavior is divided into four levels of low risk, medium risk, high risk and extremely high risk, and the corresponding action risk score is set for each risk action;

[0036] S323, combining the environmental information and the personnel action characteristics, quantifying the personnel risk level in the flood scene, and establishing a comprehensive risk score formula: RS = a x E + b x A, wherein RS is the comprehensive risk score, E is the environmental danger coefficient, A is the action danger coefficient, and a and b are weight coefficients.

[0037] Further, step S330 specifically includes the following steps:

[0038] S331, personnel risk level division: according to the comprehensive risk score, the personnel is divided into four levels of low risk, medium risk, high risk and extremely high risk;

[0039] S332, risk map generation: mapping the risk level of each personnel to the spatial coordinate system, and color coding the personnel position according to the comprehensive risk score.

[0040] The application realizes efficient and accurate personnel behavior monitoring and risk assessment in complex flood environments by integrating key frame extraction, three-dimensional pose estimation, action recognition, and risk quantification technologies. The method combines single-width flow information and multi-category behavior characteristics to significantly improve the accuracy and reliability of risk prediction, ensuring the timeliness and effectiveness of emergency response. The optimized deep learning model and efficient data processing flow ensure real-time performance and high performance when processing large-scale video data, greatly improving monitoring efficiency and resource utilization in disaster management. The intuitive risk distribution display and automatic warning mechanism enhance monitoring and response capabilities in flood disasters. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] The present application will be further described in detail below according to the drawings.

[0043] Fig. 1 is the overall architecture diagram of the risk prediction method for personnel action recognition in flood based on video key frames according to the embodiments of the present application;

[0044] Fig. 2 is the flowchart of extracting key frames of flood monitoring video according to the embodiments of the present application;

[0045] Fig. 3 is the flowchart of constructing a fast action recognition model for key frame images according to the embodiments of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0047] As Figs. 1-3As shown, a risk prediction method based on video key frame flood personnel action recognition is disclosed, comprising the following steps: first, extracting the key frame of the flood monitoring video; then, constructing a key frame image fast action recognition model, specifically, using the distribution perception single stage (DAS) model and OpenPose technology, high-precision human key point detection is performed on the extracted key frame, through recursive updating strategy and normalization flow technology, the real distribution of 3D joint position is modeled, the posture estimation accuracy in complex flood environment is improved, combined with the deep learning framework (PyTorch), the detected key point sequence is extracted using the Transformer model, the action dynamic characteristics are analyzed, a multi-class action classifier is constructed, the action type is combined with the flood situation (such as walking, falling, shaking, etc.), and the behavior classification result and the corresponding probability distribution are output. For the common personnel behavior mode in the flood, the classification model is specially optimized to adapt to the complex dynamic characteristics in the flood scene, and the personnel action recognition task in various flood scenes is realized; finally, risk level quantization is carried out based on disaster feature recognition, specifically, through media data and sensor data fusion, real-time acquisition and processing of water depth, flow rate and other environmental information are realized, and are integrated into single-width flow, and combined with human action recognition technology, a comprehensive risk score model is constructed, the model combines single-width flow with the personnel action features identified, sets reasonable weight coefficients, and dynamically calculates the comprehensive risk score of personnel, so that multi-dimensional risk level quantization is realized.

[0048] Embodiment one:

[0049] The application adopts a key frame extraction algorithm based on content change, extracts frames with significant changes as key frames by analyzing the inter-frame differences between consecutive video frames, to reduce redundant information and improve processing efficiency. The specific implementation steps are as follows:

[0050] Video frame reading: read image data from the flood monitoring video frame by frame, process in time sequence, and train the first N frames of the video to establish a background model.

[0051] Background difference: apply background difference method to each frame of image, calculate the difference between the current frame and the background model, and generate foreground mask.

[0052] Feature extraction: extract features of the foreground region, including color features, texture features, and shape features.

[0053] Content change analysis: combine the color, texture, and shape features extracted from each frame to form a feature vector, calculate the change amount of the feature vector between adjacent frames, and identify frames with significant feature changes.

[0054] Cosine similarity calculation: use cosine similarity to measure the similarity between adjacent frames.

[0055] K-means clustering: Using all frames as input, the K-means clustering algorithm is applied to group them.

[0056] Keyframe selection: Select frames near the cluster centroid as keyframes.

[0057] Keyframe storage and management: Extracted keyframes are stored in chronological order and indexed for easy subsequent processing and quick access.

[0058] Furthermore, a series of preprocessing operations are performed on the extracted keyframes to improve image quality and feature extraction effectiveness: these include denoising, image enhancement, and cropping and scaling. Denoising involves applying Gaussian filtering and median filtering techniques to remove noise and improve image clarity. Image enhancement utilizes histogram equalization and contrast stretching to enhance contrast and brightness, improving visual appeal. Cropping and scaling involve cropping key areas (such as areas of human activity) according to the specific needs of the monitored area and scaling the image to the standard 224×224 pixel model input size to ensure consistency of the input image.

[0059] Example 2:

[0060] This application employs OpenPose keypoint detection technology to perform high-precision human keypoint detection on extracted keyframes, obtaining 3D pose information of the person. Specifically, it includes keypoint detection and pose optimization. Keypoint detection uses the OpenPose model to perform human keypoint detection on preprocessed keyframes, obtaining the 3D coordinates of 17 key joints, including the head, shoulders, elbows, and knees. Pose optimization uses normalizing flow technology to model the distribution of the detected 3D joint positions, improving the accuracy and robustness of pose estimation, especially in complex dynamic environments.

[0061] This application also utilizes the Transformer deep learning model to classify human actions into multiple categories based on the detected keypoint information, thereby achieving accurate identification of various human behaviors in flood environments. Specifically, this includes temporal feature extraction and action classification. Temporal feature extraction involves inputting the keypoint coordinate sequence from consecutive keyframes into the Transformer model to extract the temporal dynamic features of the actions. Action classification involves classifying the extracted features using a multi-class classifier to identify the current behavior type of the person (such as walking, swaying, curling up, falling, etc.) and outputting the corresponding behavior category probability distribution.

[0062] Example 3:

[0063] The application combines multi-source data fusion technology to obtain real-time single-width flow information of the flood area, and pre-processes the data to ensure its accuracy and stability. As a comprehensive indicator of water depth and flow rate, single-width flow can accurately reflect the risk of floods on personnel.

[0064] The single-width flow information acquisition includes sensor data fusion and multi-source data supplementation. The flow rate and water depth of the flood are monitored by deploying flow rate sensors and water depth sensors in the flood area, and the flow rate and water depth of the flood are obtained by using media video data (for example, using a water depth estimation method that uses a reference height to estimate water depth from media video data, and for example, using media video data to track the speed of floating objects to estimate the flow rate of the flood). The measurement results of the sensors are supplemented (to improve data coverage), the water depth and flow rate information are integrated through multi-source data, and the single-width flow Q = h x u is calculated, where h is the real-time water depth in meters, and u is the water flow rate in meters per second.

[0065] The pre-processing of the data related to single-width flow includes denoising and correction, time series smoothing, synchronization and calibration. Denoising and correction: denoising the water depth and flow rate data collected by the sensor to eliminate device errors or environmental interference, and correcting the flow rate and water depth according to the actual measurement value to ensure the accuracy of the single-width flow calculation; time series smoothing: using a sliding window method to smooth the continuous time series data to reduce data fluctuations, and using Kalman filtering technology to improve the stability of the single-width flow estimation; data synchronization and calibration: synchronization through interpolation and time series analysis to align the data from different data sources in time and space, ensuring the consistency of the model input.

[0066] Uncertainty is an important issue that cannot be ignored. Since the indicators from different data sources may differ, how to effectively deal with these uncertainties is one of the keys to improving the accuracy and practicality of the model. Different data sources, such as social media (including pictures, text, videos, etc.), sensor data, and traditional physical models (such as roadblocks, traffic signs, and reference objects), often have significant differences in accuracy and data quality. Therefore, based on the pre-processing of the above data, uncertainty modeling and fusion strategies can be used, such as weighted averaging, Kalman filtering, etc. to reasonably allocate the weights of each data source, avoid over-reliance on a single data source, and further optimize the robustness of data fusion through model evaluation and dynamic adjustment mechanism, reducing the impact of low-precision data.

[0067] The application designs a comprehensive risk score model that combines single-width flow information with identified personnel action features to achieve multi-dimensional risk level quantification. First, the single-width flow level is divided, then the behavior risk level is divided, and finally the comprehensive risk score model is built.

[0068] The specific operation for single-width flow rate classification is as follows:

[0069] According to the size of the single-width flow rate, it is divided into four dynamic levels, as shown in the following table:

[0070]

[0071] According to the dynamic level of the single-width flow rate, the flow risk score is set: low flow (Q < 0.2 m² / s) corresponds to a risk score of 1-2, medium flow (0.2 ≤ Q < 0.6 m² / s) corresponds to a risk score of 3-4, high flow (0.6 ≤ Q < 1.2 m² / s) corresponds to a risk score of 5-6, and very high flow (Q ≥ 1.2 m² / s) corresponds to a risk score of 7-8.

[0072] The specific operation for behavior risk level classification is as follows:

[0073] According to the identified behavior type and frequency, the behavior is divided into four levels: low risk (still, walking), medium risk (standing still, walking in different postures), high risk (swaying, bending), and very high risk (crouching, falling, shaking, falling down), as shown in the following table:

[0074]

[0075] According to the behavior risk level, the action risk score of each action is divided: low-risk actions: still, walking, score 1 and 2; medium-risk actions: standing still, walking in different postures, score 3 and 4; high-risk actions: swaying, bending, score 5 and 6; very high-risk actions: crouching, shaking, score 7, falling, falling down, score 8.

[0076] The comprehensive risk score model is: combining environmental information (water depth, flow rate) with personnel action characteristics, quantifying the personnel risk level in the flood scene, and constructing the risk score formula RS = α × E + β × A, RS is the comprehensive risk score, E is the environmental danger coefficient, quantifying the influence of environmental characteristics such as water depth, flow rate, and floating object density on personnel risk, A is the action danger coefficient, which is set according to the identified personnel behavior (such as walking, swaying, falling, etc.), α and β are weight coefficients, which are dynamically adjusted according to the actual scene, environmental factors account for the main weight, and α = 0.6, β = 0.4 can be set.

[0077] According to the comprehensive risk score, the personnel is divided into four risk levels, so as to quickly identify the dangerous state, and the risk distribution situation is displayed through the dynamic risk map to ensure timely emergency response.

[0078] The specific operation for dividing personnel into four risk levels is as follows:

[0079] Generally, the flood water situation warning of a country is divided into four levels from low to high, which are represented by blue, yellow, orange and red respectively. In view of this, the comprehensive risk score calculated by the risk scoring model constructed above is also divided into four levels: low risk (RS < 2), medium risk (2 ≤ RS < 4), high risk (4 ≤ RS < 6) and extremely high risk (RS ≥ 6), as shown in the following table:

[0080]

[0081] The specific operation of generating the risk map is as follows:

[0082] Map each person's risk level to the spatial coordinate system, and color-code the location of each person according to the comprehensive risk score. Green: low risk, yellow: medium risk, orange: high risk, red: extremely high risk. Further, after receiving real-time video streams and sensor data, the system calculates the risk score of each person, generates a real-time updated risk distribution map, and visually displays the risk conditions in different areas.

[0083] When high-risk or extremely high-risk behavior is detected, the system automatically triggers an early warning signal to notify the emergency response team to ensure timely rescue; at the same time, according to the distribution of the risk map, the deployment of rescue resources can be dynamically adjusted to prioritize high-risk areas and personnel, improving rescue efficiency.

[0084] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A risk prediction method based on video key frame floodwater action recognition of personnel, characterized in that, Comprise the following steps: S100, extracting key frames of flood monitoring video: using content change based key frame extraction algorithm, by analyzing the interframe difference between continuous video frames, extracting the frame with significant change as key frame; S200, constructing key frame image fast action recognition model; S210, using OpenPose key point detection technology to detect the key points of the extracted key frames with high precision, and obtaining the 3D posture information of the personnel; S220, based on the detected key point information, using deep learning model Transformer to classify the action of the personnel, realizing the accurate identification of various personnel behaviors in flood environment; S300, risk level quantification based on disaster feature recognition; S310, single-width flow information acquisition and processing: combining multi-source data fusion technology, real-time acquisition of single-width flow information of flood area, and data preprocessing; The single-width flow information acquisition in step S310 is: using the deployed flow rate sensor and water depth sensor to monitor the flow rate and water depth of the flood, and using the flow rate and water depth obtained from the media video data to supplement the measurement results of the sensor, integrating the water depth and flow rate information, calculating the single-width flow Q=h×u, h is the real-time water depth, unit is meter; U is the flow velocity, unit is meter / second; S320, risk score model design: design a comprehensive risk score model, combine the single-width flow information with the identified personnel action features, realize multi-dimensional risk level quantification; Step S320 specifically comprises the following steps: S321, according to the size of single-width flow, the single-width flow is divided into low flow, medium flow, high flow and extremely high flow four dynamic levels, and the flow risk score is set according to the dynamic level of single-width flow; S322, according to the identified behavior type and frequency, the behavior is divided into low risk, medium risk, high risk and extremely high risk four levels, and the corresponding action risk score is set for each risk action; S323, combine the environmental information with the personnel action features, quantify the personnel risk level in the flood scene, and establish a comprehensive risk score formula: RS=α×E+β×A, where RS is the comprehensive risk score, E is the environmental danger coefficient, A is the action danger coefficient, and α and β are weight coefficients; S330, risk level division and early warning mechanism: according to the comprehensive risk score, the personnel is divided into multiple risk levels, and the risk distribution is displayed through the dynamic risk map to ensure timely emergency response; Step S330 specifically comprises the following steps: S331, personnel risk level division: after receiving the video stream and sensor data in real time, calculate the risk score of each personnel, according to the comprehensive risk score, divide the personnel into low risk, medium risk, high risk and extremely high risk four levels; S332, risk map generation: map the risk level of each personnel to the spatial coordinate system, and color code the personnel position according to the comprehensive risk score; S333, according to the distribution of risk map, dynamically adjust the deployment of rescue resources, and give priority to high-risk areas and personnel.

2. The risk prediction method based on the action recognition of the people in the flood from the key frame of the video according to claim 1, characterized in that, Step S100 specifically includes the following steps: S110, video frame reading: read image data from the flood monitoring video frame by frame, process in chronological order, and train the first N frames of the video to establish a background model; S120, background difference: apply background difference method to each frame of image, calculate the difference between the current frame and the background model, and generate a foreground mask; S130, feature extraction: extract features of the foreground region, including color features, texture features, and shape features; S140, content transformation analysis: combine the color, texture, and shape features extracted from each frame to form a feature vector, calculate the change amount of the feature vector between adjacent frames, and identify frames with significant feature changes; S150, cosine similarity calculation: use cosine similarity to measure the similarity between adjacent frames; S160, K-means clustering: input all frames, apply K-means clustering algorithm for grouping; S170, key frame selection: select frames near the cluster centroid as key frames; S180, key frame storage and management: store the extracted key frames in chronological order and establish an index.

3. The risk prediction method based on the action recognition of the people in the flood from the key frame of the video according to claim 2, characterized in that, Preprocessing of the extracted key frames includes denoising, image enhancement, cropping and scaling.

4. The risk prediction method based on the action recognition of the people in the flood from the key frame of the video according to claim 3, characterized in that, Step S210 specifically includes the following steps: S211, key point detection: use the OpenPose model to detect human key points in the preprocessed key frames, and obtain the three-dimensional coordinates of the human key joints; S212, pose optimization: use the normalized flow technology to model the distribution of the detected 3D joint positions, and improve the accuracy and robustness of the pose estimation.

5. The risk prediction method based on the action recognition of the people in the flood from the key frames of the video according to claim 4, characterized in that, Step S220 specifically includes the following steps: S221, time sequence feature extraction: input the key point coordinate sequence in the continuous key frames into the Transformer model to extract the time sequence dynamic features of the action; S222, action classification: classify the extracted features through a multi-class classifier, identify the current behavior type of the personnel, and output the corresponding behavior category probability distribution.

6. The risk prediction method based on the action recognition of the people in the flood from the key frame of the video according to claim 1, characterized in that, Preprocessing of single-width flow-related data includes denoising and correction, time series smoothing, synchronization and calibration of the data.

Images