Risk prediction method based on video key frame flood personnel action recognition

Abstract

The invention discloses a risk prediction method based on video key frame flood personnel action recognition. The method comprises the following steps: extracting a key frame of a flood monitoring video; constructing a key frame image quick action recognition model; and risk level quantification is carried out based on disaster feature identification. According to the invention, through integrating multiple technologies such as key frame extraction, three-dimensional attitude estimation, action recognition and risk quantification, efficient and accurate personnel behavior monitoring and risk assessment in a complex flood environment are realized; according to the method, single-width traffic information and multi-category behavior characteristics are combined, so that the accuracy and reliability of risk prediction are remarkably improved, and the timeliness and effectiveness of emergency response are ensured; an optimized deep learning model and an efficient data processing flow are adopted, the real-time performance and high performance during large-scale video data processing are guaranteed, and the monitoring efficiency and the resource utilization rate in disaster management are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a risk prediction method based on human action recognition in video key frames of floods. Background Art

[0002] With the rapid development and widespread application of computer vision and deep learning technologies, human action recognition in floods has become a new field of research, and the problems to be solved in disaster emergency response are becoming more and more complex. However, due to the diverse behavioral patterns, complex dynamic environmental characteristics and high-dimensional image information of video data in flood scenes, the processing algorithms face high computational complexity and real-time requirements, making it difficult to efficiently cope with the challenges of large-scale data and complex algorithms. In addition, the diversity of human behavior and environmental variability increase the difficulty of action recognition. Although existing technologies have made certain progress in real-time monitoring and multi-person human detection, they still have significant deficiencies in three-dimensional posture estimation, environmental adaptability and comprehensive risk quantification. An integrated, efficient and accurate method is urgently needed to improve the ability to monitor and assess risks in flood disasters. Summary of the invention

[0003] At present, the technology of human action recognition and risk prediction in floods based on video keyframes faces many challenges in practical applications. First, the amount of video data in flood scenes is huge, and contains rich dynamic information and complex environmental features, such as water flow speed, obstacle distribution, etc. This significantly increases the computational complexity and real-time requirements of existing action recognition algorithms when processing high-dimensional and diverse data. Secondly, the flood environment is highly uncertain and dynamically changeable. Traditional two-dimensional posture estimation methods are difficult to accurately capture the three-dimensional posture and action details of personnel, resulting in insufficient accuracy and robustness of behavior recognition. In addition, existing technologies mostly focus on a single task, such as only performing behavior recognition or only realizing 3D posture estimation. There is a lack of systematic solutions that organically combine multiple tasks, making it difficult to achieve comprehensive risk quantification assessment. Furthermore, the existing methods for quantifying risk levels in combination with environmental factors such as water depth are relatively limited, and cannot fully utilize the information advantages of multi-source data, limiting the accuracy and practicality of risk prediction. Finally, the development and deployment of efficient multi-task models requires deep knowledge of computer vision and deep learning, which has a high technical threshold for experts in the fields of emergency management and geoscience. Therefore, how to provide an integrated, efficient and accurate method for human action recognition and risk prediction in floods based on video key frames, which can realize real-time and highly accurate behavior recognition in complex and dynamic flood environments, and combine environmental factors for comprehensive risk quantification, is a technical problem that needs to be solved urgently.

[0004] In view of the above technical problems in the related art, the present invention provides a risk prediction method based on human action recognition in video key frame floods, which can solve the above problems.

[0005] To achieve the above technical objectives, the technical solution of the present invention is implemented as follows: A risk prediction method based on human action recognition in floods using video key frames includes the following steps: S100, extracting key frames of the flood monitoring video: using a key frame extraction algorithm based on content change, by analyzing the inter-frame differences between consecutive video frames, extracting frames with significant changes as key frames; S200, constructing a key frame image fast action recognition model; S210, using OpenPose key point detection technology to perform high-precision human key point detection on the extracted key frames to obtain 3D posture information of the person; S220, based on the detected key point information, the deep learning model Transformer is used to perform multi-category classification of personnel actions, so as to achieve accurate recognition of various personnel behaviors in the flood environment; S300, quantify risk level based on disaster feature identification; S310, single-width flow information acquisition and processing: Combine multi-source data fusion technology to obtain single-width flow information in flood areas in real time and pre-process the data; S320, Risk scoring model design: Design a comprehensive risk scoring model to combine single-width traffic information with identified personnel action characteristics to achieve multi-dimensional risk level quantification; S330, Risk level classification and early warning mechanism: Based on the comprehensive risk score, personnel are classified into multiple risk levels, and the risk distribution is displayed through a dynamic risk map to ensure timely emergency response.

[0006] Furthermore, step S100 specifically includes the following steps: S110, video frame reading: reading image data frame by frame from the flood monitoring video, processing in chronological order, and training the first N frames of the video to establish a background model; S120, background difference: applying the background difference method to each frame image, calculating the difference between the current frame and the background model, and generating a foreground mask; S130, feature extraction: extracting features of the foreground area, including color features, texture features, and shape features; S140, content transformation analysis: combining the color, texture, and shape features extracted from each frame to form a feature vector, calculating the variation of the feature vectors between adjacent frames, and identifying frames with significant feature changes; S150, cosine similarity calculation: using cosine similarity to measure the similarity between adjacent frames; S160, K-means clustering: take all frames as input and apply K-means clustering algorithm to group them; S170, key frame selection: select a frame near the cluster centroid as a key frame; S180, key frame storage and management: the extracted key frames are stored in chronological order and indexed.

[0007] Furthermore, the extracted key frames are preprocessed, including denoising, image enhancement, cropping and scaling.

[0008] Furthermore, step S210 specifically includes the following steps: S211, key point detection: Use the OpenPose model to detect the key points of the human body on the preprocessed key frames to obtain the three-dimensional coordinates of the key joints of the human body; S212, Posture Optimization: The detected 3D joint positions are distributed modeled through normalized flow technology to improve the accuracy and robustness of posture estimation.

[0009] Furthermore, step S220 specifically includes the following steps: S221, temporal feature extraction: input the key point coordinate sequence in the continuous key frames into the Transformer model to extract the temporal dynamic features of the action; S222, action classification: classify the extracted features through a multi-category classifier, identify the current behavior type of the person, and output the corresponding behavior category probability distribution.

[0010] Furthermore, in step S310, the acquisition of single-width flow information is specifically as follows: using the deployed flow velocity sensor and water depth sensor to monitor the flow velocity and water depth of the flood, and using the flood flow velocity and water depth obtained by media video data to supplement the sensor measurement results, integrating the water depth and flow velocity information through multi-source data, and calculating the single-width flow Q=h×u, where h is the real-time water depth in meters and u is the water flow velocity in meters per second.

[0011] Furthermore, preprocessing of single-width flow-related data includes data denoising and correction, time series smoothing, synchronization and calibration.

[0012] Furthermore, step S320 specifically includes the following steps: S321. According to the size of the single-width traffic, the single-width traffic is divided into four dynamic levels: low traffic, medium traffic, high traffic, and extremely high traffic, and a traffic risk score is set according to the dynamic level of the single-width traffic; S322. Classify the behaviors into four levels of low risk, medium risk, high risk and extremely high risk according to the identified behavior types and frequencies, and set corresponding action risk scores for each risk action; S323. Combine environmental information with personnel action characteristics to quantify the personnel risk level in flood scenarios and establish a comprehensive risk scoring formula: RS = α × E + β × A, where RS is the comprehensive risk score, E is the environmental hazard coefficient, A is the action hazard coefficient, and α and β are weight coefficients.

[0013] Furthermore, step S330 specifically includes the following steps: S331. Personnel risk level classification: Based on the comprehensive risk score, personnel are classified into four levels: low risk, medium risk, high risk and extremely high risk; S332. Risk map generation: Map the risk level of each person into a spatial coordinate system and color-code the person's location based on the comprehensive risk score.

[0014] Beneficial effects of the invention: This application realizes efficient and accurate personnel behavior monitoring and risk assessment in complex flood environments by integrating multiple technologies such as key frame extraction, three-dimensional posture estimation, motion recognition and risk quantification; this method combines single-width flow information with multi-category behavior characteristics, significantly improving the accuracy and reliability of risk prediction, ensuring the timeliness and effectiveness of emergency response; adopting optimized deep learning models and efficient data processing processes, it ensures real-time and high performance when processing large-scale video data, greatly improving monitoring efficiency and resource utilization in disaster management; through intuitive risk distribution display and automatic early warning mechanism, it enhances the monitoring and response capabilities in flood disasters. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] The present invention will be described in further detail below with reference to the accompanying drawings.

[0017] Figure 1 It is a general architecture diagram of the risk prediction method based on human action recognition in video key frame flood according to an embodiment of the present invention; Figure 2 is a flow chart of extracting key frames of flood monitoring video according to an embodiment of the present invention; Figure 3It is a flowchart of constructing a key frame image rapid action recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0019] like Figure 1-3 As shown, according to the present invention, a risk prediction method based on the recognition of human actions in floods based on video key frames is disclosed, comprising the following steps: first, the key frames of the flood monitoring video are extracted; then a key frame image fast action recognition model is constructed, specifically, the distributed sensing single-stage (DAS) model and OpenPose technology are used to perform high-precision human key point detection on the extracted key frames, and the real distribution of 3D joint positions is modeled through a recursive update strategy and a normalized flow technology to improve the accuracy of posture estimation in complex flood environments. In combination with a deep learning framework (PyTorch), a Transformer model is used to extract temporal features from the detected key point sequence, analyze the dynamic characteristics of the action, construct a multi-category action classifier, combine the action type with the flood situation (such as walking, falling, shaking, etc.), and output the behavior classification results and the corresponding probability distribution. According to the common human behavior patterns in floods, the classification model is specially optimized to adapt to the complex dynamic characteristics of flood scenes and realize the task of human motion recognition in various flood scenes; finally, the risk level is quantified based on the identification of disaster characteristics. Specifically, through the fusion of media data and sensor data, the real-time acquisition and processing of environmental information such as water depth and flow velocity are realized, and integrated into single-width flow. At the same time, combined with human motion recognition technology, a comprehensive risk scoring model is constructed. The model combines the single-width flow with the identified human motion characteristics, sets reasonable weight coefficients, and dynamically calculates the comprehensive risk score of the personnel, thereby realizing multi-dimensional risk level quantification.

[0020] Embodiment 1: The present invention adopts a key frame extraction algorithm based on content changes. By analyzing the inter-frame differences between consecutive video frames, frames with significant changes are extracted as key frames to reduce redundant information and improve processing efficiency. The specific implementation steps are as follows: Video frame reading: Image data is read frame by frame from the flood monitoring video, processed in chronological order, and the first N frames of the video are trained to build a background model.

[0021] Background subtraction: Apply the background subtraction method to each frame of the image, calculate the difference between the current frame and the background model, and generate a foreground mask.

[0022] Feature extraction: Extract the features of the foreground area, including color features, texture features, and shape features.

[0023] Content transformation analysis: Combine the color, texture, and shape features extracted from each frame to form a feature vector, calculate the change in the feature vector between adjacent frames, and identify frames with significant feature changes.

[0024] Cosine similarity calculation: Use cosine similarity to measure the similarity between adjacent frames.

[0025] K-means clustering: Take all frames as input and apply K-means clustering algorithm to group them.

[0026] Keyframe selection: Select frames near the cluster centroid as keyframes.

[0027] Key frame storage and management: The extracted key frames are stored in chronological order and indexed for easy subsequent processing and quick access.

[0028] Furthermore, a series of preprocessing operations are performed on the extracted key frames to improve the image quality and feature extraction effect: including denoising, image enhancement, cropping and scaling. Denoising is to apply Gaussian filtering and median filtering technology to remove noise in the image and improve image clarity; image enhancement is to enhance the contrast and brightness of the image and improve the visual effect through histogram equalization and contrast stretching; cropping and scaling is to crop the key areas in the image (such as the personnel activity area) according to the specific needs of the monitoring area, and scale the image to the standard size of 224×224 pixel model input to ensure the consistency of the input image.

[0029] Embodiment 2: This application uses OpenPose key point detection technology to perform high-precision human key point detection on the extracted key frames to obtain the 3D posture information of the person, specifically including key point detection and posture optimization. The key point detection uses the OpenPose model to perform human key point detection on the preprocessed key frames to obtain the three-dimensional coordinates of 17 key joints including the head, shoulders, elbows, knees, etc.; posture optimization uses the normalizing flow technology to distribute modeling of the detected 3D joint positions to improve the accuracy and robustness of posture estimation, especially in complex dynamic environments.

[0030] This application also uses the deep learning model Transformer to perform multi-category classification of personnel actions based on the detected key point information, thereby achieving accurate recognition of various human behaviors in flood environments, specifically including temporal feature extraction and action classification, wherein the temporal feature extraction is to input the key point coordinate sequence in continuous key frames into the Transformer model to extract the temporal dynamic characteristics of the action; the action classification is to classify the extracted features through a multi-category classifier, identify the current behavior type of the person (such as walking, shaking, curling up, falling, etc.), and output the corresponding behavior category probability distribution.

[0031] Embodiment three: This application combines multi-source data fusion technology to obtain single-width flow information in flood areas in real time and pre-process the data to ensure the accuracy and stability of the data. Single-width flow, as a comprehensive indicator of water depth and flow velocity, can accurately reflect the risk impact of floods on people.

[0032] The acquisition of single-width flow information includes sensor data fusion and multi-source data supplementation, that is, using flow velocity sensors and water depth sensors deployed in the flood area to monitor the flow velocity and water depth of the flood, and using the flood flow velocity and water depth obtained by media video data (for example, using the water depth estimation method that uses the height of a reference object using media video data, and using media video data to track the speed of floating objects to estimate the flood flow velocity), supplement the sensor measurement results (improve data coverage), integrate the water depth and flow velocity information through multi-source data, and calculate the single-width flow Q=h×u, where h is the real-time water depth in meters and u is the water flow velocity in meters per second.

[0033] The preprocessing of single-width flow-related data includes data denoising and correction, time series smoothing, synchronization and calibration. Denoising and correction: denoising the water depth and flow rate data collected by the sensor to eliminate equipment errors or environmental interference, and correcting the flow rate and water depth according to the actual measurement values ​​to ensure the accuracy of single-width flow calculation; time series smoothing: using the sliding window method to smooth the continuous time series data to reduce data fluctuations, and using Kalman filtering technology to improve the stability of single-width flow estimation for data mutation problems; data synchronization and calibration: synchronization through interpolation and time series analysis methods to achieve alignment of data from different data sources in time and space to ensure the consistency of model input.

[0034] Uncertainty is an important issue that cannot be ignored. Since indicators from different data sources may differ, how to effectively deal with these uncertainties is one of the keys to improving the accuracy and practicality of the model. Different data sources, such as social media (including pictures, texts, videos, etc.), sensor data, and traditional physical models (such as roadblocks, traffic signs, and reference objects, etc.), often have large differences in accuracy and data quality. Therefore, based on the preprocessing of the above data, uncertainty modeling and fusion strategies can be adopted, and weighted fusion can be performed using weighted averaging, Kalman filtering and other technologies to reasonably allocate the weights of each data source and avoid over-reliance on a single data source. Furthermore, through model evaluation and dynamic adjustment mechanisms, the robustness of data fusion can be continuously optimized to reduce the impact of low-precision data.

[0035] This application designs a comprehensive risk scoring model, combining single-width traffic information with identified personnel action characteristics to achieve multi-dimensional risk level quantification. First, single-width traffic level classification is performed, then behavioral risk level classification is performed, and finally a comprehensive risk scoring model is constructed.

[0036] The specific operations for single-width traffic level classification are as follows: According to the size of single-width traffic, it is divided into four dynamic levels, as shown in the following table: The flow risk score is set according to the dynamic level of single-width flow: low flow (Q<0.2 m² / s) corresponds to a risk score of 1 to 2, medium flow (0.2 ≤ Q<0.6 m² / s) corresponds to a risk score of 3 to 4, high flow (0.6 ≤ Q<1.2 m² / s) corresponds to a risk score of 5 to 6, and extremely high flow (Q ≥ 1.2 m² / s) corresponds to a risk score of 7 to 8.

[0037] The specific operations for the classification of behavioral risk levels are as follows: Based on the identified behavior types and frequencies, behaviors are divided into four levels: low risk (stationary, walking), medium risk (standing still, walking in different postures), high risk (shaking, bending) and extremely high risk (crouching, falling, shaking, falling), as shown in the following table: The action risk score of each action is divided according to the behavioral risk level: low-risk actions: stillness and walking are scored as 1 and 2; medium-risk actions: standing still and walking in different postures are scored as 3 and 4; high-risk actions: shaking and bending are scored as 5 and 6; extremely high-risk actions: curling up and shaking are scored as 7, and falling and tripping are scored as 8.

[0038] The comprehensive risk scoring model is as follows: combining environmental information (water depth, flow velocity) with personnel action characteristics, quantifying the personnel risk level in flood scenarios, and constructing a risk scoring formula RS=α×E+β×A, where RS is the comprehensive risk score, E is the environmental hazard coefficient, which quantifies the impact of environmental characteristics such as water depth, flow velocity, and floating object density on personnel risk, A is the action hazard coefficient, which is set according to the identified personnel behavior (such as walking, shaking, falling, etc.), α and β are weight coefficients, which are dynamically adjusted according to the actual scenario. Environmental factors account for the main weight, and α=0.6 and β=0.4 can be set.

[0039] This application divides personnel into four risk levels based on a comprehensive risk score to quickly identify dangerous conditions, and displays the risk distribution through a dynamic risk map to ensure timely emergency response.

[0040] The specific operations for classifying personnel into four risk levels are as follows: Generally, the country divides flood water warning into four levels from low to high, represented by blue, yellow, orange and red. In view of this, this application also divides people into four levels of low risk (RS<2), medium risk (2 ≤RS<4), high risk (4 ≤RS<6) and extremely high risk (RS ≥6) according to the comprehensive risk score calculated by the risk scoring model constructed above, as shown in the following table: The specific operations for generating the risk map are as follows: The risk level of each person is mapped to the spatial coordinate system, and the person's location is color-coded according to the comprehensive risk score. Green: low risk, yellow: medium risk, orange: high risk, red: extremely high risk. Furthermore, after receiving the video stream and sensor data in real time, the system calculates the risk score of each person and generates a real-time updated risk distribution map to intuitively display the risk status of different areas.

[0041] When high-risk or extremely high-risk behavior is detected, the system automatically triggers a warning signal and notifies the emergency response team to ensure timely rescue. At the same time, based on the distribution of the risk map, the deployment of rescue resources can be dynamically adjusted to give priority to high-risk areas and personnel, thereby improving rescue efficiency.

[0042] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A risk prediction method based on human action recognition in floods based on video key frames, characterized in that: The steps include: S100, extracting key frames of the flood monitoring video: using a key frame extraction algorithm based on content change, by analyzing the inter-frame differences between consecutive video frames, extracting frames with significant changes as key frames; S200, constructing a key frame image fast action recognition model; S210, using OpenPose key point detection technology to perform high-precision human key point detection on the extracted key frames to obtain 3D posture information of the person; S220, based on the detected key point information, the deep learning model Transformer is used to perform multi-category classification of personnel actions, so as to achieve accurate recognition of various personnel behaviors in the flood environment; S300, quantify risk level based on disaster feature identification; S310, single-width flow information acquisition and processing: Combine multi-source data fusion technology to obtain single-width flow information in flood areas in real time and pre-process the data; S320, Risk scoring model design: Design a comprehensive risk scoring model to combine single-width traffic information with identified personnel action characteristics to achieve multi-dimensional risk level quantification; S330, Risk level classification and early warning mechanism: Based on the comprehensive risk score, personnel are classified into multiple risk levels, and the risk distribution is displayed through a dynamic risk map to ensure timely emergency response.

2. According to claim 1, a risk prediction method based on human action recognition in floods using video key frames is characterized in that: Step S100 specifically includes the following steps: S110, video frame reading: reading image data frame by frame from the flood monitoring video, processing in chronological order, and training the first N frames of the video to establish a background model; S120, background difference: applying the background difference method to each frame image, calculating the difference between the current frame and the background model, and generating a foreground mask; S130, feature extraction: extracting features of the foreground area, including color features, texture features, and shape features; S140, content transformation analysis: combining the color, texture, and shape features extracted from each frame to form a feature vector, calculating the variation of the feature vectors between adjacent frames, and identifying frames with significant feature changes; S150, cosine similarity calculation: using cosine similarity to measure the similarity between adjacent frames; S160, K-means clustering: take all frames as input and apply K-means clustering algorithm to group them; S170, key frame selection: select a frame near the cluster centroid as a key frame; S180, key frame storage and management: the extracted key frames are stored in chronological order and indexed.

3. The risk prediction method based on human action recognition in flood in video key frames according to claim 2 is characterized in that: The extracted key frames are preprocessed, including denoising, image enhancement, cropping and scaling.

4. The risk prediction method based on human action recognition in flood in video key frames according to claim 3 is characterized in that: Step S210 specifically includes the following steps: S211, key point detection: Use the OpenPose model to detect the key points of the human body on the preprocessed key frames to obtain the three-dimensional coordinates of the key joints of the human body; S212, Posture Optimization: The detected 3D joint positions are distributed modeled through normalized flow technology to improve the accuracy and robustness of posture estimation.

5. The risk prediction method based on human action recognition in flood in video key frames according to claim 4 is characterized in that: Step S220 specifically includes the following steps: S221, temporal feature extraction: input the key point coordinate sequence in the continuous key frames into the Transformer model to extract the temporal dynamic features of the action; S222, action classification: classify the extracted features through a multi-category classifier, identify the current behavior type of the person, and output the corresponding behavior category probability distribution.

6. The risk prediction method based on human action recognition in flood in video key frames according to claim 1 is characterized in that: The specific method for obtaining the single-width flow information in step S310 is as follows: using the deployed flow velocity sensor and water depth sensor to monitor the flow velocity and water depth of the flood, and using the flood flow velocity and water depth obtained by media video data to supplement the sensor measurement results, integrating the water depth and flow velocity information through multi-source data, and calculating the single-width flow Q=h×u, where h is the real-time water depth in meters and u is the water flow velocity in meters per second.

7. The risk prediction method based on human action recognition in flood in video key frames according to claim 6 is characterized in that: The preprocessing of single-width flow-related data includes data denoising and correction, time series smoothing, synchronization and calibration.

8. The risk prediction method based on human action recognition in flood in video key frames according to claim 1 is characterized in that: Step S320 specifically includes the following steps: S321. According to the size of the single-width traffic, the single-width traffic is divided into four dynamic levels: low traffic, medium traffic, high traffic, and extremely high traffic, and a traffic risk score is set according to the dynamic level of the single-width traffic; S322. Classify the behaviors into four levels of low risk, medium risk, high risk and extremely high risk according to the identified behavior types and frequencies, and set corresponding action risk scores for each risk action; S323. Combine environmental information with personnel action characteristics to quantify the personnel risk level in flood scenarios and establish a comprehensive risk scoring formula: RS = α × E + β × A, where RS is the comprehensive risk score, E is the environmental hazard coefficient, A is the action hazard coefficient, and α and β are weight coefficients.

9. The risk prediction method based on human action recognition in flood in video key frames according to claim 8 is characterized in that: Step S330 specifically includes the following steps: S331. Personnel risk level classification: Based on the comprehensive risk score, personnel are classified into four levels: low risk, medium risk, high risk and extremely high risk; S332. Risk map generation: Map the risk level of each person into a spatial coordinate system and color-code the person's location based on the comprehensive risk score.

Images