Method and system for intelligently judging state of specific person based on space-time attitude graph and scene fusion

Abstract

The invention discloses a specific person state intelligent discrimination method and system based on space-time attitude graph and scene fusion, and belongs to the technical field of artificial intelligence and computer vision. The method comprises the following steps: using a multi-modal sensing module to parallelly obtain a human body key point coordinate sequence and a scene semantic graph from a video sequence; constructing a space-time attitude graph according to the key point sequence; inputting the space-time attitude graph and the scene semantic graph into a scene perception space-time graph attention network (SA-STGAN) model, extracting dynamic attitude features by using a graph convolutional network, and performing weighted fusion through a scene attention mechanism to obtain fusion features; and finally, a risk assessment and decision module is used to output a multi-dimensional state assessment vector according to the fusion features, and hierarchical response is executed. According to the method, the human body dynamic posture and the scene context information are deeply fused, so that dangerous behaviors and daily actions can be effectively distinguished, the false alarm rate and the missing report rate are remarkably reduced, the function spanning from post detection to pre-judgment is realized, and the method has high accuracy and environmental robustness.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically relating to an intelligent monitoring and behavior recognition method and system based on video analysis, and more particularly to an intelligent discrimination method and system for specific personnel states. Background Technology

[0002] With the accelerating aging of society and the increasing demand for care for special groups, the use of automation technology for safety monitoring of specific individuals such as the elderly, children, and people with mobility impairments has become an important research direction. Behavior recognition technology based on computer vision, especially fall detection, is one of the core applications in this field.

[0003] However, existing technologies still face many challenges and shortcomings in automatically identifying the status of specific individuals, especially in real and complex home or public environments. These mainly include: 1) High false alarm rate: Traditional image-based or simple pose recognition methods lack an understanding of the scene context. Therefore, normal behaviors such as lying down on a bed or sofa, or bending over to pick up an item are often misjudged as dangerous situations such as falling. Frequent false alarms seriously affect the usability of the system. 2) High false negative rate: Most existing technologies rely on analyzing the pose of a single frame image or detecting violent movements within a short period of time. For some non-sudden dangerous situations, such as slowly sliding off a chair, gradually collapsing due to exhaustion, or lingering in a specific area (such as a bathroom) for a long time, existing technologies are difficult to effectively identify, posing serious safety hazards; 3) Lack of risk prediction capability: Most existing technical solutions are "reactive," meaning they can only detect danger after a dangerous event (such as a fall) has occurred. They cannot assess risks and provide early warnings for some warning signs, such as unsteady gait, abnormal body swaying, or loitering in dangerous areas (such as stairwells), thus missing the best opportunity for proactive intervention and prevention of injury. 4) Poor environmental robustness: In real-world scenarios, factors such as changes in lighting, object occlusion, and changes in camera perspective can severely interfere with the performance of recognition algorithms. Existing methods are typically quite sensitive to these environmental changes, resulting in poor generalization ability of the model across different scenarios and unstable recognition results. Summary of the Invention

[0004] To address the problems of high false alarm rates due to lack of scene understanding, high false negative rates due to reliance on static analysis, lack of risk prediction capabilities, and poor environmental robustness in existing technologies, the present invention aims to provide a method and system for intelligent identification of specific personnel states based on spatiotemporal attitude diagrams and scene fusion.

[0005] The technical solution adopted in this invention is as follows: A method for intelligently identifying the state of a specific person based on spatiotemporal pose map and scene fusion includes the following steps: a) Using a multimodal perception module, a video sequence is acquired, and for each frame of the video sequence, human keypoint detection and scene semantic segmentation are performed in parallel to obtain a sequence of human keypoint coordinates for N consecutive frames and a scene semantic map corresponding to each frame. b) Using the spatiotemporal pose graph construction module, a spatiotemporal pose graph is constructed based on the sequence of human key point coordinates of the consecutive N frames. The nodes of the spatiotemporal pose graph are human key points, and the edges include intra-frame spatial edges that connect key points within the same frame to form the human skeleton, and inter-frame temporal edges that connect corresponding key points between consecutive frames to represent the motion flow. c) Using a graph neural network and attention fusion module, the spatiotemporal pose map and the scene semantic map are input into a preset scene-aware spatiotemporal graph attention network (SA-STGAN) model to perform fusion analysis on the state of the specific person and obtain fusion features; d) Using the risk assessment and decision-making module, based on the fusion features, output a multi-dimensional status assessment vector for the specific person, wherein the assessment vector includes at least the status category and risk level.

[0006] Furthermore, the scene-aware spatiotemporal graph attention network (SA-STGAN) model includes a dynamic pose feature extraction module built on a spatiotemporal graph convolutional network (ST-GCN) and a pose-scene fusion module built on a scene attention mechanism (SAM) connected in sequence.

[0007] Furthermore, the human keypoint detection in step a) is implemented based on the YOLOv7-pose algorithm model, and the scene semantic segmentation is implemented based on the Fast-SCNN network model.

[0008] Further, in step c), the specific process of the graph neural network and the attention fusion module performing fusion analysis includes: using the dynamic posture feature extraction module to calculate the spatiotemporal posture map and extract high-dimensional dynamic posture features containing information on human movement speed, acceleration, and joint angle changes; using the posture scene fusion module to calculate the relative spatial relationship between the key points of the human body and preset key semantic regions in the scene, such as "floor," "bed," or "stairs," based on the pixel-level positions of the key points of the human body in the scene semantic map, and generating attention weights accordingly; the posture scene fusion module uses the attention weights to weight the high-dimensional dynamic posture features to enhance features related to dangerous scene interactions and suppress features related to safe scene interactions, ultimately obtaining the fused features.

[0009] Further, in step d), the multi-dimensional state assessment vector specifically includes: a state category vector for representing the current state category, a continuous risk level value in the interval [0,1] for representing the degree of danger, and a confidence score for representing the credibility of the judgment result.

[0010] Furthermore, the state categories that the state category vector can represent include at least: normal walking, standing, lying down, bending over, slowly falling, falling rapidly, and abnormal lingering.

[0011] Furthermore, the risk assessment and decision-making module in step d) is also configured to execute an emergency alarm action when the risk level value exceeds a first preset threshold; and to execute a warning notification action when the risk level value is between a second preset threshold and a first preset threshold, which are both lower than the first preset threshold.

[0012] A specific personnel state intelligent discrimination system is used to implement the method described in any one of claims 1-7, characterized in that: the system includes a multimodal perception unit, a spatiotemporal attitude map construction unit, a graph neural network and attention fusion unit, and a risk assessment and decision-making unit connected in sequence.

[0013] The beneficial effects of this invention are as follows: This invention discloses a method and system for intelligent identification of specific personnel states based on spatiotemporal posture map and scene fusion. By introducing scene semantic segmentation and using an attention mechanism to perform weighted fusion of dynamic posture features, it achieves a deep understanding of human behavior and its environmental context, effectively distinguishing scene-related behaviors such as "lying in bed" and "falling on the floor," significantly reducing the false alarm rate. By constructing a spatiotemporal posture map and using graph convolutional networks for analysis, static posture recognition is extended to dynamic behavior pattern recognition, effectively capturing the temporal features of non-sudden events such as slow slips and staggers, greatly reducing the false alarm rate. By outputting a multi-dimensional evaluation vector containing continuous risk level values, it achieves quantitative assessment and early warning of precursory behaviors such as loitering and abnormally approaching dangerous areas, completing the functional leap from "post-event detection" to "pre-event prediction," and improving the initiative of monitoring. The posture modeling based on graph structure has natural robustness to interference such as changes in viewpoint and slight occlusion, ensuring the stability and high generalization ability of the algorithm in complex real-world environments.

[0014] Other beneficial effects of the present invention will be further explained in the specific embodiments. Attached Figure Description

[0015] Figure 1 This is a flowchart of the intelligent identification method for the specific status of personnel in this invention.

[0016] Figure 2 This is a structural block diagram of the intelligent identification system for specific personnel status in this invention.

[0017] Figure 3 This is a schematic diagram of the spatiotemporal attitude diagram construction in this invention.

[0018] Figure 4 This is a schematic diagram of the scene-aware spatiotemporal graph attention network (SA-STGAN) model structure in this invention. Detailed Implementation

[0019] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1: like Figure 1 As shown, this embodiment provides a method for intelligently identifying the state of a specific person based on the fusion of spatiotemporal attitude diagram and scene, including the following steps: S1: Build and deploy the algorithm model. This step uses artificial intelligence algorithms to build and deploy a multimodal perception module, a spatiotemporal pose graph construction module, a graph neural network and attention fusion module, and a risk assessment and decision-making module, specifically including: S1-1: A multimodal perception module is constructed using artificial intelligence algorithms. This module integrates the YOLOv7-pose algorithm model for human keypoint detection and the Fast-SCNN network model for scene semantic segmentation. The YOLOv7-pose model, as a current mainstream joint model for object detection and keypoint estimation, can efficiently complete the two tasks of human body localization and pose estimation in a single network. It has the advantages of high speed and high accuracy and is suitable for real-time monitoring scenarios. The Fast-SCNN model is a lightweight network designed for real-time semantic segmentation of high-resolution images. It adopts an encoder-decoder structure of "learning-downsampling-upsampling", which can maintain high segmentation accuracy while significantly reducing the amount of computation, thus meeting the requirements of this invention for real-time parsing of scene context. S1-2: Use artificial intelligence algorithms to build a graph neural network and attention fusion module. The core of this module is the scene-aware spatiotemporal graph attention network (SA-STGAN) model. This model includes a dynamic pose feature extraction module built on spatiotemporal graph convolutional network (ST-GCN) and a pose scene fusion module built on scene attention mechanism (SAM) connected in sequence. ST-GCN extends traditional convolutional networks to graph-structured data. By performing convolution operations simultaneously in the spatial domain (human skeleton connections) and temporal domain (inter-frame keypoint trajectories) of the spatiotemporal pose graph, it excels at capturing complex spatiotemporal dependencies in human movements. Compared to methods such as RNN or CNN, it can learn structured dynamic patterns more effectively. The SAM mechanism draws on the characteristic of the human visual system to prioritize important regions. By assigning different attention weights to pose features in different scenes, the model can focus on the most critical information for the discrimination task. For example, when the "head" keypoint is close to the "floor," its related dynamic features will be given higher importance, thereby achieving a high degree of information filtering and refinement. S2: Acquire and preprocess data. Based on the real-time video sequence, a multimodal perception module is used to acquire and preprocess data, obtaining a sequence of human keypoint coordinates and a scene semantic map for N consecutive frames. Specifically, this includes: S2-1: Acquire video streams in real time through cameras deployed in the monitoring scene, and use N consecutive frames (e.g., N=32) of images as input; S2-2: Use the YOLOv7-pose model in the multimodal perception module to process each frame and output the two-dimensional coordinates (x,y) and confidence level c of 17 key points of the human body (such as head, neck, shoulders, etc.) in the frame. S2-3: Use the Fast-SCNN model in the multimodal perception module to process each frame and output a pixel-level scene semantic map. The pixel value in the map represents the scene category of that point (such as floor, bed, chair, etc.). S3: Construct the spatiotemporal pose graph. Based on the sequence of human keypoint coordinates across N consecutive frames, use the spatiotemporal pose graph construction module to construct the spatiotemporal pose graph G=(V,E), specifically including: S3-1: Node V of the constructed graph, which consists of all valid key points in N frames with a confidence level higher than a preset threshold; S3-2: Construct the edge E of the graph, the edge includes: a) intra-frame spatial edge connecting adjacent key points in the same frame according to the human skeleton structure; b) inter-frame temporal edge connecting the same key point in two adjacent frames (such as the left shoulder in frame t and the left shoulder in frame t+1). S4: Extract high-dimensional dynamic pose features. The spatiotemporal pose map G is input into the dynamic pose feature extraction module of the SA-STGAN model, and high-dimensional dynamic pose features are obtained through multi-layer ST-GCN calculation; S5: Feature weighting by fusing scene information. This involves combining high-dimensional dynamic pose features with corresponding scene semantics. Figure 1 The input is fed into the pose scene fusion module, where the attention weights are calculated using the SAM mechanism, and the dynamic pose features are weighted to obtain the final fused features. S6: Perform state discrimination. Input the fused features into the preset classifier and regressor, and output a multi-dimensional state evaluation vector, which includes a state category vector, a risk level value in the [0,1] interval, and a confidence score; S7: Execute tiered decision-making and response. Based on the comparison results between the risk level value and the preset first and second thresholds, execute tiered responses such as emergency alarms or early warning notifications.

[0021] Example 2: like Figure 2 As shown, this embodiment provides a specific personnel state intelligent discrimination system to implement the above method. The system includes a multimodal perception unit, a spatiotemporal attitude graph construction unit, a graph neural network and attention fusion unit, and a risk assessment and decision-making unit connected in sequence. Model building unit, used for offline training and generation of YOLOv7-pose model, Fast-SCNN model and SA-STGAN model; The multimodal perception unit is used to acquire and preprocess data based on real-time video sequences by calling the YOLOv7-pose model and the Fast-SCNN model, and obtain a sequence of human keypoint coordinates and scene semantic map for N consecutive frames. The spatiotemporal pose graph construction unit is used to construct a spatiotemporal pose graph based on the sequence of human keypoint coordinates of N consecutive frames. The graph neural network and attention fusion unit is used to call the SA-STGAN model based on the spatiotemporal pose map and scene semantic map to perform feature fusion and analysis to obtain fused features; The risk assessment and decision-making unit is used to determine the state and output based on the fusion characteristics.

[0022] This invention is not limited to the optional embodiments described above, and anyone can derive other various forms of products based on the inspiration of this invention. The specific embodiments described above should not be construed as limiting the scope of protection of this invention; the scope of protection of this invention should be determined by the claims, and the specification can be used to interpret the claims.

Claims

1. A specific person state intelligent discrimination method based on spatiotemporal pose graph and scene fusion, characterized in that, The method comprises the following steps: a) using a multi-modal perception module, acquiring a video sequence, and performing human key point detection and scene semantic segmentation on each frame image in the video sequence in parallel to obtain a human key point coordinate sequence of consecutive N frames and a scene semantic map corresponding to each frame, respectively; b) using a spatio-temporal pose graph construction module, constructing a spatio-temporal pose graph according to the human key point coordinate sequence of consecutive N frames, wherein the nodes of the spatio-temporal pose graph are human key points, and the edges include intra-frame spatial edges connecting key points in the same frame to form a human skeleton, and inter-frame temporal edges connecting corresponding key points between consecutive frames to represent motion flow; c) using a graph neural network and attention fusion module, inputting the spatio-temporal pose graph and the scene semantic map into a preset scene-aware spatio-temporal graph attention network (SA-STGAN) model to perform fusion analysis on the state of the specific person to obtain a fusion feature; d) using a risk assessment and decision-making module, outputting a multi-dimensional state evaluation vector of the specific person according to the fusion feature, wherein the evaluation vector at least includes a state category and a risk level.

2. The method of claim 1, wherein: The scene-aware spatio-temporal graph attention network (SA-STGAN) model comprises a dynamic pose feature extraction module constructed based on a spatio-temporal graph convolution network (ST-GCN) and a pose-scene fusion module constructed based on a scene attention mechanism (SAM) connected in sequence.

3. The method of claim 1, wherein: The human key point detection in step a) is realized based on a YOLOv7-pose algorithm model, and the scene semantic segmentation is realized based on a Fast-SCNN network model.

4. The method of claim 2, wherein: In step c), the specific process of fusion analysis by the graph neural network and attention fusion module includes: using the dynamic pose feature extraction module to calculate the spatio-temporal pose graph and extract high-dimensional dynamic pose features containing human motion speed, acceleration, and joint angle change information; using the pose-scene fusion module to calculate the relative spatial relationship between the key points and the preset key semantic regions such as "floor", "bed", or "staircase" in the scene according to the pixel-level position of the human key points in the scene semantic map, and generating attention weights accordingly; the pose-scene fusion module uses the attention weights to weight the high-dimensional dynamic pose features to enhance the features related to dangerous scene interaction and suppress the features related to safe scene interaction, and finally obtains the fusion feature.

5. The method of claim 1, wherein: In step d), the multi-dimensional state evaluation vector specifically includes: a state category vector for representing the current state category, a continuous risk level value in the [0, 1] interval for representing the danger degree, and a confidence score for representing the credibility of the discrimination result.

6. The method of claim 5, wherein: The state category vector can represent at least the following state categories: normal walking, standing, lying, bending, slow falling, rapid falling, and abnormal staying.

7. The method of claim 5, wherein: The risk assessment and decision module in the step d) is further configured to perform an emergency alarm action when the risk level value exceeds a first preset threshold, and perform a pre-warning notification action when the risk level value is between a second preset threshold lower than the first preset threshold and the first preset threshold.

8. A specific person state intelligent discrimination system characterized by, The method comprises the following steps: a multi-modal perception unit; a spatio-temporal pose graph construction unit; a graph neural network and attention fusion unit; a risk assessment and decision unit.

9. The system of claim 8, wherein: The graph neural network and attention fusion unit specifically comprises: a dynamic pose feature extraction module constructed based on a spatio-temporal graph convolution network (ST-GCN); a pose scene fusion module constructed based on a scene attention mechanism (SAM).

10. A method for training a scene-aware spatio-temporal graph attention network (SA-STGAN) model as claimed in claim 2, characterized in that, The method comprises: a) constructing a training data set, wherein the data set contains a large number of video clips, each video clip is labeled with a continuous human key point coordinate sequence, a corresponding scene semantic graph, and a specific personnel state label corresponding to the video clip; b) taking the key point coordinate sequence and the scene semantic graph in the training data set as input, and taking the state label as expected output; c) using an end-to-end training method, iteratively optimizing the network parameters in the SA-STGAN model by minimizing the loss function between the predicted output of the model and the expected output until the loss function converges.

Images