A human pose estimation and behavior analysis method

Abstract

The application discloses a human posture estimation and behavior analysis method, comprising the following steps: S1, collecting visible light images and thermal imaging images, performing feature extraction and fusion, and generating fused semantic features; S2, obtaining a refined key point heat map and a partial affinity field according to the fused semantic features; S3, generating a key point set according to the refined key point heat map and the partial affinity field; and S4, determining an abnormal behavior according to the key point set. The application effectively overcomes the shortcomings of large pixel-level fusion calculation overhead and difficult alignment of shallow feature fusion modalities, and significantly improves the robustness and precision of human posture estimation under complex lighting conditions such as day-night alternation and backlight.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for human pose estimation and behavior analysis. Background Technology

[0002] Human pose estimation aims to locate human joints from images and construct skeletal models, serving as the foundation for applications such as behavior analysis and intelligent surveillance. With the development of deep learning, pose estimation based on visible light (RGB) images has achieved high accuracy under controlled lighting conditions, but its performance is heavily dependent on illumination. In real-world scenarios with low light levels or drastic lighting changes, such as at night, in garages, or against the light, RGB image quality degrades, texture details are lost, leading to keypoint detection failures.

[0003] Thermal imaging (infrared, IR) technology offers a potential solution. Thermal imaging sensors capture images by imprinting the thermal radiation emitted by objects, independent of ambient light sources. Therefore, they can form high-contrast human silhouettes even in complete darkness, ensuring the detection of target presence. However, thermal imaging has inherent limitations such as low spatial resolution and a lack of textural semantic information (e.g., facial and clothing details). More importantly, its imaging is susceptible to interference from environmental heat sources (e.g., heaters, electrical appliances, vehicle engines), which may lead to blurred human silhouettes or false detections, directly affecting the accuracy of keypoint localization.

[0004] Therefore, fusing complementary information from RGB and IR modalities has become a key technical approach to improve the robustness of pose estimation in all-weather environments. Existing fusion methods are mainly divided into two categories: image-level fusion and feature-level fusion. Image-level fusion aims to generate a new image that combines visible light details with infrared saliency. Common methods include multi-scale transformation, sparse representation, and methods based on generative adversarial networks (GANs). However, these methods are computationally complex, and the fused image may introduce artifacts, which can interfere with subsequent pose estimation tasks. Feature-level fusion typically involves concatenating or weighting the feature maps of the two modalities in the early or mid-stage layers of a convolutional neural network. However, due to the fundamental differences in the underlying pixel distribution between RGB and IR images (reflected light intensity vs. thermal radiation intensity), this early fusion struggles to achieve effective feature alignment and complementarity. The network requires additional capacity to learn these modal differences, resulting in low efficiency.

[0005] Furthermore, after obtaining the human skeleton, understanding the semantics of behavior is crucial for completing the intelligent perception loop. Existing methods mostly rely on deep learning models (such as 3D convolutional networks and spatiotemporal graph convolutional networks) trained on large-scale labeled data for action recognition. These models have a large number of parameters and are computationally complex, making it difficult to achieve real-time analysis on edge devices with limited computing power, thus limiting their practicality.

[0006] In summary, existing technologies have the following shortcomings: First, there is a lack of a pose estimation method that can operate stably under complex lighting conditions and efficiently utilize the complementary advantages of dual modes; second, there is a lack of a method that can achieve real-time behavior analysis based on skeleton data with extremely low computational overhead. This hinders the deployment and application of intelligent sensing technology in real-world all-weather scenarios, especially in resource-constrained environments. Summary of the Invention

[0007] To address the problems of poor robustness of existing single-modal human pose estimation methods under complex lighting conditions; the inefficiency of existing multimodal fusion methods, which are mostly computationally intensive image-level fusions or early feature fusions that are difficult to fully complement; and the computational complexity of existing deep learning-based behavior recognition methods, which makes it difficult to meet the requirements of real-time edge analysis, this invention proposes a human pose estimation and behavior analysis method.

[0008] The technical solution of the present invention is: a method for human posture estimation and behavior analysis, comprising the following steps:

[0009] S1. Acquire visible light images and thermal imaging images, extract and fuse features, and generate fused semantic features;

[0010] S2. Based on the fused semantic features, a refined key point heatmap and partial affinity fields are obtained;

[0011] S3. Generate a set of key points based on the refined key point heatmap and some affinity fields;

[0012] S4. Determine abnormal behavior based on the set of key points.

[0013] Furthermore, S1 includes the following sub-steps:

[0014] S11. Acquire visible light images and thermal imaging images, and input them into the first feature extraction network and the second feature extraction network respectively to generate preliminary key point heat maps and partial affinity fields for the RGB mode and the IR mode.

[0015] S12. Based on the preliminary key point heatmaps of the RGB mode and the preliminary key point heatmaps of the IR mode, generate the corresponding saliency energy maps;

[0016] S13. Based on the saliency energy maps corresponding to the preliminary key point heatmaps of the RGB mode and the preliminary key point heatmaps of the IR mode, generate pixel-level adaptive fusion weights for the heatmaps.

[0017] S14. Based on the pixel-level adaptive fusion weights of the heatmap, perform weighted fusion to obtain the fused heatmap features;

[0018] S15. Generate the corresponding saliency energy maps based on the partial affinity fields of the RGB modes and the partial affinity fields of the IR modes;

[0019] S16. Based on the saliency energy maps corresponding to the partial affinity fields of the RGB mode and the partial affinity fields of the IR mode, generate pixel-level adaptive fusion weights for the partial affinity fields.

[0020] S17. Perform weighted fusion based on the pixel-level adaptive fusion weights of the partial affinity field to obtain the fused partial affinity field features;

[0021] S18. Based on the fused heatmap features and the fused partial affinity field features, the fused semantic features are obtained.

[0022] Furthermore, in S11, a preliminary key point heatmap of the RGB mode. With some friendly scenes The expressions are as follows:

[0023] ;

[0024] ;

[0025] in, Represents a visible light image. The initial key point heatmap mapping function representing the RGB modes. The initial partial affinity field mapping function representing the RGB modes;

[0026] Preliminary key point heatmap of IR mode in S11 With some friendly scenes The expressions are as follows:

[0027] ;

[0028] ;

[0029] in, Represents a visible light image. The initial key point heatmap mapping function representing the RGB modes. The initial partial affinity field mapping function representing the RGB modes;

[0030] In S12, the saliency energy map corresponding to the preliminary key point heatmap of the RGB mode. The expression is:

[0031] ;

[0032] in, Indicates the total number of key points. Represents the x-coordinate on the feature map. Represents the ordinate on the feature map. Indicates the key point number;

[0033] In S12, the saliency energy map corresponding to the preliminary key point heatmap of the IR mode. The expression is:

[0034] ;

[0035] In S13, pixel-level adaptive fusion weights of the heatmap The expression is:

[0036] ;

[0037] in, Indicates a very small quantity;

[0038] In S14, the fused heatmap features The expression is:

[0039] ;

[0040] In S15, the saliency energy map corresponding to the partial affinity fields of the RGB modes. The expression is:

[0041] ;

[0042] in, Indicates the number of categories connected by the skeleton. Channel number representing a partial affinity field feature map;

[0043] In S15, the saliency energy map corresponding to the partial affinity fields of the IR modes. The expression is:

[0044] ;

[0045] In S16, pixel-level adaptive fusion weights for partial affinity fields. The expression is:

[0046] ;

[0047] In S17, the fused partial affinity field characteristics The expression is:

[0048] .

[0049] Furthermore, S2 includes the following sub-steps:

[0050] S21. Perform channel compression on the fused semantic features;

[0051] S22. Perform lightweight sensing based on the channel compression results;

[0052] S23. Generate spatial gating weights based on the lightweight perception results;

[0053] S24. Based on the channel compression results, lightweight sensing results, and spatial gating weights, output refined gating residuals;

[0054] S25. Based on the gated residual refinement, the refined key point heatmap and partial affinity field are obtained.

[0055] Furthermore, in S21, the expression for channel compression is:

[0056] ;

[0057] in, (•) represents a non-linear activation function. This represents the semantic features after fusion. This represents a 1×1 convolution operation. This indicates the channel compression result;

[0058] In S22, the expression for lightweight sensing is:

[0059] ;

[0060] in, Indicates the void ratio, This represents a 3×3 depthwise separable convolution with a hole ratio of 2. This indicates the result of lightweight perception;

[0061] In S23, spatial gating weights The expression is:

[0062] ;

[0063] in, Represents the Sigmoid function;

[0064] In S24, gated residual refinement The expression is:

[0065] ;

[0066] in, This represents element-wise multiplication;

[0067] In S25, the detailed key point heatmap The expression is:

[0068] ;

[0069] in, This represents a 1×1 convolution operation used for outputting or regressing a refined keypoint heatmap;

[0070] In S25, the refined affinity fields The expression is:

[0071] ;

[0072] in, This represents a 1×1 convolution operation used to output or regress a refined portion of the affinity field.

[0073] Furthermore, S3 includes the following sub-steps:

[0074] S31. Perform non-maximum suppression on the refined key point heatmap and part of the affinity field to obtain a set of local peaks, and remove elements with confidence scores less than or equal to a set threshold.

[0075] S32. Calculate the connectivity score based on the local peak set;

[0076] S33. Using the connection scores as edge weights, construct a bipartite graph and use the bipartite graph to generate the optimal connection set for each limb type.

[0077] S34. Generate a set of key points based on the optimal connection set for each limb type.

[0078] Furthermore, in S31, the set of local peaks The expression is:

[0079] ;

[0080] ;

[0081] ;

[0082] in, This represents the coordinates of the nth candidate point of the kth type of keypoint. This represents the confidence level of the nth candidate point of the k-th keypoint. This represents the x-coordinate of the nth candidate point of the k-th keypoint. This represents the ordinate of the nth candidate point of the k-th keypoint category. This indicates the characteristics of the merged heatmap;

[0083] In S32, the connection score The expression is:

[0084] ;

[0085] in, This represents the candidate point number of the starting key point in the limb connection. This indicates the candidate point number for the termination key point in a limb connection. Indicates the type of limb connection. This indicates the number of points sampled uniformly. Points representing uniform sampling, Indicates the unit direction. The vector representing the PAF vector field.

[0086] Furthermore, in S4, lightweight geometric computation is used to determine anomalous behavior.

[0087] The beneficial effects of this invention are:

[0088] (1) The adaptive fusion method based on high-level semantic representation proposed in this invention sets the complementary fusion of infrared and visible light information at the level of key point heat map and partial affinity field, and dynamically allocates fusion weights according to the saliency energy of dual-stream features. This method effectively overcomes the disadvantages of large computational overhead of pixel-level fusion and difficulty in mode alignment of shallow feature fusion, and realizes intelligent adaptation of "emphasizing RGB texture when the lighting is good and IR contour when the lighting is low", which significantly improves the robustness and accuracy of human posture estimation under complex lighting conditions such as day and night alternation and backlight.

[0089] (2) The lightweight abnormal behavior analysis method based on skeleton geometric features proposed in this invention relies entirely on the key point coordinates output by posture estimation. It makes judgments by calculating predefined geometric and temporal features (such as trunk velocity, aspect ratio, and joint motion variance) and applying threshold rules. This method avoids deploying spatiotemporal action recognition deep learning models with large parameters and complex calculations. It achieves real-time and accurate identification of abnormal behaviors such as falling and waving for help with extremely low computational cost. It is particularly suitable for edge computing scenarios with limited computing power and provides an efficient solution for a complete intelligent perception application closed loop. Attached Figure Description

[0090] Figure 1 A flowchart for human posture estimation and behavior analysis methods;

[0091] Figure 2 A schematic diagram of the structure for human posture estimation. Detailed Implementation

[0092] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0093] like Figure 1As shown, this invention provides a method for human posture estimation and behavior analysis, including the following steps:

[0094] S1. Acquire visible light images and thermal imaging images, extract and fuse features, and generate fused semantic features;

[0095] S2. Based on the fused semantic features, a refined key point heatmap and partial affinity fields are obtained;

[0096] S3. Generate a set of key points based on the refined key point heatmap and some affinity fields;

[0097] S4. Determine abnormal behavior based on the set of key points.

[0098] Simultaneously acquired visible light and thermal imaging images are input into a dual-stream network to extract features and generate preliminary keypoint heatmaps and partial affinity fields. Subsequently, at the semantic level, the saliency energy map output by the dual streams is calculated to generate adaptive weights, and the two feature streams are weighted and fused. The fused features are then refined by a lightweight module to obtain keypoints of the human skeleton. Skeleton analysis employs non-maximum suppression peak finding on the keypoint heatmap, combined with line integral consistency scoring of the partial affinity field and bipartite graph maximum weight matching to complete multi-person keypoint pairing and skeleton assembly. Finally, based on the geometric relationships and motion characteristics of the keypoints, abnormal behavior is analyzed in real time. This invention effectively improves the estimation robustness under complex lighting conditions and achieves efficient and accurate behavior understanding.

[0099] In this embodiment of the invention, S1 includes the following sub-steps:

[0100] S11. Acquire visible light images and thermal imaging images, and input them into the first feature extraction network and the second feature extraction network respectively to generate preliminary key point heat maps and partial affinity fields for the RGB mode and the IR mode.

[0101] S12. Based on the preliminary key point heatmaps of the RGB mode and the preliminary key point heatmaps of the IR mode, generate the corresponding saliency energy maps;

[0102] S13. Based on the saliency energy maps corresponding to the preliminary key point heatmaps of the RGB mode and the preliminary key point heatmaps of the IR mode, generate pixel-level adaptive fusion weights for the heatmaps.

[0103] S14. Based on the pixel-level adaptive fusion weights of the heatmap, perform weighted fusion to obtain the fused heatmap features;

[0104] S15. Generate the corresponding saliency energy maps based on the partial affinity fields of the RGB modes and the partial affinity fields of the IR modes;

[0105] S16. Based on the saliency energy maps corresponding to the partial affinity fields of the RGB mode and the partial affinity fields of the IR mode, generate pixel-level adaptive fusion weights for the partial affinity fields.

[0106] S17. Perform weighted fusion based on the pixel-level adaptive fusion weights of the partial affinity field to obtain the fused partial affinity field features;

[0107] S18. Based on the fused heatmap features and the fused partial affinity field features, the fused semantic features are obtained.

[0108] First, key points of the human skeleton are robustly estimated from RGB and thermal imaging images using a dual-stream adaptive fusion network. Second, based on the estimated key point coordinates, anomalous behavior is analyzed in real time using lightweight geometric and temporal rules.

[0109] A pose estimation method based on dual-stream adaptive fusion is proposed. The input to this method is a time-synchronized visible light (RGB) image and a thermal imaging (IR) image. First, a dual-stream processing architecture is employed: the RGB image is input to a first feature extraction network, and the IR image is input to a second feature extraction network with the same structure but independent parameters. The two networks extract features for their respective modalities in parallel. Subsequently, the output of each feature extraction network is fed into a pose prediction branch, which regresses and generates preliminary keypoint heatmaps and partial affinity fields for the RGB modality, and preliminary keypoint heatmaps and partial affinity fields for the IR modality, respectively. The heatmaps are used to characterize the positional probability distribution of keypoints, and the partial affinity fields are used to encode directional information about limb connections.

[0110] The core of this method is semantic layer adaptive fusion. The fusion operation is not performed on the low-level convolutional features, but rather on the high-level semantic space of the initially generated keypoint heatmap and a portion of the affinity field. Let the bimodal input be... Two structurally identical but parameter-independent feature extraction networks output two types of pose semantic tensors respectively. For the two features to be fused (taking a heatmap as an example), the channel dimension is first calculated. The norm is used to obtain the saliency energy map; then, pixel-level adaptive fusion weights are generated based on the relative proportions of the two modal energies, and weighted fusion is completed accordingly. For some affinity fields... They execute the exact same mechanism.

[0111] This mechanism allows for operation under good lighting conditions. Larger, blended results emphasize RGB texture details; in low light or complete darkness... The dominant and fusion results focus on the stable profile of the IR, thereby achieving adaptive complementarity at the semantic level.

[0112] In this embodiment of the invention, in S11, the preliminary key point heatmap of the RGB mode is... With some friendly scenes The expressions are as follows:

[0113] ;

[0114] ;

[0115] in, Represents a visible light image. The initial key point heatmap mapping function representing the RGB modes. The initial partial affinity field mapping function representing the RGB modes;

[0116] Preliminary key point heatmap of IR mode in S11 With some friendly scenes The expressions are as follows:

[0117] ;

[0118] ;

[0119] in, Represents a visible light image. The initial key point heatmap mapping function representing the RGB modes. The initial partial affinity field mapping function representing the RGB modes;

[0120] In S12, the saliency energy map corresponding to the preliminary key point heatmap of the RGB mode. The expression is:

[0121] ;

[0122] in, Indicates the total number of key points. Represents the x-coordinate on the feature map. Represents the ordinate on the feature map. Indicates the key point number;

[0123] In S12, the saliency energy map corresponding to the preliminary key point heatmap of the IR mode. The expression is:

[0124] ;

[0125] In S13, pixel-level adaptive fusion weights of the heatmap The expression is:

[0126] ;

[0127] in, Indicates a very small quantity;

[0128] To avoid division by zero, a very small amount is introduced. .

[0129] In S14, the fused heatmap features The expression is:

[0130] ;

[0131] In S15, the saliency energy map corresponding to the partial affinity fields of the RGB modes. The expression is:

[0132] ;

[0133] in, Indicates the number of categories connected by the skeleton. Channel number representing a partial affinity field feature map;

[0134] In S15, the saliency energy map corresponding to the partial affinity fields of the IR modes. The expression is:

[0135] ;

[0136] In S16, pixel-level adaptive fusion weights for partial affinity fields. The expression is:

[0137] ;

[0138] In S17, the fused partial affinity field characteristics The expression is:

[0139] .

[0140] In this embodiment of the invention, the fused semantic features (heatmap and affinity field) are fed into a lightweight gated residual refinement network (GRH-DS) to further correct the consistency between keypoint responses and limb connections. Let the fused input features be... (in or The refined network is computed along the path of "compression-sensing-gating-residual". By using gating terms, the refined network enhances the response in key point regions and suppresses noise in background regions, thereby improving spatial consistency and robustness.

[0141] S2 includes the following sub-steps:

[0142] S21. Perform channel compression on the fused semantic features;

[0143] S22. Perform lightweight sensing based on the channel compression results;

[0144] S23. Generate spatial gating weights based on the lightweight perception results;

[0145] S24. Based on the channel compression results, lightweight sensing results, and spatial gating weights, output refined gating residuals;

[0146] S25. Based on the gated residual refinement, the refined key point heatmap and partial affinity field are obtained.

[0147] In this embodiment of the invention, the expression for channel compression in S21 is:

[0148] ;

[0149] in, (•) represents a non-linear activation function. This represents the semantic features after fusion. This represents a 1×1 convolution operation. This indicates the channel compression result;

[0150] In S22, the expression for lightweight sensing is:

[0151] ;

[0152] in, Indicates the void ratio, This represents a 3×3 depthwise separable convolution with a hole ratio of 2. This indicates the result of lightweight perception;

[0153] In S23, spatial gating weights The expression is:

[0154] ;

[0155] in, Represents the Sigmoid function;

[0156] In S24, gated residual refinement The expression is:

[0157] ;

[0158] in, This represents element-wise multiplication;

[0159] In S25, the detailed key point heatmap The expression is:

[0160] ;

[0161] in, This represents a 1×1 convolution operation used for outputting or regressing a refined keypoint heatmap;

[0162] In S25, the refined affinity fields The expression is:

[0163] ;

[0164] in, This represents a 1×1 convolution operation used to output or regress a refined portion of the affinity field.

[0165] In this embodiment of the invention, S3 includes the following sub-steps:

[0166] S31. Perform non-maximum suppression on the refined key point heatmap and part of the affinity field to obtain a set of local peaks, and remove elements with confidence scores less than or equal to a set threshold.

[0167] S32. Calculate the connectivity score based on the local peak set;

[0168] S33. Using the connection scores as edge weights, construct a bipartite graph and use the bipartite graph to generate the optimal connection set for each limb type.

[0169] S34. Generate a set of key points based on the optimal connection set for each limb type.

[0170] In this embodiment of the invention, in S31, the local peak set The expression is:

[0171] ;

[0172] ;

[0173] ;

[0174] in, This represents the coordinates of the nth candidate point of the kth type of keypoint. This represents the confidence level of the nth candidate point of the k-th keypoint. This represents the x-coordinate of the nth candidate point of the k-th keypoint. This represents the ordinate of the nth candidate point of the k-th keypoint category. This indicates the characteristics of the merged heatmap;

[0175] In S32, the connection score The expression is:

[0176] ;

[0177] in, This represents the candidate point number of the starting key point in the limb connection. This indicates the candidate point number for the termination key point in a limb connection. Indicates the type of limb connection. This indicates the number of points sampled uniformly. Points representing uniform sampling, Indicates the unit direction. The vector representing the PAF vector field.

[0178] In this embodiment of the invention, a multi-person skeleton analysis algorithm based on "heatmap peak detection + PAF line integral scoring + bipartite graph matching assembly" is adopted to analyze the refined skeleton. and The decoding process yields the set of coordinates of all human skeleton key points in the image. The specific steps are as follows:

[0179] Keypoint candidate extraction (NMS peak finding): For each keypoint type In the heat map Performing nonmaximum suppression on the top yields a set of local peaks. .in For the first Coordinates of candidate points Assess its confidence level; and remove [items]. Low-confidence candidates.

[0180] Limb Connection Candidate Generation (PAF Line Integral Consistency): For each limb connection (The connection type is predefined by the skeleton topology), for any candidate pair Calculate the connection vector Unit direction Uniform sampling along this line segment. Points Take a vector in the corresponding PAF vector field Define the connection score and require that the consistency constraint be satisfied: And it meets the "positive correlation sampling ratio" requirement. This is to filter out false connections caused by overlapping limbs.

[0181] Bipartite graph matching (optimal pairing for each limb): for each connection Construct a bipartite graph Edge rights take Under the constraint that "a point participates in this type of connection at most once", the optimal connection set for this limb type can be obtained by using maximum weight matching (such as the Hungarian algorithm or a greedy approximation). .

[0182] Skeleton Assembly and Instance Output: Matching limbs according to skeleton topology The process involves progressively merging instances into a set of skeletons for multiple people; performing consistent merging when different connections share the same keypoint candidate; and allowing partial skeleton retention for instances with missing keypoints. The final output is the keypoint set for each human body instance. .

[0183] A lightweight abnormal behavior analysis method based on skeleton geometry features. This invention takes the output sequence of keypoint coordinates of the human skeleton as input and achieves real-time behavioral semantic understanding by performing lightweight geometric calculations on a general-purpose processor. This method does not rely on complex deep learning models; its core is the design of a series of judgment rules based on the spatial relationships and motion states of keypoints.

[0184] Taking fall detection as an example: The method first calculates the vertical velocity of the center point of the human torso (usually obtained by averaging the center points of the neck and hips). Simultaneously, it calculates the ratio of the height to the width of the bounding rectangle of the human body (length-to-width ratio). When the system detects that the vertical descent velocity of the center point of the torso continuously exceeds a preset threshold, and the length-to-width ratio of the human body decreases significantly within a short period (indicating a change in posture from upright to lying down), a fall event is determined to have occurred.

[0185] Taking the detection of waving for help as an example: The method continuously monitors whether the wrist key point is located above the head key point. At the same time, within a sliding time window, the variance of the horizontal motion coordinates of the wrist key point is calculated. When both conditions are met, namely "the wrist is higher than the head" and "the horizontal motion variance continuously exceeds the threshold for a certain period of time", it is determined to be a waving for help behavior.

[0186] By combining rules based on geometric and kinematic features such as distance, angle, speed, and proportion, this method can identify a variety of preset abnormal behaviors in real time and accurately with extremely low computational cost, providing direct evidence for monitoring and early warning.

[0187] In this embodiment of the invention, in S4, lightweight geometric calculations are used to determine abnormal behavior.

[0188] The complete implementation process of the method of the present invention is as follows: Figure 2 As shown, the detailed network architecture of its core fusion pose estimation step is as follows: Figure 2 As shown.

[0189] The preprocessed image pairs then proceed to the fusion pose estimation stage. This stage is comprised of... Figure 2 The example shows a dedicated neural network architecture implementation. Figure 2As shown, the network employs a dual-stream processing mechanism: RGB and IR images are processed through independent input stages and feature extraction modules to generate preliminary features for their respective modalities. These features are then fed into an L1-norm-based fusion module, which calculates the saliency of the dual-stream features and generates adaptive weights, performing weighted fusion of complementary information at the feature level. The fused features are then optimized through a refinement stage network to improve their discriminative power and spatial consistency. Finally, the network outputs accurate human pose estimation results, i.e., the set of keypoint coordinates for all human instances in the image, through post-processing operations such as keypoint grouping.

[0190] After obtaining the skeleton information, the process enters the behavior analysis phase. Let the first... The set of key points of a single human skeleton detected by the frame is ,in The x and y axes represent the horizontal and vertical coordinates, respectively, along with the confidence level. To improve stability, threshold filtering is first applied to key points with low confidence (e.g., ...). (Then it is not included in the calculation), and the key point coordinates are obtained by performing a moving average or first-order exponential smoothing over time. .

[0191] Scale-normalized geometric calculations: Calculating the height and width of the circumscribed rectangle of the human body from the maximum and minimum values ​​of key points.

[0192] ;

[0193] Calculate the aspect ratio of the human body:

[0194] ;

[0195] Calculate the center point of the torso (approximately using the average of the centers of the neck and hips):

[0196] ;

[0197] Calculate the normalized vertical descent velocity (normalized by bounding box height):

[0198] ;

[0199] Kinematic variance characteristics (sliding window): for key joints (such as left / right wrist), within a length of Within the sliding window, calculate the variance of horizontal motion:

[0200] ;

[0201] Scaling the variance (dividing by) if necessary. ).

[0202] Rule-based decision and state machine (provide implementable logic):

[0203] Fall detection: in continuous Simultaneously satisfied within the frame;

[0204] (The rate of continuous sinking exceeds the threshold);

[0205] or (A rapid change from "tall and thin" to "flat and wide");

[0206] (The height is significantly reduced relative to the initial standing height, which is optional);

[0207] Then the "Fall Down" event will be output.

[0208] Wave for help detection: using wrist points Taking the head point as an example, if the following condition is continuously met:

[0209] (When the wrist is higher than the head, and the y-axis of the image coordinate system is positive downwards, the value is "less than");

[0210] (Significant horizontal oscillation);

[0211] And lasting longer than If so, the output will be a "waving for help" event.

[0212] Output strategy: If any exception rule is triggered, generate an output containing the event type, timestamp, and trigger indicator (e.g., ...). The alarm information is displayed; otherwise, the current skeleton and the "normal / not triggered" status are output for display or storage.

[0213] Subsequently, branching processing is performed based on the results of the behavior analysis. If the current behavior is determined to be abnormal (such as meeting the rules of falling or waving for help), an alarm is triggered, generating alarm information including the event type, timestamp, and key evidence. If no abnormality is determined, the step of outputting skeleton information is executed for real-time visualization or data recording. This completes a full processing loop from perception to cognition. This process continues to process the next frame according to the settings, achieving an efficient and closed-loop analysis process.

[0214] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for human posture estimation and behavior analysis, characterized in that, Includes the following steps: S1. Acquire visible light images and thermal imaging images, extract and fuse features, and generate fused semantic features; S2. Based on the fused semantic features, a refined key point heatmap and partial affinity fields are obtained; S3. Generate a set of key points based on the refined key point heatmap and some affinity fields; S4. Determine abnormal behavior based on the set of key points.

2. The human posture estimation and behavior analysis method according to claim 1, characterized in that, S1 includes the following sub-steps: S11. Acquire visible light images and thermal imaging images, and input them into the first feature extraction network and the second feature extraction network respectively to generate preliminary key point heat maps and partial affinity fields for the RGB mode and the IR mode. S12. Based on the preliminary key point heatmaps of the RGB mode and the preliminary key point heatmaps of the IR mode, generate the corresponding saliency energy maps; S13. Based on the saliency energy maps corresponding to the preliminary key point heatmaps of the RGB mode and the preliminary key point heatmaps of the IR mode, generate pixel-level adaptive fusion weights for the heatmaps. S14. Based on the pixel-level adaptive fusion weights of the heatmap, perform weighted fusion to obtain the fused heatmap features; S15. Generate the corresponding saliency energy maps based on the partial affinity fields of the RGB modes and the partial affinity fields of the IR modes; S16. Based on the saliency energy maps corresponding to the partial affinity fields of the RGB mode and the partial affinity fields of the IR mode, generate pixel-level adaptive fusion weights for the partial affinity fields. S17. Perform weighted fusion based on the pixel-level adaptive fusion weights of the partial affinity field to obtain the fused partial affinity field features; S18. Based on the fused heatmap features and the fused partial affinity field features, the fused semantic features are obtained.

3. The human posture estimation and behavior analysis method according to claim 2, characterized in that, In S11, the preliminary key point heatmap of the RGB mode. With some friendly scenes The expressions are as follows: ； ； in, Represents a visible light image. The initial key point heatmap mapping function representing the RGB modes. The initial partial affinity field mapping function representing the RGB modes; In S11, the preliminary key point heatmap of the IR mode. With some friendly scenes The expressions are as follows: ； ； in, Represents thermal imaging images. The preliminary key point heatmap mapping function represents the IR mode. The preliminary partial affinity field mapping function represents the IR mode; In S12, the saliency energy map corresponding to the preliminary key point heatmap of the RGB mode. The expression is: ； in, Indicates the total number of key points. Represents the x-coordinate on the feature map. Represents the ordinate on the feature map. Indicates the key point number; In S12, the saliency energy map corresponding to the preliminary key point heatmap of the IR mode. The expression is: ； In S13, the pixel-level adaptive fusion weights of the heatmap The expression is: ； in, Indicates a very small quantity; In S14, the fused heatmap features The expression is: ； In S15, the saliency energy map corresponding to the partial affinity fields of the RGB modes. The expression is: ； in, Indicates the number of categories connected by the skeleton. Channel number representing a partial affinity field feature map; In S15, the saliency energy map corresponding to the partial affinity field of the IR mode. The expression is: ； In S16, the pixel-level adaptive fusion weights of partial affinity fields The expression is: ； In S17, the fused partial affinity field features The expression is: 。 4. The human posture estimation and behavior analysis method according to claim 1, characterized in that, S2 includes the following sub-steps: S21. Perform channel compression on the fused semantic features; S22. Perform lightweight sensing based on the channel compression results; S23. Generate spatial gating weights based on the lightweight perception results; S24. Based on the channel compression results, lightweight sensing results, and spatial gating weights, output refined gating residuals; S25. Based on the gated residual refinement, the refined key point heatmap and partial affinity field are obtained.

5. The human posture estimation and behavior analysis method according to claim 4, characterized in that, In step S21, the expression for channel compression is: ； in, (•) represents a non-linear activation function. This represents the semantic features after fusion. This represents a 1×1 convolution operation. This indicates the channel compression result; In step S22, the expression for lightweight sensing is: ； in, Indicates the void ratio, This represents a 3×3 depthwise separable convolution with a hole ratio of 2. This indicates the result of lightweight perception; In S23, the spatial gating weight The expression is: ； in, Represents the Sigmoid function; In S24, the gated residual is refined. The expression is: ； in, This represents element-wise multiplication; In S25, the refined key point heatmap The expression is: ； in, This represents a 1×1 convolution operation used for outputting or regressing a refined keypoint heatmap; In S25, the refined partial affinity field The expression is: ； in, This represents a 1×1 convolution operation used to output or regress a refined portion of the affinity field.

6. The human posture estimation and behavior analysis method according to claim 1, characterized in that, S3 includes the following sub-steps: S31. Perform non-maximum suppression on the refined key point heatmap and part of the affinity field to obtain a set of local peaks, and remove elements with confidence scores less than or equal to a set threshold. S32. Calculate the connectivity score based on the local peak set; S33. Using the connection scores as edge weights, construct a bipartite graph and use the bipartite graph to generate the optimal connection set for each limb type. S34. Generate a set of key points based on the optimal connection set for each limb type.

7. The human posture estimation and behavior analysis method according to claim 6, characterized in that, In S31, the set of local peaks The expression is: ； ； ； in, This represents the coordinates of the nth candidate point of the kth type of keypoint. This represents the confidence level of the nth candidate point of the k-th keypoint. This represents the x-coordinate of the nth candidate point of the k-th keypoint. This represents the ordinate of the nth candidate point of the k-th keypoint category. This indicates the characteristics of the merged heatmap; In S32, the connection score The expression is: ； in, This represents the candidate point number of the starting key point in the limb connection. This indicates the candidate point number for the termination key point in a limb connection. Indicates the type of limb connection. This indicates the number of points sampled uniformly. Points representing uniform sampling, Indicates the unit direction. The vector representing the PAF vector field.

8. The human posture estimation and behavior analysis method according to claim 1, characterized in that, In S4, lightweight geometric calculations are used to determine abnormal behavior.

Images