A Pose Feature Recognition Method Based on Video Stream

Abstract

This invention discloses a video stream-based pose feature recognition method, comprising: preprocessing a continuous video stream to obtain video frame training data; extracting keyframes and neighboring frames from each frame; constructing a feature extraction module for the human body region to obtain a global frame; using neighboring frames on the left and right sides to locally extract local frames from the human body region; constructing semantic association information for the global frame based on the temporal continuity between the local neighboring frames and the current keyframe; using a conditional feature aggregation algorithm to obtain enhanced feature representations; obtaining pose sequence data through pose detail features; establishing three-dimensional coordinates; using a human motion decoupling model to adaptively extract pose change data; predicting human pose features through a smoothing optimization strategy; and introducing a cross-attention mechanism to achieve deep fusion of spatiotemporal features, thereby enhancing the model's ability to understand complex action patterns and significantly improving the model's pose representation ability in occluded or blurred regions.

Description

Technical Field

[0001] This invention relates to the field of video pose feature recognition technology, and more specifically to a pose feature recognition method based on video streams. Background Technology

[0002] Human pose estimation uses computer vision and deep learning technologies to accurately detect and identify key points of the human body, such as the position of the head, neck, and hips, from images or videos, and then infers the human body's pose information. Human pose estimation is a fundamental technology in human motion understanding and a foundational basis for action recognition, motion prediction, and behavior understanding.

[0003] Human posture estimation technology has a wide range of applications, covering motion analysis, medical rehabilitation, and behavior recognition. In sports, dance, and fitness, human posture estimation algorithms can help analyze and evaluate athletes' movement skills and posture correctness by detecting joint positions and posture angles in real time. This provides athletes with more scientific and personalized training and guidance, thereby promoting the improvement of sports training and competitive levels. In the field of intelligent security, posture estimation technology allows security systems to analyze human postures within a safe area, detect whether someone has entered a restricted area or engaged in abnormal activities. Once the security system detects abnormal behavior or area intrusion, it can immediately trigger a security alarm and send warning information to relevant personnel or emergency agencies, and implement corresponding security measures to ensure area security and prevent potential security risks.

[0004] Compared to pose estimation, human action recognition focuses more on the sequence of human motion over a period of time. Action recognition is based on the detection and tracking of the positions of human joints in videos, based on human pose estimation, to extract and identify specific action patterns from dynamic pose sequences and determine the type of action the sequence belongs to. Action recognition has direct practical applications. However, current pose feature recognition methods based on video streams still have the following problems:

[0005] (1) In human pose estimation, rapid motion and pose occlusion in the dynamic environment of video cause degradation of visual information in video frames, making pose estimation on blurred frames, fast motion frames and occluded frames extremely difficult. However, existing human pose estimation research does not make sufficient use of global semantic information and local visual environment information, and cannot effectively solve the problem of visual degradation of video frames.

[0006] (2) In human motion recognition, when the recognition model predicts the action category of a single input motion sequence, the output is often multi-peaked, resulting in high probability values ​​in multiple action categories rather than concentrated in one category. This situation causes classification ambiguity and makes the model's accurate classification ability insufficient. Summary of the Invention

[0007] The purpose of this invention is to provide a pose feature recognition method based on video streams, in order to solve the technical problems in the prior art that the global semantic information and local visual environment information of video streams are not fully utilized, the video frame visual degradation problem cannot be effectively solved, and the human action recognition model has insufficient accurate classification ability.

[0008] To solve the above-mentioned technical problems, the present invention specifically provides the following technical solution:

[0009] This invention provides a pose feature recognition method based on video streams, comprising the following steps:

[0010] The continuous video stream is preprocessed to obtain video frame training data. Key frames and neighboring frames are extracted from each frame of the video frame training data. A target detection algorithm is used to extract the human body region in the video stream.

[0011] A feature extraction module is constructed for the human body region to obtain a global frame. Local frames are obtained by using the neighboring frames on the left and right sides to extract the human body region locally. Semantic association information is constructed for the global frame by the temporal continuity between the local neighboring frames and the current key frame.

[0012] A conditional feature aggregation algorithm is used to fuse the visual context information and semantic association information of the local frame to obtain an enhanced feature representation;

[0013] The enhanced feature representation is selectively enhanced using a deep learning algorithm to obtain pose detail features. These pose detail features are then integrated into the keyframes based on temporal features to obtain pose sequence data.

[0014] Three-dimensional coordinates are established based on the posture sequence data of each joint node of the human body. The posture change data is adaptively extracted using a human motion decoupling model, and the human posture characteristics are predicted through a smoothing optimization strategy.

[0015] As a preferred embodiment of the present invention, a continuous video stream is preprocessed to obtain video frame training data, keyframes and neighboring frames are extracted from each frame of the video frame training data, and a target detection algorithm is used to extract the human body region in the video stream, including:

[0016] The continuous video stream is sampled at a set frame rate and divided into a sequence of video frames arranged in chronological order. The video frame training data is obtained by encoding and standardizing each video frame sequence.

[0017] A time-series equal-division strategy is used to select key frames for the video frame sequence. Several neighboring frames are selected within t time windows before and after the key frame to construct a frame set centered on the key frame and containing local temporal context information.

[0018] For each frame of the video frame sequence, the Faster R-CNN detection model is used to detect human bodies, identify human targets in the video frame sequence images, and obtain the corresponding human bounding boxes.

[0019] Identify the human body region image within the human body bounding box, perform human body image cropping and normalization processing on the human body region image to obtain the human body region;

[0020] The human body region images of the keyframe and its neighboring frames are combined into multi-frame input samples, and the video frame training dataset is constructed using the annotation information of the key points corresponding to the time series.

[0021] As a preferred embodiment of the present invention, a feature extraction module is constructed for the human body region to obtain a global frame, and local frames are obtained by using neighboring frames on the left and right sides to perform local extraction on the human body region, including:

[0022] The human pose estimation algorithm is used to extract global frames of person i in the human body region from the video frame training dataset. A convolutional neural network is used to extract high-level semantic feature maps to obtain global feature representations with rich semantic information. The semantic feature maps are used as global frame features.

[0023] Keyframes are obtained from the global frame based on the character's motion features. Select the neighboring frames within t time windows before and after the keyframe, and extract the corresponding human body region images respectively.

[0024] Local visual features of each frame are extracted by a convolutional network with shared weights, and the features of neighboring frames are combined to form a set of local frame features, which models the local motion changes and visual context information between frames before and after the keyframe.

[0025] As a preferred embodiment of the present invention, semantic association information is constructed for the global frame by leveraging the temporal continuity between local neighboring frames and the current key frame, including:

[0026] From the local frame feature set, select local neighboring frames in time series and align them with the current key frame in time series. Use a variable convolution model to align the features of multiple frames within the time window t to obtain the temporal continuity between the local neighboring frames and the current key frame.

[0027] An attention mechanism is used to fuse the local motion changes and visual context information between the local neighboring frames and the key frame to obtain the semantic similarity and motion continuity of the key frame before and after the time sequence.

[0028] Based on the semantic similarity of the keyframes, local frame features are weighted and aggregated to generate enhanced context-aware features and obtain enhanced semantic association information.

[0029] As a preferred embodiment of the present invention, a conditional feature aggregation algorithm is used to fuse the visual context information of the local frame with the semantic association information, including:

[0030] The visual context information of the local frame is associated with the semantic association information by using the temporal distance between the local neighboring frames and the key frame as the correlation metric.

[0031] The global features and keyframe features are input into the shared embedding feature layer of the variable convolutional model to transform the original feature space into an embedding representation, thereby obtaining the embedding information representation of each frame. , ;

[0032] Represent the embedded information , Semantic correlation between global features and keyframe features is obtained through matrix dot product. The expression for calculating its correlation is:

[0033]

[0034] in, The function represents the feature embedding operation. This represents the matrix multiplication operation. This represents the global features of character i extracted from the global frame. Represents the keyframe features of character i.

[0035] As a preferred embodiment of the present invention, based on the semantic relevance... An enhanced feature representation is obtained by conditionally aggregating keyframe features, local features, and global features using a feature aggregation model, including:

[0036] The keyframe features, local features, and global features are input into a convolutional neural network for encoding to obtain a feature vector with a unified metric. The keyframe features are then input into a feature encoder for encoding to generate weight matrices for different regions of the human body.

[0037] Employing the semantic relevance based on time series. Local and global features are weighted separately, and then concatenated with the keyframe features according to the time series to obtain temporally adjacent local feature information;

[0038] The weight matrix is ​​processed by the Sigmoid function to obtain the enhancement points at each pixel position in the keyframe features. The local features, global features and keyframe features of each frame image are weighted and fused according to the time series to obtain fused feature information.

[0039] The fused feature information is aggregated and transformed with the original keyframe features to obtain enhanced feature vectors for different regions of the human body, generating enhanced feature representations.

[0040] As a preferred embodiment of the present invention, the enhanced feature representation is selectively enhanced using a deep learning algorithm to obtain pose detail features, including:

[0041] Visual context information of the enhanced feature representation is extracted, a convolutional neural network is used to generate a visual feature sequence, a learnable category label is introduced, and the spatial similarity within each video frame is calculated by concatenating the visual feature sequence frame by frame.

[0042] Based on the spatial similarity, the corresponding visual feature sequences are classified and labeled, and matrix multiplication is performed with the visual feature sequences to obtain the human body mask. The human body mask is then multiplied element-wise with the corresponding visual feature sequences to obtain coarse-grained features.

[0043] The keyframes are labeled using the enhanced feature representation, and then concatenated with the coarse-grained features along the time series dimension of the keypoint labels to form a multi-frame feature sequence.

[0044] The multi-frame feature sequence is input into a deep learning model for feature learning, and visual features and key point markers are separated frame by frame to obtain multi-frame features and multi-frame key point markers.

[0045] By transposing the features of multiple frames and performing matrix multiplication with the corresponding key point markers of the multiple frames, a human key point confidence map matrix is ​​generated.

[0046] The elements in the human body keypoint confidence map matrix are weighted and normalized using the softmax function to generate a keypoint mask. Enhanced pose detail features are obtained by performing element-wise multiplication of the keypoint mask with the multi-frame features.

[0047] As a preferred embodiment of the present invention, the pose detail features are integrated into the keyframes based on temporal features to obtain pose sequence data, including:

[0048] A self-attention mechanism is used to model the pose detail features of multiple consecutive frames, capture the motion patterns and evolution trends of human joints in the time dimension, and obtain dynamic feature representations with temporal relationships.

[0049] A P3D-Resnet spatiotemporal feature extraction network is constructed to map the pose detail features extracted from non-keyframes to the spatial coordinates of keyframes. The mapped pose detail features are weighted and fused with the original pose features of keyframes through an attention mechanism to enhance the pose representation ability of occluded or blurred areas in the current frame, introduce dynamic detail cues from neighboring frames, and generate enhanced keyframe pose features.

[0050] The keyframe pose features are arranged in chronological order to construct a continuous sequence of human pose data.

[0051] As a preferred embodiment of the present invention, three-dimensional coordinates are established based on the posture sequence data of each joint node of the human body, and posture change data is adaptively extracted using a human motion decoupling model, including:

[0052] Based on the human posture sequence data, extract each key node of the human body, assign initial three-dimensional coordinates to each key node of the human body, use the cross-entropy loss function to classify and identify the human posture sequence data, and optimize the three-dimensional coordinate position of each joint point by minimizing the cross-entropy loss.

[0053] A recurrent neural network is constructed to train the human posture sequence data and extract different types of motion patterns of human posture.

[0054] By introducing an attention mechanism to train the recurrent neural network, representative motion features are extracted, dynamic features are captured, posture change data is extracted, and a posture sequence dataset is generated.

[0055] As a preferred embodiment of the present invention, the pose change data between consecutive frames in the pose sequence dataset is processed by a smoothing optimization strategy to predict human pose features, including:

[0056] The pose sequence dataset is used as the pose sequence input to a convolutional neural network to extract the pose change pattern over time;

[0057] In the convolutional neural network, spatial feature information S-Transformer and temporal feature information T-Transformer are obtained;

[0058] The pose features containing coordinate and joint information and the fine-grained contour features are convolved and then input into the S-Transformer to obtain spatial features at different time scales.

[0059] Temporal pose features are input into T-Transformer, and then a cross-attention mechanism is introduced to fuse spatiotemporal features;

[0060] The corresponding pose features are fused with the appearance contour features to identify the pose features of multi-feature spatiotemporal fusion.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] This invention extracts high-level semantic feature maps as global frame features through convolutional neural networks and captures local motion changes using neighboring frames, providing rich visual contextual information. Based on temporal distance and semantic similarity, local frame features are weighted and aggregated to generate enhanced context-aware features, thereby enhancing the model's ability to understand complex action patterns.

[0063] By utilizing self-attention mechanisms and neural networks to capture the motion patterns of human joints in the temporal dimension, the temporal smoothness and consistency of the pose sequence are ensured. S-Transformer and T-Transformer are used to process spatial and temporal information respectively, and a cross-attention mechanism is introduced to achieve deep fusion of spatiotemporal features, which improves the model's ability to understand dynamic changes. The cross-entropy loss function is used to optimize the three-dimensional coordinate position of each joint point, ensuring that the pose estimation results are both consistent with actual physical constraints and as accurate as possible. It can capture different types of motion patterns, enhancing the model's ability to understand and predict human motion. Furthermore, by using the attention mechanism to perform weighted fusion of the mapped pose detail features and the original features of the keyframes, the model's ability to express pose in occluded or blurred regions is significantly improved. Attached Figure Description

[0064] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0065] Figure 1 A flowchart of a video stream-based pose feature recognition method provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] like Figure 1 As shown, this invention provides a pose feature recognition method based on video streams, comprising the following steps:

[0068] The continuous video stream is preprocessed to obtain video frame training data. Key frames and neighboring frames are extracted from each frame of the video frame training data. A target detection algorithm is used to extract the human body region in the video stream.

[0069] A feature extraction module is constructed for the human body region to obtain a global frame. Local frames are obtained by using the neighboring frames on the left and right sides to extract the human body region locally. Semantic association information is constructed for the global frame by the temporal continuity between the local neighboring frames and the current key frame.

[0070] In this embodiment, by combining the temporal continuity between local neighboring frames and key frames with the semantic association information in global frames, an enhanced feature representation containing rich visual context and global semantic information can be created, which helps to improve the robustness of the model in complex scenes, such as handling occlusion, cluttered backgrounds, and other problems.

[0071] A conditional feature aggregation algorithm is used to fuse the visual context information and semantic association information of the local frame to obtain an enhanced feature representation;

[0072] The enhanced feature representation is selectively enhanced using a deep learning algorithm to obtain pose detail features. These pose detail features are then integrated into the keyframes based on temporal features to obtain pose sequence data.

[0073] In this embodiment, a deep learning algorithm is used to selectively enhance the enhanced feature representation, which can effectively highlight those detailed features that are crucial for recognition, even if these features may be ignored or difficult to detect in the original image. This method can significantly improve the accuracy of pose estimation.

[0074] Three-dimensional coordinates are established based on the posture sequence data of each joint node of the human body. The posture change data is adaptively extracted using a human motion decoupling model, and the human posture characteristics are predicted through a smoothing optimization strategy.

[0075] In this embodiment, the human motion decoupling model is used to adaptively extract posture change data, enabling the system to better understand and predict complex motion patterns. This is particularly suitable for challenging scenarios such as multi-person interaction and rapid motion changes.

[0076] In this embodiment, by applying a smoothing optimization strategy, coherent prediction of human posture features can be achieved between consecutive frames, ensuring the temporal consistency and natural fluency of the posture sequence and reducing posture jumps caused by single-frame errors.

[0077] In this embodiment, the posture feature recognition method covers the entire process from video preprocessing to final posture prediction, including multiple sub-tasks such as target detection, feature extraction, posture estimation, 3D coordinate construction, and motion prediction. It provides a one-stop solution suitable for various practical scenarios, such as security monitoring, virtual reality, and sports analysis. It not only improves the accuracy and reliability of human posture estimation, but also enhances the adaptability and practicality of the system, providing strong support for solving the problem of human motion understanding in complex dynamic environments.

[0078] Preprocessing a continuous video stream to obtain video frame training data, extracting keyframes and neighboring frames from each frame of the video frame training data, and using an object detection algorithm to extract the human body region from the video stream, including:

[0079] The continuous video stream is sampled at a set frame rate and divided into a sequence of video frames arranged in chronological order. The video frame training data is obtained by encoding and standardizing each video frame sequence.

[0080] In this embodiment, by setting a frame rate to sample the continuous video stream and performing encoding and standardization processing on each frame, the consistency and comparability of the data are ensured, while reducing unnecessary computational burden. This approach helps to improve the efficiency of subsequent processing steps.

[0081] A time-series equal-division strategy is used to select key frames for the video frame sequence. Several neighboring frames are selected within t time windows before and after the key frame to construct a frame set centered on the key frame and containing local temporal context information.

[0082] In this embodiment, the method of selecting key frames using a time-series equal-division strategy and selecting neighboring frames within the time windows before and after them can effectively capture subtle changes and dynamic characteristics in human movements. This not only provides rich local temporal context information but also enhances the model's ability to understand complex action patterns.

[0083] For each frame of the video frame sequence, the Faster R-CNN detection model is used to detect human bodies, identify human targets in the video frame sequence images, and obtain the corresponding human bounding boxes.

[0084] In this embodiment, the Faster R-CNN detection model can efficiently and accurately locate human body regions in video frames, significantly improving detection accuracy, especially when dealing with issues such as occlusion and scale changes.

[0085] Identify the human body region image within the human body bounding box, perform human body image cropping and normalization processing on the human body region image to obtain the human body region;

[0086] The human body region images of the keyframe and its neighboring frames are combined into multi-frame input samples, and the video frame training dataset is constructed using the annotation information of the key points corresponding to the time series.

[0087] In this embodiment, the human body region images of the keyframe and its neighboring frames are combined into multi-frame input samples, and the corresponding key point annotation information is combined to construct a training dataset. This approach is beneficial for training a more robust and generalizable model, allowing the model to learn the transition details between different poses, thereby providing more accurate pose estimation results in practical applications.

[0088] A feature extraction module is constructed for the human body region to obtain global frames, and local frames are obtained by using neighboring frames on the left and right sides to perform local extraction on the human body region, including:

[0089] The human pose estimation algorithm is used to extract global frames of person i in the human body region from the video frame training dataset. A convolutional neural network is used to extract high-level semantic feature maps to obtain global feature representations with rich semantic information. The semantic feature maps are used as global frame features.

[0090] In this embodiment, high-level semantic feature maps are extracted by convolutional neural networks as global frame features, which can capture the overall structural information and contextual semantics of the human body in key frames, providing a stable and semantically rich foundation for subsequent pose estimation, and helping the model to better understand the overall human pose in the current frame.

[0091] Keyframes are obtained from the global frame based on the character's motion features. Select the neighboring frames within t time windows before and after the keyframe, and extract the corresponding human body region images respectively.

[0092] In this embodiment, the use of information from neighboring frames on the left and right sides helps to recover pose information lost due to occlusion or image quality degradation. At the same time, this joint modeling method of consecutive frames also improves the temporal consistency of pose estimation results and avoids abrupt or unnatural pose jumps.

[0093] Local visual features of each frame are extracted by a convolutional network with shared weights, and the features of neighboring frames are combined to form a set of local frame features, which models the local motion changes and visual context information between frames before and after the keyframe.

[0094] In this embodiment, neighboring frames within the time window before and after the keyframe are selected, and local visual features of each frame are extracted through a convolutional network with shared weights to construct a set of local frame features. This can effectively model the local motion changes and visual context information around the keyframe. This approach enhances the model's ability to perceive dynamic details, and performs better, especially when dealing with challenges such as occlusion and blur.

[0095] Semantic association information is constructed for the global frames by leveraging the temporal continuity between local neighboring frames and the current keyframe, including:

[0096] From the local frame feature set, select local neighboring frames in time series and align them with the current key frame in time series. Use a variable convolution model to align the features of multiple frames within the time window t to obtain the temporal continuity between the local neighboring frames and the current key frame.

[0097] In this embodiment, the deformable convolution model is used to perform feature alignment processing on local neighboring frames and key frames within a time window, which can effectively capture the motion changes and spatial displacement relationships between frames. This alignment method improves the robustness of the model in complex dynamic scenes, and is especially suitable for pose estimation under fast motion or occlusion conditions.

[0098] An attention mechanism is used to fuse the local motion changes and visual context information between the local neighboring frames and the key frame to obtain the semantic similarity and motion continuity of the key frame before and after the time sequence.

[0099] In this embodiment, an attention mechanism is used to fuse visual context information and local motion changes between local frames and keyframes, enabling the model to adaptively select the most useful information at different time points. This approach not only enhances the understanding of action details but also improves the model's performance when faced with background interference, pose ambiguity, and other situations.

[0100] Based on the semantic similarity of the keyframes, local frame features are weighted and aggregated to generate enhanced context-aware features and obtain enhanced semantic association information.

[0101] In this embodiment, local frame features are weighted and aggregated based on the semantic similarity of keyframes. This means that the model can determine which information is more important based on the "correlation" between the current frame and other frames. This mechanism is similar to the attention mechanism of humans when understanding continuous actions, which helps to improve the accuracy and stability of pose estimation.

[0102] A conditional feature aggregation algorithm is used to fuse the visual context information and semantic association information of the local frame, including:

[0103] The visual context information of the local frame is associated with the semantic association information by using the temporal distance between the local neighboring frames and the key frame as the correlation metric.

[0104] In this embodiment, by using the temporal distance between local neighboring frames and keyframes as a correlation metric, the dynamic characteristics of human movements changing over time can be captured more accurately. This method helps improve the model's ability to understand complex motion patterns, and is particularly important when dealing with fast movements or non-rigid deformations.

[0105] The global features and keyframe features are input into the shared embedding feature layer of the variable convolutional model to transform the original feature space into an embedding representation, thereby obtaining the embedding information representation of each frame. , ;

[0106] Represent the embedded information , Semantic correlation between global features and keyframe features is obtained through matrix dot product. The expression for calculating its correlation is:

[0107]

[0108] in, The function represents the feature embedding operation. This represents the matrix multiplication operation. This represents the global features of character i extracted from the global frame. Represents the keyframe features of character i.

[0109] In this embodiment, the shared embedding feature layer of the variable convolution model is used to transform the original feature space into a more abstract and higher-level embedding representation. This not only effectively integrates the rich information in global features and keyframe features, but also helps the model learn more discriminative feature representations, thereby improving the accuracy of pose estimation.

[0110] Based on the semantic relevance An enhanced feature representation is obtained by conditionally aggregating keyframe features, local features, and global features using a feature aggregation model, including:

[0111] The keyframe features, local features, and global features are input into a convolutional neural network for encoding to obtain a feature vector with a unified metric. The keyframe features are then input into a feature encoder for encoding to generate weight matrices for different regions of the human body.

[0112] In this embodiment, the three types of features—keyframes, local neighbor frames, and global frames—are uniformly encoded into feature vectors under a unified metric space. This breaks the limitations of traditional single-frame modeling and achieves a leap from "static perception" to "dynamic understanding." This multi-source information fusion strategy significantly improves the comprehensiveness and robustness of feature representation.

[0113] Employing the semantic relevance based on time series. Local and global features are weighted separately, and then concatenated with the keyframe features according to the time series to obtain temporally adjacent local feature information;

[0114] In this embodiment, local and global features are weighted based on semantic correlation of time series, enabling the model to automatically identify which neighboring frames are more important for pose estimation of the current keyframe. This approach is similar to the human ability to focus on key context frames when observing actions, thereby improving the model's discriminative ability and adaptability.

[0115] The weight matrix is ​​processed by the Sigmoid function to obtain the enhancement points at each pixel position in the keyframe features. The local features, global features and keyframe features of each frame image are weighted and fused according to the time series to obtain fused feature information.

[0116] In this embodiment, weight matrices for different regions of the human body are generated from keyframe features and then processed using the Sigmoid function as enhancement points. This achieves adaptive enhancement of key joints or occluded regions in the human pose. Compared with the overall enhancement method, this fine-grained spatial attention mechanism can better highlight local regions that are of high value for pose estimation.

[0117] In this embodiment, local, global, and keyframe features are weighted and fused based on time series in each frame image. This not only enhances the expressive power of the current frame but also ensures the smoothness and consistency of the entire pose sequence in the time dimension, avoiding inter-frame jumps and prediction jitter problems.

[0118] The fused feature information is aggregated and transformed with the original keyframe features to obtain enhanced feature vectors for different regions of the human body, generating enhanced feature representations.

[0119] In this embodiment, the fused feature information is aggregated and transformed with the original keyframe features, such as by splicing, addition, and channel attention. The feature representation is further optimized by learnable parameters, so that the model has stronger nonlinear modeling capabilities and is suitable for diverse pose estimation tasks in complex scenes.

[0120] The enhanced feature representation is selectively enhanced using a deep learning algorithm to obtain pose detail features, including:

[0121] Visual context information of the enhanced feature representation is extracted, a convolutional neural network is used to generate a visual feature sequence, a learnable category label is introduced, and the spatial similarity within each video frame is calculated by concatenating the visual feature sequence frame by frame.

[0122] In this embodiment, a convolutional neural network is used to encode the visual context information in the enhanced feature representation to generate a visual feature sequence, enabling the model to more accurately understand the spatial structure and action state of the human body in each frame of the image, providing a high-quality input foundation for subsequent key point localization and feature enhancement.

[0123] Based on the spatial similarity, the corresponding visual feature sequences are classified and labeled, and matrix multiplication is performed with the visual feature sequences to obtain the human body mask. The human body mask is then multiplied element-wise with the corresponding visual feature sequences to obtain coarse-grained features.

[0124] In this embodiment, learnable category labels are introduced, and spatial similarity is calculated in a cascaded frame-by-frame manner. This helps the model to automatically identify the semantic relevance and motion continuity between different frames. This approach enhances the model's ability to understand cross-frame information, and it performs particularly well when dealing with complex dynamic scenes.

[0125] In this embodiment, visual features are classified and labeled based on spatial similarity, and a human body mask is generated through matrix multiplication. This achieves adaptive attention to human body regions in the image. This spatial attention mechanism can effectively suppress background interference and focus on regions related to pose estimation, thereby improving the accuracy of feature extraction.

[0126] The keyframes are labeled using the enhanced feature representation, and then concatenated with the coarse-grained features along the time series dimension of the keypoint labels to form a multi-frame feature sequence.

[0127] The multi-frame feature sequence is input into a deep learning model for feature learning, and visual features and key point markers are separated frame by frame to obtain multi-frame features and multi-frame key point markers.

[0128] In this embodiment, the enhanced features of keyframes and coarse-grained features are concatenated along the time dimension to form a multi-frame feature sequence. This not only preserves the detailed information of the current frame but also integrates contextual clues from previous and subsequent frames, enhancing the model's ability to understand the action evolution process and improving the temporal consistency of pose estimation.

[0129] In this embodiment, a multi-frame feature sequence is input into a deep learning model for feature learning. Visual features and key point markers are separated frame by frame, enabling the model to learn subtle changes in pose independently and coherently in each frame, which greatly improves the ability to capture pose details.

[0130] By transposing the features of multiple frames and performing matrix multiplication with the corresponding key point markers of the multiple frames, a human key point confidence map matrix is ​​generated.

[0131] The elements in the human body keypoint confidence map matrix are weighted and normalized using the softmax function to generate a keypoint mask. Enhanced pose detail features are obtained by performing element-wise multiplication of the keypoint mask with the multi-frame features.

[0132] In this embodiment, the softmax function is used to normalize the weights of the confidence map matrix to generate a key point mask. The mask is then applied to the features of multiple frames through element-wise multiplication to further enhance the feature information related to the key points while suppressing irrelevant noise, thus achieving selective enhancement of pose details.

[0133] The pose detail features are integrated into the keyframes based on temporal features to obtain pose sequence data, including:

[0134] A self-attention mechanism is used to model the pose detail features of multiple consecutive frames, capture the motion patterns and evolution trends of human joints in the time dimension, and obtain dynamic feature representations with temporal relationships.

[0135] In this embodiment, the self-attention mechanism is used to model the pose details of multiple consecutive frames, which can effectively capture the motion patterns and evolution trends of human joints in the time dimension. This global perspective time modeling enhances the model's overall understanding of human movements, and is especially suitable for the recognition and prediction of complex and continuous movements.

[0136] A P3D-Resnet spatiotemporal feature extraction network is constructed to map the pose detail features extracted from non-keyframes to the spatial coordinates of keyframes. The mapped pose detail features are weighted and fused with the original pose features of keyframes through an attention mechanism to enhance the pose representation ability of occluded or blurred areas in the current frame, introduce dynamic detail cues from neighboring frames, and generate enhanced keyframe pose features.

[0137] In this embodiment, a P3D-ResNet spatiotemporal feature extraction network is constructed to map the pose detail features in non-key frames to the spatial coordinate system of key frames. This solves the problem of spatial misalignment between different frames due to viewpoint, displacement or deformation. This step ensures the effective fusion of information from neighboring frames and improves the accuracy of pose estimation.

[0138] The keyframe pose features are arranged in chronological order to construct a continuous sequence of human pose data.

[0139] In this embodiment, the mapped pose detail features and the original pose features of the keyframes are weighted and fused through an attention mechanism, enabling the model to automatically identify and supplement information in occluded or blurred areas. This "outside-in" enhancement method greatly improves the robustness and generalization ability of the model in complex scenes. It focuses on both the spatial structure within a single frame and the temporal evolution between frames, realizing multi-scale modeling from local details to global action patterns. It is suitable for various tasks such as static pose recognition, dynamic action classification, and long-term motion prediction.

[0140] In this embodiment, dynamic detail cues from preceding and following neighboring frames are introduced into the keyframes, which not only makes up for the limitations of single-frame information, but also provides rich local temporal context support for pose estimation. This approach helps to recover key pose information lost due to rapid motion or image quality degradation.

[0141] Three-dimensional coordinates are established based on the posture sequence data of each joint node of the human body, and posture change data is adaptively extracted using a human motion decoupling model, including:

[0142] Based on the human posture sequence data, extract each key node of the human body, assign initial three-dimensional coordinates to each key node of the human body, use the cross-entropy loss function to classify and identify the human posture sequence data, and optimize the three-dimensional coordinate position of each joint point by minimizing the cross-entropy loss.

[0143] In this embodiment, the pose sequence data is classified and identified by the cross-entropy loss function, and the loss is minimized to optimize the three-dimensional coordinate position of each joint. This ensures that the generated pose conforms to the actual physical constraints and is as accurate as possible, and can effectively improve the accuracy of pose estimation, especially in complex scenes such as occlusion and viewpoint changes, which helps to obtain a more natural and accurate human pose representation.

[0144] A recurrent neural network is constructed to train the human posture sequence data and extract different types of motion patterns of human posture.

[0145] In this embodiment, a recurrent neural network is constructed to train human posture sequence data, enabling the model to capture different types of motion patterns of human posture changing over time. This method is particularly suitable for modeling long-term and short-term dependencies and is very effective for understanding continuous action sequences.

[0146] By introducing an attention mechanism to train the recurrent neural network, representative motion features are extracted, dynamic features are captured, posture change data is extracted, and a posture sequence dataset is generated.

[0147] In this embodiment, an attention mechanism is introduced during training, which allows the model to automatically focus on the most representative motion features, thereby capturing dynamic features more effectively. This is particularly important for improving the model's performance in complex actions, as it helps the model ignore noise and focus on the truly important information.

[0148] The pose sequence dataset is processed using a smoothing optimization strategy to predict human pose features, including:

[0149] The pose sequence dataset is used as the pose sequence input to a convolutional neural network to extract the pose change pattern over time;

[0150] In the convolutional neural network, spatial feature information S-Transformer and temporal feature information T-Transformer are obtained;

[0151] In this embodiment, the S-Transformer focuses on processing information in the spatial dimension, and can capture the relative positional relationships between human joints at different points in time, maintaining high accuracy even in complex or rapidly changing movements; the T-Transformer is specifically designed to process information in the temporal dimension, and can capture the dynamic trends in posture sequences, which helps the model understand the development process of the movement, and is particularly important for predicting future posture changes.

[0152] The pose features containing coordinate and joint information and the fine-grained contour features are convolved and then input into the S-Transformer to obtain spatial features at different time scales.

[0153] In this embodiment, a cross-attention mechanism can be used to establish connections between spatial and temporal dimensions, enabling deeper feature fusion. This method allows the model to dynamically adjust its focus as needed, thereby improving its ability to understand complex actions.

[0154] Temporal pose features are input into T-Transformer, and then a cross-attention mechanism is introduced to fuse spatiotemporal features;

[0155] The corresponding pose features are fused with the appearance contour features to identify the pose features of multi-feature spatiotemporal fusion.

[0156] In this embodiment, the powerful feature extraction capabilities of Convolutional Neural Networks (CNN), the advantages of Transformers in processing long sequence data, and the flexible feature fusion strategy provided by the cross-attention mechanism are comprehensively utilized to ensure that the model can maintain high accuracy and good robustness when facing various challenges such as occlusion, blurring, and fast motion.

[0157] This invention extracts high-level semantic feature maps as global frame features through convolutional neural networks and captures local motion changes using neighboring frames, providing rich visual contextual information. Based on temporal distance and semantic similarity, local frame features are weighted and aggregated to generate enhanced context-aware features, thereby enhancing the model's ability to understand complex action patterns.

[0158] By utilizing self-attention mechanisms and neural networks to capture the motion patterns of human joints in the temporal dimension, the temporal smoothness and consistency of the pose sequence are ensured. S-Transformer and T-Transformer are used to process spatial and temporal information respectively, and a cross-attention mechanism is introduced to achieve deep fusion of spatiotemporal features, which improves the model's ability to understand dynamic changes. The cross-entropy loss function is used to optimize the three-dimensional coordinate position of each joint point, ensuring that the pose estimation results are both consistent with actual physical constraints and as accurate as possible. It can capture different types of motion patterns, enhancing the model's ability to understand and predict human motion. Furthermore, by using the attention mechanism to perform weighted fusion of the mapped pose detail features and the original features of the keyframes, the model's ability to express pose in occluded or blurred regions is significantly improved.

[0159] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A pose feature recognition method based on video stream, characterized in that, Includes the following steps: The continuous video stream is preprocessed to obtain video frame training data. Keyframes and neighboring frames are extracted from each frame of the video frame training data. A target detection algorithm is used to extract the human body region in the video stream. A feature extraction module is constructed for the human body region to obtain a global frame. Local frames are obtained by using the neighboring frames on the left and right sides to extract the human body region locally. Semantic association information is constructed for the global frame by the temporal continuity between the local neighboring frames and the current key frame. A conditional feature aggregation algorithm is used to fuse the visual context information and semantic association information of the local frame to obtain an enhanced feature representation; The enhanced feature representation is selectively enhanced using a deep learning algorithm to obtain pose detail features. These pose detail features are then integrated into the keyframes based on temporal features to obtain pose sequence data. Three-dimensional coordinates are established based on the posture sequence data of each joint node of the human body. The posture change data is adaptively extracted using a human motion decoupling model. Human posture features are predicted through a smoothing optimization strategy. Preprocessing a continuous video stream to obtain video frame training data, extracting keyframes and neighboring frames from each frame of the video frame training data, and using an object detection algorithm to extract the human body region from the video stream, including: The continuous video stream is sampled at a set frame rate and divided into a sequence of video frames arranged in chronological order. The video frame training data is obtained by encoding and standardizing each video frame sequence. A time-series equal-division strategy is used to select key frames for the video frame sequence. Several neighboring frames are selected within t time windows before and after the key frame to construct a frame set centered on the key frame and containing local temporal context information. For each frame of the video frame sequence, the Faster R-CNN detection model is used to detect human bodies, identify human targets in the video frame sequence images, and obtain the corresponding human bounding boxes. Identify the human body region image within the human body bounding box, perform human body image cropping and normalization processing on the human body region image to obtain the human body region; The human body region images of the key frame and its neighboring frames are combined into multi-frame input samples, and the annotation information of the key points corresponding to the time series is used to construct a video frame training dataset. A feature extraction module is constructed for the human body region to obtain global frames, and local frames are obtained by using neighboring frames on the left and right sides to perform local extraction on the human body region, including: The human pose estimation algorithm is used to extract global frames of person i in the human body region from the video frame training dataset. A convolutional neural network is used to extract high-level semantic feature maps to obtain global feature representations with rich semantic information. The semantic feature maps are used as global frame features. Keyframes are obtained from the global frame based on the character's motion features. Select the neighboring frames within t time windows before and after the keyframe, and extract the corresponding human body region images respectively. Local visual features of each frame are extracted by a convolutional network with shared weights, and the features of neighboring frames are combined to form a set of local frame features, which models the local motion changes and visual context information between frames before and after the key frame. Semantic association information is constructed for the global frames by leveraging the temporal continuity between local neighboring frames and the current keyframe, including: From the local frame feature set, select local neighboring frames in time series and align them with the current key frame in time series. Use a variable convolution model to align the features of multiple frames within the time window t to obtain the temporal continuity between the local neighboring frames and the current key frame. An attention mechanism is used to fuse the local motion changes and visual context information between the local neighboring frames and the key frame to obtain the semantic similarity and motion continuity of the key frame before and after the time sequence. Based on the semantic similarity of the keyframes, local frame features are weighted and aggregated to generate enhanced context-aware features and obtain enhanced semantic association information. A conditional feature aggregation algorithm is used to fuse the visual context information and semantic association information of the local frame, including: The visual context information of the local frame is associated with the semantic association information by using the temporal distance between the local neighboring frames and the key frame as the correlation metric. The global features and keyframe features are input into the shared embedding feature layer of the variable convolutional model to transform the original feature space into an embedding representation, thereby obtaining the embedding information representation of each frame. , ; Represent the embedded information , Semantic correlation between global features and keyframe features is obtained through matrix dot product. The expression for calculating its correlation is: in, The function represents the feature embedding operation. This represents the matrix multiplication operation. This represents the global features of character i extracted from the global frame. Represents the keyframe features of character i.

2. The pose feature recognition method based on video stream according to claim 1, characterized in that, Based on the semantic relevance An enhanced feature representation is obtained by conditionally aggregating keyframe features, local features, and global features using a feature aggregation model, including: The keyframe features, local features, and global features are input into a convolutional neural network for encoding to obtain a feature vector with a unified metric. The keyframe features are then input into a feature encoder for encoding to generate weight matrices for different regions of the human body. Employing the semantic relevance based on time series. Local features and global features are weighted separately, and then concatenated with the keyframe features according to the time series to obtain temporally adjacent local feature information; The weight matrix is ​​processed by the Sigmoid function to obtain the enhancement points at each pixel position in the keyframe features. The local features, global features and keyframe features of each frame image are weighted and fused according to the time series to obtain fused feature information. The fused feature information is aggregated and transformed with the original keyframe features to obtain enhanced feature vectors for different regions of the human body, generating enhanced feature representations.

3. The pose feature recognition method based on video stream according to claim 2, characterized in that, The enhanced feature representation is selectively enhanced using a deep learning algorithm to obtain pose detail features, including: Visual context information of the enhanced feature representation is extracted, a convolutional neural network is used to generate a visual feature sequence, a learnable category label is introduced, and the spatial similarity within each video frame is calculated by concatenating the visual feature sequence frame by frame. Based on the spatial similarity, the corresponding visual feature sequences are classified and labeled, and matrix multiplication is performed with the visual feature sequences to obtain the human body mask. The human body mask is then multiplied element-wise with the corresponding visual feature sequences to obtain coarse-grained features. The keyframes are labeled using the enhanced feature representation, and then concatenated with the coarse-grained features along the time series dimension of the keypoint labels to form a multi-frame feature sequence. The multi-frame feature sequence is input into a deep learning model for feature learning, and visual features and key point markers are separated frame by frame to obtain multi-frame features and multi-frame key point markers. By transposing the features of multiple frames and performing matrix multiplication with the corresponding key point markers of the multiple frames, a human key point confidence map matrix is ​​generated. The elements in the human body keypoint confidence map matrix are weighted and normalized using the softmax function to generate a keypoint mask. Enhanced pose detail features are obtained by performing element-wise multiplication of the keypoint mask with the multi-frame features.

4. The pose feature recognition method based on video stream according to claim 3, characterized in that, The pose detail features are integrated into the keyframes based on temporal features to obtain pose sequence data, including: A self-attention mechanism is used to model the pose detail features of multiple consecutive frames, capture the motion patterns and evolution trends of human joints in the time dimension, and obtain dynamic feature representations with temporal relationships. A P3D-Resnet spatiotemporal feature extraction network is constructed to map the pose detail features extracted from non-keyframes to the spatial coordinates of keyframes. The mapped pose detail features are weighted and fused with the original pose features of keyframes through an attention mechanism to enhance the pose representation ability of occluded or blurred areas in the current frame, introduce dynamic detail cues from neighboring frames, and generate enhanced keyframe pose features. The keyframe pose features are arranged in chronological order to construct a continuous sequence of human pose data.

5. The pose feature recognition method based on video stream according to claim 4, characterized in that, Three-dimensional coordinates are established based on the posture sequence data of each joint node of the human body, and posture change data is adaptively extracted using a human motion decoupling model, including: Based on the human posture sequence data, extract each key node of the human body, assign initial three-dimensional coordinates to each key node of the human body, use the cross-entropy loss function to classify and identify the human posture sequence data, and optimize the three-dimensional coordinate position of each joint point by minimizing the cross-entropy loss. A recurrent neural network is constructed to train the human posture sequence data and extract different types of motion patterns of human posture. By introducing an attention mechanism to train the recurrent neural network, representative motion features are extracted, dynamic features are captured, posture change data is extracted, and a posture sequence dataset is generated.

6. The pose feature recognition method based on video stream according to claim 5, characterized in that, The pose sequence dataset is processed using a smoothing optimization strategy to predict human pose features, including: The pose sequence dataset is used as the pose sequence input to a convolutional neural network to extract the pose change pattern over time; In the convolutional neural network, spatial feature information S-Transformer and temporal feature information T-Transformer are obtained; The pose features containing coordinate and joint information and the fine-grained contour features are convolved and then input into the S-Transformer to obtain spatial features at different time scales. Temporal pose features are input into T-Transformer, and then a cross-attention mechanism is introduced to fuse spatiotemporal features; The corresponding pose features are fused with the appearance contour features to identify the pose features of multi-feature spatiotemporal fusion.

Images