A human body posture estimation method and device based on time series image feature updating

Abstract

The present invention discloses a method and device for estimating human posture based on updating of time-series image features. The present invention first collects and preprocesses video data. Secondly, a pre-trained NanoDet model is used to detect human targets and obtain human bounding box information; then, feature extraction is performed in the detected human bounding box to generate a posture feature vector, and the vector is subdivided into 34 feature substructures. On this basis, the feature substructure of the current frame is used to update the historical feature substructure to generate 34 comprehensive feature substructures reflecting the current moment t. Finally, the comprehensive feature substructure is reorganized and the complete posture of the human body is accurately estimated through a decoder. The present invention significantly improves the accuracy and robustness of human posture estimation, and overcomes the limitation of the prior art that posture estimation relies on a single frame image.

Description

Technical Field

[0001] The present invention relates to the field of human body posture estimation, and in particular to a human body posture estimation method and device based on time series image feature updating. Background Art

[0002] Existing methods in the field of human posture estimation mainly rely on deep learning models, especially technologies based on convolutional neural networks (CNNs), which estimate human posture through image feature extraction and prediction of key human nodes. Although these methods have achieved good results to a certain extent, they often fail to provide accurate posture estimation results in complex scenes due to their reliance on single-frame images, and are subject to uncertainty interference such as occlusion, illumination changes, and human posture diversity. For example, Chinese invention patent publication number CN118351577A discloses a human posture assessment method based on a convolutional neural network, which extracts features from a single-frame image through a convolutional network model to obtain human skeleton information. Similarly, patent CN118506402A discloses a lightweight human posture estimation and pedestrian fall detection method that combines a convolutional neural network and an attention mechanism, which still relies on posture estimation of a single-frame image. Summary of the invention

[0003] In order to solve the existing problems, the present invention proposes a method and device for human posture estimation based on time series image feature update. Different from the prior art, the present invention introduces information association in the time series, uses historical posture features as reasoning evidence, and combines the single-frame posture feature information and the time correlation between consecutive frames to significantly improve the accuracy, robustness and noise resistance of posture estimation. The technical solution to achieve the purpose of the present invention is as follows:

[0004] A human body posture estimation method based on time series image feature updating, the method comprising the following steps:

[0005] S1: Collect video data through the device and perform preprocessing operations on the video data;

[0006] S2: Use the pre-trained NanoDet target detection model to perform target detection on the pre-processed video data to obtain the human body bounding box information;

[0007] S3: Extract features within the human body bounding box through the encoder to generate a posture feature vector;

[0008] S4: The posture feature vector is classified by the image feature classification module and divided into 34 feature substructures at the current time t, each feature substructure represents a local posture feature set of the human body;

[0009] S5: Design a feature update module to use the 34 feature substructures at the current time t to update the feature substructures at times t-1 and t-2, and generate 34 comprehensive feature substructures at the current time t;

[0010] S6: Recombine the 34 comprehensive feature substructures at the current time t and estimate the complete posture of the human body through the decoder.

[0011] Furthermore, the S1 collects original video data through the device end and performs preprocessing operations on the video data. The specific implementation steps are:

[0012] S1.1: Collect human motion video through the camera, use the OpenCV library to open the motion video and read the corresponding image sequence frame by frame ;

[0013] S1.2: Preprocess each frame of the image sequence, including denoising, resizing and normalization;

[0014] The method for removing noise is bilateral filtering, and the bilateral filtering algorithm formula is as follows:

[0015] (1)

[0016] in, is the input Frame image; Indicates pixels; So Centered window; is the normalized weight; is the spatial Gaussian kernel, which is used to measure the spatial distance between pixels; is the grayscale Gaussian kernel, which is used to measure the grayscale similarity between pixels; and Represents pixels and Gray value of

[0017] The resizing is to scale the length and width of the image at a fixed ratio to meet the input size requirements of the subsequent feature extraction model;

[0018] The normalization process is to normalize the pixel values ​​to the interval [0, 1] to obtain the pixel data of the tth frame. .

[0019] Furthermore, the step S2 uses the pre-trained NanoDet target detection model to perform target detection on the pre-processed video data to obtain human body bounding box information, and the specific implementation steps are:

[0020] S2.1: Preprocessed pixel data Input is a skeleton network composed of ShuffleNetV2, and features of 8x, 16x, and 32x downsampling are extracted from the skeleton network. , and input it into the Path Aggregation Network (PAN) for multi-scale feature fusion to obtain the fused features ;

[0021] S2.2: By using the detection head based on the fully convolutional single-stage target FCOS, the fused features Processing is performed to obtain the coordinate information B of the bounding box.

[0022] Furthermore, the step S3 extracts features in the human body bounding box obtained in S2 through an encoder to generate a posture feature vector, and the specific implementation steps are as follows:

[0023] S3.1: Based on the bounding box coordinate information B, the image data in the bounding box is divided into n non-overlapping fixed-size sub-blocks using convolution operations. Each sub-block is converted into a feature vector through linear embedding, and then the feature vector is linearly projected to generate a feature matrix ;

[0024] S3.2: The feature matrix Input sliding window Transformer module to generate posture features ;

[0025] S3.3: Posture feature data The spatial resolution of the features is compressed by downsampling layer by layer. Normalization is performed to generate multi-scale fused posture features .

[0026] Furthermore, the step S4 classifies the posture feature vector generated in S3 into 34 feature substructures through the image feature classification module, and the specific implementation steps are as follows:

[0027] S4.1: Multi-scale fusion of pose features using convolution Channel adjustment is performed, and then the convolutional feature map is flattened into a one-dimensional vector, and the feature dimension is further adjusted through the fully connected layer to obtain the posture feature. ;

[0028] S4.2: Pose features by stacking N mixing layers (MixerLayer) Perform nonlinear transformation to generate 34 discrete feature substructures. The hybrid layer is composed of layer normalization layer, perceptron (MLP) and jump connection. Initialize the human posture feature substructure library as the codebook.

[0029] The fully connected layer is used to calculate each feature substructure to generate the prediction score corresponding to the discrete feature category in the codebook. Finally, the activation function (softmax) is executed to convert the prediction score into a probability set, and the probability set is converted into a feature substructure as shown below:

[0030] (2)

[0031] in, The corresponding is the sequence number of the characteristic substructure, corresponds to the categories in the codebook, is the set of all categories in the codebook, The corresponding feature substructure takes the probability value of the category in the codebook.

[0032] Furthermore, the specific process of initializing the human posture feature substructure library as a code book is as follows: The human body key points are labeled to obtain a sequence of position information. Local posture features are extracted through linear layer, discard layer, mixing layer, layer normalization layer and linear projection layer in sequence. These local posture features are clustered through the K-means clustering algorithm and divided into a finite class of feature substructures. The set of feature substructures constitutes a codebook.

[0033] The codebook update process is as follows: the continuous features output by the linear projection layer are converted into discrete vector representations, and then matched with the vectors in the codebook through nearest neighbor search, the closest discrete feature substructure vector is found in the codebook through matching, the feature dot product value of the discrete vector and the discrete feature substructure vector is calculated through the EMA mechanism, and then the corresponding discrete feature substructure vector in the codebook is updated.

[0034] Furthermore, the step S5 uses the current feature substructure to update the historical feature substructure to generate a comprehensive feature substructure at the current time t, and the specific implementation steps are:

[0035] S5.1: When t = 1, assume that the comprehensive characteristic substructure evidence is about The probability of , and have ;

[0036] S5.2: When t=2, assume that the comprehensive characteristic substructure evidence is about The updated probability of , the calculation formula is as follows:

[0037] (3)

[0038] (4)

[0039] (5)

[0040] This formula indicates that when t=2, renew The updated result at time t=2 is ;in, and Comprehensive characteristic substructure reliability and the current feature substructure reliability The weight of and Comprehensive characteristic substructure reliability and the current feature substructure reliability reliability, and Calculated by the following formula:

[0041] (6)

[0042] (7)

[0043] in is the initial reliability value; is a reward and punishment factor. If the comprehensive characteristic substructure reliability The maximum value of and the current characteristic substructure confidence The maximum values ​​all point to the same category in the codebook ,but The value is 1, otherwise, The value is -1; is the reliability enhancement factor given by the similarity measure between the two credibility measures:

[0044] (8)

[0045] in, and Characteristic substructure reliability Standardized reliability , , …, The similarity is given by:

[0046] (9)

[0047] (10)

[0048] (11)

[0049] S5.3: When t ≥ 3, the characteristic substructure evidence is integrated The probability of , the specific implementation steps are:

[0050] S5.3.1: Composite feature substructure reliability Reliability of the composite feature substructure Update to obtain the historical characteristic substructure reliability , the calculation formula is as follows:

[0051] (12)

[0052] (13)

[0053] in, and Comprehensive characteristic substructure reliability and comprehensive characteristic substructure reliability The reliability of is given by formula (5);

[0054] S5.3.2: Current feature substructure reliability Confidence in historical characteristics substructure Update to get the current comprehensive feature substructure confidence , the calculation formula is as follows:

[0055] (14)

[0056] Calculated by formula (14) Enter formula (5) and replace Normalize and obtain the comprehensive feature substructure reliability and normalized reliability ;in, is the reliability of the current feature substructure The weight of and The historical characteristic substructure reliability and the current feature substructure reliability Reliability, Calculated by formula (7), Obtained by formula (5).

[0057] Furthermore, the step S6 combines the 34 comprehensive feature substructures at the current time t obtained in S5 by splicing to obtain a comprehensive feature confidence matrix, and then fuses the comprehensive feature confidence matrix with the codebook matrix by dot multiplication to obtain a comprehensive feature, and inputs the obtained comprehensive feature into a decoder composed of a linear layer, a mixing layer and a normalization layer in sequence to estimate the complete posture of the human body.

[0058] The present invention provides a human body posture estimation device based on time series image feature update, the device comprising:

[0059] Video acquisition module: collects raw video data through the device end and pre-processes the video data;

[0060] Human object detection module: Use the pre-trained NanoDet object detection model to perform object detection on the pre-processed video data to obtain human bounding box information;

[0061] Feature extraction module: extract features within the human body bounding box and generate posture feature vectors;

[0062] Feature classification module: classifies the extracted posture feature vectors into 34 feature substructures;

[0063] Feature update module: Use the 34 feature substructures at the current time t to update the feature substructures at time t-1 and t-2, and generate 34 comprehensive feature substructures at the current time t;

[0064] Decoding module: Initialize the code book of the human body posture feature substructure library, reassemble the comprehensive feature substructure and fuse it with the code book matrix by dot multiplication, and input the fusion result into the decoder to estimate the complete posture of the human body;

[0065] Result display module: used to display the final results of human posture estimation, including the location information of key points of the human body.

[0066] Beneficial effects of the present invention:

[0067] The present invention is based on the update of time-series image features, integrates the feature information in the time series, and can accurately capture the continuity of human movements. The difference from other methods is that it comprehensively utilizes single-frame and historical multi-frame data to effectively avoid the influence of uncertain interference information such as noise. In general, the present invention provides an efficient and stable posture estimation solution for the field of intelligent monitoring and behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments or technical solutions of the prior art in this specification, the relevant drawings are briefly introduced. Obviously, the following drawings only show the contents of some embodiments in this specification, and for those skilled in the art, other drawings and corresponding technical solutions can be derived based on these drawings.

[0069] Figure 1 is a step diagram showing a method for estimating a human body posture based on updating of time-series image features according to an embodiment of the present specification;

[0070] Figure 2 is a schematic diagram for explaining the extraction of key points of a human body from a two-dimensional image in this embodiment;

[0071] Figure 3 is a schematic diagram for illustrating a time-series image feature updating process in this embodiment;

[0072] Figure 4 is a schematic diagram for illustrating two stages of training a human body posture estimation model according to the present embodiment;

[0073] Figure 5 It is a block diagram of the device structure used to illustrate this embodiment. DETAILED DESCRIPTION

[0074] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the present invention is now described in more detail in conjunction with the accompanying drawings and embodiments. It should be understood that these embodiments are only used to explain the present invention, but not to limit the scope thereof. The components shown in the drawings and embodiments can be implemented in a variety of configurations and designs, and the specific described embodiments are intended to illustrate the selected implementation mode, but not to limit the scope of the present invention.

[0075] In some cases, in order to avoid obscuring the core content of the present invention, some operations are not described in detail in the specification. Those skilled in the art can fully understand and implement the relevant operations by combining the specification and their professional knowledge. Therefore, the characteristics, operations or functions described in the specification can be flexibly combined to form different implementation methods. The order or execution method of each step or operation in the method description can also be adjusted according to the judgment of the technician. Unless a certain order is specifically stated to be required, the order in the specification and the drawings is only for clear description of the specific embodiment, and it is not required to be executed in this order. The present application provides a method for human posture estimation based on time-series image feature update, which includes:

[0076] The original video data is collected and preprocessed through the device end; then, the target detection model is used to detect the human body in the processed video and extract the human body bounding box information. On this basis, the feature extraction model is used to extract key features in the bounding box to generate a posture feature vector. Then, the feature classification module classifies these posture feature vectors and divides them into 34 feature substructures, each of which represents a local posture feature set of the human body. Further, the historical feature substructures at moments t-1 and t-2 are updated using the feature substructure at the current moment t to generate 34 comprehensive feature substructures at the current moment t. Finally, the decoding layer recombines the 34 discrete comprehensive feature substructures, and fuses them with the codebook matrix to obtain comprehensive features, and the comprehensive features are input into the decoder to estimate the complete posture of the human body. The features and performance of the present invention are further described in detail below in conjunction with the embodiments and drawings.

[0077] Embodiment 1:

[0078] A preferred embodiment of the present invention provides a method for estimating human posture based on time series image feature update. For specific steps, please refer to Figure 1 , including the following steps:

[0079] S1: Collect human action video through the camera to obtain the corresponding image sequence , and preprocess each frame of the sequence including: noise removal, size adjustment and normalization;

[0080] Specifically, in this embodiment, each frame of the sequence is denoised using bilateral filtering, which combines spatial distance and pixel value differences to maintain the edges and details of the image while effectively removing noise. During the filtering process, the new value of each pixel is calculated by weighted average of neighboring pixels, with the weight determined by spatial similarity and grayscale similarity. This allows the filter to retain the edges while smoothing the image. The bilateral filtering algorithm formula is as shown in formula (1):

[0081] Specifically, in this embodiment, the input image is resized, and the length and width of the image are scaled at a fixed ratio to meet the input size requirements of the subsequent posture estimation model. The calculation formula is as follows:

[0082] (15)

[0083] in, and is the initial size of the image, is the target height.

[0084] Specifically, in this embodiment, the image is normalized to normalize the pixel values ​​to the interval [0, 1] to obtain the pixel data of the t-th frame image. The normalization method is mean-variance normalization, and the formula of mean-variance normalization is as follows:

[0085] (16)

[0086] Among them, x is the initial value of the pixel in the image, μ is the mean of the sample, and σ is the standard deviation of the sample. Through mean-variance normalization, the illumination differences and contrast changes between different images can be eliminated, making the pixel value distribution of each frame more consistent, which is crucial for the subsequent posture estimation model.

[0087] S2: Preprocessed video data through the target detection model Target detection, obtaining human body bounding box information B;

[0088] Specifically, in this embodiment, 1000 photos taken by a camera are selected and annotated for human target detection, specifically: Figure 2 The human body bounding box 200 is shown. The experimental data is divided into training set, validation set and test set in a ratio of 4:1:1. The NanoDet algorithm is used to train the object detection network. NanoDet is a lightweight and efficient object detection algorithm. The algorithm uses ShuffleNetV2 as the backbone network, path aggregation network PAN as the neck structure, NanoDetPlusHead as the main head, SimpleConvHead as the auxiliary head, and combines the allocation guidance module (AGM) and dynamic soft label assigner (DSLA) to optimize the label allocation problem. A variety of loss functions are used in model training: Quality FocalLoss (QFL) is used for classification branch with a weight of 1.0; Distribution Focal Loss (DFL) is used for bounding box accuracy regression with a weight of 0.25; GIoU Loss is used for bounding box regression with a weight of 2.0. The activation function uses LeakyReLU to output the target bounding box and the corresponding confidence. The specific parameters are: 300 training iterations, 0.001 learning rate, 416×416 crop size of training set, and Adam optimizer. The skeleton structure includes Conv1+MaxPool downsampling, multiple lightweight grouped convolution blocks for feature extraction, 1x1 convolution expansion channels, and finally classification through global pooling and fully connected layers to obtain the coordinate information of the bounding box, denoted as B;

[0089] S3.1: Based on the bounding box coordinate information B, the image data in the bounding box is divided into multiple non-overlapping fixed-size sub-blocks using convolution operations, and converted into high-dimensional feature vectors through linear embedding, which are then generated into a feature matrix with expanded dimensions ;

[0090] S3.2: The feature matrix Input sliding window Transformer module to generate posture features . The structure of the sliding window Transformer module is as follows: first, the initial feature mapping is performed through the fully connected layer (Linear Embedding), and then the following processing layers are passed in sequence: 2× sliding window Transformer blocks + block merging layer (Patch Merging) + 2× sliding window Transformer blocks + block merging layer (Patch Merging) + 6× sliding window Transformer blocks + block merging layer (Patch Merging) + 2× sliding window Transformer blocks. Each sliding window Transformer block contains two self-attention modules: window-based multi-head self-attention (W-MSA) and shifted window multi-head self-attention (SW-MSA), and each module is preceded by a normalization layer (Layer Normalization). The output of the self-attention module then passes through a structure including a normalization layer and a multi-layer perceptron (MLP). Residual connections are used in the entire block, spanning the self-attention module and the multi-layer perceptron (MLP) layer respectively.

[0091] S3.3: Posture feature data The spatial resolution of the features is compressed by downsampling layer by layer. Normalization is performed to generate multi-scale fused posture features .

[0092] S4: The feature classification module classifies the posture feature vector generated in S3 and divides it into 34 feature substructures. Each substructure represents the local posture information of the human body. The specific steps to achieve this are:

[0093] S4.1: Multi-scale fusion of pose features using convolution Perform channel adjustment, then flatten the convolutional feature map into a one-dimensional vector, and further adjust the feature dimension through a fully connected layer;

[0094] S4.2: Perform nonlinear transformation on the features obtained in S4.1 by stacking N mixing layers (MixerLayer) to generate 34 discrete feature substructures. The mixing layer is composed of layer normalization layer (Layer Normalization), perceptron (MLP) and jump connection; initialize the human posture feature substructure library as the codebook;

[0095] The fully connected layer is used to calculate each feature substructure to generate the prediction score corresponding to the discrete feature category in the codebook. Finally, the activation function (softmax) is executed to convert the prediction score into a probability set, and the probability set is converted into a feature substructure as shown in formula (2);

[0096] The specific process of initializing the human posture feature substructure library as a code book is as follows: The human body key point position information sequence is obtained by labeling, and then the local posture features are extracted through the linear layer, the dropout layer, the mixing layer, the layer normalization layer and the linear projection layer in sequence. These local posture features are clustered by the K-means clustering algorithm and divided into a finite class of feature substructures. The feature substructure set constitutes the codebook.

[0097] The codebook update process is as follows: the continuous features output by the linear projection layer are converted into discrete vector representations, and then matched with the vectors in the codebook through nearest neighbor search, the closest discrete feature substructure vector is found in the codebook through matching, the feature dot product value of the discrete vector and the discrete feature substructure vector is calculated through the EMA mechanism, and then the corresponding discrete feature substructure vector in the codebook is updated.

[0098] S5: Design a feature update module to use the current feature substructure to update the historical feature substructure and generate a comprehensive feature substructure at the current time t. For the feature update process, please refer to Figure 3 As shown;

[0099] To deepen the understanding of step S5, an example is given here. First, assume that the sample data at the three moments t=1, 2, and 3 are known, and take the first group of 34 groups of data of each sample for example, as shown in Table 1.

[0100] Table 1 Partial sample data

[0101]

[0102] According to step S5, the comprehensive characteristic substructure reliability at three moments can be given as follows:

[0103] When t=1, according to step S5.1, we can get: =0.0068400, 0.0000453, 0.0000064, …, 0.0000005, 0.0000011;

[0104] When t=2, according to step S5.2, set the reward and punishment factor =-0.5; initial reliability value =0.4, according to formula (7) we can get =0.800; According to formulas (9), (10) and (11), we can get =0.454, = 0.456;

[0105] According to formula (8), c=0.498 can be obtained; according to formula (7), =0.199, and It can be optimized by annealing algorithm, and the values ​​are 0.96 and 0.98 respectively; = =0.800; here, according to formulas (3), (4) and (5), renew , and obtain the reliability comprehensive characteristic substructure reliability at time t=2 Reliability =0.328 and =0.0065983, 0.0000434, 0.0000061, …, 0.0000006, 0.0000011;

[0106] When t=3, according to step S5.3.1, the historical feature substructure reliability can be obtained =0.0065758, 0.0000432, 0.0000061, …, 0.0000006, 0.0000011;

[0107] According to step S5.3.2, set the reward and punishment factor =0.5; According to formula (9), (10) and formula (11), we can get =0.463, =0.464; through formula (7) we can get =0.528, and we can get from formula (8) =-0.499; It can be obtained by optimizing the annealing algorithm, with a value of 0.99, and the comprehensive characteristic substructure confidence can be obtained. Reliability =0.562 and =0.0067350, 0.0000422, 0.0000060, …, 0.0000005, 0.0000011;

[0108] In this embodiment, the training of the feature extraction model, feature classification module, decoding layer and code book is mainly divided into two stages, wherein the schematic diagram of the two stages is shown in FIG. Figure 4 The data used is labeled by the labelme tool, and the specific schematic diagram is as follows Figure 2 , Figure 2 Schematic diagram of the key point position annotation of the human body, 17 key nodes (left eye key point 201, left ear key point 202, nose key point 203, right eye key point 204, right ear key point 205, left shoulder key point 206, right shoulder key point 207, left elbow key point 208, right elbow key point 209, left hand key point 210, right hand key point 211, left hip key point 212, right hip key point 213, left knee key point 124, right knee key point 215, left foot key point 216, right foot key point 217) and a human body bounding box 200.

[0109] Specifically, the first stage: training the skeleton encoder, codebook and decoding layer, and establishing the mapping relationship between the key point information of human posture and the posture coding features. Figure 4 In the part where the blue and black arrows are located, the skeleton encoder converts the key nodes of the human body posture into M feature substructures. These feature substructures are quantized by the codebook to generate comprehensive features, and finally the human body posture is restored by the decoding layer; the codebook is a human body posture feature substructure library, and each feature substructure represents a local posture feature set of the human body;

[0110] Specifically, the skeleton encoding layer converts a posture (i.e., the 2D coordinates of 17 joints of the human body, with a size of (17×2)) into 34 feature substructures, each of which encodes a substructure in the posture. The skeleton encoding layer is mainly composed of a linear layer, a dropout layer, a mixed layer (MLP-Mixer), a layer normalization layer, and a linear projection layer. The mixed layer (MLP-Mixer) is composed of layer normalization, MLP, and jump connections. It processes the relationship between each joint through interactive operations to capture local and global features in human posture.

[0111] Specifically, the codebook layer consists of an embedding vector layer and an EMA layer (Exponential Moving Average), which converts the input continuous features into discrete vector representations. The features generated by the encoding layer are matched with the vectors in the codebook through nearest neighbor search, and the closest discrete feature substructure vector is found through retrieval. EMA calculates the feature average by calculating the feature dot product before and after quantization to update the codebook.

[0112] Specifically, the decoding layer converts the discrete feature substructure generated in the skeleton encoding stage back to pose information. This process consists of a list of decoder modules consisting of mixing layers (MixerLayer), layer normalization layers, and linear layers, which gradually process and reconstruct the feature substructure to recover local and global features of the pose. Next, layer normalization (LayerNorm) is applied to ensure that the features remain stable between layers and enhance the numerical stability of the model. The decoded high-dimensional features are then mapped back to the coordinate space of the joints through a linear layer. Finally, the output of the decoding layer is the recovered joint coordinates, which correspond to the pose information originally input to the model.

[0113] The second stage: the codebook layer and decoding layer of the first stage are frozen, and the image encoder, image feature classification module and feature update module are trained, such as Figure 4 In the part where the green arrow and the black arrow are located, the image V cropped to the human body bounding box is input into the sliding window Transformer module feature extraction network to extract features. The high-dimensional feature data of spatial information and human posture information is extracted through the self-attention mechanism. The dimension is reduced through a 1×1 convolution to ensure that the dimension of the feature map is reduced to fit the feature space. The feature map is straightened into a one-dimensional feature vector, and then these features are transformed through a linear projection layer. Finally, an M×N feature substructure matrix is ​​output. The feature matrix is ​​fused with information between different feature substructures through a mixing layer (MixerLayer) and a fully connected operation to capture the dependencies between joints. After the linear classification layer, 34 features are classified. There are 2048 types of feature substructures in the codebook, so the classification result is a 34×2048 prediction score matrix. Each feature substructure (that is, each row) corresponds to an entry in the codebook. The prediction score is normalized using the activation function (Softmax) to obtain the category confidence distribution of each feature substructure. The feature fusion module is used to update the current feature substructure confidence and the fused feature substructure confidence to generate the comprehensive feature substructure confidence at the current moment. The comprehensive feature confidence obtained is multiplied by the codebook to obtain the comprehensive feature, and then the complete posture of the human body is obtained through the decoder. The key nodes of the human body after the image feature update at time t=3 obtained in this embodiment are: (948.28015,417.42545), (947.3989,

[0114] 403.78604)(941.62573406.27756),(910.6815,411.35477),

[0115] (905.63525413.28995),(894.24756,465.88718),(893.0737,476.45175),

[0116] (980.10645,480.20096),(991.67615,514.20593),(1044.3148,416.97916),

[0117] (1062.5427,445.77896),(926.5913,684.7567),(883.78394,700.50415),

[0118] (1027.1978,768.4459), (909.76074853.5393)(1026.3187,909.5581),

[0119] (880.0187,994.77466).

[0120] In the specific testing phase, 1,150 video frames shot on the device were selected for detection, and Object Keypoint Similarity (OKS) and Wing Loss were used as the criteria for measuring the model effect. OKS is used to evaluate the similarity between the key points predicted by the model and the true values, while Wing Loss is a loss function commonly used in regression problems, which can effectively reduce the deviation in key point prediction, especially for key point predictions with large errors. Through these two indicators, the accuracy and robustness of the model are more comprehensively evaluated.

[0121] Specifically, in the key point evaluation task, the key point similarity obtained by the network is not simply calculated, but a certain scale is added to calculate the similarity between the two points. The calculation formula is as follows:

[0122] (17)

[0123] in, Indicates frame, Indicates Frame No. Key points, This is the key point of current detection With the real key point The Euclidean distance between . When the value of is 1, it means that the key point is not blocked and has been marked. When it is 0, it means that the key point is blocked but marked. In this embodiment, it is unblocked and marked by default. It is The scale factor of the frame, usually the area of ​​the detection box or the length of a body part. For the The normalization factor of the key point of a class is calculated based on the standard deviation of the true value of the key point of this class in all sample sets. A larger value means that the key points of this type are more difficult to label.

[0124] Calculate the similarity between the real and detected key points , and then set a threshold T, when When it is greater than T, the estimate can be considered correct (when 17 key points appear, Represents the average of all key points ), through all frames Calculate AP:

[0125] (18)

[0126] Specifically, Wing Loss is a loss function designed for human posture estimation, which aims to improve the accuracy of key point detection, especially for key points that are difficult to predict. The smaller the Wing Loss value, the better the detection effect, indicating that the model is more accurate in dealing with key point errors, especially showing higher accuracy and robustness on difficult key points. The calculation formula is as follows:

[0127] (19)

[0128] Among them: x represents the error between the predicted key point and the true key point, usually the Euclidean distance; w is a hyperparameter used to control the error range. When the error is less than w, the logarithmic loss function is used; ϵ is a small constant to prevent the logarithmic denominator from being zero; C is a constant used to make the value of the loss function continuous when the error is large (usually .

[0129] In this embodiment, the similarity of the key points , select the standard deviation of the coordinates of the key points of the human body posture by calculating the true value =[3.68,3.69,3.74,4.34,4.04,4.14,

[0130] 4.40,3.46,4.26,2.77,2.44,2.50,4.06,3.89,4.78,2.52,4.02]. Select the diagonal length of the human body box obtained by the target detection model , where the parameter reference Figure 2 Two points in the human body bounding box and .

[0131] The key point detection model is used to predict human posture. The average precision AP and WingLoss values ​​of the present invention are compared with the results of the original PCT posture estimation model, as shown in Table 2:

[0132] Table 2: Comparison of the experimental results of the present invention and PCT

[0133]

[0134] It can be seen from the data in Table 2 that the present invention is superior to the PCT human posture estimation model in terms of the average accuracy (AP) and WingLoss of human key point detection. In experiments with different parameters T, the average accuracy of the present invention is higher than that of PCT, especially when T=0.85, the present invention achieves an accuracy of 99.98%, which is significantly better than the 99.74% of PCT. Under other parameter T values, the accuracy of the present invention is also improved. In the comparison of Wing Loss, the loss values ​​of the present invention under different parameters w are lower than those of the PCT model, indicating that the present invention performs better in handling key point detection errors. In particular, when w=10, the Wing Loss of the present invention is 39.484, which is significantly lower than 40.281 of PCT. In general, the present invention has significantly improved the average accuracy of human posture estimation and the accuracy of key point detection, especially in the robustness of key point errors.

[0135] Embodiment 2:

[0136] refer to Figure 5 The human body posture estimation device based on time series image feature update shown in the figure comprises:

[0137] Video acquisition module: collects raw video data through the device end and pre-processes the video data;

[0138] Human object detection module: Use the pre-trained NanoDet object detection model to perform object detection on the pre-processed video data to obtain human bounding box information;

[0139] Feature extraction module: extract features within the human body bounding box and generate posture feature vectors;

[0140] Feature classification module: classifies the extracted posture feature vectors into 34 feature substructures;

[0141] Feature update module: Use the 34 feature substructures at the current time t to update the feature substructures at time t-1 and t-2, and generate 34 comprehensive feature substructures at the current time t;

[0142] Decoding module: Initialize the code book of the human body posture feature substructure library, reassemble the comprehensive feature substructure and fuse it with the code book matrix by dot multiplication, and input the fusion result into the decoder to estimate the complete posture of the human body;

[0143] Result display module: used to display the final result of human posture estimation, by drawing key points and their connecting lines on the image to form a skeleton. This module uses OpenCV to draw the skeleton onto the original image for real-time display.

[0144] In summary, the present invention proposes a method for human posture estimation based on time series image feature update. By updating the historical feature information in the time series, the accuracy and robustness of posture estimation are effectively improved. Different from the prior art, the present invention not only utilizes the key point information in a single frame image, but also fully considers the temporal correlation between consecutive frames, and can capture the continuity and regularity of human movements, providing a real-time, efficient and stable posture estimation solution for the field of intelligent monitoring and behavior recognition.

[0145] The above contents are only specific implementation methods of the embodiments of this specification, and are intended to further illustrate the technical solutions of the present invention and help those skilled in the art better understand its technical concept. Methods and terms (such as algorithms, etc.) not explicitly described can be implemented by conventional techniques in the art and are not subject to specific restrictions. It should be pointed out that for those of ordinary skill in the art, various improvements and adjustments made without departing from the principles of the embodiments of this specification should be regarded as the protection scope of the embodiments of this specification.

Claims

1. A human body posture estimation method based on time series image feature updating, characterized in that: The following steps are involved: S1: Collect video data through the device and perform preprocessing operations on the video data; S2: Use the pre-trained NanoDet target detection model to perform target detection on the pre-processed video data and obtain the human body bounding box information; S3: Extract the features within the human body bounding box through the encoder to generate a posture feature vector; the specific implementation process is as follows: S3.1: Based on the bounding box coordinate information B, the image data in the bounding box is divided into n non-overlapping fixed-size sub-blocks using convolution operations. Each sub-block is converted into a feature vector through linear embedding, and then the feature vector is linearly projected to generate a feature matrix F1,t; S3.2: The feature matrix F 1,t Input sliding window Transformer module to generate posture features F 2,t ; S3.3: Posture feature data F 2,t The spatial resolution of the feature is compressed by downsampling layer by layer. 3,t Perform normalization to generate multi-scale fused posture features F 4,t ; S4: The posture feature vector is classified into 34 feature substructures at the current time t through the image feature classification module; the specific implementation process is as follows: S4.1: Using convolution to fusion multi-scale pose features F 4,t Perform channel adjustment, then flatten the convolutional feature map into a one-dimensional vector, and adjust the feature dimension through a fully connected layer; S4.2: Perform nonlinear transformation on the features obtained in S4.1 by stacking N mixing layers to generate 34 discrete feature substructures; initialize the human posture feature substructure library as the codebook; The hybrid layer is composed of layer normalization layer, perceptron and skip connection The fully connected layer is used to calculate each feature substructure to generate the prediction score corresponding to the discrete feature category in the codebook. Finally, the softmax activation function is executed to convert the prediction score into a probability set, and the probability set is converted into a feature substructure as shown below: Among them, i is the serial number of the characteristic substructure, θ j is a category in the codebook, Θ is the set of all categories in the codebook, is the probability value of the feature substructure taking the category in the codebook; S5: Design a feature update module to use the 34 feature substructures at the current time t to update the feature substructures at times t-1 and t-2, and generate 34 comprehensive feature substructures at the current time t; S6: Recombine the 34 comprehensive feature substructures at the current time t and estimate the complete posture of the human body through the decoder.

2. The method for estimating human posture based on time series image feature updating according to claim 1, characterized in that: The specific implementation process of step S1 is: S1.1: Collect human motion video through the camera and read the corresponding image sequence I1, I2, .., I frame by frame t ,..; S1.2: Preprocess each frame of the image sequence, including denoising, resizing and normalization; The denoising is implemented as bilateral filtering, and the bilateral filtering algorithm formula is as follows: Among them, I t is the t-th frame of the input image; p represents the p-th pixel; Ω is the window centered at p; W p is the normalized weight; Gσ s is the spatial Gaussian kernel, which is used to measure the spatial distance between pixels; Gσ r is the grayscale Gaussian kernel, which is used to measure the grayscale similarity between pixels; I p and I q Represent the grayscale values ​​of pixels p and q respectively; The resizing is to scale the length and width of the image at a fixed ratio to meet the input size requirements of the subsequent feature extraction model; The normalization process is to normalize the pixel value to the interval [0,1], and obtain the pixel data v of the tth frame image p,t .

3. The method for estimating human posture based on time series image feature updating according to claim 2, characterized in that: The specific implementation process of step S2 is: S2.1: The preprocessed pixel data v p,t Input the skeleton network composed of ShuffleNetV2, and extract the features f of 8x, 16x and 32x downsampling from the skeleton network. det18,t ,f det16,t ,f det32,t , and input it into the path aggregation network PAN for multi-scale feature fusion to obtain the fused feature F PAN,t ; S2.2, by using the detection head based on the fully convolutional single-stage target FCOS, the fused feature F PAN,t Processing is performed to obtain the coordinate information B of the bounding box.

4. The method for estimating human posture based on updating of temporal image features according to claim 3, characterized in that: The specific process of initializing the human posture feature substructure library as the code book is as follows: t ,....to mark and obtain the position information sequence of key points of the human body, and then extract the local posture features through linear layer, discard layer, mixing layer, layer normalization layer and linear projection layer in turn, and cluster these local posture features through K-means clustering algorithm and divide them into finite class feature substructures. The feature substructure set constitutes the codebook.

5. The method for estimating human posture based on time series image feature updating according to claim 4, characterized in that: The specific implementation process of step S5 is as follows: S5.1: When t = 1, assume that the comprehensive characteristic substructure evidence about θ j The probability of And S5.2: When t = 2, set the comprehensive characteristic substructure evidence about θ j The updated probability of The calculation formula is as follows: R′ t ＝(1-r t )(1-r′ t-1 ) (4) This formula indicates that when t = 2, renew The updated result at time t=2 is Among them, w1 and w2 are the comprehensive characteristic substructure reliability and the current feature substructure reliability The weight of r1 and r2 are the comprehensive characteristic substructure reliability and the current feature substructure reliability The reliability of the , r1 and r2 are calculated by the following formula: r1＝r0×2 (6) r t =r t-1 +τ×c×r0 (7) Where r0 is the initial value of reliability; τ is the reward and punishment factor. If the reliability of the comprehensive characteristic substructure The maximum value of and the current characteristic substructure confidence The maximum values ​​all point to the same category θ in the codebook. j , then the τ value is 1, otherwise, the τ value is -1; c is the reliability enhancement factor given by the similarity measure between the two credibility: Where T1 and T2 are the characteristic substructure reliability P and the standardized reliability W1=(1,0,...,0), W2=(0,1,…,0),…,W 2048 =(0,0,…,1), is given by: S5.3: When t ≥ 3, the characteristic substructure evidence about θ j The probability of The specific implementation steps are: S5.3.1: Composite feature substructure reliability Reliability of the composite feature substructure Update to obtain the historical characteristic substructure reliability The calculation is as follows: Among them, r′ t-2 and r′ t-1 Comprehensive characteristic substructure reliability and comprehensive characteristic substructure reliability The reliability of is given by formula (5); S5.3.2: Current feature substructure reliability Confidence in historical characteristics substructure Update to get the current comprehensive feature substructure confidence The calculation formula is as follows: Calculated by formula (14) Enter formula (5) and replace Normalize and obtain the comprehensive feature substructure reliability and normalized reliability r′ t ;in, is the reliability of the current feature substructure The weight of t-1 and r t The historical characteristic substructure reliability and the current feature substructure reliability Reliability, r t Calculated by formula (7), r′ t-1 Obtained by formula (5).

6. The method for estimating human posture based on time series image feature updating according to claim 5, characterized in that: The step S6 is specifically implemented as follows: the 34 comprehensive feature substructures at the current time t obtained in S5 are combined by splicing to obtain a comprehensive feature confidence matrix, and then the comprehensive feature confidence matrix is ​​fused with the codebook matrix by dot multiplication to obtain a comprehensive feature, and the obtained comprehensive feature is input into a decoder composed of a linear layer, a mixing layer and a normalization layer in sequence to estimate the complete posture of the human body.

7. A human body posture estimation device based on time series image feature update, used to implement the human body posture estimation method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Video acquisition module: collects raw video data through the device end and pre-processes the video data; Human object detection module: Use the pre-trained NanoDet object detection model to perform object detection on the pre-processed video data to obtain human bounding box information; Feature extraction module: extract features within the human body bounding box and generate posture feature vectors; Feature classification module: classifies the extracted posture feature vectors into 34 feature substructures; Feature update module: Use the 34 feature substructures at the current time t to update the feature substructures at time t-1 and t-2, and generate 34 comprehensive feature substructures at the current time t; Decoding module: Initialize the code book of the human body posture feature substructure library, reassemble the comprehensive feature substructure and fuse it with the code book matrix by dot multiplication, and input the fusion result into the decoder to estimate the complete posture of the human body; Result display module: used to display the position information of key points of the human body estimated by human posture.

Images