A human pose estimation behavior analysis method

Abstract

The application provides a human posture estimation behavior analysis method, comprising: processing human image information into picture data, obtaining global posture features of each group of pictures, and obtaining an image frame sequence with human two-dimensional key point information based on the global posture features; establishing a motion model according to the image frame sequence, estimating the task motion state in the image frame sequence, constructing a Transformer model, and generating posture hypotheses based on a multi-hypothesis generator of the Transformer model to regress each group of posture hypotheses to generate posture hypothesis information; constructing a behavior prediction network, learning based on pre-collected posture data to obtain optimal parameters of the network; applying the optimal parameters to the behavior prediction network, predicting the posture hypothesis information based on the behavior prediction network, and finally outputting predicted posture estimation data. The application does not require manual parameter setting, improves the efficiency and accuracy of neural network parameter searching, is simple to operate, and is convenient for staff to use.

Description

Technical Field

[0001] This invention relates to the field of posture estimation technology, and more particularly to a method for human posture estimation behavior analysis. Background Technology

[0002] Human pose estimation is an important branch of computer vision with a wide range of applications. It estimates human pose by correctly linking detected human key points in an image. Human key points typically correspond to joints with a certain degree of freedom, such as the neck, shoulder, elbow, wrist, waist, knee, and ankle. Applications include pedestrian pose detection and action prediction in street scenes in the autonomous driving industry; pedestrian re-identification in the security field; specific action monitoring in special scenarios; and movie special effects in the film industry.

[0003] Chinese patent application CN114999002A discloses a behavior recognition method that integrates human posture information. Although this invention is highly stable and overcomes the influence of the translation of skeletal point coordinates on the recognition ability of graph convolutional neural networks, and integrates information from previous and next frames of the image with human key point information, the fusion of information helps to improve the performance of action recognition, it requires manual setting of parameters, which reduces the efficiency and accuracy of the neural network in finding parameters. It requires experienced technicians to operate and is inconvenient for ordinary people to use. Summary of the Invention

[0004] This invention provides a method for human posture estimation and behavior analysis. Based on multiple uploaded sets of posture data, this invention automatically obtains the optimal parameter settings for an estimation network, and then performs posture prediction. This eliminates the need for manual parameter setting, improves the efficiency and accuracy of the neural network in finding parameters, and is simple to operate and convenient for staff to use.

[0005] The technical means employed in this invention are as follows:

[0006] A human pose estimation behavior analysis method includes:

[0007] S1. Process the human image information into image data. After preprocessing the image data, obtain the global pose features of each group of images. Based on the global pose features, obtain an image frame sequence with human two-dimensional key point information.

[0008] S2. Offline processing of single-camera video or image sequence frames with a fixed frame rate, establishing a motion model, estimating the task motion state in the image frame sequence to obtain 2D pose data, constructing a Transformer model and processing the 2D pose data based on the Transformer model's multi-hypothesis generator to generate pose hypotheses, and performing regression on each set of pose hypotheses to generate pose hypothesis information.

[0009] S3. Construct a behavior prediction network and learn from multiple pre-collected pose data to obtain the optimal parameters of the network;

[0010] S4. Apply the optimal parameters to the behavior prediction network, and predict the attitude assumption information based on the behavior prediction network, and finally output the predicted attitude estimation data.

[0011] Further, the image data is preprocessed, including:

[0012] The image data of each group is transformed from the image space to the frequency space by using Fourier transform, and its high-frequency components are filtered to reduce noise interference.

[0013] Then, the filtered image data is converted from frequency space to image space using inverse Fourier transform.

[0014] Further, global pose features of each group of images are obtained, and two-dimensional key points of the human body are obtained based on the global pose features, including:

[0015] The global pose features of each group of image data were obtained by collecting multiple ShuffleBlocks from the network.

[0016] The global pose features are regressed onto the keypoint feature map through deconvolution.

[0017] The key point feature map is decoded, and the generated two-dimensional key points of the human body are collected.

[0018] Furthermore, a motion model is established based on the image frame sequence, the task motion state in the image frame sequence is estimated to obtain 2D pose data, a Transformer model is constructed, and a multi-hypothesis generator based on the Transformer model processes the 2D pose data to generate pose hypotheses. Regression is performed on each set of pose hypotheses to generate pose hypothesis information, including:

[0019] S201. Process the current person's video information or image sequence frames offline, calculate and record the interval time of the actual video frames, and then establish a motion model based on Kalman filtering theory.

[0020] S202. Assign an ID to all people in the image information. After the assignment is completed, define the motion state of the people in the video frame based on the linear motion assumption of the people through the motion model. Collect the motion state of each person in the current video frame and construct a prediction equation to estimate the motion state of each tracked target in the next video frame to obtain 2D pose data.

[0021] S203. Construct a Transformer model and input 2D pose data into the Transformer model. In the Transformer model, a multi-hypothesis generator receives each set of 2D pose data and generates different representations of pose hypotheses at different layers of the model. Then, single hypothesis dependencies are modeled through multiple parallel self-attention blocks to form self-hypothesis communication.

[0022] S204. The Hybrid Hypothesis MLP extracts the features of each hypothesis from the concatenation and segments them to obtain each corrected hypothesis. The cross hypothesis interactor performs interactive modeling of information from different hypotheses. Finally, the Transformer model regression module regresses each group of pose hypotheses to obtain the final 3D pose data.

[0023] Furthermore, a behavior prediction network is constructed and learned based on multiple pre-collected sets of pose data to obtain the optimal parameters of the network, including:

[0024] S301. The behavior prediction network collects multiple sets of pose data that have been pre-uploaded, selects one set as validation data, and then fits the remaining data into a set of test models. The detection accuracy of the test model is then verified using the validation data. After that, the validation data is replaced and the verification is repeated until all pose data has been verified.

[0025] S302. Initialize the parameter range and list all possible data results according to the preset learning rate and step size. For each set of data, select any subset as the test set and the remaining subset as the training set to train the test model. After training, predict the test set and calculate the root mean square error of the test results.

[0026] S303. Simultaneously, replace the test set with another subset, and then take the remaining subset as the training set. Calculate the root mean square error again until all data have been predicted once. Select the combination parameters corresponding to the minimum root mean square error as the optimal parameters within the data interval.

[0027] Furthermore, the optimal parameters are applied to the behavior prediction network, and the pose assumption information is predicted based on the behavior prediction network. Finally, the predicted pose estimation data is output, including:

[0028] S401. The behavior prediction network receives 3D pose data generated by the Transformer model, replaces the original parameters with the optimal parameters, and then imports the key point information of each person in the current image information into the behavior prediction network.

[0029] S402. Divide the key point information of each person in the current image information into a training set and a test set, and standardize the training set. Then, import the training samples generated by the standardization process into the behavior prediction network. At the same time, train the behavior prediction network using a long-term iterative method, and input the test set into the trained model. Output the prediction percentage of 3D pose data, and output the highest 3D pose data as the prediction result.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. Compared with previous behavior analysis methods, the human posture estimation behavior analysis method provided by this invention collects multiple sets of posture data uploaded by staff through a behavior prediction network, selects one set as validation data, fits the remaining data into a test model and verifies its detection accuracy, then lists all possible data results based on manually set or system default settings of learning rate and step size, predicts each set of data results and records the root mean square error of each set of data, and selects the combination of parameters with the smallest root mean square error as the optimal parameters. Then, the key point information of each person in the current image information and the 3D posture data generated by the Transformer model are imported into the behavior prediction network for posture prediction, and the predicted percentage of 3D posture data is output, with the highest 3D posture data as the prediction result. No manual parameter setting is required, which improves the efficiency and accuracy of the neural network in finding parameters. It is simple to operate and convenient for staff to use.

[0032] 2. The human pose estimation behavior analysis method provided by this invention calculates and records the interval time of actual video frames, then establishes a motion model based on Kalman filtering theory to define the motion state of the person in the video frame, and constructs a prediction equation to estimate the motion state of each tracked target in the next video frame to obtain 2D pose data. The 2D pose data is then input into a Transformer model. The multi-hypothesis generator in this model receives each set of 2D pose data and generates different representations of pose hypotheses at different layers. Then, multiple parallel self-attention blocks are used to model the dependence of a single hypothesis, extract the features of each hypothesis, and segment them to obtain each corrected hypothesis. The information of different hypotheses is interactively modeled. Finally, the Transformer model regression module regresses each set of pose hypotheses to obtain the final 3D pose data. Obtaining the 3D pose prediction data of each person through the Transformer model allows staff to view the estimation results more intuitively, improves the user experience, and is beneficial for subsequent prediction of person poses, effectively improving the estimation accuracy. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of a human posture estimation behavior analysis method according to the present invention.

[0035] Figure 2 This is a flowchart of the algorithm for a human posture estimation behavior analysis method according to the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0037] like Figure 1-2 As shown in the figure, an embodiment of the present invention discloses a human posture estimation behavior analysis method, the specific steps of which are as follows:

[0038] S1. Collect image information of people and extract key points.

[0039] Specifically, the acquired human image information is extracted frame by frame according to manually set time frames to obtain corresponding image data. The number of blocks is determined based on the image display ratio, where the image display ratio is the height-to-width ratio in pixels, determined based on the image's height and width according to user settings or system defaults. Then, the images are divided into blocks based on the number of blocks. A forward Fourier transform is used to convert each block of image data from image space to frequency space, and the high-frequency components are filtered to reduce noise interference. Then, an inverse Fourier transform is used to convert each block of image data from frequency space back to image space. The data is then processed through multiple ShuffleBlock operations by the acquisition network to obtain the global pose features F of each block of image data. mid Then, the global pose feature F is made more accurate through deconvolution. mid The process is then redirected to the key point feature map F0, which is then decoded, and the resulting two-dimensional human body key points are collected.

[0040] In this embodiment, the specific formulas for the forward and inverse Fourier transforms are as follows:

[0041]

[0042] Where u and v are frequency variables, x and y are the coordinates of each pixel corresponding to each image data, N represents the Fourier transform coefficients, formula (1) is the forward Fourier transform, and formula (2) is the inverse Fourier transform.

[0043] Global pose feature F mid The specific formula is as follows:

[0044] F mid =ρ(M,w,b) (3)

[0045] In the formula, ρ(,) represents multiple ShuffleBlocks, M represents the input image, and w and b represent the learnable convolution kernel and bias.

[0046] The specific formula for deconvolution is as follows:

[0047] F0 = f -1 (F mid ,w,b) (4)

[0048] In the formula, F0 represents a tensor with dimensions (17, n, m), where 17 represents the number of key points, and n and m represent the width and height of the feature map;

[0049] The specific formula for decoding is as follows:

[0050] J i =Max(F0(i)) (5)

[0051] In the formula, J i F0(i) represents the i-th key point and F0(i) represents the i-th key point feature map.

[0052] It should be further explained that ShuffleBlock mainly includes Channel Split and ChannelShuffle. Channel Split is a channel separation operation, and Channel Shuffle is a channel shuffling operation. The key points of the character include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0053] S2. Obtain attitude hypothesis information based on the current image information.

[0054] For details, please refer to Figure 2It is known that offline processing of single-camera video or image sequences with a fixed frame rate captures motion impact information of the target task within these frames. The interval time between actual video frames is calculated and recorded. Then, a motion model is established based on Kalman filtering theory. Each person in the image information is assigned a unique ID. After assignment, the motion model, based on the linear motion assumption of the people, defines their motion state within the video frame. The motion state of each person in the current video frame is collected, and a prediction equation is constructed to estimate the motion state of each tracked target in the next video frame to obtain 2D pose data. This prediction equation is:

[0055]

[0056] In the formula, This represents the mean motion state of the tracked target predicted by the linear motion model in the next video frame. A represents the best estimate of the motion state of the tracked target in the current video frame. k+1 P represents the state transition matrix at time k+1. k+1,k P represents the predicted motion state covariance matrix of the tracked target in the next video frame. k,k Q represents the covariance matrix of the tracking target's motion state in the current video frame. k+1 The noise matrix of the motion model at time k+1.

[0057] A Transformer model is constructed, and 2D pose data is input into it. A multi-hypothesis generator within the model receives the 2D pose data from various sets and generates different representations of pose hypotheses at different layers of the model. Then, multiple parallel self-attention blocks are used to model the dependencies of single hypotheses, forming self-hypothesis communication. The generated pose hypotheses are used to obtain the positions of keypoints on each human body, thus confirming the hypothetical features. A hybrid hypothesis MLP extracts the concatenated features of each hypothesis and segments them to obtain a revised hypothesis. A cross-hypothesis interactor models the information from different hypotheses interactively. Finally, the Transformer model regression module regresses the pose hypotheses to obtain the final 3D pose data.

[0058] S3. Construct a behavior prediction network and find the optimal parameters.

[0059] Specifically, the behavior prediction network collects multiple sets of pose data uploaded by staff, selects one set as validation data, and then fits the remaining data into a test model. The detection accuracy of the test model is then verified using the validation data. This process is repeated with different validation data until all pose data has been validated. The parameter range is initialized, and the learning rate and step size are set manually or by system default. All possible data results are listed. For each set of data, any subset is selected as the test set, and the remaining subsets are used as the training set to train the test model. After training, predictions are made on the test set, and the root mean square error (RMSE) of the test results is calculated. The test set is then replaced with another subset, and the remaining subsets are used as the training set. The RMSE is calculated again until all data has been predicted. The optimal parameters within the data range are selected based on the combination of parameters with the smallest RMSE.

[0060] S4. Estimate the pose of the current character.

[0061] Specifically, the behavior prediction network receives 3D pose data generated by the Transformer model, replaces the original parameters with optimal parameters, and then imports the key point information of each person in the current image information into the behavior prediction network. The key point information of each person in the current image information is divided into training set and test set, and the training set is standardized. The training samples generated by the standardized process are then imported into the behavior prediction network. At the same time, the behavior prediction network is trained using a long-term iterative method, and the test set is input into the trained model. The network outputs the predicted percentage of 3D pose data and outputs the highest 3D pose data as the prediction result.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human posture estimation and behavior analysis, characterized in that, include: S1. Process the human image information into image data. After preprocessing the image data, obtain the global pose features of each group of images. Based on the global pose features, obtain an image frame sequence with human two-dimensional key point information. S2. Offline processing of single-camera video or image sequence frames with a fixed frame rate, establishing a motion model, estimating the task motion state in the image frame sequence to obtain 2D pose data, constructing a Transformer model and processing the 2D pose data based on the Transformer model's multi-hypothesis generator to generate pose hypotheses, interactively modeling the information of different hypotheses through a cross-hypothesis interactor, and regressing each group of pose hypotheses to generate pose hypothesis information. S3. Construct a behavior prediction network and learn it based on multiple pre-collected pose data. Select the combination parameters corresponding to the minimum root mean square error as the optimal parameters within the data interval to obtain the optimal parameters of the network. S4. Apply the optimal parameters to the behavior prediction network, and predict the attitude assumption information based on the behavior prediction network, and finally output the predicted attitude estimation data.

2. The human posture estimation behavior analysis method according to claim 1, characterized in that, Preprocessing the image data includes: The image data of each group is transformed from the image space to the frequency space by using Fourier transform, and its high-frequency components are filtered to reduce noise interference. Then, the filtered image data is converted from frequency space to image space using inverse Fourier transform.

3. The human posture estimation behavior analysis method according to claim 2, characterized in that, Obtain global pose features from each group of images, and obtain 2D key points of the human body based on the global pose features, including: The global pose features of each group of image data were obtained by collecting multiple ShuffleBlocks from the network. The global pose features are regressed onto the keypoint feature map through deconvolution. The key point feature map is decoded, and the generated two-dimensional key points of the human body are collected.

4. The human posture estimation behavior analysis method according to claim 1, characterized in that, A motion model is established based on the image frame sequence. The task motion state in the image frame sequence is estimated to obtain 2D pose data. A Transformer model is constructed, and a multi-hypothesis generator based on the Transformer model processes the 2D pose data to generate pose hypotheses. Regression is performed on each set of pose hypotheses to generate pose hypothesis information, including: S201. Process the current person's video information or image sequence frames offline, calculate and record the interval time of the actual video frames, and then establish a motion model based on Kalman filtering theory. S202. Assign an ID to all people in the image information. After the assignment is completed, define the motion state of the people in the video frame based on the linear motion assumption of the people through the motion model. Collect the motion state of each person in the current video frame and construct a prediction equation to estimate the motion state of each tracked target in the next video frame to obtain 2D pose data. S203. Construct a Transformer model and input 2D pose data into the Transformer model. In the Transformer model, a multi-hypothesis generator receives each set of 2D pose data and generates different representations of pose hypotheses at different layers of the model. Then, single hypothesis dependencies are modeled through multiple parallel self-attention blocks to form self-hypothesis communication. S204. The Hybrid Hypothesis MLP extracts the features of each hypothesis from the concatenation and segments them to obtain each corrected hypothesis. The cross hypothesis interactor performs interactive modeling of information from different hypotheses. Finally, the Transformer model regression module regresses each group of pose hypotheses to obtain the final 3D pose data.

5. The human posture estimation behavior analysis method according to claim 1, characterized in that, A behavior prediction network is constructed and learned based on multiple pre-collected pose data sets to obtain the optimal parameters of the network, including: S301. The behavior prediction network collects multiple sets of pose data that have been pre-uploaded, selects one set as validation data, and then fits the remaining data into a set of test models. The detection accuracy of the test model is then verified using the validation data. After that, the validation data is replaced and the verification is repeated until all pose data has been verified. S302. Initialize the parameter range and list all possible data results according to the preset learning rate and step size. For each set of data, select any subset as the test set and the remaining subset as the training set to train the test model. After training, predict the test set and calculate the root mean square error of the test results. S303. Simultaneously, replace the test set with another subset, and then take the remaining subset as the training set. Calculate the root mean square error again until all data have been predicted once. Select the combination parameters corresponding to the minimum root mean square error as the optimal parameters within the data interval.

6. The human posture estimation behavior analysis method according to claim 1, characterized in that, The optimal parameters are applied to the behavior prediction network, and the pose assumption information is predicted based on the behavior prediction network. The final output is the predicted pose estimation data, including: S401. The behavior prediction network receives 3D pose data generated by the Transformer model, replaces the original parameters with the optimal parameters, and then imports the key point information of each person in the current image information into the behavior prediction network. S402. Divide the key point information of each person in the current image information into a training set and a test set, and standardize the training set. Then, import the training samples generated by the standardization process into the behavior prediction network. At the same time, train the behavior prediction network using a long-term iterative method, and input the test set into the trained model. Output the prediction percentage of 3D pose data, and output the highest 3D pose data as the prediction result.

Images