Human body key point analysis training and reasoning method and device based on video stream

Abstract

The present invention discloses a method and device for human key point analysis training and reasoning based on video streams. The method comprises the following steps: using all human key points and occlusion marks of historical continuous video frames as sample training data to train a human key point analysis model; using all human key points and occlusion marks of current continuous video frames as original data to initialize the human key point analysis model; using the human key point analysis model to infer the position of the human key point corresponding to each occlusion mark of the current continuous video frame to obtain the human posture of the current continuous video frame; the present invention introduces occlusion marks and integrates occlusion marks into human key point analysis training and reasoning, thereby greatly improving the recognition accuracy of human key points.

Description

Technical Field

[0001] The present invention relates to the technical field of human key point analysis, and in particular to a human key point analysis training and reasoning method and device based on video stream. Background Art

[0002] Traditional human keypoint analysis algorithms primarily take images as input, analyze them, and output the keypoints of the human body. However, when keypoints are obscured, the algorithms often struggle to analyze and infer their correct locations.

[0003] However, traditional human key point analysis algorithms do not consider the video information of previous and next frames when analyzing human key points in video streams. The algorithm has limited accuracy, and during training and inference, it does not fully utilize the information of unobstructed key points to improve the accuracy of key recognition, which often leads to a large difference between the correct position of the inferred key points and the actual one. Summary of the Invention

[0004] The purpose of the present invention is to provide a human body key point analysis training and reasoning method based on video stream, introduce occlusion markers, and integrate occlusion markers into human body key point analysis training and reasoning, thereby greatly improving the recognition accuracy of human body key points.

[0005] In order to achieve the above purpose, the present invention discloses a human body key point analysis training and reasoning method based on video stream, which comprises the following steps:

[0006] S1. Use all human key points and occlusion marks of historical continuous video frames as sample training data to train the human key point analysis model;

[0007] S2. Initializing the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data;

[0008] S3. Inferring the position of the human key point corresponding to each of the occlusion marks in the current continuous video frame through the human key point analysis model to obtain the human body posture of the current continuous video frame.

[0009] Preferably, the step (1) specifically includes:

[0010] S11. Randomly intercept the historical continuous video frames according to a first preset number of frames to obtain temporary historical continuous video frames;

[0011] S12, performing bidirectional recurrent neural network analysis training on the human body key points and occlusion marks of the temporary historical continuous video frames to obtain the positions of the human body key points corresponding to all occlusion marks of the temporary historical continuous video frames;

[0012] S13, updating the temporary historical continuous video frames according to the positions of the human body key points corresponding to all the occlusion marks of the temporary historical continuous video frames;

[0013] S14, updating the new temporary historical continuous video frame to the historical continuous video frame;

[0014] S15. Iteratively train the human body key point analysis model based on the new historical continuous video frames.

[0015] Specifically, the occlusion mark in any video frame in each of the temporary historical continuous video frames needs to have at least one corresponding unoccluded human key point in the remaining video frames in the temporary historical continuous video frames.

[0016] Preferably, the step (3) specifically includes:

[0017] S31, dividing the current continuous video frames into a plurality of current continuous video frame units according to a second preset number of frames;

[0018] S32, performing bidirectional recurrent neural network reasoning on the human body key points and occlusion marks of each of the current continuous video frame units to obtain the positions of the human body key points corresponding to the occlusion marks of each of the current continuous video frame units;

[0019] S33: Combining the positions of the human body key points and the human body key points corresponding to the occlusion marks of all current continuous video frame units, and analyzing to obtain the human body posture corresponding to the current continuous video frame.

[0020] Specifically, the step (32) further comprises:

[0021] S321, calculating the number of occlusion marks in each of the current continuous video frame units;

[0022] S322, calculating the difference between the number of occlusion marks in each of the current continuous video frame units and a preset threshold;

[0023] S323: Determine whether to perform bidirectional recurrent neural network inference on the human body key points and occlusion marks of the corresponding current continuous video frame unit based on each of the differences.

[0024] Specifically, the step (323) includes:

[0025] S3231: If the difference is greater than or equal to the preset threshold, performing bidirectional recurrent neural network inference on the human body key points and occlusion marks of the current continuous video frame unit;

[0026] S3232: If the difference is less than the preset threshold, ignore the position of the human body key point corresponding to the occlusion mark of the current continuous video frame unit.

[0027] Preferably, the occlusion mark in any video frame in each of the current continuous video frame units needs to have at least one corresponding unoccluded human key point in the remaining video frames in the current continuous video frame unit.

[0028] Preferably, before step (1), the step further includes:

[0029] S101, obtaining the historical continuous video frames;

[0030] S102, performing a human body key point algorithm analysis on the historical continuous video frames to obtain human body key point feature values ​​of the historical video frames;

[0031] S103: Analyze the characteristic values ​​of the human body key points of the historical video frames to obtain the human body key points and occlusion marks of the historical video frames.

[0032] Preferably, the step (2) further includes:

[0033] S201, obtaining the current continuous video frames;

[0034] S202, performing a human body key point algorithm analysis on the current continuous video frame to obtain a human body key point feature value of the current video frame;

[0035] S203: Analyze the characteristic values ​​of the human body key points of the current video frame to obtain the human body key points and occlusion marks of the current video frame.

[0036] Accordingly, the present invention also provides a human body key point analysis training and reasoning device based on video stream, which comprises:

[0037] A training module is used to train a human key point analysis model using all human key points and occlusion marks in historical continuous video frames as sample training data;

[0038] An initialization module is used to initialize the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data;

[0039] The inference module is used to infer the position of the human key point corresponding to each of the occlusion marks in the current continuous video frame through the human key point analysis model to obtain the human posture of the current continuous video frame.

[0040] Compared with the prior art, the present invention uses all human key points and occlusion marks of historical continuous video frames as sample training data to train a human key point analysis model, and uses the human key point analysis model to infer the position of the human key point corresponding to each occlusion mark of the current continuous video frame to obtain the human posture of the current continuous video frame. On the one hand, by introducing occlusion marks to train the human key point analysis model, the human key point analysis model has a higher ability to recognize the position of the human key points corresponding to the occlusion marks, thereby improving the ability of the human key point analysis model to recognize the position of the human key points corresponding to the occlusion marks; on the other hand, continuous frames are used for training and reasoning, and the correlation between the human key points and occlusion marks of the previous and next frames is fully utilized, thereby further improving the ability of the human key point analysis model to recognize the position of the human key points corresponding to the occlusion marks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the human body key point analysis training and reasoning method based on video stream of the present invention;

[0042] Figure 2 It is a flowchart of step (1) in the human body key point analysis training and reasoning method based on video stream of the present invention;

[0043] Figure 3 It is an execution flow chart of the human body key point analysis model of historical continuous video frames in step (1) of the human body key point analysis training and reasoning method based on video stream of the present invention;

[0044] Figure 4 It is a flowchart of step (3) in the human body key point analysis training and reasoning method based on video stream of the present invention;

[0045] Figure 5 It is an execution flow chart of inferring the positions of the human key points corresponding to the occlusion marks in the current continuous video frames in steps (2) and (3) of the human key point analysis training and reasoning method based on video stream of the present invention;

[0046] Figure 6 It is a structural block diagram of the human body key point analysis training and reasoning device based on video stream of the present invention. DETAILED DESCRIPTION

[0047] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the embodiments and the accompanying drawings.

[0048] See also Figure 1-Figure 3As shown, the human body key point analysis, training, and reasoning method based on video streams of this embodiment can perform model learning and reasoning on the human body key points and occlusion positions of continuous video frames through a bidirectional recurrent neural network, effectively improving the recognition accuracy of human body key points in video streams. The human body key point analysis, training, and reasoning method based on video streams of this embodiment includes the following steps:

[0049] S1. Use all human key points and occlusion marks of historical continuous video frames as sample training data to train the human key point analysis model;

[0050] S2. Initializing the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data;

[0051] S3. Inferring the position of the human key point corresponding to each of the occlusion marks in the current continuous video frame through the human key point analysis model to obtain the human body posture of the current continuous video frame.

[0052] It is understood that the historical continuous video frames and current continuous video frames mentioned here are continuous video frames from different time periods of the same video stream. The continuous video frames here refer to all continuous images that constitute the video stream of the corresponding time period. For example, under normal circumstances, a video stream with a duration of one second consists of 24 continuous images. Of course, the historical continuous video frames and current continuous video frames can also be continuous video frames of the same type from different video streams to meet more diverse material training and inference requirements.

[0053] For ease of understanding, Figure 3 A flowchart of the human key point analysis model of historical continuous video frames in step (1) is given.

[0054] The target value p of algorithm training * It is a heat map of key points generated by Gaussian distribution based on the key point positions. The loss function L used in algorithm training is:

[0055]

[0056] p * i,j,c,t Represents the target probability value at coordinate position i, j and channel c at time t, p i,j,c Represents the predicted probability value at coordinate position i, j and channel c at time t. That is, the mean square error of the predicted probability value of each point and the probability value generated by the Gaussian distribution and the mean square error of the occlusion mark are calculated.

[0057] Preferably, the step (1) specifically includes:

[0058] S11. Randomly intercept the historical continuous video frames according to a first preset number of frames to obtain temporary historical continuous video frames;

[0059] S12, performing bidirectional recurrent neural network analysis training on the human body key points and occlusion marks of the temporary historical continuous video frames to obtain the positions of the human body key points corresponding to all occlusion marks of the temporary historical continuous video frames;

[0060] S13, updating the temporary historical continuous video frames according to the positions of the human body key points corresponding to all the occlusion marks of the temporary historical continuous video frames;

[0061] S14, updating the new temporary historical continuous video frame to the historical continuous video frame;

[0062] S15. Iteratively train the human body key point analysis model based on the new historical continuous video frames.

[0063] It can be understood that since different historical continuous video frames have different numbers of video frames, and video streams generally have a large number of video frames, in order to ensure the hardware requirements and software requirements of model training, the efficiency and stability of model training, each time the model is trained with temporary historical continuous video frames with a first preset number of frames. The first preset number of frames here can be twenty-four, or thirty or other values. The setting of this value is set according to the actual hardware and software parameters.

[0064] In addition, the temporary historical continuous video frames used each time are randomly extracted to meet the randomness of model training and facilitate the improvement of subsequent reasoning accuracy.

[0065] Specifically, the occlusion mark in any video frame in each of the temporary historical continuous video frames needs to have at least one corresponding unobstructed human key point in the remaining video frames in the temporary historical continuous video frames. It is understandable that the preceding and following frames of the continuous video are limited to have at least one corresponding unobstructed human key point, so that the corresponding human key point can be used as a reference when identifying and inferring the position of the occlusion mark, thereby improving the recognition and inference accuracy of the human key point analysis model for the occlusion mark.

[0066] Preferably, before step (1), the step further includes:

[0067] S101: Acquire the historical continuous video frames.

[0068] S102: Performing a human body key point algorithm analysis on the historical continuous video frames to obtain human body key point feature values ​​of the historical video frames.

[0069] S103: Analyze the characteristic values ​​of the human body key points of the historical video frames to obtain the human body key points and occlusion marks of the historical video frames.

[0070] It can be understood that step (103) is to obtain the human key point feature values ​​of the historical video frame through a human key point analysis algorithm. The human key point analysis algorithm can adopt an existing human key point analysis algorithm or be replaced by other algorithms.

[0071] Step (103) is to obtain the human body key points and occlusion marks of the historical video frame through bidirectional recurrent neural network (LSTM) reasoning. The bidirectional recurrent neural network includes a forward LSTM layer and a reverse LSTM layer. The forward LSTM layer deduces and trains the historical continuous video frames in forward time sequence, while the reverse LSTM layer deduces and trains the historical continuous video frames in reverse time sequence. The advantage of using a bidirectional recurrent neural network is that it can perform forward and reverse deduction and training on the same video frame in combination with the previous frame and the next frame to improve the reasoning and recognition effect.

[0072] See also Figure 1-Figure 5 As shown, for ease of understanding, Figure 5 The flowchart of inferring the position of the human key points corresponding to the occlusion mark in the current continuous video frames through the human key point analysis model in step (2) and step (3) is given.

[0073] Preferably, the step (3) specifically includes:

[0074] S31. Divide the current continuous video frames into a plurality of current continuous video frame units according to a second preset number of frames.

[0075] S32. Perform bidirectional recurrent neural network reasoning on the human body key points and occlusion marks of each of the current continuous video frame units to obtain the positions of the human body key points corresponding to the occlusion marks of each of the current continuous video frame units.

[0076] S33: Combining the positions of the human body key points and the human body key points corresponding to the occlusion marks of all current continuous video frame units, and analyzing to obtain the human body posture corresponding to the current continuous video frame.

[0077] It can be understood that since the number of video frames of different current continuous video frames is different, and the video stream generally has a large number of video frames, in order to ensure the efficiency and stability of the hardware requirements and software requirements of model training, the model training is performed each time with a temporary historical continuous video frame with a second preset number of frames. The second preset number of frames here can be twenty-four, or thirty or other values. The setting of this value is set according to the actual hardware and software parameters.

[0078] By dividing the current continuous video frame into multiple current continuous video frame units, each current continuous video frame unit is independently inferred to obtain the position of the human body key point corresponding to the occlusion mark of each current continuous video frame unit, thereby reducing the model training load.

[0079] Since step (32) has completed the inference of the positions of the human body key points corresponding to the occlusion marks of all current continuous video frame units, step (33) only needs to jointly calculate the positions of the human body key points and the human body key points corresponding to the occlusion marks of all current continuous video frame units to obtain the human body posture corresponding to the current continuous video frame.

[0080] The target value p of algorithm training * It is a heat map of key points generated by Gaussian distribution based on the key point positions. The loss function L used in algorithm training is:

[0081]

[0082] p * i,j,c,t Represents the target probability value at coordinate position i, j and channel c at time t, p i,j,c Represents the predicted probability value at coordinate position i, j and channel c at time t. That is, the mean square error of the predicted probability value of each point and the probability value generated by the Gaussian distribution and the mean square error of the occlusion mark are calculated.

[0083] Specifically, the step (32) further comprises:

[0084] S321: Calculate the number of occlusion marks in each of the current continuous video frame units.

[0085] S322: Calculate the difference between the number of occlusion marks in each of the current continuous video frame units and a preset threshold.

[0086] S323: Determine whether to perform bidirectional recurrent neural network inference on the human body key points and occlusion marks of the corresponding current continuous video frame unit based on each of the differences.

[0087] It is understandable that since bidirectional recurrent neural network reasoning on the human body key points and occlusion marks of the current continuous video frame unit requires a certain amount of computation and time, the significance of the above steps is to determine whether it is necessary to perform bidirectional recurrent neural network reasoning on the current continuous video frame unit based on the number of occlusion marks in the current continuous video frame unit. This judgment is generally used as an arbitration analysis. Since there are cases where the number of occlusion marks in a certain current continuous video frame unit is very small or zero, when the current continuous video frame unit is subjected to bidirectional recurrent neural network reasoning, the actual position of the human body key points corresponding to the occlusion marks that can be solved by reasoning is limited, and the amount of computation and time for reasoning of this part is similar to that of the rest of the current continuous video frame units, ignoring the reasoning of the occlusion marks of this part has a very small impact on the overall reasoning. Therefore, by introducing the arbitration analysis mechanism, the amount of computation and time for reasoning can be saved, and the recognition accuracy of the overall reasoning can be improved.

[0088] Specifically, the step (323) includes:

[0089] S3231: If the difference is greater than or equal to the preset threshold, perform bidirectional recurrent neural network inference on the human body key points and occlusion marks of the current continuous video frame unit.

[0090] S3232: If the difference is less than the preset threshold, ignore the position of the human body key point corresponding to the occlusion mark of the current continuous video frame unit.

[0091] The arbitration analysis determines whether the key points of the human body are blocked based on the output of the previous step. The activation strategy is as follows. Set V k (t) indicates whether the key point k of the human body is blocked at time t. The formula for determining whether it is blocked is as follows:

[0092]

[0093] The above condition states that if at least w / a frames out of the total w video frames are not occluded, the strategy is enabled. The parameters w and a can be adjusted based on the actual situation.

[0094] Preferably, the occlusion mark in any video frame in each of the current continuous video frame units needs to have at least one corresponding unoccluded human key point in the remaining video frames in the current continuous video frame unit.

[0095] It can be understood that here the front and rear frames of the continuous video are limited to have at least one corresponding unobstructed human key point, so that when identifying and inferring the position of the occlusion mark, there is a corresponding human key point as a reference basis, so as to improve the recognition and reasoning accuracy of the human key point analysis model for the occlusion mark.

[0096] Preferably, the step (2) further includes:

[0097] S201: Acquire the current continuous video frames.

[0098] S202: Perform human body key point algorithm analysis on the current continuous video frame to obtain human body key point feature values ​​of the current video frame.

[0099] S203: Analyze the characteristic values ​​of the human body key points of the current video frame to obtain the human body key points and occlusion marks of the current video frame.

[0100] See also Figure 6 As shown, accordingly, the present invention also provides a human body key point analysis training and reasoning device based on video stream, which includes:

[0101] A training module 10 is used to train a human key point analysis model using all human key points and occlusion marks of historical continuous video frames as sample training data;

[0102] Initialization module 20, for initializing the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data;

[0103] The inference module 30 is configured to infer the position of the human key point corresponding to each of the occlusion marks in the current continuous video frame using the human key point analysis model to obtain the human posture of the current continuous video frame.

[0104] Combine Figures 1-6 The present invention uses all human key points and occlusion marks of historical continuous video frames as sample training data to train a human key point analysis model, and uses the human key point analysis model to infer the position of the human key point corresponding to each occlusion mark of the current continuous video frame to obtain the human posture of the current continuous video frame. On the one hand, by introducing occlusion marks to train the human key point analysis model, the human key point analysis model has a higher ability to recognize the position of the human key points corresponding to the occlusion marks, thereby improving the ability of the human key point analysis model to recognize the position of the human key points corresponding to the occlusion marks; on the other hand, continuous frames are used for training and reasoning, and the correlation between the human key points and occlusion marks of the previous and next frames is fully utilized, thereby further improving the ability of the human key point analysis model to recognize the position of the human key points corresponding to the occlusion marks.

[0105] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope of the present invention.

Claims

1. A human body key point analysis training and reasoning method based on video stream, characterized in that: The steps include: All human key points and occlusion marks in historical continuous video frames are used as sample training data to train the human key point analysis model; Initializing the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data; Inferring the position of the human key point corresponding to each of the occlusion marks in the current continuous video frames using the human key point analysis model to obtain the human body posture of the current continuous video frames; The method uses all human key points and occlusion marks of historical continuous video frames as sample training data to train the human key point analysis model, specifically including: Randomly intercepting the historical continuous video frames according to a first preset number of frames to obtain temporary historical continuous video frames; Performing bidirectional recurrent neural network analysis and training on the human body key points and occlusion marks of the temporary historical continuous video frames to obtain the positions of the human body key points corresponding to all occlusion marks of the temporary historical continuous video frames; Updating the temporary historical continuous video frames according to positions of human body key points corresponding to all occlusion marks of the temporary historical continuous video frames; Updating the new temporary historical continuous video frame to the historical continuous video frame; Iteratively training the human body key point analysis model based on the new historical continuous video frames; The method uses all human key points and occlusion marks of historical continuous video frames as sample training data to train the human key point analysis model, which also includes: Acquire the historical continuous video frames; Performing a human body key point algorithm analysis on the historical continuous video frames to obtain human body key point feature values ​​of the historical continuous video frames; The human body key points feature values ​​of the historical continuous video frames are analyzed to obtain the human body key points and occlusion marks of the historical continuous video frames.

2. The human body key point analysis training and reasoning method based on video stream according to claim 1, characterized in that: Each occlusion mark in any video frame in the temporary historical continuous video frames needs to have at least one corresponding unoccluded human key point in the remaining video frames in the temporary historical continuous video frames.

3. The human body key point analysis training and reasoning method based on video stream according to claim 1, characterized in that: The method of inferring the position of the human key point corresponding to each of the occlusion marks in the current continuous video frames by using the human key point analysis model to obtain the human posture of the current continuous video frames specifically includes: Dividing the current continuous video frames into a plurality of current continuous video frame units according to a second preset number of frames; Performing bidirectional recurrent neural network reasoning on the human body key points and occlusion marks of each of the current continuous video frame units to obtain the positions of the human body key points corresponding to the occlusion marks of each of the current continuous video frame units; The positions of the human body key points and the human body key points corresponding to the occlusion marks of all current continuous video frame units are combined to analyze and obtain the human body posture corresponding to the current continuous video frame.

4. The human body key point analysis training and reasoning method based on video stream according to claim 3, characterized in that: The performing of bidirectional recurrent neural network reasoning on the human body key points and occlusion marks of each current continuous video frame unit to obtain the position of the human body key points corresponding to the occlusion marks of each current continuous video frame unit further includes: Calculating the number of occlusion marks in each of the current continuous video frame units; Calculating the difference between the number of occlusion marks in each of the current continuous video frame units and a preset threshold; Based on each of the differences, it is determined whether to perform bidirectional recurrent neural network inference on the human body key points and occlusion marks of the corresponding current continuous video frame unit.

5. The human body key point analysis training and reasoning method based on video stream according to claim 4, characterized in that: The determining, based on each of the differences, whether to perform bidirectional recurrent neural network inference on the human body key points and occlusion marks of the corresponding current continuous video frame unit specifically includes: If the difference is greater than or equal to the preset threshold, performing bidirectional recurrent neural network inference on the human body key points and occlusion marks of the current continuous video frame unit; If the difference is smaller than the preset threshold, the position of the human body key point corresponding to the occlusion mark of the current continuous video frame unit is ignored.

6. The human body key point analysis training and reasoning method based on video stream according to claim 3, characterized in that: Each occlusion mark in any video frame in the current continuous video frame unit needs to have at least one corresponding unoccluded human body key point in the remaining video frames in the current continuous video frame unit.

7. The human body key point analysis training and reasoning method based on video stream according to claim 1, characterized in that: The method further includes: initializing the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data; Obtaining the current continuous video frames; Performing a human body key point algorithm analysis on the current continuous video frames to obtain human body key point feature values ​​of the current continuous video frames; The human body key point feature values ​​of the current continuous video frames are analyzed to obtain the human body key points and occlusion marks of the current continuous video frames.

8. A human body key point analysis training and reasoning device based on video stream, characterized in that: The method for implementing the human body key point analysis training and reasoning method based on video stream according to any one of claims 1 to 7 comprises: A training module is used to train a human key point analysis model using all human key points and occlusion marks in historical continuous video frames as sample training data; An initialization module is used to initialize the human key point analysis model using all human key points and occlusion marks of the current continuous video frame as raw data; An inference module is configured to infer the position of the human key point corresponding to each of the occlusion marks in the current continuous video frame using the human key point analysis model to obtain the human body posture of the current continuous video frame; The method uses all human key points and occlusion marks of historical continuous video frames as sample training data to train the human key point analysis model, specifically including: Randomly intercepting the historical continuous video frames according to a first preset number of frames to obtain temporary historical continuous video frames; Performing bidirectional recurrent neural network analysis and training on the human body key points and occlusion marks of the temporary historical continuous video frames to obtain the positions of the human body key points corresponding to all occlusion marks of the temporary historical continuous video frames; Updating the temporary historical continuous video frames according to positions of human body key points corresponding to all occlusion marks of the temporary historical continuous video frames; Updating the new temporary historical continuous video frame to the historical continuous video frame; The human body key point analysis model is iteratively trained based on the new historical continuous video frames.

Images