Real-time human motion counting method based on spatiotemporal optimization of skeleton points

Abstract

The present invention provides a method for real-time human action counting based on spatiotemporal domain optimization of skeleton points, which is used in intelligent analysis of multimedia real-time video. The method of the present invention comprises: generating a number of action dictionary entries according to standard demonstration videos of different actions, each entry comprising an action index, a mapping function of skeleton point-action state value, an ideal time-action state waveform, and an expected key frame sequence of different dense samplings; identifying each action video for the input video stream, identifying the action type for the action video, extracting key frames from the video according to the key frame sequence, and dynamically adjusting the expected key frame sequence used according to the state value of the extracted key frame, and finally judging whether the action is qualified based on the extracted key frame; counting the action until the video stream is interrupted or terminated, and outputting the counting result. The present invention can realize real-time human action counting with known action content, and ensures sufficiently high accuracy, and the recognized action types have high scalability.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence and multimedia applications, relates to real-time video analysis of multimedia content analysis, and specifically relates to a real-time human motion counting method based on spatiotemporal optimization of skeleton points. Background Art

[0002] Daily exercise is an important part of people's healthy life. With the advent of the artificial intelligence era, fitness training based on artificial intelligence has gradually developed into a promising field. For most people, it may not be possible to achieve the best results without guidance when exercising, and there is even a risk of injury. However, to seek experienced guidance, we usually need to go to the gym or gym for help, which incurs a lot of extra time and money. Therefore, in order to make daily exercise more scientific and convenient, fitness training based on artificial intelligence has shown its unique advantages.

[0003] With the development of mobile computing power, a variety of AI-based fitness training applications have emerged in the market. However, after our investigation, their accuracy is extremely low. Ideally, exercisers can get the count results of repeated exercise movements from these applications after selecting the action type. But in most cases, wrong or random movements, or even trembling due to fatigue, may be incorrectly counted as repetitions. This is due to the simple models used by these applications when trying to provide real-time services. Therefore, we found that in addition to strict response latency, a scientific and controllable method to understand the details of the action content and evaluate the effectiveness of the action is equally important for the human action counting (HAC) task.

[0004] Existing studies have proposed methods to achieve content-agnostic repetition counting or use expensive and inconvenient equipment for counting. Repetition estimation is a task defined by Levy et al., which aims to count the number of repetitions of approximately the same action in the input video. However, this goal is coarse-grained. It is agnostic to the content of the action when counting any form of periodic action. Therefore, it cannot meet the requirements of content awareness. Sensor-based action counting can perform limited analysis of the action content through inertial and position information. However, their requirements for special equipment may be high, and the inconvenience of wearing sensors will seriously affect the motion experience. Therefore, they are also not suitable for traditional human action counting scenarios.

[0005] In recent years, research based on visual models has achieved high accuracy in HAC tasks. However, these methods require frame-by-frame extraction of skeleton coordinates and even more redundant reasoning processes. Through experiments on several skeleton extraction algorithms running on the RTX 3060 GPU, we found that the time overhead of analyzing the video frame by frame far exceeds the real-time limitation. Considering that it is difficult for mobile computing power to surpass the above computing power in the short term, it can be considered impractical to achieve real-time reasoning through frame-by-frame pattern analysis. Therefore, the existing high-precision action recognition schemes involving action counting lack real-time reasoning efficiency, so it is necessary to find an optimization solution. Summary of the invention

[0006] In view of the above problems, in order to realize real-time and high-precision human motion counting with known action content, the present invention proposes a real-time human motion counting method based on spatiotemporal optimization of skeleton points, which can achieve sufficiently high action counting accuracy and real-time or quasi-real-time reasoning under limited computing power while analyzing the action content.

[0007] A method for real-time human motion counting based on spatiotemporal optimization of skeleton points of the present invention comprises the following steps:

[0008] Step 1, constructing an action dictionary; the action dictionary contains multiple action dictionary entries, one action dictionary entry is generated by a standard demonstration video of an action, and contains the following features: an action index for action type recognition, formed by selecting human skeleton points of video frames at K moments, K>1; a mapping function M for mapping skeleton points to action state values; a curve for recording the change of action state values ​​over time in an action cycle, and constructing three key frame sequences W0, W1 and W2 with different densities according to time-action state values;

[0009] Step 2, receiving the video stream, eliminating the redundant frames between two actions in the video by residual calculation, comparing the action video V with the action index in the action dictionary, identifying the action type, and setting the initial key frame sequence W of the action video V to W1 in the corresponding action dictionary entry;

[0010] Step 3: extract key frames from the action video V according to the time sequence in the key frame sequence W, identify the human skeleton points in the key frames and calculate the action state value according to the corresponding mapping function, and convert the obtained time-action state value c i (x i ,y i ) is stored in array C, x i and i Represents the time point and action state value of the i-th key frame;

[0011] Step 4: The time-action state value of the i-th key frame expected by W and the actual c i (x i ,y i ), dynamically adjust the expected key frame sequence W; the adjustment includes time axis scaling, density switching and reconstruction of the key frame sequence; during dynamic adjustment, set the minimum adjustment threshold t and the maximum adjustment threshold T, calculate yi and Then execute: (1) If the difference is not greater than t, keep the current sampling density and continue to step 3 to extract the next key frame; (2) If the difference is greater than T, set W to W2, and then start from the time point greater than Start with the video frame of , and go to step 3 to extract the i-th key frame; (3) If the difference is greater than t and not greater than T, increase the sampling density by one level, compress the time axis of the key frame sequence proportionally, and then resample; If in the cases of (2) and (3), the current W is already W2 and the density cannot be increased any further, then re-record a new densely sampled key frame sequence W2′;

[0012] Step 5: Determine whether the action end signal of the action video V is detected. If not, proceed to step 3 to extract the next key frame. If yes, calculate the time-action state waveform corresponding to the array C and the ideal waveform. If the error exceeds the error threshold, the current action is judged to be unqualified, otherwise the current action is judged to be qualified;

[0013] Step 6, determine whether there is a newly recorded key frame sequence W2'. If not, continue to step 2 and perform action detection on the next action video; if it exists, when the action judgment is unqualified, discard W2'. When the action judgment is qualified, use W2' to update W2, and resample W0 and W1 from W2', then go to step 2 and perform action detection on the next action video. Output the human action counting result until the video stream is interrupted or the video ends.

[0014] In step 1, in each action dictionary entry, a static region of interest RoI is obtained according to the maximum X-axis and Y-axis coverage of the spatial area covered by the action performer in the standard demonstration video, and a mapping function R is constructed. action , records the change process of the action performer from the coverage of the initial frame to the static RoI; the mapping function R action The input is the video frame at the start of the action, and the output is a static RoI.

[0015] The step 3 uses a spatial optimization algorithm to determine the static RoI of the action using the static RoI mapping function corresponding to the action type in the action dictionary for the action video V, and uses the static RoI to crop the extracted key frame, eliminate the background part, and then identify the human skeleton points. The step 3 also uses frame-level image residual calculation on the action video V, deletes the frames of the static part of the performer in the video, eliminates redundant frames, and then extracts the key frame.

[0016] Compared with the prior art, the present invention has the following advantages and positive effects:

[0017] (1) The method of the present invention utilizes an action dictionary to effectively handle the recognition and counting of different actions, and the implemented system can be highly modularized with low coupling between modules, so that it can arbitrarily adapt to the optimization of different reasoning steps, can realize the structure of efficient analysis of action content, and has high scalability with flexible adaptation and optimization for the identification of action types.

[0018] (2) The method of the present invention uses a dynamically adjusted expected key frame sequence based on a time domain optimization algorithm for action recognition, which fundamentally solves the efficiency problem of the HAC task. Among the various methods of the HAC task, the computational overhead of converting video content into skeleton points accounts for the vast majority of the entire algorithm, and the simple skeleton point extraction model will cause the performance of the overall method to drop sharply. Therefore, adaptively adjusting the video key frames selected by the algorithm is the core breakthrough in solving the reasoning efficiency of the entire task. Through this dynamic mechanism, the method of the present invention eliminates a large number of redundant frame recognitions while ensuring the accuracy of the identified key frames, thereby greatly improving the action count and ensuring a sufficiently high accuracy.

[0019] (3) The spatial domain optimization algorithm of the method of the present invention is particularly suitable for situations where the efficiency and accuracy of the skeleton point extraction model decrease significantly when the video content is complex. It can effectively eliminate background noise, adapt to any known skeleton point extraction model, and provide better efficiency and accuracy performance for real-time human motion counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of an implementation of the method for counting real-time human motions based on the spatiotemporal optimization of skeleton points of the present invention;

[0021] Figure 2 A schematic diagram of a formalized pseudo code of a time domain dynamic frame screening algorithm implemented in an embodiment of the present invention;

[0022] Figure 3 This is a comparison chart of the time consumption for single video action inference according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0024] The present invention provides a real-time human motion counting method based on spatiotemporal optimization of skeleton points, which can be implemented as a real-time human motion counting system. The first goal of the present invention is to reduce the total energy consumption caused by wireless communication through reasonable scheduling while ensuring the performance loss of upper-layer applications. The total energy consumption includes the energy consumption generated by data transmission and the additional tail energy consumption in the cellular network environment. The second goal is to reasonably weigh the relationship between performance loss and energy consumption when the performance loss tolerance of upper-layer applications changes dynamically, so as to ensure that the scheduling scheme can dynamically adapt to the dynamic changes of such performance guarantee requirements. In order to achieve this goal, the real-time human motion counting scheme of the present invention mainly proposes the following three improvements:

[0025] (1) The present invention proposes a human motion counting scheme combined with a special data structure "motion dictionary", which has a structure for efficiently analyzing motion content and high scalability that can be flexibly adapted and optimized.

[0026] (2) The present invention proposes a time domain optimization algorithm, which can eliminate redundant information in the video and improve the reasoning efficiency to a great extent by dynamically screening the video frames to be analyzed.

[0027] (3) The present invention proposes a spatial domain optimization algorithm, which can filter out confusing video backgrounds by constructing a static region-of-interest (RoI), thereby further improving the reasoning efficiency and accuracy of the system.

[0028] In order to achieve the improvement point 1, the present invention designs an action dictionary data structure. An action dictionary contains multiple action dictionary entries. Each action dictionary entry is generated by a standard demonstration video definition of a corresponding action. The generated action dictionary entry mainly retains a series of artificial features of a specific action:

[0029] (a) Action indexing for action type recognition based on human skeleton points in certain frames;

[0030] (b) a mapping function that maps a human skeleton point at a certain moment to an action state value;

[0031] (c) Standard action waveform recording the change of action state value over time in an action cycle.

[0032] The present invention combines the above-mentioned action dictionary structure to realize a real-time human action counting method based on the spatiotemporal optimization of skeleton points for action content analysis and counting, referred to as SRHAC. The framework of SRHAC runs in a loop mode, and each action in the video corresponds to an outer loop of the algorithm. At the beginning of the outer loop, the frame residual calculation is first used to eliminate the redundant frames where the performer remains motionless between two actions. When the system analyzes the first action, the action dictionary entry is indexed according to the (a) action index in the action dictionary and the skeleton point data of the performer in the video to determine the action type. Then, the inner loop of the system is entered, and the next video frame to be analyzed is selected according to the key frame sequence saved in the system, while the baseline system analyzes all frames of the video. Next, the video frame to be analyzed is pruned by the temporal optimization RoI algorithm of the present invention, and the skeleton points in the video frame are extracted using any skeleton extraction model. Next, the action state value corresponding to the skeleton point is calculated using the (b) mapping function in the action dictionary entry; and it is saved as the basis for judging the action qualification after the end of the loop. At the same time, the system of the present invention dynamically adjusts the key frame sequence in the system according to the action state value at the current moment, enters the next inner loop of the system, and selects the next video frame to be analyzed. After detecting that the action state value returns to zero, the system considers that the action cycle is over, and compares the saved time-action state value with the (c) standard action waveform in the action entry to determine the effectiveness of the action. Then the system enters the next outer loop and starts analyzing the next action in the video.

[0033] To achieve improvement point 2, the present invention designs a dynamic frame screening algorithm in the time domain. The dynamic frame screening algorithm dynamically adjusts the density of key frame selection and the expected state value of the key frame. The algorithm derives three key frame sequences with different densities from the (c) standard action waveform in the action dictionary entry, and uses the centered density list as the initial key frame sequence of the system. During the system analysis process, the dynamic frame screening algorithm dynamically adjusts the key frame sequence according to the action state value expected to appear in the key frame sequence and the action state value obtained in the actual analysis. The main adjustments include time axis scaling, density switching and reconstruction of the key frame sequence. Through this dynamic mechanism, the system can ensure the accuracy of the identified key frames while eliminating a large number of redundant frame recognitions, thereby achieving a sufficiently high reasoning accuracy while greatly improving the system reasoning efficiency.

[0034] To achieve improvement point 3, the present invention designs a static RoI algorithm in the airspace, which generates an RoI generation function by gathering all the areas where the performer appears in the standard action video. When the function recognizes the initial action of the performer, it takes the area where the performer appears as input to obtain the total area where the performer may appear in all action cycles. And before the subsequent video frame analysis, the background part outside the area is segmented and eliminated. This airspace algorithm can effectively eliminate background noise in the video, including the chaotic background and other exercisers captured.

[0035] The method for real-time human motion counting based on spatiotemporal optimization of skeleton points in an embodiment of the present invention comprises the following 7 steps.

[0036] Step 1) defining an action dictionary entry according to a standard demonstration video of a certain action, including the following steps 11 to 15.

[0037] Step 11) The standard demonstration video contains a complete action cycle, and the limb bone points of the action performer and the area covered by the performer in the video frame are extracted frame by frame from the standard demonstration video.

[0038] Step 12) Select the skeleton point at the start time of the action and the skeleton point at the moment when the action is in progress and has action characteristics as the action index I of the action action .

[0039] Step 13) Customize the bone point-state value mapping function M according to the angle and position changes of the limb bone points with action characteristics action , maps the human skeleton points to action state values, with a value range of [0, 1]. The skeleton point-state value mapping function is user-defined and is used to identify the action posture state. Generally, the starting action posture state can be set to 0, and the state with the highest action amplitude can be set to 1. For example, when doing push-ups, the state when you push up is 0, and the state when you push down to the lowest point is 1.

[0040] Step 14) According to the maximum X-axis and Y-axis coverage of the spatial area covered by the actor in the entire video, a static RoI is obtained and a mapping function R is constructed. action Record the change process from the coverage of the initial frame to the total range RoI.

[0041] Step 15) Construct the time-state value curve of the ideal action with the time axis as the X-axis and the action state value as the Y-axis And record the reasonable number of frames required to perform a standard action The time-state values ​​of the action state values ​​{0, 1, 0} and {0, 0.5, 1, 0.5, 0} are stored in sequence as the density initial values ​​W0 and W1 of the two dynamic frame lists, and are stored as the key frame sequence initial value W2 with the highest density at a frequency of sampling 1 frame every 4 frames. In the embodiment of the present invention, the reasonable number of frames required to perform a standard action is set is the total number of frames of the standard action demonstration video. The key frame sequences W0, W1 and W2 are video frame sequences corresponding to the action state values, such as W0 is a sequence of three video frames corresponding to the action state values ​​{0, 1, 0}, and W2 is a video frame sequence sampled at the set frequency. The key frame sequences W0, W1 and W2 represent peak, base and dense sampling, respectively, and can be fitted into a desired time-action state waveform.

[0042] Step 2) Receive the input video / video stream, start action counting, compare each action video according to the initial posture and subsequent posture of the action and the action index I of various actions in the action dictionary, determine the type of action being performed and set the initial key frame sequence W of the input video according to W1 in the action dictionary entry. W also represents the expected time-action state waveform fitted by the current key frame sequence.

[0043] In the embodiment of the present invention, the action type can be determined by generally identifying the starting posture of the action. If there is an action with the same starting posture, the subsequent postures with action characteristics are compared, and each action in the input video stream is identified by the action index. After the action type is determined, the key frame sequence W1 of the standard action in the corresponding action dictionary is set as the initial key frame sequence W of the input action video. Figure 1 In, W E is the desired time-action state waveform, corresponding to the embodiment of the present invention, namely W, which will be dynamically adjusted by the time domain dynamic key frame screening algorithm according to the analyzed video frames.

[0044] Step 3) Use frame-level image residual calculation to delete the frames of the static part of the performer, eliminate redundant frames between two actions in the video, and reduce the overhead of skeleton point recognition. The time point when the inter-frame residual value exceeds a certain threshold is considered to be the time point when the action actually starts.

[0045] Step 4) Sample the current i-th video key frame from the video to be detected according to the current key frame sequence W, extract the coordinates of the human skeleton points from the key frame, calculate the action state value yi through the skeleton point-action state value mapping function M, and sum it with the time point x i Composition time-action state value c i (x i ,y i ) is stored in array C.

[0046] According to the mapping function Raction Determine the static region of interest in the currently analyzed action video, then crop the extracted video frames and filter out confusing video backgrounds to further improve the system's reasoning efficiency and accuracy.

[0047] Step 5) Based on the expected time-state value and the actual c i (x i ,y i ), dynamically adjust the key frame sequence W, and the specific adjustment algorithm is formally described as follows Figure 2 As shown, the input data includes: three desired time-action state waveforms W θ , θ=0, 1, 2, corresponding to three densities, namely peak, basic and dense sampling; skeleton extraction algorithm E; skeleton point-state value mapping function M; inverse function of current expected time-action state waveform W -1 ; The activation thresholds T and t of the two different adjustment mechanisms; After executing the adjustment algorithm, the validity of the current action and the expected time-state waveform are output. Under the premise of judging that the action is valid, the dynamic adjustment algorithm of the key frame sequence is executed, including the following steps 51 to 53.

[0048] Step 51) If yi and If the difference is greater than the minimum adjustment threshold t and less than or equal to T, the sampling density θ is increased by one level, and the time axis X axis of the key frame sequence is compressed proportionally, and then resampled; if the highest density 2 is reached before the density is increased, a new default dense sampling key frame sequence W2′ is recorded instead.

[0049] In the embodiment of the present invention, when the current sampling density of W is less than 2, θ is increased by one level, and the current desired time-action state waveform W is set to W θ+1 , compress the time axis proportionally, the compression ratio is Coming soon θ+1 Zhongzai Subsequent expected time-action state value The time point in According to the expected time-state waveform W after compressing the time axis θ+1 Set the key frame sequence W, and then continue with the next sampling. If the current sampling density of W is already 2, then record a new dense sampling key frame sequence W2′. W2′ is the key frame sequence in W2. Replace with

[0050] Figure 2 In the i-th ideal data Mark as Actual dataci (x i ,y i ) is marked as Similarly, is the jth ideal time-state value, is the kth ideal time-state value.

[0051] Step 52) If yi and If the difference is greater than the resampling threshold T, it is directly switched to the highest sampling density 2 and the sampling is restarted, and the current action video is sampled according to W2 to obtain the key frame sequence W. If the current sampling density is already the highest density 2, a new dense sampling key frame sequence W2′ is recorded as in step 51.

[0052] Step 53) If y i and If the difference is less than or equal to t, the current sampling density θ is maintained and the next sampling will be continued.

[0053] Step 6) Return to step 4 and continue sampling the next key frame according to the current W until the current action end signal is detected. Usually the state value decreases to 0, or decreases to a lower value and then starts to increase again, thereby detecting whether the action is a result. After the current action ends, the stored C is compared with the ideal time-state value curve Whether the difference exceeds the custom error threshold, judge whether the current action is qualified. Fit the actual time-action state curve to array C and calculate the ideal curve corresponding to the standard action For example, the mean square error or the root mean square error is calculated as the error value. When the error value exceeds the threshold, it is judged that the current action is unqualified, otherwise it is judged that the current action is qualified.

[0054] Step 7) If there is a newly stored densely sampled key frame sequence W2', and the action is considered qualified, then use W2' to update the default value of the key frame sequence W2, and resample based on W2' to obtain the default sequences W0 and W1 as the new initial sampling sequence for the corresponding action. After updating the default key frame sequence, continue to read the next action video and return to step 3 to start the next action detection until a video stream interruption or video end signal is detected, then terminate the action counting and output the human action counting result.

[0055] The method of the present invention is compared with the existing method on the human action recognition dataset (HAC) and the UI-PRMD dataset, and the results are shown in Table 1. Table 1 also subdivides the comparison results of action count recognition of three action types, pull-up, push-up, and sit-up, on the dataset HAC.

[0056] Table 1 Comparative experimental results of each method on two datasets

[0057]

[0058] The comparison methods include: AI-Fit, Fourier and SRHAC-static. SRHAC-static modifies the method of the present invention to count human bodies using only sequences with density 1, and the expected time-state waveform remains unchanged (only the period is modified proportionally), and the others are the same. AI-Fit is a technical solution in reference [1] (Fieraru M, Zaniir M, Pirlea SC, et al. AIFit: Automatic 3D Human-Interpretable Feedback Models for FitnessTraining[J]. IEEE, 2021.). Fourier is a technical solution in reference [2] (Hsu YC, Zhang Q, Tsougenis E, et al. Viewpoint-Invariant Exercise Repetition Counting[J]. 2021.). The comparison indicators are MAE (mean absolute error) and OBOA (Off-By-One Accuracy). The smaller the MAE, the better the performance of the solution, and the larger the OBOA, the better the performance of the solution. The comparison indicators are calculated as follows:

[0059]

[0060] Where P is the number of videos, o i is the actual number of action executions in the i-th video, is the counting result obtained by inferring the i-th video using the action counting method.

[0061] like Figure 3 As shown, the time consumption of using the method of the present invention and frame-by-frame reasoning to identify actions is compared. The vertical axis represents the average duration of an action / reasoning time, and the horizontal axis represents each video sample. The blue color represents the average single video reasoning time consumption using the method of the present invention, and the orange color represents the actual time of single video action execution. The comparison scheme is frame-by-frame reasoning to identify actions. It can be seen that the use of the method of the present invention reduces the reasoning time and can meet the needs of real-time recognition of human action counting.

[0062] In summary, the present invention realizes a real-time human motion counting method based on the spatiotemporal domain optimization of skeleton points, which can fully analyze the content characteristics of the executed action and complete the action type judgment. At the same time, in the process of human motion counting with action content perception, the degree of standardization of the action can be identified. Under the premise of ensuring high precision that is better than the most advanced work, the dynamic frame screening algorithm can be used to greatly improve the reasoning efficiency, thereby saving computing resources and realizing real-time human motion counting.

Claims

1. A method for real-time human motion counting based on spatiotemporal optimization of skeleton points, characterized in that: The steps include: Step 1, construct an action dictionary; the action dictionary contains multiple action dictionary entries, one action dictionary entry is generated by a standard demonstration video of an action, and contains the following features: an action index for action type recognition, formed by selecting human skeleton points of video frames at K moments, K>1; a mapping function M for mapping skeleton points to action state values; an ideal waveform for recording the change of action state values ​​over time in an action cycle Construct three key frame sequences W0, W1 and W2 with different density sampling according to the time-action state value; Step 2, receiving the video stream, eliminating the redundant frames between two actions in the video by residual calculation, comparing the current action video V with the action index in the action dictionary, identifying the action type, and initially setting the expected key frame sequence W of the action video V to W1 in the corresponding action dictionary entry; Step 3: extract the i-th key frame from the action video V according to the current time point W, where i is initially 1, identify the human skeleton points in the key frame and calculate the action state value according to the corresponding mapping function M, and convert the obtained time-action state value c i (x i ,y i ) is stored in array C, x i and i Represents the time point and action state value of the i-th key frame; Step 4: The desired time-action state value of the i-th key frame and the actual c i (x i ,y i ), dynamically adjust the expected key frame sequence W; wherein, set the minimum adjustment threshold t and the maximum adjustment threshold T, calculate y i and Then execute: (1) If the difference is not greater than t, keep the current sampling density; (2) If the difference is greater than T, set W to W2, and then start from the time greater than t. Start with the video frame of , and go to step 3 to extract the i-th key frame; (3) If the difference is greater than t and not greater than T, increase the sampling density by one level, compress the time axis of the key frame sequence proportionally, and then resample; If in the cases of (2) and (3), the current W is already W2 and the density cannot be increased, then re-record a densely sampled key frame sequence W2′; Step 5: Determine whether the action end signal of the action video V is detected. If not, proceed to step 3 to extract the next key frame. If yes, calculate the time-action state waveform corresponding to the array C and the ideal waveform. If the error exceeds the error threshold, the current action is judged to be unqualified, otherwise the current action is judged to be qualified; Step 6, determine whether there is a newly recorded key frame sequence W2′. If so, when the action judgment is unqualified, discard W2′. When the action judgment is qualified, use W2′ to update W2, and resample W0 and W1 from W2′, then take the next action video and go to step 2 for execution; if it does not exist, directly take the next action video and go to step 2 to start action detection on the next action video; until the video stream is interrupted or the video ends, output the human action counting result.

2. The method according to claim 1, characterized in that In step 1, the steps of generating an action dictionary entry for a standard demonstration video of an action include: (11) The standard demonstration video contains a complete action cycle, and the human skeleton points of the performer in the video and the area covered by the performer in the video frame are extracted frame by frame; (12) Select the skeleton point at the start of the action and the skeleton point at the moment when the action is in progress and has action characteristics as the action index I of the action action ; (13) According to the angle and position changes of the skeleton points with action characteristics, a mapping function M between skeleton points and action state values ​​is customized to map the skeleton points to action state values, and the value range of the action state values ​​is [0, 1]; (14) According to the maximum X-axis and Y-axis coverage of the spatial area covered by the action performer in the standard demonstration video, a static region of interest RoI is obtained and a mapping function R is constructed. action Record the change process of the actor from the coverage of the initial frame to the static RoI; mapping function R action The input is the video frame at the start of the action, and the output is a static RoI; (15) The time axis is the X axis and the action state value is the Y axis to construct the time-action state value waveform of the ideal action. Record the reasonable number of frames required to perform a standard action Three key frame sequences W0, W1 and W2 with different densities are constructed according to the time-action state value, where W0 is the video frame sequence corresponding to the action state value of {0, 1, 0}, W1 is the video frame sequence corresponding to the action state value of {0, 0.5, 1, 0.5, 0}, and W2 is the video frame sequence sampled at a frequency of 1 frame every 4 frames.

3. The method according to claim 1, characterized in that In the step 3, the action video V is firstly calculated using the frame-level image residual, the frames of the static part of the performer in the video are deleted, and then the key frames are extracted.

4. The method according to claim 1, 2 or 3, characterized in that: In step 3, for the action video V, the static RoI mapping function R corresponding to the action type in the action dictionary is used action Determine the static RoI of the action, use the static RoI to crop the extracted key frame, eliminate the background part, and then identify the human skeleton points.

5. The method according to claim 1 or 2, characterized in that: In step 4, in (3), let the current sampling density be θ, and find the inverse function W of the time-action state waveform fitted by the current expected key frame sequence W. -1 , calculate the time axis compression ratio as For the key frame sequence W θ+1 The time axis in is compressed by this ratio. Subsequent sampling time points Compress to Set W to the W after compressing the timeline θ+1 .

6. The method according to claim 1 or 2, characterized in that: In step 4, the newly recorded densely sampled key frame sequence W2′ is converted into the expected time-action state value in the key frame sequence W2. Replace with get.

Images