Action counting method, apparatus, device, and storage medium

Abstract

Embodiments of the present application relate to the technical field of video recognition, and disclose a motion counting method and device, equipment and a storage medium to solve the technical problems of poor robustness and inaccurate counting of existing motion counting. In the present application, motion counting includes: using a video frame motion recognition model obtained by pre-training to recognize a video to be counted to obtain a Gaussian regression output sequence; performing Gaussian modeling according to the Gaussian regression output sequence to obtain a Gaussian model; and performing motion counting according to the number of Gaussian distributions in the Gaussian model.

Description

Technical Field

[0001] This application relates to the field of video recognition technology, and in particular to an action counting method, apparatus, device, and storage medium. Background Technology

[0002] Video motion counting refers to the process of counting the number of times a specific repetitive action occurs in a given video. Video motion counting has important applications in industrial production, agricultural production, and daily life. For example, in industrial manufacturing processes, the completion level of certain steps directly affects the quality of the final product; too few or too many actions in a step can lead to poor product quality or even render the product unusable. In sports activities, motion counting is necessary when assessing athletes in events such as rope skipping, sit-ups, and pull-ups.

[0003] However, existing methods for counting actions in videos either rely solely on the periodicity of actions or rely solely on single-frame images for action classification and recognition to achieve action counting.

[0004] Although both methods can achieve action counting, the periodicity, frequency, and completeness of the same repetitive action can vary greatly in practical applications. This results in incomplete features extracted according to a fixed period, leading to a lack of robustness in the action counting scheme based solely on periodicity. On the other hand, action classification and recognition based on single-frame images is prone to misjudgment and inaccurate counting because the key action postures cannot fully define the complete action type. Summary of the Invention

[0005] The purpose of this application is to provide an action counting method, apparatus, device, and storage medium to solve the above-mentioned technical problems.

[0006] To address the aforementioned technical problems, embodiments of this application provide an action counting method, comprising:

[0007] The pre-trained video frame action recognition model is used to identify the video to be counted, and a Gaussian regression output sequence is obtained.

[0008] Gaussian modeling is performed based on the Gaussian regression output sequence to obtain the Gaussian model;

[0009] Action counting is performed based on the number of Gaussian distributions in the Gaussian model.

[0010] To achieve the above objectives, embodiments of this application also provide an action counting device, comprising:

[0011] The network model inference module is used to identify the video frame action recognition model to be counted using a pre-trained video frame action recognition model, and obtain the Gaussian regression output sequence.

[0012] The Gaussian modeling processing module is used to perform Gaussian modeling based on the Gaussian regression output sequence to obtain a Gaussian model.

[0013] The action counting module is used to count actions based on the number of Gaussian distributions in the Gaussian model.

[0014] To achieve the above objectives, embodiments of this application also provide an action counting device, comprising:

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the action counting method as described above.

[0017] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the action counting method described above.

[0018] The action counting method, apparatus, device, and storage medium proposed in this application, from a probabilistic and statistical perspective, use a Gaussian distribution to represent a complete action interval. Based on this characteristic, a video frame action recognition model is pre-trained to identify the Gaussian regression value of each video frame in the video to be counted. Then, when counting actions in the video to be counted, the video frame action recognition model is used to identify the video, which yields a Gaussian regression output sequence that represents the entire video. Furthermore, Gaussian modeling is performed based on the Gaussian regression output sequence to obtain a Gaussian model that records the Gaussian distribution corresponding to each action interval in the video. Finally, by counting the number of Gaussian distributions in the Gaussian model, the number of Gaussian distributions is taken as the number of actions included in the video, thus achieving action counting. Because this method counts for each complete action interval, it has better robustness and more accurate counting compared to existing methods that simply use the periodicity of actions or single-frame images for action counting.

[0019] Furthermore, the action counting method, apparatus, device, and storage medium proposed in this application, based on the Gaussian distribution action counting method, can output the number of actions and also provide the start and end time points of the actions according to the Gaussian distribution, which has important guiding significance for the positioning of time-series actions. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0021] Figure 1 This is a flowchart of the action counting method provided in the first embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the network structure of the video frame action recognition model involved in the action counting method provided in the first embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the Gaussian model obtained by Gaussian modeling in the action counting method provided in the first embodiment of this application;

[0024] Figure 4 This is a flowchart of the action counting method provided in the second embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the action counting device provided in the third embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the action counting device provided in the fourth embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0028] The first embodiment of this application relates to an action counting method. This method first uses a pre-trained video frame action recognition model to identify the video to be counted, obtaining a Gaussian regression output sequence. Then, Gaussian modeling is performed based on the Gaussian regression output sequence to obtain a Gaussian model. Finally, action counting is performed based on the number of Gaussian distributions in the Gaussian model. Because this method performs counting for each complete action interval, it is more robust and accurate than existing methods that simply use the periodicity of actions or single-frame images for action counting.

[0029] The implementation details of the action counting method in this embodiment are described below. The following implementation details are provided for ease of understanding only and are not necessary for implementing this solution.

[0030] The action counting method provided in this embodiment is specifically applied to any terminal device capable of executing the method. The terminal device can be a client device, such as a personal computer, tablet computer, or smartphone, or a server device, such as a server. This embodiment does not impose any restrictions on this.

[0031] Furthermore, for ease of explanation, this embodiment uses the application of the action counting method in industrial process production to count the actions performed by operators on a certain process as an example for specific explanation.

[0032] The specific process of this embodiment is as follows: Figure 1 As shown, the specific steps include:

[0033] Step 101: Use the pre-trained video frame action recognition model to identify the video to be counted and obtain the Gaussian regression output sequence.

[0034] Specifically, in practical applications, in order to ensure the smooth progress of action counting, it is necessary to pre-train the video frame action recognition model.

[0035] To make the action count more accurate based on the video frame action recognition model obtained from training, this embodiment takes a probabilistic approach and uses a Gaussian distribution to represent a complete action interval. By counting each complete action interval, the action count of the video to be counted can be achieved.

[0036] Therefore, the network model structure on which the video frame action recognition model trained in this embodiment is based must at least include a Gaussian regression output branch.

[0037] Regarding the training based on this network model structure, the details are as follows:

[0038] (1) Obtain motion video samples.

[0039] Understandably, in actual training, the obtained motion video samples are videos with a known number of actions, as well as the start frame time, end frame time, key frame time, and temporal length factor for each action.

[0040] (2) The motion intervals in the motion video samples are labeled with a Gaussian distribution.

[0041] Specifically, in order to ensure that the 3σ (Laida) regions of the Gaussian distribution are all within the action interval, this embodiment uses the Laida 3σ criterion of the Gaussian distribution to label the action intervals in the action video samples with a Gaussian distribution.

[0042] The operation of Gaussian distribution annotation on the action intervals in the aforementioned action video samples is as follows:

[0043] The start time of the action in the action video sample is marked as t. s The time of the action termination frame is marked as t. e The key frame time is marked as t m The time scale factor is denoted as s.

[0044] Based on the above annotation information, this embodiment makes the Gaussian distribution value of the action satisfy the following formula (1):

[0045]

[0046] Where μ = st m ,

[0047] Therefore, after completing the annotation of the action video samples in steps (1) and (2) above, we can proceed to the model training stage mentioned in step (3).

[0048] (3) Using the labeled action video samples, the pre-designed network model structure is iteratively trained until the preset convergence condition is met, and the video frame action recognition model is obtained.

[0049] Understandably, since action video samples are often composed of multiple frames, in order to achieve recognition training for each frame of the action video sample while avoiding inputting the entire action video sample into the network model structure for training at once, which would affect the training speed, this embodiment uses the labeled action video samples to iteratively train a pre-designed network model structure, i.e., a network model structure including a Gaussian regression output branch, specifically through the following process:

[0050] First, select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence.

[0051] To facilitate understanding, the following examples will be used to illustrate the point:

[0052] For example, for an action video sample consisting of 100 video frames, it is stipulated that 32 consecutive video frames are read each time, which are used as the continuous video frame sequence of the Gaussian regression output branch in the input network model structure.

[0053] The first video frames read from the labeled motion video samples are frames 0 to 31, which are 32 consecutive video frames.

[0054] Then, the continuous video frame sequence is input into the Gaussian regression output branch in the network model structure.

[0055] Specifically, in practical applications, each group of consecutive video frame sequences can be sequentially input into the Gaussian regression output branch in the network model structure, and the Gaussian regression output branch can analyze and process each video frame in each group of consecutive video frame sequences.

[0056] Furthermore, in practical applications, a buffer queue can be pre-set in the network model structure. Each set of consecutive video frame sequences is added to the buffer queue in sequence. After the Gaussian regression output branch completes the analysis and processing of a set of consecutive video frame sequences, the next set of consecutive video frame sequences is retrieved from the buffer queue in sequence for analysis and processing, thereby avoiding thread blocking and affecting the processing speed.

[0057] Finally, the starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch is iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thus obtaining the video frame action recognition model.

[0058] Specifically, the preset convergence condition can be set according to actual business needs. For example, for business scenarios that require high convergence, the convergence condition can be set to a higher number of training iterations.

[0059] Correspondingly, for business scenarios with relatively low convergence requirements, the convergence condition can be set to a relatively low number of training iterations.

[0060] In this embodiment, in order to ensure both convergence and training speed, the number of training iterations is set to 200,000. That is, after iteratively training the Gaussian regression output branch using the stochastic gradient descent method with momentum for 200,000 iterations, the current network model structure can be used as a video frame action recognition model.

[0061] It should be understood that the above examples are merely examples listed for the purpose of better understanding the technical solutions of this embodiment, and are not intended to be the only limitation of this embodiment.

[0062] Therefore, in practical applications, by inputting the video to be counted (which can also be a series of video frames read from the video to be counted according to a preset length) into the video frame action recognition model trained in the above manner, the output result after processing by the video frame action recognition model is the Gaussian distribution value of each video frame. By combining the Gaussian distribution values ​​in sequence, the Gaussian regression output sequence mentioned in this embodiment can be obtained.

[0063] Furthermore, in practical applications, in order to minimize the training difficulty of the Gaussian regression output branch and accelerate network convergence, the network model structure used to train the video frame action recognition model can also include an action classification output branch.

[0064] Specifically, the action classification output branch is mainly used to determine whether a video frame belongs to an action.

[0065] Specifically, in practical applications, it can be determined whether a video frame belongs to an action by judging the confidence level of each video frame. For example, if the confidence level is set to be higher than a certain threshold, it can be determined that the video frame belongs to an action.

[0066] In addition, in practical applications, besides using the action classification output branch to determine whether a video frame belongs to an action, the action classification output branch can also be used to determine the specific action type of the video frame, such as running, jumping, walking, etc. These will not be listed here, and this embodiment does not impose any restrictions on them.

[0067] Accordingly, for cases where the network model structure includes an action classification output branch and a Gaussian regression output branch, the step of iteratively training the pre-designed network model structure using the labeled action video samples until the network model structure satisfies the preset convergence condition, thereby obtaining the video frame action recognition model, specifically involves:

[0068] First, select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence.

[0069] Then, the continuous video frame sequence is input into the Gaussian regression output branch in the network model structure.

[0070] Next, the Gaussian distribution label corresponding to each video frame in the continuous video frame sequence is converted into a binary label, and the converted continuous video frame sequence is input into the action classification output branch in the network model structure.

[0071] That is, before training the action classification output branch, it is necessary to convert the Gaussian label corresponding to each video frame in the action video sample (that is, the Gaussian distribution label labeled with Gaussian distribution) into a binary label.

[0072] Specifically, it can be defined that if the value of the Gaussian label is greater than 0, it indicates an action, and the instruction outputs 1; otherwise, the instruction outputs 0. Therefore, by identifying whether the binary label is 0 or 1, it is possible to determine whether the video frame is an action.

[0073] Finally, the starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch and the action classification output branch are iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thus obtaining the video frame action recognition model.

[0074] That is, for a network model structure that includes an action classification output branch and a Gaussian regression output branch, when inputting continuous video frames into the network model structure, they are specifically input to these two branches respectively. In this way, the training process of the Gaussian regression output branch can refer to the action classification output branch, but it can also avoid relying entirely on the output results of the action classification output branch for training, which would lead to problems such as poor convergence and robustness of the Gaussian regression output branch due to abnormal action classification output results.

[0075] Furthermore, in practical applications, in order to address the problem that existing action counting schemes use spatial RGB graphics and temporal optical flow sequence maps for temporal feature extraction during deep feature extraction, resulting in weak extraction capabilities and high counting complexity, the network model structure used to train the video frame action recognition model can also include a 3D convolutional trunk.

[0076] Specifically, the 3D convolutional trunk is used for temporal feature extraction.

[0077] To better understand the process of training a video frame action recognition model based on a network model structure including a 3D convolutional branch, an action classification output branch, and a Gaussian regression output branch in this embodiment, the following will be combined with... Figure 2 Explanation:

[0078] Specifically, in practical applications, after Gaussian distribution labeling of the motion video samples according to the sample labeling methods given in steps (1) and (2) above, a sequence of consecutive video frames of a preset length is first selected from the labeled motion video samples to obtain a sequence of consecutive video frames; then, the sequence of consecutive video frames is input into... Figure 2 The 3D convolutional backbone in the process extracts temporal features, and the extracted temporal features are then used as inputs to the relevant systems. Figure 2 The Gaussian regression output branch and the action classification output branch are selected in the video frame sequence. Finally, the starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch and the action classification output branch are iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, so as to obtain the video frame action recognition model.

[0079] Understandably, since the training of the action classification output branch is based on binary labels, the continuous video frame sequence output by the 3D convolution trunk also needs to undergo the operation of converting Gaussian labels to binary labels as mentioned above when inputting into the action classification output branch.

[0080] As described above, the final continuous video frame sequence input to the Gaussian regression output branch and the action classification output branch is a continuous video frame sequence processed by the 3D convolution trunk. Based on the strong extraction capability and low computational complexity of 3D convolution, the complexity of the features extracted when training the video frame action recognition model and when using the trained video frame action recognition model for recognition is greatly reduced, thereby reducing the final computational complexity.

[0081] Understandably, in practical applications, the specific number of residual network layers in the 3D convolutional backbone can be determined based on actual business needs. For example, for situations requiring high convergence and robustness but with low training time requirements, a residual network with relatively more layers can be selected, while a residual network with relatively fewer layers can be selected.

[0082] Combining the above two points, this embodiment selects an 18-layer residual network 3D convolutional version, namely ResNet8-3D, as the 3D convolutional backbone for temporal feature privilege.

[0083] Furthermore, it is understandable that in practical applications, both the action classification output branch and the Gaussian regression output branch include fully connected layers, and a loss function is used to process them accordingly after the fully connected layers.

[0084] Specifically, for the action classification output branch, the loss function used is the softmax cross-entropy loss function; for the Gaussian regression output branch, in order to facilitate subsequent processing based on the Gaussian distribution value output by the Gaussian regression output branch, this embodiment specifies that the output range of the Gaussian regression output branch is between 0.0 and 1.0. Therefore, for the Gaussian regression output branch, the loss function used is the sigmoid cross-entropy loss function.

[0085] Furthermore, it should be noted that the action classification output branch is intended to convert Gaussian distributed labels ( Figure 3 Convert the “о” in the text to a binary label ( Figure 3 The Gaussian distribution value output by the Gaussian regression output branch in this embodiment is limited to the range of 0.0 to 1.0. Therefore, it can be set that when the value of the Gaussian label is greater than 0, it represents an action, and the result of the action classification output branch is 1; otherwise, the result is 0.

[0086] It should be understood that the above-described network model structures, including specific trunks and branches, are designed for three business needs. In practical applications, those skilled in the art can select appropriate network model structures to train and obtain video frame action recognition models, so that the video frame action recognition models can achieve low complexity, high robustness, convergence, and accurate calculation.

[0087] Step 102: Perform Gaussian modeling based on the Gaussian regression output sequence to obtain the Gaussian model.

[0088] Specifically, in practical applications, fitting the Gaussian regression output sequence using the global least squares method can easily lead to local optima, resulting in Gaussian fitting failure. Therefore, this embodiment, when performing Gaussian fitting, i.e., Gaussian modeling, specifically uses a heuristic method based on piecewise Gaussian fitting. Gaussian modeling is performed based on the Gaussian regression output sequence to obtain the following... Figure 3 The Gaussian model shown.

[0089] Regarding the heuristic method based on piecewise Gaussian fitting, the operation of Gaussian modeling based on the Gaussian regression output sequence is as follows:

[0090] a) Given the Gaussian regression output sequence y(t), set the minimum Gaussian threshold ε. val =0.1, minimum Gaussian sampling point threshold ε num =4, initial number of Gaussians k=0, Gaussian parameters

[0091] b) Calculate the maximum value y of the sequence y(t). max And record the index number corresponding to the maximum value as t. max ;

[0092] c) If y max <ε val End piecewise Gaussian fitting, return parameters k and Ω, otherwise proceed to step d);

[0093] d) with t max Starting from this point, search to the left for the first value in y(t) that is less than ε. val The value of is denoted as its corresponding index number t. l ;

[0094] e) with t max Starting from this point, search to the right for the first value in y(t) that is less than ε. val The value of is denoted as its corresponding index number t. r ;

[0095] f) If |t r -t l |<εnum Then y(t) l :t r Set y(t) to 0, then return to step b); otherwise, set y(t) to 0. l :t r Perform single Gaussian fitting, setting the number of Gaussians k = k + 1, and the Gaussian parameter (μ) k ,σ k ), the Gaussian parameter set Ω=Ω∪(μ k ,σ k ), y(t l :t r Set to 0, then return to step b).

[0096] The specific process for the single Gaussian fitting mentioned above is as follows:

[0097] For a single Gaussian model, it can be expressed as the following formula (2):

[0098]

[0099] Where (μ,σ) are the parameters to be estimated, taking the logarithm of both sides yields:

[0100]

[0101] in, Parameter estimates can be obtained using the least squares method.

[0102] Thus, the operation of performing Gaussian modeling based on the Gaussian regression output sequence to obtain a Gaussian model is realized, such as based on... Figure 3 The Gaussian model obtained by performing Gaussian modeling on the motion video samples shown is... Figure 3 Medium Gaussian fitting results ( Figure 3 (in the "...")

[0103] Step 103: Count the actions based on the number of Gaussian distributions in the Gaussian model.

[0104] Specifically, from a probability and statistics perspective, a Gaussian distribution—that is, a sequence of events from the lowest point to the highest point and back to the lowest point—can represent a complete action interval, and a complete action interval corresponds to one action. Therefore, this embodiment, based on... Figure 3 When counting actions using the number of Gaussian distributions in the Gaussian model shown, it is essentially by estimating the number of complete Gaussian distributions in the Gaussian model. Figure 3 The "..." in the text is used to count the number of Gaussian distributions obtained from the statistics, and then the number of actions included in the video to be counted is used as the number of actions included in the video to be counted, so as to realize the action counting for the video to be counted.

[0105] Still with Figure 3 For example, statistics can determine that Figure 3 The Gaussian model shown includes four complete Gaussian distributions, so the final number of actions is 4.

[0106] As can be seen from the above description, the action counting method provided in this embodiment, from a probabilistic statistical perspective, uses a Gaussian distribution to represent a complete action interval. Based on this characteristic, a video frame action recognition model is pre-trained to identify the Gaussian regression value of each video frame in the video to be counted. Then, when counting actions in the video to be counted, the video frame action recognition model is used to identify the video, which yields a Gaussian regression output sequence that represents the entire video. Furthermore, by performing Gaussian modeling based on the Gaussian regression output sequence, a Gaussian model is obtained that records the Gaussian distribution corresponding to each action interval in the video. Finally, by counting the number of Gaussian distributions in the Gaussian model, the number of Gaussian distributions is taken as the number of actions included in the video, thus achieving action counting. Because this method counts for each complete action interval, compared to existing methods that simply use the periodicity of actions or single-frame images for action counting, the video frame action recognition model has better robustness and more accurate counting.

[0107] Furthermore, the action counting method provided in this embodiment, based on a Gaussian distribution, not only outputs the number of actions but also describes any complete action as N(μ) according to a Gaussian distribution. i ,σ i If 1 ≤ i ≤ k, then the starting time of the action is μ. i -3σ i The termination time of the action is μ. i +3σ i .

[0108] In other words, the action counting method provided in this embodiment can not only accurately predict the number of actions based on the number of fitted Gaussian pairs, but also give the start and end time points of the actions based on the Gaussian distribution, which has important guiding significance for the localization of time-series actions.

[0109] The second embodiment of this application relates to an action counting method. The second embodiment further improves upon the first embodiment, mainly by: using a video frame action recognition model capable of determining whether a video frame is an action, and recognizing the video to be counted according to a preset overlap strategy to ensure the accuracy of the recognition results, thereby ensuring the accuracy of the final action counting result.

[0110] like Figure 4 As shown, the action counting method in the second embodiment includes the following steps:

[0111] Step 401: According to the preset overlap strategy, the video frame action recognition model obtained through pre-training is used to recognize the video to be counted, and a Gaussian regression output sequence is obtained.

[0112] Specifically, the overlap strategy described in this embodiment stipulates that the (N-1)th consecutive video frame sequence of length T contains L video frames that are identical to those in the Nth consecutive video frame sequence of length T, and L and T satisfy the following relationship: 0 <L<T。

[0113] The specific steps for using the pre-trained video frame action recognition model to identify the video to be counted, according to the above overlap strategy, to obtain the Gaussian regression output sequence are as follows:

[0114] (1) Select consecutive video frames of fixed length T from the video to be counted to obtain N consecutive video frame sequences of length T.

[0115] (2) Input N consecutive video frame sequences of length T into the video frame action recognition model in sequence to obtain the action classification output result and Gaussian distribution value of each video frame in each consecutive video frame sequence of length T. The action classification output result is 1 or 0, and the Gaussian distribution value is between [0, 1].

[0116] Understandably, since the action classification output branch in the trained video frame action recognition model is trained based on binary labels, while the Gaussian regression output branch is trained based on Gaussian distribution labels, when the continuous video frames that need to be counted are input into the trained video frame action recognition model, the action classification output branch in the video frame action recognition model will output a corresponding binary number for each video frame, that is, either 0 or 1; while the Gaussian distribution value output by the Gaussian regression output branch will be distributed between [0, 1].

[0117] (3) For each video frame, determine whether the corresponding action classification output result is 1.

[0118] (4) If it is 1, determine that the video frame is an action and obtain the confidence level of the action classification output result of the video frame.

[0119] Understandably, when training the video frame action recognition model, it is stipulated that for video frames corresponding to actions, the Gaussian distribution value output by the Gaussian regression output branch is between [0, 1], while the result output by the action classification output branch is 1. Conversely, when the video frame is not an action, the result output by the action classification output branch is 0.

[0120] Therefore, based on this relationship and combined with the confidence value of the action classification output, the Gaussian distribution value of the repeated video frames is determined, which effectively ensures the accuracy of the Gaussian distribution value and thus increases the accuracy of the action counting results of the Gaussian model built based on the Gaussian distribution value.

[0121] (5) Compare the confidence of the action classification output results of the same video frames in the (N-1)th consecutive video frame sequence of length T with that in the Nth consecutive video frame sequence of length T.

[0122] (6) Based on the comparison results, select the action classification result with higher confidence as the target action classification result of the video frame identified by the video frame recognition model.

[0123] (7) The Gaussian distribution value corresponding to the target action classification result is used as the target Gaussian distribution value of the video frame identified by the video frame recognition model.

[0124] (8) Arrange the target Gaussian distribution values ​​in order according to the time when the video frame appears in the video to be counted, and obtain the Gaussian regression output sequence.

[0125] To better understand the operations in steps (1) to (8) above, the following examples will be used to illustrate the process:

[0126] Assume the video to be counted consists of 100 video frames, with a fixed length T = 32, meaning that 32 consecutive video frames are read from the video to be counted each time, and an overlap length L = 16, meaning that 16 video frames are the same in two adjacent sets of consecutive video frame sequences.

[0127] Based on the above regulations, the first consecutive video frame sequence read is the video frames from frame 0 to frame 31, the second consecutive video frame sequence read is the video frames from frame 16 to frame 47, the third consecutive video frame sequence read is the video frames from frame 32 to frame 63, and so on. That is, the consecutive video frame sequence read in the Nth time contains the same L video frames as the consecutive video frame sequence read in the N-1th time.

[0128] For the same L-frame video frame in two consecutive consecutive video frame sequences, select the action classification output result and Gaussian distribution value with higher confidence between the two action classification output results as the target action classification output result and target Gaussian distribution value corresponding to this L-frame video frame.

[0129] Finally, by arranging the target Gaussian distribution values ​​of each video frame in the video to be counted in order, the Gaussian regression output sequence for constructing the Gaussian model can be obtained.

[0130] Step 402: Perform Gaussian modeling based on the Gaussian regression output sequence to obtain a Gaussian model.

[0131] Step 403: Count the actions based on the number of Gaussian distributions in the Gaussian model.

[0132] It is not difficult to see that steps 402 and 403 in this embodiment are roughly the same as steps 102 and 103 in the first embodiment, so they will not be described again here.

[0133] Therefore, the action counting method provided in this embodiment, when using a pre-trained video frame action recognition model to identify the video to be counted and obtain a Gaussian regression output sequence, selects the input video frame action recognition model from the video to be counted based on an overlapping strategy of overlapping single-frame prediction. Finally, by selecting the recognition result with higher confidence in the action classification output result of the overlapping video frames as the prediction result of that video frame, it can not only correct recognition errors, but also minimize misidentification, thereby further ensuring the accuracy of the final action counting result of the video to be counted.

[0134] Furthermore, the action counting method provided in this embodiment has no limitation on the length of the video to be counted. By cyclically inputting a fixed number of frames into the video frame action recognition model, it is possible to complete the dense prediction of all video frames in the video to be counted. The entire counting process is convenient, simple, and easy to implement, thus making it better adaptable to various practical application scenarios.

[0135] Furthermore, it should be understood that the step divisions of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0136] The third embodiment of this application relates to an action counting device, such as... Figure 5 As shown, it includes: a network model inference module 501, a Gaussian modeling processing module 502, and an action counting module 503.

[0137] The network model inference module 501 is used to identify the video to be counted using a pre-trained video frame action recognition model to obtain a Gaussian regression output sequence; the Gaussian modeling processing module 502 is used to perform Gaussian modeling based on the Gaussian regression output sequence to obtain a Gaussian model; and the action counting module 503 is used to count actions based on the number of Gaussian distributions in the Gaussian model.

[0138] In another example, the action counting device also includes a network model training module.

[0139] Specifically, the network model training module is used to iteratively train the video frame action recognition model based on the pre-designed network model structure and labeled action video samples.

[0140] In another example, the video frame action recognition model obtained by training the network model can be used only to achieve the recognition of Gaussian distribution values.

[0141] Therefore, in this case, the designed network model structure can include only the Gaussian regression output branch.

[0142] Accordingly, the network model training module is specifically used to train and obtain the video frame action recognition model according to the following process:

[0143] Obtain motion video samples;

[0144] The motion intervals in the motion video samples are labeled with a Gaussian distribution;

[0145] Using the labeled action video samples, the pre-designed network model structure is iteratively trained until the preset convergence condition is met, thus obtaining the video frame action recognition model.

[0146] Furthermore, in another example, the Gaussian distribution annotation of the action intervals in the action video samples specifically involves:

[0147] The start time of the action in the action video sample is marked as t. s The time of the action termination frame is marked as t. e The key frame time is marked as t m The time scale factor is denoted as s;

[0148] Let the Gaussian distribution of the action satisfy the following formula:

[0149]

[0150] Where μ = st m ,

[0151] In another example, the network model training module is specifically used for:

[0152] Select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence;

[0153] The continuous video frame sequence is input into the Gaussian regression output branch in the network model structure;

[0154] The starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch is iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thereby obtaining the video frame action recognition model.

[0155] In another example, to improve the convergence of the video frame action recognition model and reduce the training difficulty of the Gaussian regression output branch during training, the pre-designed network model structure can also include an action classification output branch.

[0156] Specifically, the action classification output branch is used to determine whether a video frame belongs to an action.

[0157] Accordingly, the network model training module is specifically used to train and obtain the video frame action recognition model according to the following process:

[0158] Select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence;

[0159] The continuous video frame sequence is input into the Gaussian regression output branch in the network model structure;

[0160] The Gaussian distribution label corresponding to each video frame in the continuous video frame sequence is converted into a binary label, and the converted continuous video frame sequence is input into the action classification output branch in the network model structure.

[0161] The starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch and the action classification output branch are iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thereby obtaining the video frame action recognition model.

[0162] In another example, to further reduce the complexity of training a video frame action recognition model, the pre-designed network model structure can also include a 3D convolutional trunk.

[0163] Accordingly, the network model training module is specifically used to train and obtain the video frame action recognition model according to the following process:

[0164] Select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence;

[0165] The continuous video frame sequence is input into the 3D convolutional trunk in the network model structure, and the 3D convolutional trunk performs temporal feature extraction. The extracted temporal features are then used as the continuous video frame sequence that needs to be input into the Gaussian regression output branch and the action classification output branch in the network model structure, respectively.

[0166] The continuous video frame sequence output by the 3D convolution trunk is respectively input into the Gaussian regression output branch and the action classification output branch in the network model structure;

[0167] The starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch and the action classification output branch are iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thereby obtaining the video frame action recognition model.

[0168] Furthermore, in another example, to ensure the accuracy of the final action count, the network model inference module 501, when using the pre-trained video frame action recognition model to identify the video to be counted and obtain the Gaussian regression output sequence, specifically does the following:

[0169] According to the preset overlap strategy, the video frame action recognition model obtained through pre-training is used to identify the video to be counted, and a Gaussian regression output sequence is obtained.

[0170] It should be noted that, in this embodiment, the overlap strategy stipulates that the (N-1)th consecutive video frame sequence of length T contains L video frames that are identical to those in the Nth consecutive video frame sequence of length T. <L<T。

[0171] In another example, the network model inference module 501, following a preset overlap strategy, uses a pre-trained video frame action recognition model to identify the video to be counted, obtaining a Gaussian regression output sequence. Specifically:

[0172] Select consecutive video frames of fixed length T from the video to be counted to obtain N consecutive video frame sequences of length T;

[0173] The video frame action recognition model is input sequentially with N consecutive video frame sequences of length T to obtain the action classification output result and Gaussian distribution value of each video frame in each consecutive video frame sequence of length T. The action classification output result is 1 or 0, and the Gaussian distribution value is between [0, 1].

[0174] For each video frame, determine whether the corresponding action classification output result is 1;

[0175] If the value is 1, the video frame is determined to be an action, and the confidence level of the action classification output result of the video frame is obtained;

[0176] The confidence scores of the action classification outputs of the same video frames in the (N-1)th consecutive video frame sequence of length T are compared with those of the Nth consecutive video frame sequence of length T.

[0177] Based on the comparison results, the action classification result with higher confidence is selected as the target action classification result of the video frame identified by the video frame recognition model.

[0178] The Gaussian distribution value corresponding to the target action classification result is used as the target Gaussian distribution value of the video frame identified by the video frame recognition model.

[0179] Based on the time when the video frames appear in the video to be counted, the target Gaussian distribution values ​​are arranged sequentially to obtain the Gaussian regression output sequence.

[0180] That is, there are partially identical video frames in two consecutive video frame sequences of the same length. By performing overlap prediction on the given video frames, the actual Gaussian distribution value of the overlapping video frames is determined, and a Gaussian regression output sequence that can accurately reflect the actual situation is obtained.

[0181] Furthermore, in another example, to avoid the problem of easily getting trapped in the optimal solution and thus causing Gaussian fitting failure when simply using Gaussian fitting, the Gaussian modeling processing module 502 performs Gaussian modeling based on the Gaussian regression output sequence. Specifically, when obtaining the Gaussian model, it does so as follows:

[0182] Based on a heuristic method of piecewise Gaussian fitting, Gaussian modeling is performed on the Gaussian regression output sequence to obtain a Gaussian model.

[0183] Furthermore, in another example, when the action counting module 503 counts actions based on the number of Gaussian distributions in the Gaussian model, it specifically does so as follows:

[0184] Count the number of complete Gaussian distributions in the Gaussian model.

[0185] The number of Gaussian distributions is taken as the number of actions included in the video to be counted.

[0186] Based on this, the action counting device provided in this application has at least the following advantages compared with the prior art:

[0187] (1) The network model structure of the 3D convolution trunk, action classification output branch and Gaussian regression output branch designed in this application uses the 3D convolution trunk to extract temporal features, which not only has strong extraction ability, but also greatly reduces the computational complexity. The introduction of the action classification output branch can greatly reduce the training difficulty of the Gaussian regression output branch and accelerate network convergence.

[0188] (2) This application starts from the perspective of probability and statistics, uses Gaussian distribution to represent a complete action interval, and the number of Gaussian distributions represents the number of actions. Then, an efficient piecewise Gaussian fitting algorithm is used to perform Gaussian fitting to obtain the number of Gaussian distributions. Compared with existing methods, this application is more accurate and robust in implementing scene action counting applications.

[0189] (3) Thanks to the novel Gaussian distribution action representation method, this application can output the number of actions and also give the start time and end time of the action according to the 3σ criterion of Gaussian distribution, which has important guiding significance for the positioning of time-series actions.

[0190] (4) When performing network model inference, this application can correct identification errors and reduce misidentification by using a strategy based on overlapping single-frame prediction. On the other hand, it can complete the dense prediction of all video frames by cyclically inputting a fixed number of frames into the network model. Therefore, this application does not impose any restrictions on the length of the video.

[0191] Furthermore, it is readily apparent that this embodiment is a device embodiment corresponding to the first or second embodiment, and this embodiment can be implemented in conjunction with the first or second embodiment. The relevant technical details mentioned in the first or second embodiment remain valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the first or second embodiment.

[0192] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0193] The fourth embodiment of this application relates to an action counting device, such as... Figure 6 As shown, it includes: at least one processor 601; and a memory 602 communicatively connected to at least one processor 601; wherein the memory 602 stores instructions executable by at least one processor 601, the instructions being executed by at least one processor 601 to enable at least one processor 601 to perform the action counting method described in the above method embodiments.

[0194] The memory 602 and processor 601 are connected via a bus, which may include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 601 and memory 602 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 601 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 601.

[0195] Processor 601 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 602 can be used to store data used by processor 601 during operation.

[0196] The fifth embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the action counting method described in the above-described method embodiments.

[0197] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0198] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for counting actions, characterized in that, include: The video frame action recognition model obtained through pre-training is used to identify the video to be counted, and a Gaussian regression output sequence is obtained. The Gaussian regression output sequence is the Gaussian distribution value corresponding to each video frame in sequence. Gaussian modeling is performed based on the Gaussian regression output sequence to obtain the Gaussian model; Action counting is performed based on the number of Gaussian distributions in the Gaussian model. The process of using a pre-trained video frame action recognition model to identify the video to be counted and obtaining a Gaussian regression output sequence includes: Select a fixed length from the video to be counted. Continuous video frames, obtained A length of A continuous sequence of video frames; In sequence A length of A continuous sequence of video frames is input into the video frame action recognition model to obtain each frame with a length of... The action classification output result and Gaussian distribution value of each video frame in a continuous video frame sequence, wherein the action classification output result is 1 or 0, and the Gaussian distribution value is between [0, 1]. For each video frame, determine whether the corresponding action classification output result is 1; If the value is 1, the video frame is determined to be an action, and the confidence level of the action classification output result of the video frame is obtained; The first A length of A sequence of consecutive video frames and the first A length of The confidence scores of action classification outputs for the same video frame in a continuous video frame sequence are compared. Based on the comparison results, the action classification result with higher confidence is selected as the target action classification result of the video frame recognized by the video frame action recognition model. The Gaussian distribution value corresponding to the target action classification result is used as the target Gaussian distribution value of the video frame identified by the video frame action recognition model. Based on the time when the video frames appear in the video to be counted, the target Gaussian distribution values ​​are arranged sequentially to obtain the Gaussian regression output sequence.

2. The action counting method as described in claim 1, characterized in that, The training of the video frame action recognition model includes: Obtain motion video samples; The motion intervals in the motion video samples are labeled with a Gaussian distribution; Using the labeled action video samples, the pre-designed network model structure is iteratively trained until a preset convergence condition is met, thus obtaining the video frame action recognition model. The network model structure includes a Gaussian regression output branch.

3. The action counting method as described in claim 2, characterized in that, The step of labeling the motion regions in the motion video samples using a Gaussian distribution includes: The start time of the action in the action video sample is marked as... The action termination frame time is marked as Keyframe moments are marked as The time scale factor is labeled as ; Let the Gaussian distribution of the action satisfy the following formula: in, , .

4. The action counting method as described in claim 2, characterized in that, The step of iteratively training a pre-designed network model structure using the labeled action video samples until a preset convergence condition is met, to obtain the video frame action recognition model, includes: Select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence; The continuous video frame sequence is input into the Gaussian regression output branch in the network model structure; The starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch is iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thereby obtaining the video frame action recognition model.

5. The action counting method according to any one of claims 2 to 4, characterized in that, The network model structure also includes an action classification output branch; The step of iteratively training a pre-designed network model structure using the labeled action video samples until the network model structure meets a preset convergence condition to obtain the video frame action recognition model includes: Select consecutive video frames of a preset length from the labeled action video samples to obtain a consecutive video frame sequence; The continuous video frame sequence is input into the Gaussian regression output branch in the network model structure; The Gaussian distribution label corresponding to each video frame in the continuous video frame sequence is converted into a binary label, and the converted continuous video frame sequence is input into the action classification output branch in the network model structure. The starting position of the continuous video frame sequence is randomly selected, and the Gaussian regression output branch and the action classification output branch are iteratively trained using the stochastic gradient descent method with momentum until the preset convergence condition is met, thereby obtaining the video frame action recognition model.

6. The action counting method as described in claim 5, characterized in that, The network model structure also includes a 3D convolutional trunk; Before inputting the continuous video frame sequence into the Gaussian regression output branch of the network model structure, converting the Gaussian distribution label corresponding to each video frame in the continuous video frame sequence into a binary label, and inputting the converted continuous video frame sequence into the action classification output branch of the network model structure, the method further includes: The continuous video frame sequence is input into the 3D convolutional trunk in the network model structure, where temporal features are extracted. The extracted temporal features are then input into the Gaussian regression output branch and the action classification output branch in the network model structure, respectively.

7. The action counting method as described in claim 6, characterized in that, The step of performing Gaussian modeling based on the Gaussian regression output sequence to obtain a Gaussian model includes: Based on the heuristic method of piecewise Gaussian fitting, Gaussian modeling is performed according to the Gaussian regression output sequence to obtain the Gaussian model; The heuristic method based on piecewise Gaussian fitting specifically includes: Obtain the maximum value of the Gaussian regression output sequence and the first index number corresponding to the maximum value; If the maximum value is greater than or equal to the minimum Gaussian threshold, starting from the first index number, search to the left for the first corresponding second index number in the Gaussian regression output sequence whose value is less than the minimum Gaussian threshold, and search to the right for the first corresponding third index number in the Gaussian regression output sequence whose value is less than the minimum Gaussian threshold. If the absolute value of the difference between the third index number and the second index number is greater than or equal to the minimum Gaussian sampling point threshold, perform single Gaussian fitting on the Gaussian regression output sequence between the third index number and the second index number, increase the number of Gaussians by one, and set the Gaussian regression output sequence between the third index number and the second index number to 0; or, if the absolute value of the difference between the third index number and the second index number is less than the minimum Gaussian sampling point threshold, obtain the next maximum value and the next first index number, and set the Gaussian regression output sequence between the third index number and the second index number to 0. The piecewise Gaussian fitting ends when the maximum value is less than the minimum Gaussian value threshold.

8. The action counting method as described in claim 6, characterized in that, The step of counting actions based on the number of Gaussian distributions in the Gaussian model includes: Count the number of complete Gaussian distributions in the Gaussian model. The number of Gaussian distributions is taken as the number of actions included in the video to be counted.

9. An action counting device, characterized in that, include: The network model inference module is used to identify the video to be counted using a pre-trained video frame action recognition model, and to obtain a Gaussian regression output sequence, wherein the Gaussian regression output sequence is the Gaussian distribution value corresponding to each video frame in sequence. The Gaussian modeling processing module is used to perform Gaussian modeling based on the Gaussian regression output sequence to obtain a Gaussian model. The action counting module is used to count actions based on the number of Gaussian distributions in the Gaussian model. The network model inference module is also used for: Select a fixed length from the video to be counted. Continuous video frames, obtained A length of A continuous sequence of video frames; In sequence A length of A continuous sequence of video frames is input into the video frame action recognition model to obtain each frame with a length of... The action classification output result and Gaussian distribution value of each video frame in a continuous video frame sequence, wherein the action classification output result is 1 or 0, and the Gaussian distribution value is between [0, 1]. For each video frame, determine whether the corresponding action classification output result is 1; If the value is 1, the video frame is determined to be an action, and the confidence level of the action classification output result of the video frame is obtained; The first A length of A sequence of consecutive video frames and the first A length of The confidence scores of action classification outputs for the same video frame in a continuous video frame sequence are compared. Based on the comparison results, the action classification result with higher confidence is selected as the target action classification result of the video frame recognized by the video frame action recognition model. The Gaussian distribution value corresponding to the target action classification result is used as the target Gaussian distribution value of the video frame identified by the video frame action recognition model. Based on the time when the video frames appear in the video to be counted, the target Gaussian distribution values ​​are arranged sequentially to obtain the Gaussian regression output sequence.

10. An action counting device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the action counting method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the action counting method according to any one of claims 1 to 8.

Images