Repeated action counting method and device based on multi-structure information perception network

Abstract

The present invention relates to the field of computer vision technology, and provides a method and device for counting repeated actions based on a multi-structure information perception network. The multi-structure information perception network used includes a structure information extraction module, a structure information fusion module, and a repetition counting module. The structure information extraction module extracts the structure information in each video frame of the video to be counted for auxiliary discrimination, which can improve the performance of the multi-structure information perception network. The structure information fusion module adaptively captures the relevance of structure information through a multiple attention mechanism, mines the local features of the structure information through multiple convolution operations, and pays attention to the changes in local details. The two complement each other to ensure the accuracy of the number of repetitions of each action category. The repetition counting module can achieve accurate counting of repeated actions in the video to be counted by applying the threshold corresponding to each action category.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for counting repeated actions based on a multi-structure information perception network. Background Art

[0002] With the increasing development of artificial intelligence technology, the field of video analysis has also ushered in a revolution. One of them is the application of video repetition counting. Repetition counting is a technology that uses video capture technology to count the number of repetitions of specific actions. This technology has great potential in evaluating the training effect of athletes and monitoring and judging their physical condition. It can also be used in the field of fitness to help individuals track their progress and measure their fitness intensity.

[0003] Existing repeated action counting methods can be mainly divided into two categories: traditional methods and computer vision based methods.

[0004] Traditional methods mainly include manual counting and sensor-assisted counting. Manual counting requires a dedicated recorder, which is labor-intensive and often difficult to accurately count for certain fast-frequency movements. There are counting errors caused by delayed reactions, and there may also be counting errors due to recorder fatigue. The sensor-assisted counting method generally installs infrared sensors, pressure sensors, etc. on the sports field, or allows athletes to wear corresponding sensors, and then analyzes the sensor data information to achieve repeated action counting. Although this method is highly accurate, the equipment is complex, and the sensors used for different actions are not the same, and the layout cost is high. In addition, wearing sensors is very likely to affect performance or cause safety accidents.

[0005] Computer vision-based methods can overcome the shortcomings of traditional methods such as low efficiency and contact characteristics. This type of method solves the above problems in a data-driven way through context perception or time correlation modeling, thereby achieving repeated counting in general scenarios. However, its counting accuracy is far from meeting the actual application requirements in physical fitness test scenarios. Existing methods focus on global spatial information by treating each frame of the video as a whole, lacking the ability to distinguish local area features with periodic motion, making it difficult to identify fine-grained local periodic motion, which in turn leads to large repeated counting errors. Summary of the invention

[0006] The present invention provides a repeated action counting method and device based on a multi-structure information perception network, so as to solve the defects existing in the prior art.

[0007] The present invention provides a repeated action counting method based on a multi-structure information perception network, comprising:

[0008] Get the video to be counted;

[0009] Inputting the video to be counted into a multi-structure information perception network to obtain the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network;

[0010] Wherein, the multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module;

[0011] The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes position information of each joint point, angle information of a specified joint point and distance information between target joint point pairs;

[0012] The structural information fusion module is used to fuse the structural information based on multiple attention mechanisms and multiple convolution operations to obtain fused features, and based on the fused features, obtain the score of each action category in each video frame of the video to be counted;

[0013] The repetition counting module is used to count the repetitive actions in the video to be counted based on the score of each action category in each video frame of the video to be counted and by applying the threshold value corresponding to each action category.

[0014] The present invention also provides a repeated action counting device based on a multi-structure information perception network, comprising:

[0015] A video acquisition module is used to acquire the video to be counted;

[0016] A repeated action counting module, used for inputting the video to be counted into a multi-structure information perception network, and obtaining the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network;

[0017] Wherein, the multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module;

[0018] The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes position information of each joint point, angle information of a specified joint point and distance information between target joint point pairs;

[0019] The structural information fusion module is used to fuse the structural information based on multiple attention mechanisms and multiple convolution operations to obtain fusion features, and obtain the scores of each action category in each video frame of the video to be counted based on the fusion features;

[0020] The repetition counting module is used to count the repetitive actions in the video to be counted based on the score of each action category in each video frame of the video to be counted and by applying the threshold value corresponding to each action category.

[0021] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-described XXXX methods when executing the program.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described repeated action counting methods based on a multi-structure information perception network.

[0023] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for counting repeated actions based on a multi-structure information perception network as described in any one of the above is implemented.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The method and device for counting repeated actions based on a multi-structure information perception network provided by the present invention first obtain a video to be counted; then input the video to be counted into the multi-structure information perception network to obtain the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network. The multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module. The structure information extraction module extracts the structure information in each video frame of the video to be counted for auxiliary discrimination, which can improve the performance of the multi-structure information perception network. The structure information fusion module adaptively captures the relevance of the structure information through a multiple attention mechanism, mines the local features of the structure information through multiple convolution operations, and pays attention to the changes in local details. The two complement each other to ensure the accuracy of the number of repetitions of each action category. The repetition counting module can accurately count the repeated actions in the video to be counted by applying the thresholds corresponding to each action category. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without paying any creative work.

[0027] Figure 1 It is a flow chart of a repeated action counting method based on a multi-structure information perception network provided by the present invention;

[0028] Figure 2 It is a structural schematic diagram of a multi-structure information perception network in a repeated action counting method based on a multi-structure information perception network provided by the present invention;

[0029] Figure 3 It is a schematic diagram of each joint point in each video frame of the video to be counted in the repeated action counting method based on the multi-structure information perception network provided by the present invention;

[0030] Figure 4 It is a schematic diagram of angle information of a specified joint point in each video frame of a video to be counted in the repeated action counting method based on a multi-structure information perception network provided by the present invention;

[0031] Figure 5 It is a schematic diagram of distance information between target joint point pairs in each video frame of a video to be counted in a repeated action counting method based on a multi-structure information perception network provided by the present invention;

[0032] Figure 6 It is a structural schematic diagram of a multiple attention module of a multi-structure information perception network in a repeated action counting method based on a multi-structure information perception network provided by the present invention;

[0033] Figure 7 It is a structural schematic diagram of each attention module of the multi-structure information perception network in the repeated action counting method based on the multi-structure information perception network provided by the present invention;

[0034] Figure 8 It is a structural schematic diagram of a structural feature mining module of a multi-structure information perception network in a repeated action counting method based on a multi-structure information perception network provided by the present invention;

[0035] Fig. 9 It is a schematic diagram of the scores of each action category in each video frame in the repeated action counting method based on the multi-structure information perception network provided by the present invention;

[0036] Fig.10 It is a structural schematic diagram of a repetition counting module of a multi-structure information perception network in a repetition action counting method based on a multi-structure information perception network provided by the present invention;

[0037] Fig.11 It is a structural schematic diagram of a repeated action counting device based on a multi-structure information perception network provided by the present invention;

[0038] Fig.12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Figure 1 Schematic diagram of a flow chart of a repeated action counting method based on a multi-structure information perception network provided in an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0041] S1, obtain the video to be counted;

[0042] S2, inputting the video to be counted into a multi-structure information perception network, and obtaining the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network;

[0043] Wherein, the multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module;

[0044] The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes position information of each joint point, angle information of a specified joint point and distance information between target joint point pairs;

[0045] The structural information fusion module (MIF-Module) is used to fuse the structural information based on multiple attention mechanisms and multiple convolution operations to obtain fusion features, and based on the fusion features, obtain the score of each action category in each video frame of the video to be counted;

[0046] The repetition counting module is used to count the repetitive actions in the video to be counted based on the score of each action category in each video frame of the video to be counted and by applying the threshold value corresponding to each action category.

[0047] Specifically, the repeated action counting method based on a multi-structure information perception network provided in an embodiment of the present invention, its executor is a repeated action counting device based on a multi-structure information perception network, and the device can be configured in a computer, and the computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0048] First, step S1 is performed to obtain a video to be counted. The video to be counted may include multiple video frames, and each video frame may include one or more action categories.

[0049] Then, step S2 is executed to input the video to be counted into a multi-structure information perception network (MIA-Net), and the multi-structure information perception network outputs the number of repetitions of each action category in the video to be counted.

[0050] like Figure 2 As shown, the multi-structure information perception network may include a structure information extraction module, a structure information fusion module, and a repetition counting module connected in sequence.

[0051] The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes the position information of each joint point, the angle information of the specified joint point and the distance information between the target joint point pairs.

[0052] The structural information extraction module can extract the position information of each joint point in each video frame of the video to be counted through the human posture tracking algorithm. Figure 3 As shown, head joints, shoulder joints, elbow joints, hip joints, knee joints and foot joints may be included.

[0053] The human posture tracking algorithm can be the Blazepose algorithm. That is,

[0054] ;

[0055] ;

[0056] Among them, V represents the video to be counted, P represents the joint point information of the video to be counted, and the joint point is the feature point of the significant part. represents the i-th video frame in the video to be counted, Indicates the number of channels, usually three channels. Indicates height, Indicates width, Indicates the frame number. Represents the joint point information in the i-th video frame. In order to represent the joint point information in each video frame, the sequence To express. Represents the dimension of each joint point, which is generally three-dimensional, including two-dimensional position information and one-dimensional depth information. Indicates the number of joint points.

[0057] Since different action categories will lead to different changes in the angle information and distance information between joints during the movement, the hidden structural information such as angle information and distance information can help distinguish different action categories. Figure 4As shown, according to the real scene, the changes of the left elbow angle θ11, the left shoulder angle θ12, the left hip angle θ13, the left knee angle θ14, the right elbow angle θ21, the right shoulder angle θ22, the right hip angle θ23 and the right knee angle θ24 are the most obvious in most actions, and the changes of these angles are highly representative for specific actions. Therefore, the structural information extraction module can calculate the angle information of the specified joint point as auxiliary structural information. Here, the specified joint point can include each joint point, and can also include an elbow joint point, a shoulder joint point, a hip joint point and a knee joint point.

[0058] The angle information of the specified joint point can be determined by the position information of the specified joint point and its adjacent joint points in each video frame of the video to be counted. The mathematical expression for calculating the angle information of the specified joint point includes:

[0059] ;

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] Among them, B represents the specified joint point, A and C are the adjacent joint points of the specified joint point B, is the location information of A, is the location information of B, is the location information of C, is the line vector between A and B, is the line vector between B and C, is the angle information of B.

[0065] like Figure 5 As shown, in most actions, the distance information between the left wrist joint and the shoulder joint , the distance information between the right wrist joint and the shoulder joint , the distance information between the left wrist joint and the hip joint , the distance information between the right wrist joint and the hip joint , the distance information between the left shoulder joint and the knee joint , the distance information between the right shoulder joint and the knee joint , the distance information between the left head joint and the knee joint The distance information between the right head joint and the knee joint ,The distance information between these joint points also plays a key role in the ,judgment of repeated actions.

[0066] Therefore, the structural information extraction module uses the distance information between the target joint points as additional structural information, the purpose of which is to enhance and assist the coordinate information in the discrimination of repeated actions. The target joint point pairs may include wrist joint points and shoulder joint points, wrist joint points and hip joint points, shoulder joint points and knee joint points, head joint points and knee joint points.

[0067] The distance information between the target joint point pairs can be determined by the position information of each target joint point in the target joint point pair. The mathematical expression for calculating the distance information between the target joint point pairs can be expressed as:

[0068] .

[0069] in, is the distance information between the target joint point pair formed by the target joint points E and F, is the location information of E, is the location information of F.

[0070] Based on this, the position information set of each joint point in each video frame of the video to be counted , the angle information set of the specified joint point in each video frame of the video to be counted And the distance information set between the target joint point pairs in each video frame of the video to be counted It can be expressed as:

[0071] ;

[0072] ;

[0073] ;

[0074] in, , , They respectively represent the position information of the nth joint point in a certain video frame in the video to be counted, the angle information of the mth specified joint point, and the distance information between the tth target joint point pair. , , They respectively represent the number of joint points in a certain video frame in the video to be counted, the number of specified joint points, and the number of target joint point pairs.

[0075] The structural information fusion module can use multiple attention mechanisms and multiple convolution operations to fuse structural information and obtain fused features. The multiple attention mechanisms can establish long-distance dependencies on structural information globally, and the multiple convolution operations can capture local subtle changes in structural information, thereby obtaining fused features.

[0076] The structural information fusion module can also establish a mapping relationship between fusion features and action categories with the help of the fully connected layer, input the fusion features into the fully connected (FC) layer, and obtain the scores of each action category in each video frame of the video to be counted.

[0077] The repetition counting module can use the scores of each action category in each video frame of the video to be counted, and through the action trigger, apply the threshold corresponding to each action category to count the repetitive actions in the video to be counted. Each action category can correspond to an entry threshold and an exit threshold, the entry threshold is the score of the starting posture of the action category, and the exit threshold is the score of the ending posture of the action category.

[0078] When the entry threshold and exit threshold corresponding to a certain action category are triggered continuously, it means that the action category is generated once. Then, the number of consecutive triggering of the entry threshold and exit threshold corresponding to each action category in the video to be counted can be counted to obtain the number of repetitions of each action category.

[0079] The repeated action counting method based on the multi-structure information perception network provided in the embodiment of the present invention first obtains the video to be counted; then the video to be counted is input into the multi-structure information perception network to obtain the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network. The multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module. The structure information extraction module extracts the structure information in each video frame of the video to be counted for auxiliary discrimination, which can improve the performance of the multi-structure information perception network. The structure information fusion module adaptively captures the relevance of the structure information through a multiple attention mechanism, mines the local features of the structure information through multiple convolution operations, and pays attention to the changes in local details. The two complement each other to ensure the accuracy of the number of repetitions of each action category. The repetition counting module can accurately count the repeated actions in the video to be counted by applying the thresholds corresponding to each action category.

[0080] On the basis of the above embodiment, the structural information fusion module includes an information fusion and embedding module (IFE-Module), a multiple attention module (MA-Module), a structural feature mining module (SFM-Module) and a feature mapping module;

[0081] The information fusion and embedding module is used to splice the structural information in each video frame of the video to be counted to obtain a splicing result, and embed the splicing result into the feature space of the multiple attention module to obtain an embedded feature;

[0082] The multiple attention module is used to establish a long-distance dependency relationship between the embedded features globally based on the multiple attention mechanism to obtain global features;

[0083] The structural feature mining module is used to perform local feature mining and fusion on the global features based on multiple convolution modules to obtain the fusion features;

[0084] The feature mapping module is used to perform classification prediction on the fusion feature based on a fully connected layer to obtain the score of each action category in each video frame of the video to be counted.

[0085] Specifically, in order to obtain multiple structural information, the information fusion and embedding module first splices the structural information in each video frame to obtain the splicing result ,in is the number of information after splicing. Splicing result It can be expressed as:

[0086] ;

[0087] Among them, Concat is a concatenation operation.

[0088] Subsequently, the information fusion and embedding module uses the embedding layer to embed the concatenation result into the feature space of the multiple attention modules to obtain high-dimensional embedded features. ,in is the number of dimensions of each joint point after embedding. Here, the embedding layer may include a batch normalization layer (Batch Norm, BN) and two linear modules (LBR), and the linear module includes a linear layer (Linear), a batch normalization layer and an activation layer (ReLU).

[0089] Embedding Features It can be expressed as:

[0090] .

[0091] Embedding Features It is passed to the multi-attention module, which adaptively updates the weight of each feature and learns the correlation between each feature, thereby generating global features with higher representativeness and more key information. In order to reduce information loss and learn more complex feature representations without increasing the number of network parameters, this module connects the output features of the attention module through a residual structure.

[0092] like Figure 6 As shown, the multiple attention module includes a first attention module, a first superposition layer, a second attention module, a second superposition layer, a third attention module and a third superposition layer connected in sequence, and the input end of the first attention module, the input end of the first superposition layer and the input end of the third superposition layer are all used to connect with the output end of the information fusion and embedding module for inputting the embedding feature The input end of the second stacking layer is also used to connect to the output end of the first stacking layer. Finally, the output end of the third stacking layer outputs the global feature :

[0093] ;

[0094] ;

[0095] ;

[0096] in, is the output of the first attention module, is the output of the second attention module, Operation for the attention mechanism.

[0097] It is understandable that if Figure 7 As shown, each attention module can generate a query vector W by passing the input through three linear layers. Q , key vector W K Sum value vector W V , by taking the key vector W K With the value vector W V Multiply them, and pass the product through the normalization layer (Softmax) to obtain the normalized features. Q Multiply it with the normalized feature, and superimpose the product result with the input twice to get the output.

[0098] In order to more effectively extract representative global features The most significant local features are extracted from the structure feature mining module, and the global features are used As input, a new round of feature extraction and splicing is performed through multiple convolution modules. The purpose is to fuse information from multiple aspects and obtain more significant detail information to improve the representation of significant features and obtain fused features. .

[0099] Here, if Figure 8As shown in the figure, the structural feature mining module specifically includes the first convolution module, the first concatenation layer (Concat1), the second convolution module, the second concatenation layer (Concat2), the third convolution module, the third concatenation layer (Concat3), the fourth concatenation layer (Concat4) and the fourth convolution module which are connected in sequence. The structural feature mining module can reduce the loss of effective information, and learn more significant and richer high-level features, providing the most representative input for subsequent feature mapping.

[0100] The input end of the first convolution module is used to connect to the output end of the multi-attention module to input the global features ; The input end of the second splicing layer is also used to connect to the output end of the first splicing layer; The input end of the third splicing layer is also used to connect to the output end of the multi-attention module and also to input the global feature , output fusion features .

[0101] The first convolution module may include one convolution block, the second convolution module may include two convolution blocks, the third convolution module may include one convolution block, and the fourth convolution module may include four convolution blocks. Here, the convolution block may include a 1×1 convolution layer (Conv), a batch normalization layer (BetchNorm), and an activation layer (ReLU), and the convolution block may be represented by CBR. Through the convolution block, the dimension of the network can be gradually reduced, and the parameter parameters can be kept unchanged.

[0102] Based on this, the operation of the structural feature mining module can be expressed by the following formula:

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] .

[0110] .

[0111] in, is the output of the first convolutional module, is the output of the first concatenation layer, is the output of the second convolutional module, is the output of the second concatenation layer, is the output of the third convolutional module, is the output of the third concatenation layer, is the output of the fourth concatenation layer, is the number of features after multiple feature extraction and fusion.

[0112] It is understandable that the fourth convolution module can constitute a classification decoder to output the fusion feature .

[0113] The feature mapping module is used to classify and predict the fused features based on the fully connected layer to obtain the scores of each action category in each video frame of the video to be counted. Here, the feature mapping module may include a flattening layer and a fully connected layer. The flattening layer can convert the fused features into one-dimensional features, and the fully connected layer can convert the one-dimensional features into the scores of each action category in each video frame of the video to be counted. That is:

[0114] .

[0115] in, is the score of a certain action category.

[0116] like Fig. 9 , which is a schematic diagram of the scores of each action category in each video frame. Fig. 9 In the example, the score of each action category in each video frame can be represented by a cuboid, and the higher the cuboid, the higher the score.

[0117] In the embodiment of the present invention, the score of each action category in each video frame of the video to be counted is determined through the specific structure of the structural information fusion module, so that the accuracy of the score can be guaranteed due to the subsequent accurate recording of the number of repetitions of each action category.

[0118] Based on the above embodiment, the repetition counting module is specifically used for:

[0119] For any action category, based on the score of any action category in each video frame of the video to be counted, apply the first threshold and the second threshold corresponding to any action category, determine the number of times the first threshold and the second threshold are triggered continuously in sequence, and use the number as the number of repetitions of any action category.

[0120] Specifically, Fig.10As shown, for any action category, all video frames in the video to be counted can be traversed to determine the score of any action category in all video frames, and the first threshold and the second threshold corresponding to any action category can be applied to determine the number of times the first threshold and the second threshold are triggered continuously in sequence, and this number is used as the number of repetitions of any action category.

[0121] Here, the first threshold may be smaller than the second threshold. The first threshold may be the exit threshold of any action category, which is the score of the ending posture of any action category. The second threshold may be the entry threshold of the action category, which is the score of the starting posture of any action category. Fig.10 In , the horizontal axis is the serial number of the video frame, and the vertical axis is the score of any action category in each video frame. Fig.10 It can be seen that the number of repetitions of any action category is 7.

[0122] In the embodiment of the present invention, by introducing the first threshold and the second threshold to determine the number of repetitions of the action category, the counting process can be simplified and the counting efficiency can be improved.

[0123] Based on the above embodiment, the multi-structure information perception network is trained based on the following steps:

[0124] Input each video frame sample in the video sample into the initial perception network, obtain the sample features of each video frame sample obtained by the structural information fusion module in the initial perception network and the sample action category of each video frame sample output by the initial perception network; each video frame sample includes an anchor point sample, a positive sample and a negative sample;

[0125] Based on the sample features, calculating a first feature distance between the anchor point sample and the positive sample and a second feature distance between the anchor point sample and the negative sample, and calculating a triple boundary loss based on the first feature distance and the second feature distance;

[0126] Calculating a binary cross entropy loss based on the sample action category and the action category label carried by each video frame sample;

[0127] Based on the triple boundary loss and the binary cross entropy loss, a comprehensive loss is calculated, and based on the comprehensive loss, the structural parameters of the initial perception network are iteratively optimized to obtain the multi-structure information perception network.

[0128] Specifically, during the training process of the multi-structure information perception network, each video frame sample in the video sample can be first input into the initial perception network. The structure of the initial perception network is the same as that of the multi-structure information perception network. The difference is that the structural parameters of the initial perception network are initialized, and the structural parameters of the multi-structure information perception network are obtained after optimization through the training process.

[0129] The initial perception network also includes a structure information extraction module, a structure information fusion module and a repetition counting module. After each video frame sample in the video sample is input into the initial perception network, the sample features of each video frame sample output by the initial fusion module in the initial perception network and the sample action category of each video frame sample output by the initial perception network are obtained.

[0130] Each video frame sample may include anchor samples, positive samples and negative samples. Anchor samples refer to real samples with real action categories, positive samples refer to samples with the same action category as real samples, and negative samples refer to samples with different action categories from real samples.

[0131] After that, the first feature distance between the anchor sample and the positive sample and the second feature distance between the anchor sample and the negative sample can be calculated using the sample features, and the triplet margin loss (Triplet Margin Loss) can be calculated using the first feature distance and the second feature distance. .

[0132] That is:

[0133] ;

[0134] Among them, a is the anchor sample, p is the positive sample, and n is the negative sample. is the first characteristic distance, is the second characteristic distance, Is a fixed value. It stands for cosine similarity, which is used to measure the similarity between features.

[0135] Using the sample action category and the action category label carried by each video frame sample, calculate the binary cross entropy loss (Binary Cross Entropy Loss) , that is:

[0136] ;

[0137] ;

[0138] in, represents the batch size, where each frame constitutes a batch, so N is the number of video frames in the video to be counted, Represents the number of categories. represents the jth action category label in the i-th video frame, is the jth sample action category in the i-th video frame.

[0139] Triple Boundary Loss The first feature distance between the anchor sample and the positive sample can be reduced, while the second feature distance between the anchor sample and the negative sample can be increased. The purpose of this is to better distinguish each action category, thereby improving performance. Binary Cross Entropy Loss Each action category can be classified into two categories.

[0140] Finally, triple boundary loss is used With binary cross entropy loss The weighted sum of , that is:

[0141] ;

[0142] in, To control the triple frontier loss and binary cross entropy loss This ensures that the triple boundary loss is used during network training. and binary cross entropy loss The relative importance of is within the same range.

[0143] Utilizing comprehensive loss , the structural parameters of the initial perception network are iteratively optimized until the comprehensive loss converges or the preset number of iterations is reached, and a multi-structure information perception network is obtained.

[0144] During the training process, the PyTorch-Lightning framework is used in an embodiment of the present invention to train a multi-structure information perception network. The PyTorch-Lightning framework performs a training step before officially starting training to monitor the changes in the comprehensive loss in the batch processing so as to automatically select the initial optimal learning rate. In addition, after each round of traversal (ie, epoch) is completed, a verification will be performed. If the comprehensive loss is not reduced after 6 consecutive epochs, the learning rate will be automatically adjusted. In addition, in an embodiment of the present invention, the optimizer is set to Adam, and the overall architecture is trained using Triplet MarginLoss and BCELoss on the NVIDIA PCle A100 GPU.

[0145] Compared with the traditional video-level method, the posture-level method focuses on the changes in the coordinates of the human body joints, which significantly improves the performance. However, the posture-level method ignores the hidden structural information between the human body joints during the motion process. Therefore, in the embodiment of the present invention, based on the posture-level method, further research is conducted on the structural information that is significant and distinguishable for the changes in human body posture during the motion process, and these important structural information are used to supplement the simple coordinate information.

[0146] The main metrics for evaluating model performance are Mean Absolute Error (MAE) and Off-Band Error (OBO). MAE represents the mean absolute error between the model prediction and the actual situation. On the other hand, OBO is defined as if the network's predicted value differs from the true value by no more than 1 (usually less than or equal to 1), then the network's predicted value of the count is considered correct. They can be defined as follows:

[0147] ;

[0148] ;

[0149] in, is the true value, is the predicted value, and N is the number of video samples.

[0150] As shown in Table 1, the multi-structure information perception network MIA-Net provided in the embodiment of the present invention is compared with some conventional methods on the RepCount-pose dataset, and the best result is MAE of 0.203 and OBO of 0.592. Compared with conventional video-level methods, MIA-Net reduces MAE by 18.1% and improves OBO by 20.6%.

[0151] In addition, compared with the latest method PoseRAC on pose keypoints, MIA-Net reduces MAE by 3.3% and improves OBO by 3.2%. Experimental results show that MIA-Net effectively learns the relationship between multi-structure information and establishes a good mapping relationship between fused features and action categories, which improves the performance of MIA-Net.

[0152] Table 1 Comparison of two key objective indicators between MIA-Net algorithm and existing algorithms on RepCount-pose dataset

[0153]

[0154] like Fig.11 As shown, based on the above embodiment, an embodiment of the present invention provides a repeated action counting device based on a multi-structure information perception network, including:

[0155] The video acquisition module 111 is used to acquire the video to be counted;

[0156] A repeated action counting module 112 is used to input the video to be counted into a multi-structure information perception network to obtain the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network;

[0157] Wherein, the multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module;

[0158] The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes position information of each joint point, angle information of a specified joint point and distance information between target joint point pairs;

[0159] The structural information fusion module is used to fuse the structural information based on multiple attention mechanisms and multiple convolution operations to obtain fusion features, and obtain the scores of each action category in each video frame of the video to be counted based on the fusion features;

[0160] The repetition counting module is used to count the repetitive actions in the video to be counted based on the score of each action category in each video frame of the video to be counted and by applying the threshold value corresponding to each action category.

[0161] Based on the above embodiment, the structure information extraction module is specifically used for:

[0162] Based on the human body posture tracking algorithm, extract the position information of each joint point in each video frame of the video to be counted;

[0163] Determine angle information of the designated joint point in each video frame of the video to be counted based on position information of the designated joint point and its adjacent joint points in each video frame of the video to be counted;

[0164] Based on the position information of the target joint point pairs in each video frame of the video to be counted, the distance information between the target joint point pairs in each video frame of the video to be counted is determined.

[0165] On the basis of the above embodiment, the structural information fusion module includes an information fusion and embedding module, a multiple attention module, a structural feature mining module and a feature mapping module;

[0166] The information fusion and embedding module is used to splice the structural information in each video frame of the video to be counted to obtain a splicing result, and embed the splicing result into the feature space of the multiple attention module to obtain an embedded feature;

[0167] The multiple attention module is used to establish a long-distance dependency relationship between the embedded features globally based on the multiple attention mechanism to obtain global features;

[0168] The structural feature mining module is used to perform local feature mining and fusion on the global features based on multiple convolution modules to obtain the fusion features;

[0169] The feature mapping module is used to perform classification prediction on the fusion feature based on the fully connected layer to obtain the score of each action category in each video frame of the video to be counted.

[0170] On the basis of the above embodiment, the structural feature mining module specifically includes a first convolution module, a first splicing layer, a second convolution module, a second splicing layer, a third convolution module, a third splicing layer, a fourth splicing layer and a fourth convolution module connected in sequence;

[0171] The input end of the first convolution module is used to connect to the output end of the multiple attention module; the input end of the second splicing layer is also used to connect to the output end of the first splicing layer; the input end of the third splicing layer is also used to connect to the output end of the multiple attention module.

[0172] Based on the above embodiment, the multiple attention module includes a first attention module, a first superposition layer, a second attention module, a second superposition layer, a third attention module and a third superposition layer connected in sequence;

[0173] The input end of the first attention module, the input end of the first superimposed layer and the input end of the third superimposed layer are all used to connect with the output end of the information fusion and embedding module; the input end of the second superimposed layer is also used to connect with the output end of the first superimposed layer.

[0174] Based on the above embodiment, the repetition counting module is specifically used for:

[0175] For any action category, based on the score of any action category in each video frame of the video to be counted, apply the first threshold and the second threshold corresponding to any action category, determine the number of times the first threshold and the second threshold are triggered continuously in sequence, and use the number as the number of repetitions of any action category.

[0176] Based on the above embodiment, a training module is further included, which is used for:

[0177] Input each video frame sample in the video sample into the initial perception network, obtain the sample features of each video frame sample output by the structural information fusion module in the initial perception network and the sample action category of each video frame sample output by the initial perception network; each video frame sample includes an anchor point sample, a positive sample and a negative sample;

[0178] Based on the sample features, calculating a first feature distance between the anchor point sample and the positive sample and a second feature distance between the anchor point sample and the negative sample, and calculating a triple boundary loss based on the first feature distance and the second feature distance;

[0179] Calculating a binary cross entropy loss based on the sample action category and the action category label carried by each video frame sample;

[0180] Based on the triple boundary loss and the binary cross entropy loss, a comprehensive loss is calculated, and based on the comprehensive loss, the structural parameters of the initial perception network are iteratively optimized to obtain the multi-structure information perception network.

[0181] Specifically, the functions of each module in the repeated action counting based on the multi-structure information perception network provided in the embodiment of the present invention correspond one-to-one to the operation process of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0182] Fig.12 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.12 As shown, the electronic device may include: a processor 121, a communication interface 122, a memory 123 and a communication bus 124, wherein the processor 121, the communication interface 122 and the memory 123 communicate with each other through the communication bus 124. The processor 121 may call the logic instructions in the memory 123 to execute the repeated action counting method based on the multi-structure information perception network provided in the above embodiments.

[0183] In addition, the logic instructions in the above-mentioned memory 123 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0184] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the repeated action counting method based on a multi-structure information perception network provided in the above-mentioned embodiments.

[0185] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the repeated action counting method based on a multi-structure information perception network provided in the above-mentioned embodiments.

[0186] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A repeated action counting method based on a multi-structure information perception network, characterized in that: include: Get the video to be counted; Inputting the video to be counted into a multi-structure information perception network to obtain the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network; Wherein, the multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module; The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes position information of each joint point, angle information of a specified joint point and distance information between target joint point pairs; The structural information fusion module is used to fuse the structural information based on multiple attention mechanisms and multiple convolution operations to obtain fusion features, and obtain the scores of each action category in each video frame of the video to be counted based on the fusion features; The repetition counting module is used to count the repetitive actions in the video to be counted based on the score of each action category in each video frame of the video to be counted and applying the threshold value corresponding to each action category; The structural information fusion module includes an information fusion and embedding module, a multiple attention module, a structural feature mining module and a feature mapping module; The information fusion and embedding module is used to splice the structural information in each video frame of the video to be counted to obtain a splicing result, and embed the splicing result into the feature space of the multiple attention module to obtain an embedded feature; The multiple attention module is used to establish a long-distance dependency relationship between the embedded features globally based on the multiple attention mechanism to obtain global features; The structural feature mining module is used to perform local feature mining and fusion on the global features based on multiple convolution modules to obtain the fusion features; The feature mapping module is used to perform classification prediction on the fusion feature based on the fully connected layer to obtain the score of each action category in each video frame of the video to be counted.

2. The method for counting repeated actions based on a multi-structure information perception network according to claim 1, characterized in that: The structural information extraction module is specifically used for: Based on the human body posture tracking algorithm, extract the position information of each joint point in each video frame of the video to be counted; Determine angle information of the designated joint point in each video frame of the video to be counted based on position information of the designated joint point and its adjacent joint points in each video frame of the video to be counted; Based on the position information of the target joint point pairs in each video frame of the video to be counted, the distance information between the target joint point pairs in each video frame of the video to be counted is determined.

3. The method for counting repeated actions based on a multi-structure information perception network according to claim 1, characterized in that: The structural feature mining module specifically includes a first convolution module, a first splicing layer, a second convolution module, a second splicing layer, a third convolution module, a third splicing layer, a fourth splicing layer and a fourth convolution module connected in sequence; The input end of the first convolution module is used to connect to the output end of the multiple attention module; the input end of the second splicing layer is also used to connect to the output end of the first splicing layer; the input end of the third splicing layer is also used to connect to the output end of the multiple attention module.

4. The method for counting repeated actions based on a multi-structure information perception network according to claim 1, characterized in that: The multiple attention module includes a first attention module, a first superposition layer, a second attention module, a second superposition layer, a third attention module and a third superposition layer connected in sequence; The input end of the first attention module, the input end of the first superimposed layer and the input end of the third superimposed layer are all used to connect with the output end of the information fusion and embedding module; the input end of the second superimposed layer is also used to connect with the output end of the first superimposed layer.

5. The method for counting repeated actions based on a multi-structure information perception network according to claim 1, characterized in that: The repeat counting module is specifically used for: For any action category, based on the score of any action category in each video frame of the video to be counted, apply the first threshold and the second threshold corresponding to any action category, determine the number of times the first threshold and the second threshold are triggered continuously in sequence, and use the number as the number of repetitions of any action category.

6. The method for counting repeated actions based on a multi-structure information perception network according to any one of claims 1 to 5, characterized in that: The multi-structure information perception network is trained based on the following steps: Input each video frame sample in the video sample into the initial perception network, obtain the sample features of each video frame sample output by the structural information fusion module in the initial perception network and the sample action category of each video frame sample output by the initial perception network; each video frame sample includes an anchor point sample, a positive sample and a negative sample; Based on the sample features, calculating a first feature distance between the anchor point sample and the positive sample and a second feature distance between the anchor point sample and the negative sample, and calculating a triple boundary loss based on the first feature distance and the second feature distance; Calculating a binary cross entropy loss based on the sample action category and the action category label carried by each video frame sample; Based on the triple boundary loss and the binary cross entropy loss, a comprehensive loss is calculated, and based on the comprehensive loss, the structural parameters of the initial perception network are iteratively optimized to obtain the multi-structure information perception network.

7. A repeated action counting device based on a multi-structure information perception network, characterized in that: include: A video acquisition module is used to acquire the video to be counted; A repeated action counting module, used for inputting the video to be counted into a multi-structure information perception network, and obtaining the number of repetitions of each action category in the video to be counted output by the multi-structure information perception network; Wherein, the multi-structure information perception network includes a structure information extraction module, a structure information fusion module and a repetition counting module; The structural information extraction module is used to extract structural information in each video frame of the video to be counted; the structural information includes position information of each joint point, angle information of a specified joint point and distance information between target joint point pairs; The structural information fusion module is used to fuse the structural information based on multiple attention mechanisms and multiple convolution operations to obtain fusion features, and obtain the scores of each action category in each video frame of the video to be counted based on the fusion features; The repetition counting module is used to count the repetitive actions in the video to be counted based on the score of each action category in each video frame of the video to be counted and applying the threshold value corresponding to each action category; The structural information fusion module includes an information fusion and embedding module, a multiple attention module, a structural feature mining module and a feature mapping module; The information fusion and embedding module is used to splice the structural information in each video frame of the video to be counted to obtain a splicing result, and embed the splicing result into the feature space of the multiple attention module to obtain an embedded feature; The multiple attention module is used to establish a long-distance dependency relationship between the embedded features globally based on the multiple attention mechanism to obtain global features; The structural feature mining module is used to perform local feature mining and fusion on the global features based on multiple convolution modules to obtain the fusion features; The feature mapping module is used to perform classification prediction on the fusion feature based on the fully connected layer to obtain the score of each action category in each video frame of the video to be counted.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the repeated action counting method based on a multi-structure information perception network as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the repeated action counting method based on a multi-structure information perception network as described in any one of claims 1 to 6 is implemented.

Images