Video action recognition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a video action recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a target video clip, wherein the target video clip is determined based on target video division; inputting each target video clip into a target behavior recognition main network, and obtaining an initial recognition result output by the target behavior recognition main network; and according to the time sequence of the target video clips, the initial recognition result corresponding to each target video clip is processed based on a confidence evaluation network to obtain a target recognition result of the target video, and the target recognition result is used for representing the behavior category of the target video. By adopting the method, the accuracy of action recognition and the generalization ability of the model in a complex scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video action recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of human-computer interaction, video content understanding and intelligent perception technologies, video action recognition, as one of the core research directions in the field of computer vision, has demonstrated wide application value in many practical scenarios such as intelligent monitoring, behavior analysis, motion capture and human-computer collaborative systems.

[0003] Although traditional action recognition methods based on RGB video sequences have good information integrity, their recognition performance is often significantly limited when faced with problems such as complex background interference, partial body occlusion, and changes in appearance features. Summary of the Invention

[0004] Based on this, it is necessary to provide a video action recognition method, device, computer equipment, computer-readable storage medium and computer program product that can improve recognition accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for video action recognition, the method comprising:

[0006] Obtaining a target video segment, wherein the target video segment is determined based on the target video segmentation;

[0007] Inputting each target video clip into a target behavior recognition main network and obtaining an initial recognition result output by the target behavior recognition main network, wherein the initial recognition result is used to represent the behavior classification probability vector corresponding to the target video clip; the target behavior recognition main network is determined by distilling the initial behavior recognition main network as a student model through a skeleton-guided network as a teacher model;

[0008] According to the time sequence of the target video segments, the initial recognition results corresponding to each target video segment are processed based on the confidence evaluation network to obtain a target recognition result of the target video, wherein the target recognition result is used to characterize the behavior category of the target video.

[0009] In one embodiment, the training process of the target behavior recognition main network includes:

[0010] Obtaining a first sample video segment and a true label of the first sample video segment; the first sample video segment is determined based on the first sample video segmentation;

[0011] Inputting the first sample video clip into the initial action recognition main network to obtain a first prediction result output by the initial action recognition main network; determining the initial network loss based on the true label and the first prediction result;

[0012] Obtaining an error between a distillation layer of the skeleton-guided network and a corresponding layer of the initial behavior recognition main network, and determining a distillation loss based on the error;

[0013] Determining a total loss based on the initial network loss, the distillation loss, and a distillation loss weight;

[0014] The initial behavior recognition main network is adjusted according to the total loss to obtain the target behavior recognition main network.

[0015] In one embodiment, obtaining an error between a distillation layer of the skeleton-guided network and a corresponding layer of the initial behavior recognition main network, and determining a distillation loss based on the error, includes:

[0016] Obtaining a first feature of a distillation layer in the skeleton-guided network and a second feature of a corresponding layer in the initial action recognition main network; wherein the corresponding layer is a layer corresponding to the distillation layer; the first feature represents high-level semantic features of key points of a human body in the first sample video clip; and the second feature represents visual features of the first sample video clip;

[0017] The distillation loss is determined based on a mean square error between the first feature and the second feature.

[0018] In one embodiment, the method for determining the distillation loss weight includes:

[0019] When a second prediction result of the first sample video segment is the same as the true label of the first sample video segment, determining the distillation loss weight to be a first target value; wherein the second prediction result is output by the skeleton guidance network according to the first feature of the distillation layer;

[0020] When the second prediction result of the first sample video segment is different from the true label of the first sample video segment, the distillation loss weight is determined to be a second target value.

[0021] In one embodiment, the confidence assessment network is used to process the initial recognition results corresponding to the target video segments to obtain the target recognition results of the target video, including:

[0022] For each of the initial recognition results, performing confidence calculation on the initial recognition result using the confidence evaluation network to obtain each confidence score;

[0023] Normalizing each of the confidence scores to obtain a weight corresponding to each of the initial recognition results;

[0024] Based on the weights and the initial recognition results, a target recognition result of the target video is obtained.

[0025] In one embodiment, the confidence assessment network is trained by:

[0026] Obtaining a training sample, where the training sample includes a sample initial recognition result corresponding to the second sample video clip and a label corresponding to the sample initial recognition result, where the label indicates whether the sample initial recognition result is correct or incorrect;

[0027] The initial confidence assessment network is trained using the training samples to obtain a confidence assessment network.

[0028] In a second aspect, the present application further provides a video action recognition device, the device comprising:

[0029] A video segment acquisition module is used to acquire a target video segment, wherein the target video segment is determined based on the target video segmentation;

[0030] An initial result acquisition module is configured to input each target video clip into a target behavior recognition main network and obtain an initial recognition result output by the target behavior recognition main network, wherein the initial recognition result is used to represent the behavior classification probability vector corresponding to the target video clip; the target behavior recognition main network is determined by distilling the initial behavior recognition main network as a student model through a skeleton-guided network as a teacher model;

[0031] The target result determination module is used to process the initial recognition results corresponding to each target video segment based on the confidence assessment network according to the time sequence of the target video segment to obtain the target recognition result of the target video, wherein the target recognition result is used to characterize the behavior category of the target video.

[0032] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0034] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0035] The above-mentioned video action recognition method, device, computer equipment, computer-readable storage medium and computer program product, on the one hand, input each target video clip into the target behavior recognition main network determined by distilling the initial behavior recognition main network as the student model through the skeleton guidance network as the teacher model, and obtain the initial recognition result output by the target behavior recognition main network; the distilled target behavior recognition main network has the ability to perceive the skeletal features of the teacher model, and effectively copes with problems such as complex background interference and partial body occlusion; on the other hand, since the target video is divided into multiple target video clips, the recognition results of each target video clip may have different distinguishing capabilities, and the traditional average pooling method fails to effectively measure the importance of the target video clip. The confidence weighted aggregation strategy is introduced to assign weights to each target video clip, thereby improving the utilization rate of key clips in the target video, thereby improving the accuracy of the target recognition results of the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 1 is a flow chart of a method for video action recognition according to an embodiment;

[0038] Figure 2 A schematic diagram of a target behavior recognition main network structure in one embodiment;

[0039] Figure 3 Schematic diagram of a process flow for training a target behavior recognition main network in one embodiment;

[0040] Figure 4 FIG1 is a flow chart of obtaining a target recognition result of a target video based on a confidence assessment network in one embodiment;

[0041] Figure 5 is a structural block diagram of a video action recognition device in one embodiment;

[0042] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] In one embodiment, Figure 1 As shown, a video action recognition method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S106. Among them:

[0045] Step S102: obtaining a target video segment, where the target video segment is determined based on the target video segmentation.

[0046] Optionally, the terminal obtains the complete target video and divides it into several non-overlapping or partially overlapping target video segments according to a fixed frame length (for example, 16 frames). Each target video segment undergoes normalization, including scaling (for example, to 112×112 pixels), pixel normalization, and center cropping, to maintain consistent data distribution with the training phase. A target video segment refers to time-series data consisting of multiple consecutive RGB image frames. An RGB image is an image consisting of a pixel matrix of three channels: red (R), green (G), and blue (B).

[0047] In step S104, each target video clip is input into the target behavior recognition main network, and an initial recognition result output by the target behavior recognition main network is obtained.

[0048] Among them, the initial recognition result is used to characterize the behavior classification probability vector corresponding to the target video clip; the target behavior recognition main network is determined by distilling the initial behavior recognition main network as the student model through the skeleton guidance network as the teacher model.

[0049] Optionally, before training the target behavior recognition main network, the terminal trains a skeleton guidance network as a teacher model, and the skeleton guidance network uses human skeleton features as the network input.

[0050] In an optional embodiment, a method for a terminal to train a skeleton-guided network includes: obtaining an initial training sample, the initial training sample being labeled with human skeleton key points; filtering the initial training sample to obtain a second training sample, the second training sample being labeled with human skeleton key points; and training the initial skeleton-guided network based on the second training sample to obtain the skeleton-guided network. The initial training sample is an initial sample video clip, and the second training sample is a second sample video clip.

[0051] Furthermore, the terminal uses tool chains such as MMPose / OpenPose to perform enhanced posture annotation on the original RGB video clips of the original RGB video as the initial training samples. The annotation range covers the main structures of the human body (head, torso, upper and lower limbs) as well as the key points of the face and hands to construct a complete human skeleton label image; the annotated posture data is filtered according to the preset screening rules, and the annotation data with poor quality is eliminated to obtain the second training sample. After the initial training sample is processed, the foreground contrast of the human body structure is significantly improved, and the interference of irrelevant factors such as clothing and background on action modeling is effectively suppressed. The initial skeleton guidance network adopts a feature modeling architecture based on the Spatio-Temporal Graph Convolutional Network (ST-GCN), which models the relationship between human body action and spatial posture structure by constructing a graph structure of human key points in the spatiotemporal dimension; the second sample video clip is used to perform end-to-end training on the ST-GCN model, and the loss function adopts the MSE loss function. The fully trained ST-GCN model can obtain the ability to perform action recognition based on skeleton information. Figure 2 The skeleton-guided network in

[15] .

[0052] In an optional embodiment, the terminal trains the initial behavior recognition main network in a manner including: obtaining a first sample video clip, and using the first sample video clip to train the original behavior recognition main network to obtain the initial behavior recognition main network. The initial behavior recognition main network includes a 3D convolution and Transformer RGB main network, such as Figure 2 The network structure in .

[0053] For example, a target video clip is input into the initial action recognition main network. The data processing process of the initial action recognition main network includes: obtaining the target video clip through the input layer; obtaining a fused spatiotemporal tensor through the feature extraction layer; using the sliding window layer to apply a self-attention mechanism to the fused spatiotemporal tensor within each sliding window to output an enhanced feature map; and outputting the initial recognition result based on the enhanced feature map through the output layer. The feature extraction layer is the feature extraction front end, and the sliding window layer is the feature extraction back end.

[0054] Among them, the fused spatiotemporal tensor is obtained through the feature extraction layer, including: performing convolution processing on the target video segment through the convolution layer to obtain a spatiotemporal feature map; dividing the spatiotemporal feature map into a target number of feature segments in the time dimension through the segmentation and splicing layer; and splicing the target number of feature segments in the spatial dimension to obtain a fused spatiotemporal tensor.

[0055] Furthermore, the feature extraction front-end processing includes: the terminal performs 3D convolution on the target video segment to capture low-level dynamic information between consecutive frames. Two 3D convolution operations are used, with convolution kernels of 7 and 3 time dimensions, respectively, to form a temporal window with a receptive field of 11 frames. Two-stage 3D convolution is used to effectively capture short-range and long-range temporal dependencies. The output spatiotemporal feature map is divided into four feature segments according to the time dimension using formula (1).

[0056] Formula (1)

[0057] Where fi represents the segmented features after time division; H represents the original height of the input video frame; W represents the original width of the input video frame; D represents the channel dimension; represents the set of real numbers.

[0058] The terminal concatenates the four feature segments in the spatial dimension to obtain a fused spatiotemporal tensor, as shown in formula (2).

[0059] Formula (2)

[0060] Where, f W Represents the fused space-time tensor.

[0061] Through the processing of the feature extraction front end, that is, through spatiotemporal decomposition and reorganization, the computational complexity is significantly reduced while maintaining the feature expression capability.

[0062] The feature extraction backend includes: The terminal uses a SwinTransformer structure based on a sliding window (Shifted Window) to divide the fused spatiotemporal tensor into non-overlapping local windows; self-attention is only calculated within each window. This significantly reduces computational complexity. For each window, multi-head self-attention calculation is performed within the window, as shown in Equation (3).

[0063] Formula (3)

[0064] Where, are query, key, and value matrices respectively, is the channel dimension, is the position encoding bias, is the window size.

[0065] While simple window division reduces computational complexity, it also disrupts the flow of information between windows. Therefore, the terminal implements cross-region information exchange between windows in successive Transformer blocks, generating recombined feature maps. These recombined feature maps are used to model local-global long-range dependencies in the spatial dimension. The feature vector at each position incorporates rich local and global spatial context, enabling the model to demonstrate strong pose structure modeling capabilities.

[0066] The terminal will use the skeleton guidance network of the teacher model to perform distillation training on the initial behavior recognition main network of the student model to obtain the distilled target behavior recognition main network.

[0067] It should be noted that the network structure of the target behavior recognition main network is exactly the same as that of the initial behavior recognition main network.

[0068] Each processed segment is fed into the main target action recognition network as an independent target video segment. The network extracts features from the target video segment through a feature extraction front-end and back-end. The output layer determines the initial recognition result based on the probability distribution of the extracted features. The distilled main target action recognition network is capable of perceiving skeletal features, effectively addressing complex background interference and partial body occlusion.

[0069] Step S106 : processing the initial recognition results corresponding to the target video segments based on the confidence evaluation network according to the time sequence of the target video segments to obtain the target recognition results of the target video.

[0070] Among them, the target recognition result is used to characterize the behavior category of the target video.

[0071] In practical applications, since the entire video is divided into multiple segments, the recognition results of each segment may have different distinguishing capabilities. The traditional average pooling method fails to effectively measure the importance of the segment. Therefore, this paper introduces a confidence-weighted aggregation strategy and introduces a segment confidence network as shown in Formula (4).

[0072] Formula (4)

[0073] Where c i represents the target video clip, f(c i ) represents the initial recognition result output by the target behavior recognition main network for the target video clip, h() is the confidence evaluation network; s i Represents the confidence score output by the confidence assessment network.

[0074] Optionally, the terminal processes the initial recognition results corresponding to each target video segment based on the confidence assessment network according to the time sequence of the target video segment in the target video, obtains the weight of each target video segment, and outputs the category corresponding to the maximum probability as the target recognition result.

[0075] In the above-mentioned video action recognition method, on the one hand, each target video clip is input into a target action recognition main network, which is determined by distilling the initial action recognition main network (student model) through a skeleton-guided network (teacher model), and obtaining the initial recognition results output by the target action recognition main network. The distilled target action recognition main network has the teacher model's ability to perceive skeletal features, effectively dealing with problems such as complex background interference and partial body occlusion. On the other hand, because the target video is divided into multiple target video segments, the recognition results of each target video segment may have different distinguishing capabilities. Traditional average pooling methods fail to effectively measure the importance of target video segments. In this way, a confidence-weighted aggregation strategy is introduced to assign weights to each target video segment, thereby improving the utilization of key segments in the target video and thus the accuracy of target recognition results. Because the student model has the teacher model's ability to perceive skeletal features, the student model maintains low computational overhead and saves computer resources.

[0076] In an exemplary embodiment, Figure 3 As shown, the training process of the target behavior recognition main network includes the following steps S302 to S310.

[0077] Step S302: Obtain a first sample video clip and a true label of the first sample video clip.

[0078] The first sample video segment is determined based on the first sample video division.

[0079] Optionally, the terminal obtains each first sample video segment and a true label of each first sample video segment.

[0080] Step S304: input the first sample video clip into the initial behavior recognition main network to obtain a first prediction result output by the initial behavior recognition main network; and determine the initial network loss based on the true label and the first prediction result.

[0081] Optionally, the terminal inputs each first sample video clip as an independent sample into the initial behavior recognition main network, and obtains the corresponding first prediction result output by the initial behavior recognition main network based on each first sample video clip. The terminal determines the initial network loss based on the true label and the first prediction result, which is recorded as .

[0082] Step S306: Obtain the error between the distillation layer of the skeleton guidance network and the corresponding layer of the initial behavior recognition main network, and determine the distillation loss based on the error.

[0083] Optionally, the terminal obtains the error between the distillation layer of the skeleton guidance network and the corresponding layer of the initial behavior recognition main network, and determines the distillation loss based on the error, which is recorded as . The error can be the mean square error.

[0084] Step S308: determining the total loss based on the initial network loss, the distillation loss, and the distillation loss weight.

[0085] Among them, the distillation loss weight is recorded as , characterizing the selection of the distillation path. Considering that the skeleton-guided network may make incorrect predictions for certain complex samples, directly uniformly distilling all segments may introduce noise. Therefore, the distillation loss weight represents a conditionally activated distillation path control mechanism.

[0086] Optionally, the terminal calculates the initial network loss according to formula (5). , distillation loss and the distillation loss weight to determine the total loss, denoted as .

[0087] Formula (5)

[0088] Where, is an adjustable hyperparameter.

[0089] Step S310: Adjust the initial behavior recognition main network according to the total loss to obtain the target behavior recognition main network.

[0090] Optionally, the terminal is based on the total loss The initial behavior recognition main network is adjusted to obtain the target behavior recognition main network.

[0091] In this embodiment, by introducing full-body posture annotation data covering key areas such as the face, hands, and torso, a skeleton-guided network with significant posture understanding capabilities is trained. This network is used as a teacher model to transfer high-level action semantic representations to a student network that only accepts RGB inputs, and implicit injection of posture perception capabilities is achieved in the form of a total loss function. Through the total loss function, knowledge transfer is achieved.

[0092] In an exemplary embodiment, the error between the distillation layer of the skeleton-guided network and the corresponding layer of the initial behavior recognition main network is obtained, and the distillation loss is determined based on the error, including: obtaining a first feature of the distillation layer in the skeleton-guided network and a second feature of the corresponding layer in the initial behavior recognition main network; and determining the distillation loss based on the mean square error between the first feature and the second feature.

[0093] Among them, the corresponding layer is a layer corresponding to the distillation layer; the first feature represents the high-level semantic features of the key points of the human body in the first sample video clip; and the second feature represents the visual features of the first sample video clip.

[0094] Optionally, in order to transfer the posture perception capability of the skeleton-guided network to the recognition network that only takes RGB as input, the “Feature-Level Knowledge Distillation” mechanism is used to establish a cross-modal high-level semantic representation alignment. Therefore, the terminal obtains the first feature of the distillation layer in the skeleton-guided network and records it as , and the second feature of the corresponding layer in the initial behavior recognition main network is recorded as ; Based on formula (6), the mean square error between the first feature and the second feature is calculated and the distillation loss is determined as . During training The minimum is the goal.

[0095] Formula (6)

[0096] Where M is the number of training samples. i represents the index of the target video segment; Represents the square of the L2 norm.

[0097] In this embodiment, by aligning high-level semantic representations across modalities, a distillation loss can be determined to force the student model to imitate the intermediate-level feature representations of the teacher model.

[0098] In an exemplary embodiment, a method for determining the distillation loss weight includes: determining the distillation loss weight as a first target value when the second prediction result of the first sample video clip is the same as the true label of the first sample video clip; wherein the second prediction result is output by the skeleton guidance network according to the first feature of the distillation layer; and determining the distillation loss weight as a second target value when the second prediction result of the first sample video clip is different from the true label of the first sample video clip.

[0099] The first target value is greater than the second target value, for example, the first target value is 1 and the second target value is 0. The distillation loss weight is used to control the strength of teacher knowledge transfer.

[0100] The second prediction result is the output of the skeleton guidance network based on the first feature of the distillation layer, which is the category with the highest prediction probability.

[0101] Considering that the skeleton-guided network may make mistakes in predicting some complex samples, directly uniformly distilling all the clips may introduce noise, that is, propagating wrong knowledge. For this reason, the distillation loss weight represents a conditionally activated distillation path control mechanism. When the second prediction result of the first sample video clip is the same as the true label of the first sample video clip, it indicates that the teacher model prediction is correct, that is, the teacher model is reliable; the distillation loss weight is determined to be the first target value, such as 1, to open the knowledge transfer channel, and the student model learns high-level feature representations from the teacher model. At this time, the total loss contains .

[0102] Otherwise, that is, when the second prediction result of the first sample video clip is different from the true label of the first sample video clip, it indicates that the teacher model prediction is wrong, that is, the teacher model is unreliable; the distillation loss weight is determined to be the second target value, such as 0, to close the knowledge transfer channel, and the student model only learns from the true label. At this time, the total loss does not include . As shown in formula (7).

[0103] Formula (7)

[0104] Where, Represents the distillation loss weight; is the true label of the fragment; Represents the skeleton-guided network (teacher model) on the fragment prediction; argmax represents the category with the largest predicted probability.

[0105] In this embodiment, through adaptive knowledge transfer, distillation is fully applied to simple samples and disabled for complex samples to avoid misleading information. The intensity of teacher knowledge transfer can be adaptively controlled.

[0106] In an exemplary embodiment, Figure 4 As shown, the initial recognition results corresponding to each target video segment are processed based on the confidence evaluation network to obtain the target recognition result of the target video, including steps S402 to S406.

[0107] Step S402 : For each initial recognition result, the confidence of the initial recognition result is calculated by a confidence evaluation network to obtain each confidence score.

[0108] Optionally, the terminal processes each initial recognition result through a confidence network as shown in formula (4) and outputs a confidence score. This confidence score represents the probability that the initial recognition result is correct when the target behavior recognition main network outputs it.

[0109] Formula (4)

[0110] Where c i represents the target video clip, f(c i ) represents the initial recognition result output by the target behavior recognition main network for the target video clip, h() is the confidence evaluation network; s i Represents the confidence score output by the confidence assessment network.

[0111] Step S404: normalize each confidence score to obtain a weight corresponding to each initial recognition result.

[0112] Optionally, to enhance the influence of high-confidence segments, the terminal uses a quadratic normalization function to stretch the segment scores, as shown in formula (8), to obtain the weight corresponding to the initial recognition result: The confidence scores of different videos are mapped to a unified interval [0, 1]. This eliminates differences in confidence scales between target video segments, making the weights comparable. A quadratic operation implements nonlinear amplification, exponentially increasing the weights of high-confidence segments and exponentially decreasing the weights of low-confidence segments, highlighting the role of key target video segments.

[0113] Formula (8)

[0114] Where, Representative The weight of the confidence score, N represents the number of target video clips, s i Represents the confidence score output by the confidence assessment network, min represents the minimum value, and max represents the maximum value.

[0115] Step S406: Obtain a target recognition result of the target video based on the weights and the initial recognition results.

[0116] Optionally, the terminal obtains a target recognition result of the target video based on each weight and each initial recognition result through formula (9).

[0117] Formula (9)

[0118] Where, Represents the target recognition result.

[0119] In this example, traditional average pooling assumes that all segments are of equal quality and importance, which is not realistic. By dynamically adjusting the importance of different time segments through confidence, the model focuses more on reliably classified video segments, significantly improving the classification accuracy of long videos.

[0120] In an exemplary embodiment, a confidence assessment network training method includes: obtaining a training sample, the training sample including a sample initial recognition result corresponding to a second sample video clip and a label corresponding to the sample initial recognition result, the label indicating whether the sample initial recognition result is correct or incorrect; training an initial confidence assessment network using the training sample to obtain a confidence assessment network.

[0121] Optionally, the terminal obtains a training sample, the training sample including a sample initial recognition result corresponding to the second sample video clip and a label corresponding to the sample initial recognition result, the label indicating whether the sample initial recognition result is correct or incorrect, such as ,in, It is an action recognition model for video clips The output sample initial recognition result; wherein, the behavior recognition model may include the initial behavior recognition main network, the target behavior recognition main network or other classification models. is the label corresponding to the initial recognition result of the sample, which indicates whether the initial recognition result of the sample is correct or incorrect. M represents the number of training samples. Calculated by formula (10). =1, it indicates that the classification is correct. =0, it indicates misclassification.

[0122] Formula (10)

[0123] Where, is the true value corresponding to the second sample video clip.

[0124] The terminal trains the initial confidence assessment network using the above training samples and the MSE loss function to obtain a confidence assessment network.

[0125] In this embodiment, a mapping relationship between prediction and authenticity can be established through training samples, so that the confidence network can learn to judge whether the prediction of the classifier is credible.

[0126] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0127] Based on the same inventive concept, the present application also provides a video action recognition device for implementing the aforementioned video action recognition method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following video action recognition device embodiments can be found in the aforementioned limitations on the video action recognition method, and will not be further elaborated here.

[0128] In an exemplary embodiment, Figure 5 As shown, a video action recognition device is provided, comprising: a video clip acquisition module 501, an initial result acquisition module 502 and a target result determination module 503, wherein:

[0129] The video segment acquisition module 501 is used to acquire a target video segment. The target video segment is determined based on the target video segmentation.

[0130] The initial result acquisition module 502 is used to input each target video clip into the target behavior recognition main network and obtain the initial recognition result output by the target behavior recognition main network, wherein the initial recognition result is used to characterize the behavior classification probability vector corresponding to the target video clip; the target behavior recognition main network is determined by distilling the initial behavior recognition main network as the student model through the skeleton guidance network as the teacher model.

[0131] The target result determination module 503 is used to process the initial recognition results corresponding to each target video segment based on the confidence evaluation network according to the time sequence of the target video segment to obtain the target recognition result of the target video, wherein the target recognition result is used to characterize the behavior category of the target video.

[0132] In an exemplary embodiment, a video action recognition device further includes:

[0133] The first training module is used to obtain a first sample video clip and a true label of the first sample video clip; the first sample video clip is determined based on the first sample video division; the first sample video clip is input into the initial behavior recognition main network to obtain a first prediction result output by the initial behavior recognition main network; the initial network loss is determined based on the true label and the first prediction result; the error between the distillation layer of the skeleton-guided network and the corresponding layer of the initial behavior recognition main network is obtained, and the distillation loss is determined based on the error; the total loss is determined based on the initial network loss, the distillation loss and the distillation loss weight; the initial behavior recognition main network is adjusted based on the total loss to obtain the target behavior recognition main network.

[0134] In an exemplary embodiment, the first training module is also used to obtain the first feature of the distillation layer in the skeleton-guided network and the second feature of the corresponding layer in the initial behavior recognition main network; wherein the corresponding layer is the layer corresponding to the distillation layer; the first feature represents the high-level semantic features of the key points of the human body in the first sample video clip; the second feature represents the visual features of the first sample video clip; and the distillation loss is determined based on the mean square error between the first feature and the second feature.

[0135] In an exemplary embodiment, the first training module is further used to determine the distillation loss weight as a first target value when the second prediction result of the first sample video clip is the same as the true label of the first sample video clip; wherein the second prediction result is output by the skeleton guidance network according to the first feature of the distillation layer; and when the second prediction result of the first sample video clip is different from the true label of the first sample video clip, determine the distillation loss weight as a second target value.

[0136] In an exemplary embodiment, the target result determination module 503 is also used to calculate the confidence of each initial recognition result through a confidence evaluation network to obtain each confidence score; normalize each confidence score to obtain the weight corresponding to each initial recognition result; and obtain the target recognition result of the target video based on each weight and each initial recognition result.

[0137] In an exemplary embodiment, a video action recognition device further includes: a second training module, used to obtain training samples, the training samples including a sample initial recognition result corresponding to a second sample video clip and a label corresponding to the sample initial recognition result, the label indicating whether the sample initial recognition result is correct or incorrect; and training an initial confidence assessment network through the training samples to obtain a confidence assessment network.

[0138] Each module in the video action recognition device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store video clip data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video action recognition method is implemented.

[0140] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0141] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0143] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0144] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0145] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0146] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A video action recognition method, characterized in that: The method comprises: Obtaining a target video segment, wherein the target video segment is determined based on the target video segmentation; Inputting each target video clip into a target behavior recognition main network and obtaining an initial recognition result output by the target behavior recognition main network, wherein the initial recognition result is used to represent the behavior classification probability vector corresponding to the target video clip; the target behavior recognition main network is determined by distilling the initial behavior recognition main network as a student model through a skeleton-guided network as a teacher model; According to the time sequence of the target video segments, the initial recognition results corresponding to each target video segment are processed based on the confidence evaluation network to obtain a target recognition result of the target video, wherein the target recognition result is used to characterize the behavior category of the target video.

2. The method according to claim 1, characterized in that The training process of the target behavior recognition main network includes: Obtaining a first sample video segment and a true label of the first sample video segment; the first sample video segment is determined based on the first sample video segmentation; Inputting the first sample video clip into the initial action recognition main network to obtain a first prediction result output by the initial action recognition main network; determining the initial network loss based on the true label and the first prediction result; Obtaining an error between a distillation layer of the skeleton-guided network and a corresponding layer of the initial behavior recognition main network, and determining a distillation loss based on the error; Determining a total loss based on the initial network loss, the distillation loss, and a distillation loss weight; The initial behavior recognition main network is adjusted according to the total loss to obtain the target behavior recognition main network.

3. The method according to claim 2, characterized in that Obtaining an error between a distillation layer of the skeleton guidance network and a corresponding layer of the initial behavior recognition main network, and determining a distillation loss according to the error, includes: Obtaining a first feature of a distillation layer in the skeleton-guided network and a second feature of a corresponding layer in the initial action recognition main network; wherein the corresponding layer is a layer corresponding to the distillation layer; the first feature represents high-level semantic features of key points of a human body in the first sample video clip; and the second feature represents visual features of the first sample video clip; The distillation loss is determined based on a mean square error between the first feature and the second feature.

4. The method according to claim 3, characterized in that The method for determining the distillation loss weight includes: When a second prediction result of the first sample video segment is the same as the true label of the first sample video segment, determining the distillation loss weight to be a first target value; wherein the second prediction result is output by the skeleton guidance network according to the first feature of the distillation layer; When the second prediction result of the first sample video segment is different from the true label of the first sample video segment, the distillation loss weight is determined to be a second target value.

5. The method according to claim 1, wherein The confidence evaluation network is used to process the initial recognition results corresponding to the target video segments to obtain target recognition results of the target video, including: For each of the initial recognition results, performing confidence calculation on the initial recognition result using the confidence evaluation network to obtain each confidence score; Normalizing each of the confidence scores to obtain a weight corresponding to each of the initial recognition results; Based on the weights and the initial recognition results, a target recognition result of the target video is obtained.

6. The method according to claim 1, characterized in that The training method of the confidence assessment network includes: Obtaining a training sample, where the training sample includes a sample initial recognition result corresponding to the second sample video clip and a label corresponding to the sample initial recognition result, where the label indicates whether the sample initial recognition result is correct or incorrect; The initial confidence assessment network is trained using the training samples to obtain a confidence assessment network.

7. A video action recognition device, characterized in that: The device comprises: A video segment acquisition module is used to acquire a target video segment, wherein the target video segment is determined based on the target video segmentation; An initial result acquisition module is configured to input each target video clip into a target behavior recognition main network and obtain an initial recognition result output by the target behavior recognition main network, wherein the initial recognition result is used to represent the behavior classification probability vector corresponding to the target video clip; the target behavior recognition main network is determined by distilling the initial behavior recognition main network as a student model through a skeleton-guided network as a teacher model; The target result determination module is used to process the initial recognition results corresponding to each target video segment based on the confidence assessment network according to the time sequence of the target video segment to obtain the target recognition result of the target video, wherein the target recognition result is used to characterize the behavior category of the target video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images