An abnormal action positioning method, device and storage medium for network live video

Abstract

Embodiments of the present application provide an abnormal action positioning method, device and storage medium for network live video, belonging to the technical field of video. The abnormal action positioning method comprises: performing feature extraction on the network live video based on a multi-fiber network to generate a segment-level video feature sequence of the network live video; processing the segment-level video features to output a category activation sequence and an attention branch for anchor actions in the network live video; and calculating a weighted result of the category activation sequence and the attention branch based on a multi-instance learning method to generate a timing abnormal action nomination for the anchor actions, wherein the timing abnormal action nomination is used to indicate an abnormal action positioning result for the anchor actions of the network live video. Embodiments of the present application do not require a large amount of video labeling, adopt a weakly supervised learning method, reduce data labeling cost and time, meet the real-time positioning demand for timing abnormal actions in the video, and realize rapid positioning of abnormal actions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video technology, in particular to a method and device for locating abnormal action in a network live video and a storage medium. BACKGROUND

[0002] At present, network live video is becoming a more and more popular live video mode, which refers to real-time live video by using Internet technology. Compared with traditional television live video, network live video has the characteristic properties of network video and television live video, for example, self-media, which actually provides an opportunity for the traditional "audience" to perform.

[0003] Generally, for ordinary network video, manual review is needed before playing to the public, so as to effectively monitor ordinary network video. However, compared with ordinary network video, network live video has the characteristic of real-time playing, and it is difficult to use manual pre-audit method, which causes great difficulty in network monitoring of this kind of video. In view of this situation, there are currently two methods:

[0004] One is to use intelligent analysis method to identify the anchor action in network anchor video. However, since the anchor action is a subjective behavior that is not easy to predict, for example, the anchor has no action for a period of time, at this time, action recognition cannot be used for effective monitoring. Or when a sudden abnormal event occurs, action recognition cannot be found and blocked immediately, resulting in more serious adverse consequences.

[0005] The other is to use deep learning technology to identify illegal content in live video. However, in actual application, in order to avoid examination, some illegal anchors will change the video after being banned by the monitoring agency and appear in a new form, so that the performance form of anchor illegal activity in network live video is various and changes quickly. Since deep learning technology relies on a large number of labeled data to produce excellent feature representation ability, it is difficult to collect a sufficient number of illegal videos with complete labels in a short time for network live video, so that this method cannot meet the demand of timely blocking of illegal videos.

[0006] Therefore, there is an urgent need for a method for locating abnormal action of anchor in network live video to provide effective monitoring means and technical support for network monitoring departments. SUMMARY

[0007] The purpose of the embodiments of the present application is to provide a method and device for locating abnormal action in network live video and a storage medium, which at least partially solve the above-mentioned technical problems.

[0008] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for locating abnormal actions in live-streamed videos. This method includes: extracting features from the live-streamed video using a multi-fiber network to generate a segment-level video feature sequence; processing the segment-level video features to output a category activation sequence and an attention branch for the broadcaster's actions in the live-streamed video; and calculating a weighted result of the category activation sequence and the attention branch based on a multi-instance learning method to generate a temporal abnormal action nomination for the broadcaster's actions, wherein the temporal abnormal action nomination is used to indicate the abnormal action location result for the broadcaster's actions in the live-streamed video.

[0009] Optionally, before performing feature extraction on the live web video based on the multi-fiber network, the abnormal action localization method further includes pre-training the multi-fiber network in the following manner: using cropped live web videos from the dataset, and removing the background of the initial videos in the dataset to generate diverse video samples after background removal; and performing data augmentation on the initial videos and the diverse video samples after background removal, inputting them into the initial multi-fiber network to extract features, then feeding them into a classifier, and training the video action recognition task based on metric learning loss and cross-entropy loss to obtain a feature extraction backbone network that is sensitive to the actions of live web videos.

[0010] Optionally, background removal is performed on the videos in the dataset to generate diverse video samples after background removal, including: randomly sampling one frame of the initial video in the dataset to obtain the corresponding sampled frame image; weighting the initial video and the sampled frame image and fusing them into each frame of the initial video to obtain the diverse video samples after background removal.

[0011] Optionally, the segment-level video feature sequence is processed to output a corresponding category activation sequence, including: embedding the segment-level video feature sequence to obtain an embedded segment-level video feature sequence; classifying the embedded segment-level video feature sequence using a segment-level classifier to obtain a category activation sequence; and weighting the category activation sequence using an attention branch to output a weighted category activation sequence.

[0012] Optionally, the generation of temporal anomalous action nominations is based on multi-instance learning to calculate category activation sequences, including: calculating category activation sequences based on multi-instance learning, obtaining confidence scores for each category of the video under test, and comparing the confidence scores with preset category confidence thresholds to filter candidate sequences for the temporal anomalous action nominations; calculating the confidence scores of the candidate sequences using an internal-external comparison function; and determining candidate bounding boxes for anomalous action localization based on the confidence scores of the candidate sequences, the anomalous action categories, and the start and end times.

[0013] Optionally, candidate boxes are screened based on nonmaximum suppression to determine the temporal anomalous action nomination, including: sorting the confidence scores of the candidate boxes, calculating the intersection-union ratio (IU) of the candidate boxes in the video dimension; deleting candidate boxes whose IU with high-confidence candidate boxes is greater than a preset threshold in the video dimension, and using the remaining candidate boxes as the temporal anomalous action localization results.

[0014] Optionally, the abnormal action localization method further includes: constructing a loss function optimization model to optimize the localization performance of abnormal actions in the live video.

[0015] Secondly, embodiments of the present invention provide an abnormal action localization device, the abnormal action localization device comprising: a memory storing a program capable of running on a processor; and the processor configured to implement the abnormal action localization method for live online video described in the first aspect when executing the program.

[0016] Thirdly, embodiments of the present invention provide a machine-readable storage medium storing instructions that cause a machine to perform the abnormal action localization for live online video described in the first aspect of this application.

[0017] Through the above technical solution, this invention adopts a weakly supervised learning approach. Compared to traditional fully supervised methods, which require extensive and costly annotation of video samples before training the neural network for abnormal action localization, this application eliminates the need for extensive annotation, reduces the cost of learning new abnormal action categories, and is easier to deploy. Furthermore, it abandons the optical flow modality in traditional frameworks, requiring only input of the live video stream for segment-level video feature extraction using MF-Net. Then, by processing these segment-level video features and employing a multi-instance learning approach for abnormal action nomination, it effectively reduces inference time, meets the real-time requirements for anchor's temporal abnormal action localization, and achieves rapid abnormal action localization in live video streams.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0020] Figure 1 This is a schematic flowchart illustrating an abnormal action localization method according to an exemplary embodiment;

[0021] Figure 2 This is a schematic diagram illustrating a pre-trained MF-Net network process according to an exemplary embodiment;

[0022] Figure 3 This is a schematic diagram illustrating a process for generating diverse video samples based on background erasure, according to an exemplary embodiment.

[0023] Figure 4 This is a schematic diagram illustrating an abnormal action localization process according to an exemplary embodiment;

[0024] Figure 5 This is a schematic diagram illustrating a process for generating diverse video samples based on background erasure, according to an exemplary embodiment.

[0025] Figure 6 This is a schematic diagram illustrating a method for generating category activation sequences according to an exemplary embodiment;

[0026] Figure 7 This is a schematic flowchart illustrating an abnormal action nomination method according to an exemplary embodiment;

[0027] Figure 8 This is a flowchart illustrating a method for determining candidate bounding boxes for abnormal action localization according to an exemplary embodiment;

[0028] Figure 9 This is a schematic flowchart illustrating a method for generating abnormal action localization results according to an exemplary embodiment; and

[0029] Figures 10A to 10C This is a schematic diagram illustrating the result of abnormal action localization according to an exemplary embodiment. Detailed Implementation

[0030] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0031] Example 1

[0032] This invention provides an abnormal action localization method that employs weakly supervised learning, eliminating the need for extensive annotation of video samples. Figure 1 As shown, the abnormal action localization method includes:

[0033] Step S110: Extract features from the live web video based on a multi-fiber network to generate a segment-level video feature sequence of the live web video.

[0034] Specifically, Multi-Fiber Networks (MF-Nets) can break down complex networks into lightweight network ensembles, effectively recognizing specific categories of objects or backgrounds, especially target actions. Therefore, this embodiment of the invention uses MF-Net to extract features from live web videos, making the generated segment-level video features more motion-sensitive.

[0035] Specifically, in this embodiment of the invention, the video is divided into segments of 16 frames each. If the last segment has fewer than 16 frames, the last frame is added to make the segment 16 frames. Further, after segmenting, MF-Net is used to extract segment-level features for each segment, and global average pooling is used to compress its spatial and temporal dimensions. Finally, the segment-level features extracted from each video segment are concatenated along the segment dimensions to obtain the corresponding feature sequence. This feature sequence is the segment-level video feature for the live video stream.

[0036] Step S120: Process the segment-level video features and output the category activation sequence and attention branch for the anchor's actions in the live video.

[0037] Specifically, after extracting the segment-level video features of the live stream, the extracted features need to be classified and weighted by attention. On the one hand, a classifier is used to classify the features related to the streamer's actions in the extracted segment-level video features, resulting in a category activation sequence. On the other hand, attention weighting is used to further measure the probability that the segment-level video features contain actions, resulting in a weighted category activation sequence.

[0038] Step S130: Calculate the weighted result of the category activation sequence and the attention branch based on the multi-instance learning method to generate a temporal abnormal action nomination for the anchor's actions, wherein the temporal abnormal action nomination is used to indicate the abnormal action localization result for the anchor's actions in the live video.

[0039] Specifically, a weighted category activation sequence is calculated based on a multi-instance learning approach to obtain the confidence score of each category in the video under test. The confidence score is then compared with a preset category confidence threshold to filter candidate sequences for the nomination of the temporal abnormal action. An internal-external comparison function is used to calculate the confidence score of the candidate sequences. Based on the confidence score of the candidate sequences, the abnormal action category, and the start and end times, candidate boxes for abnormal action localization are determined. The confidence scores of the candidate boxes are sorted, and the intersection-union ratio (IUR) of each candidate box in the video dimension is calculated. Candidate boxes with an IUR greater than a preset threshold with high confidence scores are deleted, and the remaining candidate boxes are used as the temporal abnormal action localization results.

[0040] As can be seen from the above embodiments, this invention adopts a weakly supervised learning approach. Compared with traditional fully supervised methods, which require extensive and costly annotation of video samples before training the neural network for abnormal action localization, this application eliminates the need for extensive annotation and reduces the cost of learning new abnormal action categories. Furthermore, it abandons the optical flow modality in traditional frameworks, requiring only input of the live video stream for segment-level video feature extraction using MF-Net. Then, by processing these segment-level video features and employing a multi-instance learning approach for abnormal action nomination, it effectively reduces inference time, meets the real-time requirements for anchor's temporal abnormal action localization, and achieves rapid abnormal action localization in live video streams.

[0041] Example 2

[0042] To enable MF-Net to better handle temporal anomaly localization tasks, it needs to be pre-trained to improve the network's accuracy and sensitivity in action localization. The following section provides a detailed explanation of the MF-Net pre-training process.

[0043] In a preferred embodiment, such as Figure 2 As shown, the pre-training of the multi-fiber network includes the following steps:

[0044] Step S210: Use the cropped live domain videos from the dataset and perform background erasure on the initial videos in the dataset to generate diverse video samples after background erasure.

[0045] Specifically, the dataset used in this embodiment of the invention is the BJUT-SAD-DA dataset. By selecting 2619 cropped videos from the live domain in the BJUT-SAD-DA dataset, and dividing the video samples in the dataset into a test set and a training set, MF-Net is trained.

[0046] Preferred, such as Figure 3As shown, the step of removing the background from the videos in the dataset to generate diverse video samples after background removal includes:

[0047] Step S310: Randomly sample one frame from the initial video in the dataset to obtain the corresponding sampled frame image.

[0048] Step S320: The initial video and the sampled frame images are weighted and fused into each frame of the initial video to obtain the diverse video samples after background erasure.

[0049] Specifically, such as Figure 4 As shown, before inputting the videos from the dataset into the MF-Net network for pre-training, the videos first need to undergo background erasure to generate diverse video samples, thereby increasing the video's focus on motion information. Figure 5 As shown, first along the time dimension T For the input video Randomly sample a frame to obtain the frame image. :

[0050] (1)

[0051] Secondly, based on the random generation of weight coefficients after sampling For live video V and frame image V t Weighting:

[0052] (2)

[0053] Finally, the weighted frame images are fused into each frame of the live video to obtain diverse video samples after background erasure. :

[0054] (3)

[0055] Step S220: The initial video and the diverse video samples after background erasure are augmented with data and input into the initial multifiber network to extract features. Then, the network is fed into the classifier and trained on video action recognition task based on metric learning loss and cross-entropy loss to obtain a feature extraction backbone network that is more sensitive to the actions in live online videos.

[0056] Specifically, during the pre-training phase, the images of each frame of the video are first processed by scaling and cropping. For example, each frame is scaled to... Size, then randomly crop to During sampling, similar to the TSN method, the video is randomly divided into 16 segments, with one frame randomly sampled from each segment and spliced ​​along the video dimension.

[0057] Furthermore, to ensure the richness of the training data, data augmentation can be performed on the input videos and diverse video samples. For example, video sample V and diverse video samples can be combined. Data augmentation strategies including random cropping, color jittering, horizontal flipping, and downsampling were implemented to reduce feature similarity, resulting in augmented sample V1 and V2. :

[0058] (4)

[0059] (5)

[0060] Furthermore, the enhanced video sample V1 and The data is input into the MF-Net network for training, fragment-level features are extracted, and then their temporal and spatial dimensions are compressed using global average pooling to obtain the corresponding feature representations F and F. :

[0061] (6)

[0062] (7)

[0063] As can be seen, by inputting the diverse videos generated after erasing the background and the video itself into the MF-Net network for training and generating feature representations, the MF-Net network is able to extract fragment-level features with sufficient motion sensitivity.

[0064] Furthermore, the feature representation F obtained in the above implementation and After concatenation along the batch dimension, the data is fed into a classifier consisting of fully connected layers, and the prediction result is normalized using the pred algorithm.

[0065] Softmax (8)

[0066] The cross-entropy loss Lce is further calculated using the following formula:

[0067] (9)

[0068] Where class_num represents the total number of categories. j These represent the channels output by the classifier, and their total number is the same as the number of categories. i Represents the correct category. y i 'pred' represents the true label, and 'pred' represents the predicted result.

[0069] By optimizing the cross-entropy loss, MF-Net can be made to have a strong discriminative ability for the input video features.

[0070] Based on the video feature representation and the diversified video feature representation after background erasure, a metric learning loss is obtained.

[0071] Specifically, the metric learning loss L m It was calculated in the following way:

[0072] (10)

[0073] in, N ij This represents the number of positive sample pairs. s ij This represents the similarity between positive samples. s ik This represents the similarity between negative samples.

[0074] The similarity calculation in the above formula (10) is implemented using cosine similarity. Taking positive samples as an example, the calculation method is as follows:

[0075] (11)

[0076] Optimizing the metric learning loss can increase the similarity of feature representations from the same category of video features, while decreasing the similarity between feature representations from different categories of video features.

[0077] By combining the cross-entropy loss and the metric learning loss, the final loss L for the pre-training stage of the multi-fiber network is obtained, thereby optimizing the multi-fiber network.

[0078] Specifically, the total loss during the pre-training phase is calculated using the following formula:

[0079] (12)

[0080] in, As hyperparameters, in the embodiments of the present invention Set it to 0.2.

[0081] The pre-training method employed in this embodiment of the invention enables the MF-Net network to extract action-sensitive features, and significantly reduces the memory requirements compared to the end-to-end approach.

[0082] Example 3

[0083] This embodiment of the invention provides a further detailed description of the generation of category activation sequences in Embodiment 1. For example... Figure 6 As shown, based on the segment-level video features, the corresponding category activation sequences are obtained, including:

[0084] Step S410: Perform feature embedding on the segment-level video feature sequence to obtain the embedded segment-level video feature sequence.

[0085] Specifically, such as Figure 7 As shown, after extracting motion-sensitive feature representations F using the pre-trained MF-Net network, further feature embedding of the fragment-level features is required. Specifically, this is achieved using Conv... embed The embedding layer, constructed with the LeakyReLU activation function, is further optimized to extract the embedded segment-level video features F. embed :

[0086] (13)

[0087] Step S420: Use a segment-level classifier to classify the embedded segment-level video feature sequence to obtain the category activation sequence and attention branch.

[0088] Specifically, such as Figure 7 As shown, a fragment-level feature classifier is constructed to classify each fragment-level feature. In order to facilitate subsequent optimization, this embodiment of the invention adds a prediction output to the classifier used for foreground category prediction, so as to predict whether the fragment-level feature belongs to the background feature. That is, this embodiment of the invention treats the background fragment as a category.

[0089] Furthermore, by using a segment-level feature classifier composed of fully connected layers to classify each segment, the class activation sequence P is obtained:

[0090] (14)

[0091] Step S430: The category activation sequence is weighted using attention branches, and the weighted category activation sequence is output.

[0092] Specifically, such as Figure 7 As shown, to better locate abnormal actions, this embodiment of the invention introduces an attention branch to assist in distinguishing whether a segment contains an action. Specifically, the attention branch in this embodiment of the invention uses a convolutional layer. atn Implemented with the Softmax function, where the convolutional layer is responsible for dividing the segment into three sets, and the Softmax function is used to normalize the attention, making the attention represent the model's prediction confidence that a segment belongs to each set:

[0093] (15)

[0094] Among them, attention Foreground attention A ins Action context attention Acon and background attention A bak It is constructed by splicing.

[0095] We use attention to weight the category activation sequences to obtain the output:

[0096] (16)

[0097] Among them, P ins Foreground attention-weighted category activation sequences, P con P is a category activation sequence weighted for action context attention. bak Category activation sequences weighted for background attention.

[0098] This invention introduces an attention branch to weight the category activation sequence, assisting in the determination and localization of subsequent abnormal actions.

[0099] Example 4

[0100] This embodiment of the invention provides a further detailed description of the abnormal action nomination process in Embodiment 1. For example... Figure 8 As shown, based on the aforementioned category activation sequence, an internal / external comparison function is used to generate temporal anomalous action nominations, including:

[0101] Step S510: Calculate the weighted category activation sequence based on the multi-instance learning method, obtain the confidence score of each category of the video to be tested, and compare the confidence score with the preset category confidence threshold to filter the candidate sequence of the temporal abnormal action nomination.

[0102] Specifically, first, find the previous category along the time dimension. k The index corresponding to the weighted category activation sequence score. a :

[0103] (17)

[0104] in, P a This represents one of the three weighted results.

[0105] Using index a Sample category activation sequence score P a The corresponding features are used to calculate the average value, and after normalization, the confidence score (pred) for the abnormal action category of the entire video is obtained. a :

[0106] (18)

[0107] Furthermore, considering that the probability of abnormal action categories with low confidence levels is relatively small, this embodiment of the invention uses a category confidence threshold to filter out each possible abnormal action, in order to avoid abnormal actions with low confidence levels affecting subsequent localization results. The formula is as follows:

[0108] (19)

[0109] in, The sequence represents the filtered sequence, and cls_thres represents the confidence threshold for the abnormal action category. In this embodiment of the invention, cls_thres is set to 0.25.

[0110] Step S520: The confidence score of the candidate sequence is calculated using the internal and external comparison function.

[0111] Specifically, for sequences that, after filtering, consist of a series of consecutive segments... i Assuming the starting time is t s i The end time is t e i The abnormal action category of this sequence is c i The confidence score is calculated based on the internal and external comparison function. i :

[0112] (20)

[0113] The first term in the formula represents the score within the segment, which is the average score of that category from the start time to the end time. The second term represents the score outside the segment, which is the average score over a period of time before the start time and a period of time after the end time. t l i The calculation formula is as follows:

[0114] (twenty one)

[0115] Step S530: Based on the confidence score, abnormal action category, and start and end time of the candidate sequence, determine the candidate box for abnormal action localization.

[0116] Specifically, by combining the start and end times of each sequence t s i and t e i Abnormal Action Categories c iand confidence score i This can generate preliminary candidate box proposals. i ={ t s i , t e i , c i score i}

[0117] In a preferred embodiment, such as Figure 9 As shown, the method of filtering candidate boxes based on nonmaximum suppression to determine the temporal anomalous action nomination includes:

[0118] Step S610: Sort the confidence scores of the candidate boxes and calculate the intersection-union ratio of the candidate boxes in the video dimension.

[0119] Step S620: Delete candidate boxes whose intersection-union ratio with high-confidence candidate boxes in the video dimension is greater than a preset threshold, and use the remaining candidate boxes as the result of the temporal abnormal action localization.

[0120] Specifically, considering the potential overlap between different candidate boxes, non-maximum suppression (NMS) is used to filter them. This process first sorts the confidence scores of the candidate boxes, then calculates the temporal intersection of union (t-IoU) of each candidate box in the video dimension, and finally retains candidate boxes with smaller t-IoU values ​​according to a threshold, while deleting candidate boxes with higher t-IoU values ​​than those with high confidence scores, thereby generating the anchor's temporal anomaly action localization results.

[0121] Preferably, in the above-mentioned abnormal action localization model based on weakly supervised learning, in order to improve the accuracy of abnormal action nomination and prevent misjudgment, the embodiments of the present invention use a loss function to train the model.

[0122] Specifically, embodiments of the present invention optimize the localization performance of abnormal actions in the live video by constructing a loss function optimization model. The loss function optimization model of the present invention includes cross-entropy loss, attention-guided loss, abnormal action feature separation loss, and sparse attention loss.

[0123] Specifically, such as Figure 7 As shown, the cross-entropy loss value corresponding to the confidence scores of the three videos is calculated through weakly supervised annotation, which enables the model to pay attention to features sensitive to abnormal action categories and avoids misclassification of some segments:

[0124] (twenty two)

[0125] in, i Represents the correct category. y i For the ground truth labels, ∑ represents the category dimension. For the foreground attention branch, the foreground anomalous action label in the ground truth labels matches the anomalous action label contained in the video, and the background label is set to 0; for the anomalous action context branch, the foreground anomalous action label is the same, and the background label is set to 1; for the background attention branch, both foreground anomalous action labels are 0, and the background label is 1. After obtaining the cross-entropy loss for each branch, they are summed to obtain the total cross-entropy loss term L. ce :

[0126] (twenty three)

[0127] Attention-guided loss L gui The discriminativeness of the foreground attention branch is improved by constraining the relationship between the background class prediction results in the foreground attention and the category activation sequence weighted by foreground attention:

[0128] (twenty four)

[0129] Here, the average Mean is applied to the video dimension, and bg_cls represents the background category.

[0130] The abnormal action feature separation loss constrains the feature relationships belonging to the foreground, the abnormal action context, and the background, respectively. Among these, component L... emb_ins The amplitude of the foreground feature is higher than that of the context feature corresponding to the abnormal action; component L emb_con The context features promoting abnormal actions have higher amplitudes compared to background features; component L emb_bak Reduce the amplitude of background features. By constraining the relationship between the three types of features, the embedding layer becomes more sensitive to abnormal actions:

[0131] (25)

[0132] in, m As a margin, it was set to 50 in the experiment. The summation of the three components of the above formula yields the abnormal action feature separation loss L. emb :

[0133] (26)

[0134] Sparse attention loss L spa Used to constrain the foreground attention branch and the abnormal action context branch, preventing the model from predicting too many segments as foreground:

[0135] (27)

[0136] By combining all the above loss terms, the final loss value is obtained:

[0137] (28)

[0138] in, These are all hyperparameters, and in this embodiment of the invention, they are respectively set to... , and .

[0139] To more intuitively illustrate the effectiveness of the proposed method in temporal action localization tasks, Figures 10A to 10C Subjective results are presented on the BJUT-SAD-DA dataset. Figure 10A For videos that include the action of eating, Figure 10B For videos that include clapping, Figure 10C For videos featuring actions such as climbing stairs and walking. Figures 10A to 10C In each graph, from top to bottom, the video frame, the attention scores of the three branches, the category activation sequence after foreground attention weighting, and the ground truth label of the action location are represented. For the attention score curves, the three curves represent the foreground attention score, the action context score, and the background attention score, respectively. It's important to note that the reason for the long stretch of completely consistent predictions in the latter half of the video is that during the testing phase, to ensure a consistent video length, the last segment was repeatedly added until 100 segments were reached. Since the receptive field of the embedding layer includes multiple segments, segments with all features set to 0 were added when extracting features from the last segment. Therefore, the prediction result for the last segment may differ from the previous parts, causing changes in the curve. By comparing the prediction results with the ground truth labels, it can be seen that the abnormal action localization method of this embodiment can accurately locate the time period of the action and identify the action that occurred.

[0140] In summary, the abnormal action localization method for live online videos provided by the embodiments of the present invention has the following advantages:

[0141] 1) No optical flow modalities are required as auxiliary learning video features, which meets the real-time requirement for locating abnormal actions in the broadcaster's time sequence;

[0142] 2) By adopting a weakly supervised learning method, only a small amount of labeling cost is required to realize the task of locating time-series abnormal actions, which effectively reduces the cost of data collection and enables rapid deployment for new tasks.

[0143] 3) Improve the performance in the task of locating abnormal actions of anchors by using background erasure technology to generate diverse video samples, pay more attention to the temporal characteristics of actions, and reduce the demand for video memory.

[0144] Accordingly, this embodiment of the invention also provides an abnormal action localization device for live online videos, characterized in that the abnormal action localization device includes a memory storing a program that can run on a processor; and the processor is configured to implement the abnormal action localization method for live online videos described in the above embodiments when executing the program.

[0145] This invention also provides a machine-readable storage medium storing instructions that cause a machine to perform any of the above-mentioned abnormal action localization for live online videos.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for locating abnormal actions in live online videos, characterized in that, The method for locating abnormal actions includes: Feature extraction is performed on the live web video based on a multi-fiber network to generate a segment-level video feature sequence of the live web video; The segment-level video features are processed to output category activation sequences and attention branches for the broadcaster's actions in the live stream video; and Based on the weighted result of the category activation sequence and the attention branch, a temporal abnormal action nomination for the anchor's actions is generated, wherein the temporal abnormal action nomination is used to indicate the abnormal action localization result for the anchor's actions in the live video. The generation of the time-series abnormal action nomination includes: The weighted result of the category activation sequence and the attention branch is calculated based on the multi-instance learning method to obtain the confidence score of each category of the video under test; The confidence score is compared with a preset category confidence threshold to filter candidate sequences for the proposed time-series abnormal actions; The confidence score of the candidate sequence is calculated using the following formula: In the formula, for a sequence composed of consecutive segments i score i This represents the confidence score. t s i Indicates the start time. t e i Indicates the end time. c i Abnormal action categories, t l i This indicates a duration between the start time and the end time. This represents the filtered sequence; Based on the confidence score, abnormal action category, and start and end times of the candidate sequences, candidate bounding boxes are determined for abnormal action localization; and Filter the candidate box to determine the nomination of the time-series abnormal action.

2. The abnormal action localization method according to claim 1, characterized in that, Before performing feature extraction on the live video based on the multi-fiber network, the abnormal action localization method further includes pre-training the multi-fiber network in the following manner: Using cropped live domain videos from the dataset, and background erasure of the initial videos in the dataset, diverse video samples with background erasure are generated; The initial video and the diverse video samples after background erasure are data augmented, input into an initial multifiber network to extract features, and then fed into a classifier. The classifier is trained on video action recognition task based on metric learning loss and cross-entropy loss to obtain a feature extraction backbone network that is sensitive to the actions of live online videos.

3. The abnormal action localization method according to claim 2, characterized in that, Background removal is performed on the videos in the dataset to generate diverse video samples with background removal, including: Randomly sample one frame from the initial video in the dataset to obtain the corresponding sampled frame image; The initial video and the sampled frame images are weighted and fused into each frame of the initial video to obtain diverse video samples after background erasure.

4. The abnormal action localization method according to claim 1, characterized in that, The segment-level video feature sequence is processed to output the corresponding category activation sequence, including: The segment-level video feature sequence is embedded to obtain the embedded segment-level video feature sequence; A segment-level classifier is used to classify the embedded segment-level video feature sequence to obtain the category activation sequence and attention branch; The attention branch is used to weight the category activation sequence, and the weighted category activation sequence is output.

5. The abnormal action localization method according to claim 1, characterized in that, The candidate boxes are filtered based on a nonmaximum suppression method to determine the proposed time-series anomalous actions, including: The confidence scores of the candidate boxes are sorted, and the intersection-union ratio of the candidate boxes in the video dimension is calculated. Candidate boxes with an intersection-union ratio greater than a preset threshold in the video dimension that are related to high-confidence candidate boxes are deleted, and the remaining candidate boxes are used as the results of the temporal abnormal action localization.

6. The abnormal action localization method according to claim 1, characterized in that, The abnormal action localization method also includes: A loss function optimization model is constructed to optimize the localization performance of abnormal actions in the live video.

7. An abnormal action localization device for live online videos, characterized in that, The abnormal action location device includes: Memory that stores programs that can run on a processor; and The processor is configured to implement the abnormal action localization method for live online video as described in any one of claims 1-6 when executing the program.

8. A machine-readable storage medium storing instructions for causing a machine to perform the abnormal action localization method for live web video as described in any one of claims 1-6.

Images