Action Detection of Video Frames Using a Gated Past

Abstract

An exemplary solution for action detection of video frames uses a gated past and includes the following: receiving a video stream including a plurality of video frames; grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame; determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how much useful information the video frames contain for predicting an action in the current video frame; weighting the set of past video frames with the set of attention weights to create a weighted set of past video frames; and generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames.

Description

Background Art

[0001] Detecting actions in real time in streaming video has applications in multiple scenarios, such as content moderation (e.g., proactively detecting suspicious / harmful behavior), public safety (e.g., detecting abnormal events in security videos), retail (e.g., loss prevention), etc. Action detection in video frames is difficult because future video frames cannot be utilized at any time to predict actions within the current video frame. Due to the limited context of the current video frame, the false positive rate of "no action" video frames is high (e.g., "no action" video frames are misidentified as "action" video frames).

[0002] Existing solutions for action detection in video frames generally use past video frames within a network model. However, not all past video frames contain useful information to guide the identification and classification of actions (or the absence thereof) that may occur in subsequent video frames. In some scenarios, the use of past frames that do not contain useful information may actually worsen the false positive rate.

Summary of the Invention

[0003] The disclosed examples are described in detail below with reference to the accompanying drawings listed below. The following summary is provided to explain some of the examples disclosed herein. However, it is not intended to limit all examples to a specific configuration or a series of operations.

[0004] Exemplary solutions for action detection in video frames use a gated past and include: receiving a video stream including a plurality of video frames, grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame, determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how much information the video frames contain for predicting an action in the current video frame, weighting the set of past video frames with the set of attention weights to create a weighted set of past video frames, and generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames.

[0005] The disclosed examples are described in detail below with reference to the accompanying drawings listed below.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0007] Corresponding reference numerals indicate corresponding parts throughout the drawings. With reference to the accompanying drawings, various examples are described in detail. Preferably, the same reference numerals may be used to refer to the same or similar parts throughout the drawings. References made throughout this disclosure in connection with specific examples and implementations are for illustrative purposes only and are not meant to limit all examples unless otherwise indicated.

[0008] Online action detection is the task of predicting an action as soon as it occurs in a streaming video, such as when a golfer starts to swing a club or a person starts to speak. In contrast, in action detection in an offline setting, the entire untrimmed video is observable at any instant, making it considerably easier to detect the specific frame at which the action starts. A major challenge in online action detection is that, when making a prediction, the prediction is based only on observations of the contextual past (i.e., the video frames observed up to that point) without accessing future video frames.

[0009] A major challenge in leveraging the past for action detection is that, in the case of a long untrimmed video, its length becomes unmanageable as time passes. One option is to limit the past to only the most recent frames, e.g., within one minute. Unfortunately, the past containing useful information may be discarded and provide no value (i.e., does not improve the probability of making a correct action / no-action prediction). Not all past frames contain useful information and are beneficial, and if some past containing little useful information is used, it may actually degrade the accuracy.

[0010] Therefore, in online action detection using an untrimmed streaming video, it is valuable to emphasize a portion of the past that contains more useful information for the prediction of the current frame. The present disclosure presents a location-guided gated cross-attention mechanism that enhances or suppresses a portion of the past based on how much useful information it contains for the current frame prediction. In some examples, when subsequent observed frames are available, they are used to further render the past features to contain more useful information.

[0011] The present disclosure integrates the capabilities of a transformer for long-range temporal modeling and a regression model for selectively encoding relevant information. In some examples, a background suppression objective is also introduced to further reduce false positive background frames that closely resemble action frames. Additionally, the flow-free version can achieve higher or near accuracy at a higher frame rate compared to previous solutions that require both color pixel information ("RGB") and optical flow information for prediction.

[0012] An exemplary solution for action detection in video frames (also known as online action detection) uses a gated past and includes: receiving a video stream including a plurality of video frames; grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame; determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how much useful information the video frames contain for predicting an action in the current video frame; weighting the set of past video frames with the set of attention weights to create a weighted set of past video frames; and generating an action prediction for the current video frame based at least on the weighted set of past video frames and the set of current video frames.

[0013] Aspects of the present disclosure improve the operation of a computing device, for example, at least by weighting a set of past video frames with a set of attention weights indicating how much useful information the video frames contain for predicting actions in a current video frame, thereby improving the accuracy and / or speed of action detection in video frames. In some examples, the advantages of selective encoding of a long short-term memory (LSTM) regression neural network (NN) are combined with the long-range modeling of a transformer to better utilize frames containing useful information over a long past. Practical applications include safety, surveillance, content moderation, augmented reality (AR), self-driving vehicles, and autonomous vehicles.

[0014] FIG. 1A shows an exemplary architecture 100 that advantageously provides action detection for video frames using a gated past. A video stream 102 has a set 106 of current video frames including a current video frame 102q, and a set 104 of past video frames that together form a long past. For example, while the current video frame is the current frame at present, the set of current video frames includes not only the current frame at present (or the current frame), but also one or more immediately preceding current frames. In other words, the set of current video frames includes the most immediate frame relative to the current video frame at a given point in time. (See FIG. 5A for further details). The video stream 102 is provided to a feature extractor 500 that includes a future-augmented past (FAH) 502 component. A past encoder 400 that includes a gated past unit (GHU) 300 determines a set of attention weights and uses that set to weight the set of past video frames 104 (and also some or all of the set 106 of current video frames). This creates a weighted set of past video frames that is provided to a current decoder 600.

[0015] In parallel, the feature extractor 500 also encodes the current set of video frames 106 and provides it as an additional input to the current decoder 600. The current decoder 600 has a set of attention networks and a classifier 620 that outputs an action prediction 610 for at least the current video frame 102q. The GHU 300 is described in more detail in relation to FIG. 3. The past encoder 400 is described in more detail in relation to FIG. 4. The feature extractor 500 and the FAH are described in more detail in relation to FIG. 5. The current decoder 600 and background suppression are described in more detail in relation to FIG. 7. The video stream 102, the current set of video frames 106, and the past set of video frames 104 are shown in more detail in relation to FIG. 7. A more detailed operation of the architecture 100 is provided after describing the various components in each of FIGS. 2-7.

[0016] FIGS. 2A and 2B show two of many practical uses for using the architecture of FIG. 1. In FIG. 2A, an action detection scenario 200a for a video frame depicts a video camera 216 that captures a scene 218 and outputs a video stream 102 that receives a broadcast delay 202. The video stream 102 is also provided to an architecture 100 that outputs an action prediction 610. The action prediction 610 is provided to an action response 204 that responds to the action prediction 610 based on the predicted action class and the specific use (e.g., security, surveillance, content moderation, AR, self-driving vehicle, autonomous vehicle, or others) employed for action detection of the video frame. In the current scenario 200a, the predicted action is represented as an annotation 210 overlaid on the current video frame 102q.

[0017] Architecture 100 operates in real time, and annotation 210 is prepared during broadcast delay 202, which can be on the order of several seconds (e.g., 7 seconds in some examples). Broadcast function 206 transmits current video frame 102q to display 208, and on display 208, current video frame 102q is displayed together with annotation 210. In this illustrated example, annotation 210 includes object 212 involved in action prediction 610, e.g., the outline of a golfer swinging a golf club. In this scenario, action prediction 610 is not undesirable; instead, it is the type of action where the viewer's attention is desired.

[0018] In FIG. 2B, video frame action detection scenario 200b depicts AR engine 222 that generates video stream 102 by, for example, mixing live scene capture by a camera (e.g., video camera 216 and scene 218 of scenario 200a). Video stream 102 is provided to architecture 100 that outputs action prediction 610. Action prediction 610 is provided to action response 204 that responds to action prediction 610 based on the specific application employed for action detection of the video frame. In current scenario 200a, the action is also annotation 210 overlaid on current video frame 102q.

[0019] Architecture 100 operates in real time, and annotation 210 is prepared at approximately the same time that AR engine 222 can insert icons (e.g., AR object 214) and / or virtual objects into current video frame 102q. AR engine 222 provides current video frame 102q to display 208, where current video frame 102q is displayed together with annotation 210. In this illustrated example, annotation 210 includes brightening pixels within the region of current video frame 102q that are proximate to object 212 involved in action prediction 610. Other possibilities for annotation include obscuring objects involved in action prediction, or blanking out the current video frame if, for example, the predicted action is of a class indicating something undesirable.

[0020] There are further a plurality of practical applications for architecture 100. For example, architecture 100 may be deployed and operated within a customer premise, such as the Internet of Things (IoT) and edge devices, including training and testing. For example, architecture 100 may be deployed in a retail store where one or more surveillance cameras capture video feeds over a long period of time. The video feeds may be annotated and used to train architecture 100 for online action detection of customer behavior and actions / events, including suspicious or criminal behavior. Once trained, the architecture 100 model may be deployed to on-premises IoT devices for online detection of actions and events.

[0021] As a further development, scenarios of edge artificial intelligence (AI) can be utilized, such as on-premises devices and cloud services. Instead of predicting actions for the currently observed frame, Architecture 100 can perform the action prediction task by training the model to predict actions for frames that will be observed in the future, after some time. This can be useful in scenarios such as content moderation of live video streams, where the model predicts whether any harmful or inappropriate behavior is likely to occur and a time buffer can be provided to block the video feed without delay.

[0022] This application may also be useful in autonomous driving, where it can help predict the trajectories of vehicles and pedestrians on the road in advance and ensure making driving decisions in a state with a lot of useful information. Furthermore, it may also be useful in augmented reality (AR) scenarios where the user's intentions and behaviors are predicted to improve the user experience.

[0023] Architecture 100 performs online action detection when future frames are not available to the model, but Architecture 100 can also perform frame-by-frame action prediction in an offline setting where access to all frames of the video stream is available. To do this, the current frame is set as an intermediate frame in the current decoder of the model, and the frames observed subsequently after the current frame can be considered "future" frames. In this way, the model can utilize future frames available in the offline setting for frame-by-frame action prediction. The ability to perform frame-by-frame action prediction in an offline setting is beneficial in scenarios where time is not critical, and the main purpose is to improve the accuracy of action prediction across the entire video stream.

[0024] Some examples of such scenarios include generating highlights of a sports event for later broadcast or assisting a coach in evaluating a player's performance. Other scenarios include analyzing videos uploaded by a user and evaluating actions that occur at unknown locations within the video stream.

[0025] In some examples, the architecture 100 can take in inputs from multiple modalities and process features simultaneously to improve the performance of action detection. Modalities can include RGB-based appearance, optical flow / motion, depth data from time-of-flight sensors, audio data, string and / or language data, data from sensors such as accelerometers, gyroscopes, magnetometers, etc. This can support several multimodal user scenarios such as moderation of audio-visual content, autonomous driving using multiple sensors, AR / metaverse using both RGB and depth sensors. The architecture 100 can support multimodal scenarios for various tasks such as online action detection, action prediction, and frame-by-frame offline action detection.

[0026] FIG. 3 shows the GHU300, a location-guided gated cross-attention component of the architecture 100 that enhances or suppresses frames of the video stream 102 (e.g., frames of the set 104 of past video frames) depending on how much useful information each frame contains for predicting an action for the current frame 103q. Here, an introduction to the components of the GHU300 is provided, and further details regarding the operation of the GHU300 are provided after various other components of the architecture 100 are similarly introduced in FIGS. 4-7.

[0027] The query (Q302), key (K304), and value (V308) are provided by the feature extractor 500. The gating score (G306) is a learned distinct encoding in the range from negative infinity to e (2.78) in some examples. Q302 and K304 are given to the matrix multiplication 312 and scaled by the scalar 314. G306 is given to the sigmoid function 316, the output is multiplied by the logarithmic function 318, and added to itself by the addition 320. This is gated by the gate 322 and added to the output of the scalar 314 by the soft gating addition 324. The softmax 326 is applied and the result is given to the matrix multiplication 328 with V308. This creates a set of attention weights 310 for at least the set 104 of past video frames (and in some examples, in addition, the set 106 of current video frames).

[0028] Figure 4 shows the past encoder 400. A set of encoded features (described later in relation to, for example, Figure 5) is given to the GHU 300 together with Q302. The output of the GHU 300 (for example, the set of attention weights 310) is given to the self-attention network 402 that outputs a weighted set of past video frames 410.

[0029] Figure 5A shows further details regarding the feature extractor 500 and the video stream 102. The video stream 102 is shown as including a plurality of video frames, namely video frames 102a to 102q. The set 104 of past video frames includes video frame 102a, video frame 102b, video frame 102c, video frame 102d, video frame 102e, video frame 102f, video frame 102g, video frame 102h, video frame 102i, video frame 102j, video frame 102k, video frame 102l, video frame 102m, video frame 102n, and video frame 102o.

[0030] The current set of video frames 106 is a set of the most recent video frames, including the current video frame 102q which is the last frame (or the current frame at present). The current set of video frames 106 also includes the video frame 102o and the video frame 102p which are the current frames immediately preceding the current video frame 102q. The video frame 102o is also included in the set of past video frames 104, giving an overlap between the set of past video frames 104 and the current set of video frames 106. The future video frame 102r is not yet available. These numbers of video frames are for illustrative purposes, and it should be understood that some examples of the architecture 100 can use a much larger number of video frames.

[0031] The current video frame 102q does not have future video frames (e.g., the video frame 102r) available for refining action prediction, but the video frames within the set of past video frames 104 have available "future" video frames. A glance at Figure 5B shows the idea of "future" video frames for past video frames.

[0032] For any given video frame within the set of past video frames 104, a video frame can be designated as a past frame. For each individual past frame, within the video stream 102, there is a set of subsequently observed video frames that are temporally newer than that individual past frame. For example, for the video frame 102c, the video frames 102d, 102e, 102f are newer, for the video frame 102d, the video frames 102e, 102f, 102g are newer, and for the video frame 102e, the video frames 102f, 102g, 102h are newer. From the perspective of any given past frame, the set of subsequently observed video frames represents the "future" video frames with respect to that given past frame. This is because such "future" video frames are temporally later than that past frame.

[0033] This can be used to improve the set 104 of past video frames and provide future-enhanced encoding by FAH502. In other words, FAH502 utilizes hindsight to provide "future" frames for past frames in order to improve past encoding for current frame prediction. FAH502 aggregates the observed "future" information into the feature quantities of past frames, making it possible to recognize the "future" observable so far. For each new time step when a new frame is observed, FAH502 feeds forward the feature extraction backbone twice to extract the feature quantities of the new frame.

[0034] Returning to FIG. 5A, the video stream 102, including the set 104 of past video frames and the set 106 of current video frames, is weighted by the attention weights 508. The video frames of the set 104 of past video frames and the video frames of the set 106 of current video frames have already passed through the architecture 100 as the current video frame at that time. In this way, since it is evaluated for each video frame whether there was an action or not, the attention weights 508 can be derived by remembering this behavior for each video frame.

[0035] FAH502 extracts feature quantities only from the video frames containing the most useful information, creating the feature quantities 504a and 504b for the set 104 of past video frames and the feature quantity 504s for the set 106 of current video frames. The feature quantities 504a - 504c are encoded by the encoder 506 and become the encoded feature quantities 510 provided to the past encoder 400 of FIG. 4. The feature quantity 504c is encoded by the encoder 506 and becomes the encoded feature quantity 512 provided to the current decoder 600 of FIG. 6.

[0036] As described below, some examples of the feature extractor 500 use optical flow to improve accuracy, but at the cost of increased execution time. Optical flow estimation identifies the pixel-level movement of objects (e.g., object 212 in FIG. 2) between consecutive video frames. Some examples of the feature extractor 500 do not use optical flow and instead use only red, green, and blue (RGB) pixel information to improve the computation time, e.g., reducing the execution time by up to two-thirds. Some examples of the architecture 100 may not use optical flow because they operate in time-critical applications and / or on low-performance computing platforms.

[0037] FIG. 6 shows a current decoder 600 that correlates a small set of the most recent video frames (e.g., set 106 of the current video frame) with the past (e.g., at least set 104 of the past video frames) of the video stream 102 to perform current frame prediction. The encoded feature set 512 and the weighted set 410 of past video frames are provided to a set of attention networks. For example, the encoded feature set 512 is provided to a self-attention network 602 with a causal mask, and the output thereof is provided to a cross-attention network 604 together with the weighted set 410 of past video frames.

[0038] The output of the cross-attention network 604 is provided to both another self-attention network 606 and another cross-attention network 608 together with the output of the self-attention network 606. The output of the cross-attention network 608 is provided to a classifier 620 that outputs an action prediction 610. In some examples, the action prediction 610 includes both an action class 612 (e.g., “no action” or one of a predefined set of action classes) and a confidence score 614 that is a measure of how reliable the classifier 620 is with respect to the action prediction 610.

[0039] Some examples of the current decoder 600 also use background suppression 622 to reduce false positive predictions of background frames that closely resemble action frames. Background suppression 622 uses a loss function 624 and is applied as a loss to the classifier 620. Background suppression 622 emphasizes the low-confidence predictions that occur between video frames without action and video frames with action so that the classifier 620 is better trained, motivating it to correctly learn these frames. In other words, background suppression 622 modifies the confidence of action predictions by weighing more heavily the low-confidence video frames using separate emphasis for the action class and the background class for the classifier 620 that generates action predictions.

[0040] Figure 7 shows video frames without action, low-confidence action video frames, and high-confidence action video frames that may be encountered when using an example of architecture 100. The action prediction 610m for video frame 102m indicates that the action class 612m is "no action" and the confidence 612m is relatively high. The action prediction 610n for video frame 102n indicates that the action class 612n is "golf swing" and the confidence 612n is relatively low. The action prediction 610o for video frame 102o indicates that the action class 612o is "golf swing" and the confidence 612o is relatively high. The action prediction 610p for video frame 102p indicates that the action class 612p is "golf swing" and the confidence 612p is relatively high. The action prediction 610q for the current video frame 102q indicates that the action class 612q is "golf swing" and the confidence 612q is relatively high.

[0041] Therefore, video frame 102m is an actionless video frame, video frame 102n is a low-confidence action video frame, and video frames 102o to 102q are high-confidence action video frames. Background suppression 622 modifies the confidence, for example, using loss function 624, to emphasize video frame 102n when training classifier 620. In some examples, the confidence value ranges from zero to one, [0,1], and represents the probability of predicting the correct action.

[0042] Next, further details of the operation of architecture 100 are provided. Architecture 100 includes GHU300, FAH502, and background suppression 622, enabling improvement in accuracy and / or speeding up the execution time by approximately three times. GHU300 provides position-guided gated cross-attention that explicitly enhances or suppresses a portion of the video past depending on how much useful information it contains for predicting an action for the current frame. FAH502 extracts features from past frames using the frames subsequently observed to enhance past encodings. Background suppression 622 reduces false positive predictions of background frames that closely resemble action frames.

[0043] Streaming video sequence

[0044]

Number

[0045]

Number

[0046]

Number

[0047] The T past frames,

[0048]

Number

[0049] Specifically, as a streaming sequence of T past frames ending at the current frame h0

[0050]

Number

[0051]

Number

[0052]

Number

[0053]

Number

[0054]

Mathematics

[0055]

Mathematics

[0056]

Mathematics

[0057]

Mathematics

[0058]

Mathematics

[0059]

Mathematics

[0060] The position-guided gating score G is obtained for h as follows: z g = σ(z h W g ) Equation (1) G = log(z g ) + z g Equation (2) Here

[0061]

Number

[0062]

Number

[0063] In GHU300, the gating score (G308) of the past frame is

[0064]

Number

[0065]

Number

[0066]

Number

[0067]

Number

[0068] From equations (1) and (2), z g It can be seen that each scalar of is in [0, 1] by the sigmoid. This means that each gating score of G exists in [-∞, 1]. Thus, the softmax function in equation (3) can calibrate the attention weights of each past frame by factors in [0, e] such that factors in [0, 1) suppress a given past frame and factors in (1, e] enhance a given past frame. This enables GHU300 to learn an explicit ability to calibrate the attention weights of past frames based on how much useful information the past frames contain for the prediction of the current frame h0.

[0069] Therefore, G is input-dependent and learns based on the past frames and the position of the past frames relative to the current frame. This allows GHU300 to evaluate how much useful information each frame contains based on the feature representation of each past frame and the relative position from the current frame h0. The output of GHU300 is fed into a series of N self-attention layers (self-attention network 402) to obtain the final past encoding output from the past encoder 400 (i.e., the set 410 of weighted past video frames).

[0070] FAH502 utilizes hindsight to provide "future" frames for past frames in order to improve the past encoding for current frame prediction. (See reference to Figure 5B). FAH502 aggregates the observed "future" information into the features of past frames, enabling the recognition of the currently observable "future". For past frame h t and feature extraction backbone u, when the "future" past frame for h t with respect to t f can be observed, FAH502 selects a set of frames

[0071]

Number

[0072]

Number

[0073]

Number

[0074]

Number

[0075]

Number

[0076]

Number

[0077] To correlate the current and the past to perform current frame prediction, the most recent past frame of t pr

[0078]

Number

[0079] [Number] is applied to each of the frame features, which is then passed through multi-head self-attention with a causal mask. The causal mask restricts the influence to only previous frames for a given frame. pr The output from the self-attention is cross-attentioned with the past encoding from the past encoder 400. This is repeated, but for the second time, the self-attention does not require a causal mask. t

[0080] The output corresponding to each frame is fed to a classifier layer (e.g., classifier 620) for prediction. pr The output corresponding to each frame is fed to a classifier layer (e.g., classifier 620) for prediction.

[0081] The background ("no action") video frames can be anything from completely blank at the start of the video stream 102 to something that closely resembles an action frame but is actually not an action frame (e.g., aiming before taking a billiard shot). The latter scenario is a common cause of false positives (e.g., classifying a "no action" video frame as an "action" video frame). To reduce false positives, background suppression 622 applies emphasis to low-confidence action predictions and background predictions during the training of classifier 620, increasing the margin between action and background ("no action") video frames.

[0082] For frame h t the objective (loss) function L t (loss function 524) is defined as follows:

[0083]

Number

[0084] Some examples use optical flow in addition to RGB to capture fine-grained motion between frames. However, calculating optical flow requires more computational time than feature extraction or model inference alone, and in time-critical applications (e.g., self-driving cars and autonomous vehicles), it may be too slow in some scenarios. Therefore, some examples do not use optical flow.

[0085] To capture motion using only RGB frames without using optical flow, in some examples, multiple temporal resolutions using a spatio-temporal backbone are used. In some examples, two feature vectors are extracted for frame h t by encoding a frame sequence sampled at a higher frame rate covering a shorter period and another frame sequence sampled at a lower frame rate covering a longer period. The two feature vectors are concatenated.

[0086] In one example, the video stream 102 is sampled at 24 frames per second (FPS), and frames are extracted at 4 FPS for training and evaluation. The size of the set 104 of past video frames is set to 1024, and the size of the set 106 of current video frames is set to the most recently observed 8 frames. This is at 4 FPS, covering 256 seconds and 2 seconds respectively. To extract frame-level RGB and optical flow features, a two-stream temporal segment network (TSN) is used. The RGB and optical flow features are concatenated along the channel dimension before being fed into a linear encoding layer. The period of FAH502 is the past t ps frames and the future t f frames are set to be 1 second and 2 seconds respectively.

[0087] In the version without using optical flow, the optical flow features are replaced by features obtained from additional multi-frame inputs of RGB frames uniformly sampled over a period of 2 seconds. Training is run for 10 epochs with a weight decay of 5E-5 and a batch size of 50. D is set to 1024, the latent encoding size is 16, and 2 layers are used in the past decoder. Each attention layer has 16 heads (N HEADS = 16), and γ a = 0.6, γ b = 0.2 for background suppression.

[0088] FIG. 8 shows a flowchart 800 illustrating exemplary operations that may be performed by architecture 100. In some examples, the operations described with respect to flowchart 800 are performed by computing device 1000 of FIG. 10. Flowchart 800 begins, at operation 802, by receiving a video stream 102 that includes a plurality of video frames. When a streaming video sequence, such as video stream 102, is provided, the task is to identify whether an action occurs in the current frame and what action occurs. There may be labels for background frames without action (i.e., “no action” video frames) available for use by a classifier such as classifier 620, as well as labels for any number of different action classes.

[0089] Operation 804 groups a plurality of video frames 102a - 106q (of video stream 102) into a current set of video frames 106 (including current video frame 102q) and a past set of video frames 104. In some examples, the current set of video frames 106 and the past set of video frames 104 overlap. Since future frames are not yet accessible, the process makes a prediction of the current frame using the most recent frames observed up to the current frame. The most recent frames may be large in an untrimmed video stream, but not all frames observed in the previous past necessarily contain useful information for predicting the current frame in the same way. Operation 806 subsequently weights the set of video frames to be observed with attention weights determined previously.

[0090] Operation 808 extracts feature quantities 504a and 504b from a set 104 of past video frames based on at least a set of past frames and a set of video frames subsequently observed thereof. In some examples, extracting the feature quantities uses optical flow. In some examples, extracting the feature quantities does not use optical flow. Operation 810 encodes the extracted feature quantities 504a and 504b. In some examples, this operation includes determining a potential encoding for each video frame in the set 104 of past video frames. Some examples of operation 810 include operation 812 of encoding the extracted feature quantities 504a and 504b using cross-attention to project variable-length portions of multiple video frames into fixed-length learned potential encodings.

[0091] In some examples, in order to accurately make a prediction for the current frame based on past frames, the transformer encodes the past of the video sequence (e.g., at least the set 106 of past video frames of the video stream 102), and then associates the current frame with the encoding for prediction. The past encoder 400 uses cross-attention to project the variable-length past into a fixed-length learned potential encoding. Specifically, each frame is encoded by a linear encoding layer following a feature quantity extraction backbone. The output is applied to a learnable positional encoding for the current frame. Thereafter, the encoding of the learnable potential query is cross-attended with each frame.

[0092] Operation 814 determines a set 310 of attention weights for a set 104 of past video frames. The set 310 of attention weights indicates how much useful information a video frame contains for predicting an action in the current video frame 102q, and reinforces or suppresses the video frames of the set 104 of past video frames based at least on how much useful information the video frames contain for predicting an action in at least the current video frame 102q. In some examples, each attention weight of the set 310 of attention weights is a positive value. In some examples, the set 310 of attention weights is within a range from zero to a positive maximum value. A value between 0 and 1 suppresses attention to the video frame, and a value between 1 and the positive maximum value reinforces attention to the video frame. In some examples of operation 814, for each video frame of the set 104 of past video frames, an operation 816 of determining a position-guided gating score G306 is used. In one example, the position-guided gating score is obtained for each frame using a matrix that projects each past frame to a scalar, and after applying a sigmoid function, a sequence of scalars for the past frames is generated. Thereby, the softmax function can calibrate the attention weights of each past frame to suppress or reinforce a given past frame, and the GHU300 realizes the ability to learn to calibrate the attention weights of the past frames based on how much useful information the past frames contain for the prediction of a given current frame. This demonstrates that the position-guided gating score is input-dependent and learns based on the past frames and the position of the past frames relative to the current frame. Thereby, the GHU300 can evaluate how much useful information a frame contains based on the feature representation of each past frame and the relative position from the current frame.

[0093] Operation 818 weights a set 104 of past video frames with a set 310 of attention weights to create a weighted set 410 of past video frames. For example, the output of GHU300 is fed into a series of self-attention layers such as self-attention network 402 to obtain a final past encoding such as a weighted set 410 of past video frames output from past encoder 400. Operation 820 generates an action prediction 610 for current video frame 102q based on at least the weighted set 410 of past video frames and the set 106 of current video frames. Operation 820 uses operations 822-826. Operation 822 generates an action prediction 610 for current video frame 102q by cross-attention between an encoded past based on at least the set of weighted past video frames 410 and a self-attention network output based on the encoded extracted feature amounts of at least the set 106 of current video frames. In some examples, action prediction 610 includes a no-action prediction determined in operation 824, or a prediction of an action class 612 selected from a plurality of action classes. In some examples, action prediction 610 includes a confidence 614 determined in operation 826.

[0094] Operation 828 performs background suppression to modify confidence 614 by, for example, generating a loss function 624 that weights less reliable video frames more heavily using a separate emphasis for the action class and the background class for classifier 620 that generates action prediction 610 using operation 830. In some examples, the training of classifier 620 continues during operation.

[0095] The determination operation 832 determines whether the current video frame 102q is classified as an action frame. If so, based at least on the action prediction 610 for the current video frame 102q, the operation 834 generates an annotation 210 for the current video frame 102q. In some examples, the annotation 210 for the current video frame 102q includes an annotation selected from a list consisting of the contour of the object 212 involved in the action prediction 610, brightening pixels within the region of the current video frame 102q proximate to the object 212 involved in the action prediction 610, obscuring the object 212 involved in the action prediction 610, and blanking the current video frame 102q. The operation 836 displays the current video frame 102q according to the annotation 210 for the current video frame 102q. Otherwise, the operation 838 displays the current video frame 102q as normal.

[0096] FIG. 9 shows a flowchart 900 illustrating exemplary operations that may be performed by the architecture 100. In some examples, the operations described with respect to the flowchart 900 are performed by the computing device 1000 of FIG. 10. The flowchart 900 begins with an operation 902 that includes receiving a video stream that includes a plurality of video frames. The operation 904 includes grouping the plurality of video frames into a set of current video frames and a set of past video frames, where the set of current video frames includes the current video frame.

[0097] Operation 906 includes determining a set of attention weights for a set of past video frames, the set of attention weights indicating how much useful information the video frames contain for predicting an action in the current video frame. Operation 908 includes weighting the set of past video frames with the set of attention weights to produce a weighted set of past video frames. Operation 910 includes generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames. Further examples The exemplary system includes a processor and a computer-readable medium, the computer-readable medium storing instructions that, when executed by the processor, operate to: receive a video stream including a plurality of video frames; group the plurality of video frames into a set of current video frames and a set of past video frames, the set of current video frames including the current video frame; determine a set of attention weights for the set of past video frames, the set of attention weights indicating how much useful information the video frames contain for predicting an action in the current video frame; weight the set of past video frames with the set of attention weights to produce a weighted set of past video frames; and generate an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames.

[0098] One example of a computerized method includes: receiving a video stream including a plurality of video frames; grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame; determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how much useful information the video frames contain for predicting an action in the current video frame; weighting the set of past video frames with the set of attention weights to create a weighted set of past video frames; and generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames.

[0099] One or more example computer storage devices have computer-executable instructions stored thereon that, when executed by a computer, cause the computer to perform operations including: receiving a video stream including a plurality of video frames; grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame; determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how much useful information the video frames contain for predicting an action in the current video frame; weighting the set of past video frames with the set of attention weights to create a weighted set of past video frames; and generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames.

[0100] Alternatively, or in addition to other examples described herein, examples include any of the following combinations: - Generating an annotation for a current video frame based at least on an action prediction for the current video frame; - Displaying the current video frame according to an annotation for the current video frame; - Determining a set of attention weights includes determining a location-guided gating score for each video frame of a set of past video frames; - A plurality of video frames includes a set of past frames and a set of video frames that are successively observed for each past frame within the set of past frames; - The set of video frames that are successively observed are newer than the past frames; - Extracting feature quantities from a set of past video frames based at least on the set of past frames and the set of video frames that are successively observed thereafter; - Encoding the extracted feature quantities; - Extracting the feature quantities does not use optical flow; - Performing background suppression; - The action prediction includes a confidence level; - Performing background suppression includes weighting video frames with low confidence more heavily using separate emphasis for the action class and the background class for a classifier that generates an action prediction; - The action prediction includes a prediction of no action or an action class selected from a plurality of action classes; - The set of current video frames and the set of past video frames overlap; - Determining a latent encoding for each video frame within the set of past video frames; - Each attention weight of the set of attention weights is set to a positive value; - The set of attention weights is within the range from zero to a positive maximum value, where values between 0 and 1 suppress attention to the video frame, and values between 1 and the positive maximum value enhance attention to the video frame. - The set of attention weights enhances or suppresses video frames of a set of past video frames based on how much useful information the video frames contain, at least for predicting action predictions for the current video frame. - The annotation for the current video frame includes an annotation selected from the list consisting of outlining objects involved in action prediction, brightening pixels within the region of the current video frame proximate to objects involved in action prediction, obscuring objects involved in action prediction, and blanking out the current video frame. - A set of subsequently observed video frames is weighted with previously determined attention weights. - Weight subsequent observed video frames with previously determined attention weights. - Encoding the extracted feature amounts includes using cross-attention to project variable-length portions of multiple video frames onto a fixed-length learned latent encoding. - Generating an action prediction for the current video frame includes cross-attention between an encoded past based at least on a set of weighted past video frames and a self-attention network output based at least on the encoded extracted feature amounts of a set of current video frames. - Extracting feature amounts uses optical flow.

[0101] Although aspects of the present disclosure have been described from various example aspects in conjunction with associated operations, those skilled in the art will understand that combinations of operations from any number of different examples are also within the scope of the aspects of the present disclosure. Exemplary Operating Environments FIG. 10 is a block diagram of an exemplary computing device 1000 (e.g., a computer storage device) for implementing the aspects disclosed herein, and is generally referred to as computing device 1000. In some examples, one or more computing devices 1000 are provided for an on-premises computing solution. In some examples, one or more computing devices 1000 are provided as a cloud computing solution. In some examples, a combination of on-premises and cloud computing solutions is used. Computing device 1000 is an example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein, whether used alone or as part of a larger set.

[0102] Computing device 1000 should not be construed as having any dependency or requirement related to any one or combination of the illustrated components / modules. The examples disclosed herein may be described in the general context of computer-executable instructions, such as program components, including computer code or machine-usable instructions that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components, including routines, programs, objects, components, data structures, etc., refer to code that performs a particular task or implements a particular abstract data type. The disclosed examples may be implemented in various system configurations, including personal computers, laptop computers, smartphones, mobile tablets, handheld devices, home electronics, special computing devices, etc. The disclosed examples may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network.

[0103] Computing device 1000 includes a bus 1010 that directly or indirectly couples the following devices: computer storage memories 1012, one or more processors 1014, one or more presentation components 1016, input / output (I / O) ports 1018, I / O components 1020, a power supply 1022, and a network component 1024. Although computing device 1000 appears to be depicted as a single device, multiple computing devices 1000 can operate together and share the depicted device resources. For example, memory 1012 may be distributed across multiple devices, or processors 1014 may be housed in different devices.

[0104] Bus 1010 represents what can be one or more buses (such as an address bus, a data bus, or a combination thereof). Although the various blocks in FIG. 10 are shown using lines for clarity, depicting the various components can be achieved with alternative representations. For example, presentation components such as display devices are in some examples I / O components, and some examples of processors have their own memory. Categories such as "workstation," "server," "laptop," "handheld device," etc. are not distinguished as all are assumed to be within the scope of what is referred to as a "computing device" in FIG. 10 and this specification. Memory 1012 can take the form of a computer storage medium as referred to below and functionally provides storage for computer-readable instructions, data structures, program modules, and other data for computing device 1000. In some examples, memory 1012 stores one or more of an operating system, a universal application platform, or other program modules and program data. Thus, memory 1012 can store and access data 1012a and instructions 1012b that are executable by processor 1014 and configured to perform the various operations disclosed herein. In some examples, memory 1012 includes a computer storage medium. Memory 1012 can include any amount of memory related to or accessible by computing device 1000. Memory 1012 can be internal to computing device 1000 (as shown in FIG. 10), external to computing device 1000 (not shown), or both (not shown). Additionally, or alternatively, memory 1012 can be distributed across multiple computing devices 1000, for example, in a virtualized environment where instruction processing is executed on multiple computing devices 1000.For the purposes of this disclosure, "computer storage medium", "computer storage memory", "memory", and "memory device" are synonymous terms for computer storage memory 1012, and none of these terms include a carrier wave or propagated signaling.

[0105] Processor 1014 can include any number of processing units that read data from various entities such as memory 1012 or I / O component 1020. Specifically, processor 1014 is programmed to execute computer-executable instructions for implementing aspects of the present disclosure. The instructions may be executed by the processor, by multiple processors within computing device 1000, or by a processor external to client computing device 1000. In some examples, processor 1014 is programmed to execute instructions as shown in the flowcharts described below and depicted in the accompanying drawings. Further, in some examples, processor 1014 represents an implementation of analog techniques for performing the operations described herein. For example, the operations may be performed by an analog client computing device 1000 and / or a digital client computing device 1000. Presentation component 1016 presents data indications to a user device or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, and the like. Those skilled in the art will understand and appreciate that computer data can be presented in many ways, visually in a graphical user interface (GUI), auditorily through speakers, wirelessly between computing devices 1000, via a wired connection, or otherwise. I / O port 1018 enables computing device 1000 to be logically coupled to other devices including I / O component 1020, some of which may be built-in. Examples of I / O component 1020 include, but are not limited to, microphones, joysticks, game pads, satellite antennas, scanners, printers, wireless devices, and the like.

[0106] Computing device 1000 can operate in a networked environment via network component 1024 using a logical connection to one or more remote computers. In some examples, network component 1024 includes a network interface card and / or computer-executable instructions (e.g., a driver) to operate the network interface card. Communication between computing device 1000 and other devices can occur using any protocol or mechanism via a wired or wireless connection. In some examples, network component 1024 is operable to communicate data wirelessly between devices, or in combination thereof, publicly, privately, or in a hybrid (public and private) manner using a transfer protocol and a short-range communication technology (e.g., near field communication (NFC), Bluetooth (trademark) brand communication, etc.). Network component 1024 communicates with remote resources 1028 (e.g., cloud resources) via network 1030 via wireless communication link 1026 and / or wired communication link 1026a. Various different examples of communication links 1026 and 1026a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed through the Internet.

[0107] Although described in conjunction with the exemplary computing device 1000, the examples of the present disclosure are capable of implementation in a number of other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and / or configurations suitable for use in aspects of the present disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld devices or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set-top boxes, programmable household electronics, cellular phones, mobile computing and / or communication devices in the form of wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, and the like. Such systems or devices can receive input from a user in any manner, including via gesture input, via proximity input (such as by hovering), and / or via voice input, including input from input devices such as a keyboard or pointing device.

[0108] Examples of the present disclosure may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices in software, firmware, hardware, or combinations thereof. Computer-executable instructions may be organized into one or more computer-executable components or modules. In general, program modules include routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types, without limitation. Aspects of the present disclosure may be implemented with any number and arrangement of such components or modules. For example, aspects of the present disclosure are not limited to particular computer-executable instructions, or to particular components or modules shown in the drawings and described herein. Other examples of the present disclosure may include various computer-executable instructions or components with more or less functionality than illustrated and described herein. In examples involving general-purpose computers, aspects of the present disclosure transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

[0109] By way of example and not limitation, computer-readable media includes computer storage media and communication media. Computer storage media includes volatile and non-volatile memory, removable and non-removable memory implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media is tangible and does not include communication media. Computer storage media is implemented in hardware, excluding carrier waves and propagated signals. For the purposes of this disclosure, computer storage media is not the signal itself. Exemplary computer storage media includes the following: hard disks, flash drives, solid state memories, phase change random access memories (PRAMs), static random access memories (SRAMs), dynamic random access memories (DRAMs), other types of random access memories (RAMs), read only memories (ROMs), electrically erasable programmable read only memories (EEPROMs), flash memories or other memory technologies, compact disc read only memories (CD-ROMs), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media typically embodies computer-readable program instructions, data structures, program modules, etc. as a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media.

[0110] The order of execution or performance of operations in the examples of the present disclosure illustrated and described herein is not essential, and in various examples, they may be performed in various order patterns. For example, it is contemplated that performing or implementing a particular operation before, simultaneously with, or after other operations is within the scope of aspects of the present disclosure. When introducing elements of an aspect of the present disclosure or examples thereof, the articles "a", "an", "the", and "said" are intended to mean that one or more of those elements are present. The terms "comprising", "including", and "having" are intended to be inclusive and mean that there may be additional elements other than the recited elements. The term "exampleary" is intended to mean "an example of". The phrase "one or more of A, B, and C" means "at least one of A, and / or at least one of B, and / or at least one of C".

[0111] Although aspects of the present disclosure have been described in detail, it will be apparent that modifications and variations are possible without departing from the scope of the aspects of the present disclosure as defined in the appended claims. Since various changes can be made to the above-described structures, products, and methods without departing from the scope of the aspects of the present disclosure, all matters included in the above description and shown in the accompanying drawings are intended to be construed in an illustrative, rather than a limiting, sense.

Claims

1. Processor and Computer-readable media for storing instructions and A system including the above, wherein the instruction, when executed by the processor, The steps include receiving a video stream containing multiple video frames, A step of grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame, A step of determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how useful the video frames are for predicting the action in the current video frame. The steps include: weighting the aforementioned set of past video frames with the aforementioned set of attention weights to create a weighted set of past video frames; The steps of generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames, It operates in such a way that it does the following: system.

2. The system according to claim 1, wherein the instruction is The steps include generating annotations for the current video frame based on the action prediction for at least the current video frame, The steps include: displaying the current video frame according to the annotations for the current video frame; The system then operates to perform further actions.

3. The system according to claim 1, wherein the step of determining the set of attention weights is: For each video frame in the aforementioned set of past video frames, the step of determining a position-guided gating score. A system that includes this.

4. The system according to claim 1, wherein the plurality of video frames include a set of past frames and, for each individual past frame in the set of past frames, a set of consecutively observed video frames, wherein the set of consecutively observed video frames is newer than the individual past frame, and the instruction is A step of extracting features from the set of past video frames based on at least the set of past frames and the set of video frames observed thereafter, The steps include encoding the extracted feature quantities, The system then operates to perform further actions.

5. A system according to claim 4, wherein the step of extracting features does not use optical flow.

6. The system according to claim 1, wherein the instruction is Steps to perform background suppression The process further operates to perform the action prediction, the action prediction includes confidence, and the step of performing the background suppression is, The step of modifying the confidence of the classifier that generates the action prediction by applying separate emphasis to the action class and the background class, and weighting the less confident video frames more heavily. A system that includes this.

7. The system according to claim 1, wherein the action prediction includes a prediction of no action or a prediction of a class of action selected from a plurality of classes of actions.

8. A computerized method, The steps include receiving a video stream containing multiple video frames, A step of grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame, A step of determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how useful the video frames are for predicting the action in the current video frame. The steps include: weighting the aforementioned set of past video frames with the aforementioned set of attention weights to create a weighted set of past video frames; The steps of generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames, A method that includes this.

9. The method according to claim 8, The steps include generating annotations for the current video frame based on the action prediction for at least the current video frame, The steps of displaying the current video frame in accordance with the annotations for the current video frame, and A method that further includes this.

10. The method according to claim 8, wherein the step of determining the set of attention weights is: For each video frame in the aforementioned set of past video frames, the step of determining a position-guided gating score. Methods that include...

11. The method according to claim 8, wherein the plurality of video frames include a set of past frames and, for each past frame in the set of past frames, a set of consecutively observed video frames, wherein the set of consecutively observed video frames is newer than the past frames, and the method A step of extracting features from the set of past video frames based on at least the set of past frames and the set of video frames observed thereafter, The steps include encoding the extracted features and A method that further includes this.

12. A method according to claim 11, wherein the step of extracting features does not use optical flow.

13. The method according to claim 8, Steps to perform background suppression The further includes, the action prediction includes confidence, and the step of performing background suppression is, The classifier that generates the action predictions is weighted more heavily on the less confident video frames by using separate emphasis on the action class and the background class. Methods that include...

14. A method according to claim 8, wherein the action prediction includes a no-action prediction or a prediction of a class of actions selected from a plurality of classes of actions.

15. One or more computer memory devices storing computer executable instructions, wherein, when executed by a computer, the instructions cause the computer to perform an action, and the action is, The steps include receiving a video stream containing multiple video frames, A step of grouping the plurality of video frames into a set of current video frames and a set of past video frames, wherein the set of current video frames includes the current video frame, A step of determining a set of attention weights for the set of past video frames, wherein the set of attention weights indicates how useful the video frames are for predicting the action in the current video frame. The steps include: weighting the aforementioned set of past video frames with the aforementioned set of attention weights to create a weighted set of past video frames; The steps include generating an action prediction for the current video frame based on at least the weighted set of past video frames and the set of current video frames, One or more computer storage devices, including [the specified element].

16. One or more computer storage devices according to claim 15, wherein the operation is: The steps include generating annotations for the current video frame based on the action prediction for at least the current video frame, The steps include: displaying the current video frame according to the annotations for the current video frame; One or more computer storage devices, further including the following:

17. One or more computer storage devices according to claim 15, wherein the step of determining the set of attention weights is: For each video frame in the aforementioned set of past video frames, the step of determining a position-guided gating score. One or more computer storage devices, including [the specified element].

18. One or more computer storage devices according to claim 15, wherein the plurality of video frames include a set of past frames and, for each past frame in the set of past frames, a set of consecutively observed video frames, wherein the set of consecutively observed video frames is newer than the past frames, and the operation is A step of extracting features from the set of past video frames based on at least the set of past frames and the set of video frames observed thereafter, The steps include encoding the extracted features and One or more computer storage devices, further including the following:

19. One or more computer storage devices according to claim 15, wherein the operation is: Steps to perform background suppression The further includes, the action prediction includes confidence, and the step of performing background suppression is, The classifier that generates the action predictions is weighted more heavily on the less confident video frames by using separate emphasis on the action class and the background class. One or more computer storage devices, including [the specified element].

20. One or more computer storage devices according to claim 15, wherein the action prediction includes a no-action prediction or a prediction of a class of action selected from a plurality of classes of actions.

Images