Adaptive motion recognizer for video

Abstract

An adaptive action recognizer for video is provided that performs a multiscale spatio-temporal decomposition on a video to generate low complexity videos. The adaptive action recognizer has multiple processing pipelines, one for each video complexity level, each with a different computational cost. The adaptive action recognizer applies a decision scheme that supports using low average computational cost while maintaining high accuracy.

Description

[0001] Cross-referencing related applications

[0002] This application claims priority to U.S. Patent Application No. 62 / 848,535, filed May 15, 2019, entitled “GESTURE AND ACTION RECOGNITION IN VIDEOS WITH A RECEPTIVE FIELD ENHANCING ENSEMBLE NETWORK (RFEEN),” and also claims priority to U.S. Patent Application No. 16 / 874,478, filed May 14, 2020, entitled “ADAPTIVE ACTION RECOGNIZER FOR VIDEO,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application generally relates to video analytics, and more specifically, to an adaptive action recognizer for video. Background Technology

[0004] Automatic motion recognition in videos is crucial for many digital video applications in robotics, manufacturing, industrial applications, and human-machine interface (HMI) applications that involve motion recognition. However, existing technologies require significant resources and may not be suitable for resource-constrained scenarios. Therefore, more efficient technologies are needed for automatic motion recognition in videos. Summary of the Invention

[0005] This application provides an adaptive action recognizer for video, which performs multi-scale spatiotemporal decomposition on the video to generate low-complexity video. The adaptive action recognizer has multiple processing paths, one for each video complexity level, and each processing path has a different computational cost. The adaptive action recognizer applies a decision-making scheme that supports high accuracy while using low average computational cost.

[0006] The adaptive action recognizer can be implemented using a neural network. The multiple processing pathways can include a coarse pathway (C), a medium (or medium-fine or finer) pathway (M), and a fine (or finest) processing pathway, providing coarse-to-fine (C2F) processing pathways for different video complexity levels. Video complexity can vary according to resolution and / or frame rate. The neural network can thus be considered a multi-resolution or multi-complexity network. Each processing pathway includes a deep convolutional neural network (CNN) that performs action recognition on a video. Depending on the pathway, the pathway can also include a video decomposition module for spatial and / or temporal decomposition and a feature fusion module. A decision scheme for action recognition supports using a coarse processing pathway with low average computational cost during inference and using a fine processing pathway with high average computational cost in a data-dependent manner for action recognition to maintain accuracy while reducing computational cost.

[0007] Generating coarse videos and using a trained deep CNN for action recognition on these coarse videos allows generating activation maps with large effective receptive fields in the deep CNN of a high resolution pathway to complement activation maps with small effective receptive fields in a low resolution pathway. The pathways can be combined in an ensemble, which can be implemented as a relatively simple but effective end-to-end learnable ensemble that combines the multiple processing pathways. The decision scheme directs the network to move to a more fine, computationally more expensive pathway only when needed.

[0008] The adaptive action recognizer can operate at different computational cost budgets. A controller can provide a control scheme that controls a coarse-exit hyperparameter for determining a processing pass used by the adaptive action recognizer, such that the adaptive action recognizer operates at a budgeted computational cost allocated from outside by a device or system using the adaptive action recognizer. A separate moving average based prediction method can be used to estimate a complexity level of a video that the adaptive action recognizer can process, use a regression predictor to predict an optimal value of the coarse-exit hyperparameter, and continuously allocate the optimal coarse-exit hyperparameter to the adaptive action recognizer, so that the adaptive action recognizer operates at a budgeted computational cost that is close to the input budgeted computational cost. The control scheme can be used in a single device or extended to a group of multiple devices, such as in an Internet-of-Things (IoT) connected device environment, such as in a case where a phone, a laptop, and a TV are all connected, the required computation in the TV is done in the laptop and the phone, and then sent back to the TV. This provides a control system to operate the adaptive action recognizer across multiple devices.

[0009] According to a first embodiment of the first aspect of the application, there is provided a method for recognizing actions in a video. A high complexity video is processed to generate a medium complexity video and a low complexity video. A first action recognizer processes the low complexity video to recognize actions in the low complexity video. When it is determined that a probability associated with a highest probability action recognized by the first action recognizer is less than a threshold, a second action recognizer processes the medium complexity video to recognize actions in the medium complexity video. When it is determined that a probability associated with a highest probability action recognized by the second action recognizer is less than the threshold, a third action recognizer processes the high complexity video to recognize actions in the high complexity video.

[0010] In some or all examples of the first embodiment of the first aspect, the first action recognizer, the second action recognizer, and the third action recognizer each outputs a set of probabilities, one probability associated with each action in a class of actions.

[0011] In some or all examples of the first embodiment of the first aspect, the method further comprises: when determining that the probability associated with the highest probability action identified by the second action recognizer is less than the threshold, combining the probability sets respectively output by the first action recognizer, the second action recognizer and the third action recognizer to generate a probability set; when determining that the probability associated with the highest probability action in the probability set is equal to or greater than the threshold, outputting the highest probability action in the probability set.

[0012] In some or all examples of the first embodiment of the first aspect, the first action recognizer, the second action recognizer and the third action recognizer all comprise a deep convolutional neural network (CNN) comprising a plurality of fully connected layers, and the combining the probability sets respectively output by the first action recognizer, the second action recognizer and the third action recognizer to generate a probability set comprises: concatenating the probability sets respectively output by the first action recognizer, the second action recognizer and the third action recognizer; and a reparameterized fully connected layer generating the probability set from the concatenated probability sets.

[0013] In some or all examples of the first embodiment of the first aspect, the method further comprises: when determining that the probability of the highest probability action identified by the first action recognizer is equal to or greater than the threshold, outputting the probability set identified by the first action recognizer; and when determining that the probability of the highest probability action identified by the second action recognizer is equal to or greater than the threshold, outputting the probability set identified by the second action recognizer.

[0014] In some or all examples of the first embodiment of the first aspect, the threshold is a hyperparameter T based on a budgeted computational resource B.

[0015] In some or all examples of the first embodiment of the first aspect, the hyperparameter T is optimized according to the following equation:

[0016]

[0017] wherein, B∈[f C ,f C2F ],f C represents an actual computational cost of the first action recognizer generating the probability set for a single identification, f C2F represents an actual computational cost of the second action recognizer generating the probability set for a single identification, f av represents a running average computational resource, T avThis represents the average value of the recognitions up to r recognitions.

[0018] In some or all examples of the first embodiment of the first aspect, the first action recognizer, the second action recognizer and the third action recognizer are respectively used to recognize a class of actions in a video, and the first action recognizer, the second action recognizer and the third action recognizer are used to recognize a class of actions in a video at low video complexity level, medium video complexity level and high video complexity level.

[0019] In some or all examples of the first embodiment of the first aspect, the threshold is a hyperparameter T based on budgeted computational resources B, which are determined based on the available computational resources of a group of devices jointly performing the method, one of which is included in the group of devices, and the available computational resources of the device are included in the budgeted computational resources when it is determined that the available computational resources of the device are equal to or greater than the computational cost of the first action recognizer.

[0020] In some or all examples of the first embodiment of the first aspect, the first action recognizer, the second action recognizer and the third action recognizer all include a deep convolutional neural network (CNN).

[0021] In some or all examples of the first embodiment of the first aspect, the high-complexity video is spatiotemporally decomposed to generate the medium-complexity video and the low-complexity video.

[0022] In some or all examples of the first embodiment of the first aspect, the spatiotemporal decomposition includes: downsampling the high-complexity video to generate the medium-complexity video and the low-complexity video, which simultaneously have low resolution and low frame rate.

[0023] In some or all examples of the first embodiment of the first aspect, the downsampling includes: reducing the high-complexity video by a factor of 2 using bilinear interpolation to generate the medium-complexity video, and reducing the medium-complexity video by a factor of 2 using the bilinear interpolation to generate the low-complexity video.

[0024] In some or all examples of the first embodiment of the first aspect, the resolution of the high-complexity video is width (W) pixels × height (H) pixels, and the frame rate is X frames per second (FPS); the resolution of the medium-complexity video is W / 2 pixels × H / 2 pixels, and the frame rate is X / 2 FPS; the resolution of the medium-complexity video is W / 4 pixels × H / 4 pixels, and the frame rate is X / 4 FPS.

[0025] According to a second embodiment of the first aspect of this application, a method is provided for analyzing a video to identify actions in the video. The actions may include gestures, etc. The method includes: receiving the video, processing the video to generate a coarse video, and processing the video to generate a finest video. The method further includes: using a first action recognizer to identify actions, such as gestures, in the coarse video; determining a first confidence level that the first action recognizer has identified an action; and when the first confidence level is determined to be no greater than a threshold, using a finest action recognizer to identify actions in the finest video. The method further includes: determining a second confidence level that the finest action recognizer has identified an action; and when the second confidence level is determined to be greater than the threshold, providing an indication of the action.

[0026] In some or all examples of the second embodiment of the first aspect, the first action recognizer includes a first neural network comprising a plurality of fully connected layers. In some or all examples, the first neural network includes a deep convolutional neural network.

[0027] In some or all examples of the second embodiment of the first aspect, the coarse video includes a video with relatively low resolution, which includes frames with a size of 32 pixels × 32 pixels or smaller.

[0028] In some or all examples of the second embodiment of the first aspect, the coarse video includes video with relatively low frames per second, which includes video with 8 frames per second or less.

[0029] In some or all examples of the second embodiment of the first aspect, the finest video includes a relatively high-resolution video comprising frames of size 64 pixels × 64 pixels or smaller.

[0030] In some or all examples of the second embodiment of the first aspect, the finest video includes video with relatively high frames per second, which includes video with 16 frames per second.

[0031] According to another aspect of this application, a computing device is provided, the computing device including a processing system comprising at least one processor and a memory. The memory tangibly stores executable instructions. When the processing system executes the executable instructions, it causes the computing device to perform the method described above.

[0032] According to another aspect of this application, a non-transitory machine-readable medium is provided, tangibly storing executable instructions for execution by a processing system of a computing device. The processing system includes at least one processor. When the processing system executes the executable instructions, it causes the processing system to perform the method described above.

[0033] According to another aspect of this application, an adaptive action recognizer is provided. The adaptive action recognizer includes: a video decomposer for processing high-complexity video to generate medium-complexity video and low-complexity video; a first action recognizer for processing the low-complexity video to recognize actions in the low-complexity video; a second action recognizer for processing the medium-complexity video to recognize actions in the medium-complexity video when the probability associated with the highest probability action recognized by the first action recognizer is less than a threshold; and a third action recognizer for processing the high-complexity video to recognize actions in the high-complexity video when the probability associated with the highest probability action recognized by the second action recognizer is less than the threshold.

[0034] In some or all examples of the adaptive action recognizer, the first action recognizer, the second action recognizer, and the third action recognizer each output a probability set, with one probability associated with each action in a class of actions.

[0035] In some or all examples of the adaptive action recognizer, the adaptive action recognizer further includes: a controller configured to: output the highest probability action recognized by the first action recognizer, the second action recognizer, or the third action recognizer when it is determined that the probability associated with the highest probability action recognized by the first action recognizer, the second action recognizer, or the third action recognizer is equal to or greater than the threshold.

[0036] In some or all examples of the adaptive action recognizer, the high-complexity video is spatiotemporally decomposed to generate the medium-complexity and low-complexity videos.

[0037] In some or all examples of the adaptive action recognizer, the first action recognizer, the second action recognizer, and the third action recognizer all include a deep convolutional neural network (CNN).

[0038] Other aspects and features of this application will be apparent to those skilled in the art when the following description of specific implementations of this application is reviewed in conjunction with the accompanying drawings. Attached Figure Description

[0039] Figure 1 A processing unit suitable for practicing the teachings of this application is shown.

[0040] Figure 2 An adaptive action recognizer including an analysis module is shown in an embodiment of this application.

[0041] Figure 3 The embodiments provided in this application are shown. Figure 2 The hierarchical action recognizer in the analysis module.

[0042] Figure 4 This application illustrates the use of one embodiment of the present application. Figure 2 The method of adaptive action recognizer for recognizing actions.

[0043] Figure 5A An embodiment of this application is shown for implementing Figure 2 An adaptive action recognizer with enhanced receptive field ensemble network.

[0044] Figure 5B An embodiment of this application is shown for implementing Figure 2 An adaptive action recognizer with enhanced receptive field ensemble network.

[0045] Figure 6 The operational features of the adaptive action recognizer provided in this application are shown.

[0046] Figure 7 Exemplary operational features of the adaptive action recognizer provided in this application are shown.

[0047] Figure 8A The comparison between the theoretical receptive field and the effective receptive field of the video at its original resolution is shown.

[0048] Figure 8B It shows Figure 8A A comparison between the theoretical receptive field and the effective receptive field of the video after downsampling by a factor of 2.

[0049] Figure 9 An optimization provided by one embodiment of this application is illustrated. Figure 2 The method of operating the adaptive action recognizer.

[0050] Figure 10This application illustrates a workflow for determining budget computing resources based on the computing status of multiple devices, according to one embodiment of the present application.

[0051] Figure 11 The workflow of multi-device scheduling provided in one embodiment of this application is illustrated.

[0052] Figure 12 A table is shown comparing the performance of various implementations of the adaptive action recognizer of this application with state-of-the-art methods based on known datasets.

[0053] Figure 13 The controller in the adaptive action recognizer is able to maintain a cost graph based on the Something-something V2 dataset using hyperparameter T. Detailed Implementation

[0054] This application is made with reference to the accompanying drawings, which illustrate embodiments. However, many different embodiments may be used, and therefore the description should not be construed as limiting the embodiments set forth herein. Rather, these embodiments are provided to make this application comprehensive and complete. Wherever possible, the same reference numerals are used in the drawings and the following description to refer to the same elements, and apostrophes are used to indicate similar elements, operations, or steps in alternative embodiments. The separate blocks of the systems and devices shown or the separate functional elements shown do not necessarily require that these functions be physically independent, as communication between these elements can be achieved through message sending and receiving, function calls, shared storage, etc., without any such physical independence. Therefore, functions do not need to be implemented in physically or logically independent platforms, but are shown separately herein for ease of explanation. Different devices may have different designs, so while some devices implement some functions in fixed-function hardware, others may implement those functions in a programmable processor with code obtained from a machine-readable medium. Finally, an element may be one or more unless the context explicitly or inherently indicates otherwise.

[0055] The terms "gesture" and "action" are used in this invention. A gesture is a subclass of an action. An action can be a gesture, and all gestures are actions. A gesture is an action performed by a person for communication (e.g., for communication with an HMI). Gestures are performed by body parts, such as the hand, head, leg, foot, etc. Examples of gestures include giving a thumbs-up and nodding. Actions do not need to be performed by a person, nor do they need to be performed for communication. Examples of actions include walking, pouring a glass of water, riding a bicycle, driving, etc.

[0056] Figure 1A processing unit 100 suitable for practicing the teachings of this application is shown. The processing unit 100 includes a processing system comprising at least one processor 102 interacting with a memory 104. The memory 104 stores received video to be analyzed and data for use by an adaptive motion recognizer 200. Figure 2 Code 106. It should be understood that the video to be analyzed may have been previously received and stored in memory 104 for later processing. Alternatively, the video to be analyzed may be received during processing for real-time processing.

[0057] The processing unit 100 is implemented in a host computing device. Examples of host computing devices include, but are not limited to, security cameras or security video systems, line scan cameras, personal cameras or camera peripherals, smartphones, tablets, personal computers such as desktop or laptop computers, smart glasses or other smart displays mounted on head-mounted devices, smart speakers, robots or unmanned aerial vehicles (UAVs), etc.

[0058] Figure 2 An implementation of an adaptive motion recognizer 200 implemented when processor 102 executes code 106 is shown. The adaptive motion recognizer 200 has multiple processing paths, which vary depending on the video complexity processed by each path. Video complexity can vary based on one or both of image resolution and frame rate. The video received by the adaptive motion recognizer 200 can be spatially and / or temporally decomposed from an initial high resolution and an initial high (or normal) frame rate into one or more low-complexity videos with low resolution and / or low frame rate. For each video complexity level, a dedicated motion recognizer is provided, trained to identify the actions corresponding to each video complexity level.

[0059] The adaptive motion recognizer 200 includes a controller 212, a video decomposer 214, and an analysis module 216 including a hierarchical motion recognizer 228. Optionally, the controller 212 can be considered distinct from but coupled to the adaptive motion recognizer 200. The adaptive motion recognizer 200 may also include a scheduler 218. The video decomposer 214 receives input video, such as video in RGB (red, green, blue) format. The video decomposer 214 includes multiple video generation modules for generating videos with different levels of video complexity based on the input video. As mentioned above, the video complexity can vary depending on one or both of the image resolution and frame rate. The number of video generation modules and the number of video complexity levels can vary. The video generation modules perform downsampling operations to generate videos with low resolution and / or low frame rates. The downsampling operation is performed by scaling down the input video by a scaling factor using bilinear interpolation or other suitable techniques. The downsampling operation can be repeated n–1 times, resulting in n videos with different resolutions. The video generation modules perform downsampling operations starting from either the input video or the output of another video generation module.

[0060] exist Figure 2 In the illustrated embodiment, the video decomposer 214 includes two (2) video generation modules to perform spatiotemporal decomposition on the original input video and generate downsampled videos with two video complexity levels. The first video generation module 202 is used to generate a medium-resolution video with medium frames per second (FPS) from the original input video. For example, a medium-resolution video is a video comprising frames with a resolution of W / 2 pixels × H / 2 pixels, where W × H are the width (W) and height (H) of the original input video in pixels. For example, a medium-FPS video is a video with a frame rate of X / 2, where X is the frame rate of the original input video in FPS. The second video generation module 204 is used to generate a low-resolution video with low FPS from the original input video. For example, a low-resolution video is a video comprising frames with a resolution of W / 4 pixels × H / 4 pixels, where W × H is the resolution of the original input video in pixels. For example, a low-FPS video is a video with a frame rate of X / 4, where X is the frame rate of the original input video in FPS.

[0061] Video generation modules 202 and 204 perform downsampling operations to generate videos with both low resolution and low frame rate. The downsampling operation is performed by reducing the input video by a factor of 2 using bilinear interpolation. Video generation modules 202 and 204 can be cascaded, where the output of the first video generation module 202 is the input of the second video generation module 204, allowing for progressive downsampling of the high-resolution, high-frame-rate input video to generate a low-complexity video with both low resolution and low frame rate.

[0062] In other embodiments, more than two video generation modules may be provided. For example, in another embodiment, five video generation modules may exist. A third video generation module may be used to generate a medium-resolution, low-FPS video. A fourth video generation module may be used to generate a medium-resolution, normal-FPS video based on the received video. A fifth video generation module may be used to generate a high-resolution (i.e., original or initial resolution) and low-FPS video based on the received video. Similar to the above, these video generation modules may operate in series, where the output of one video generation module is the input of another, so that the high-resolution, high-frame-rate input video is progressively downsampled to generate a low-complexity video with low resolution and / or low frame rate.

[0063] Figure 3 Detailed illustration Figure 2 The hierarchical motion recognizer 228 is located in the analysis module 216. The hierarchical motion recognizer 228 includes a motion recognizer decision unit 302 and is used to access the input video and the low-complexity decomposed video with low resolution and / or low frame rate generated by the video decomposer 214, as well as to receive video from the controller 212. The hierarchical motion recognizer 228 selectively accesses the input video and the decomposed video generated as needed. Optionally, the controller 212 can be used to access the input video and the decomposed video generated by the video decomposer 214, and selectively provide video to the hierarchical motion recognizer 228 as needed.

[0064] Action recognizer decision unit 302 can be connected to provide received video to one action recognizer selected from a plurality of action recognizers. Each video generation module and each video complexity level provides an action recognizer. In the illustrated embodiment, the plurality of action recognizers includes a first action recognizer 304, a second action recognizer 308, and a final action recognizer 312. These action recognizers can be implemented using neural networks such as deep convolutional neural networks (CNNs). Action recognizer decision unit 302 is used to receive the probability value or highest probability value of each action in a class of actions and a class of actions with the highest probability value, and to output an indication of the action based on the action with the highest probability value. As mentioned above, such actions can include or consist of gestures, such as swiping the hand from left to right, swiping the hand from right to left, raising a thumb, and rotating the hand counterclockwise, etc. Therefore, the indication of the action can be an indication of one of these gestures.

[0065] Figure 4 An embodiment of this application is shown, provided by Figure 2The method for automatically recognizing actions is executed by the analysis module 216 in the adaptive action recognizer 200. In operation 402, the first action recognizer 304 receives low-complexity video, such as low-resolution and low-FPS video, through the first video generation module 204 and attempts to recognize an action, such as a gesture. The first action recognizer 304 generates a probability value p for the action with the highest probability. The probability value p is provided by the first action recognizer 304 to the action recognizer decision unit 302.

[0066] In operation 404, the action recognition decision unit 302 determines whether the first action recognition unit 304 has recognized an action by judging whether the probability value p generated by the first action recognition unit 304 is equal to or greater than a threshold T. The threshold T can be controlled by the controller 212, etc., which will be described in more detail below. When it is determined that the first action recognition unit 304 has recognized an action, in operation 418, the action recognition decision unit 302 outputs an indication of the recognized action. The indication of the recognized action can be used by the host computing device in further processing, for example, as part of the HMI processing logic.

[0067] When it is determined that the first action recognizer 304 has not yet recognized an action (i.e., the probability value p output by the first action recognizer 304 is not equal to or greater than the threshold T), in operation 406, the second action recognizer 306 receives a high-complexity video, such as a medium-resolution and medium-FPS video, through the second video generation module 204, and attempts to recognize an action, such as a gesture. The second action recognizer 306 generates a probability value p for the action with the highest probability. The probability value p is provided by the second action recognizer 306 to the action recognizer decision unit 302.

[0068] In operation 408, the action recognizer decision-maker 302 determines whether the probability value p generated by the second action recognizer 306 is equal to or greater than the threshold T, thus determining whether the second action recognizer 306 has recognized an action. When it is determined that the second action recognizer 306 has recognized an action, in operation 418, the action recognizer decision-maker 302 outputs an indication of the recognized action.

[0069] When it is determined that the second action recognizer 306 has not yet recognized an action (i.e., the probability value p output by the second action recognizer 306 is not equal to or greater than the threshold T), in operation 410, the final action recognizer 308 receives another high-complexity video, such as a high-resolution and high-FPS input video, and attempts to recognize an action, such as a gesture. The final action recognizer 308 generates a probability value p for the action with the highest probability. The probability value p is provided by the final action recognizer 308 to the action recognizer decision unit 302.

[0070] In operation 412, the action recognizer decision-maker 302 determines whether the probability value p generated by the final action recognizer 308 is equal to or greater than the threshold T, thus determining whether the final action recognizer 308 has recognized an action. When it is determined that the final action recognizer 308 has recognized an action, in operation 418, the action recognizer decision-maker 302 outputs an indication of the recognized action.

[0071] When it is determined that the final action recognizer 308 has not yet recognized an action (i.e., the probability value p output by the final action recognizer 308 is not equal to or greater than the threshold T), in operation 420, the action recognizer decision unit 302 outputs an indication that no action has been recognized. This indication that no action has been recognized can be used by the host computing device in further processing, for example, as part of the HMI processing logic.

[0072] In other embodiments, there may be more than three motion recognizers. In these embodiments, there are multiple intermediate motion recognizers, each of which processes video with increasing video complexity when processing occurs between a first motion recognizer 302 that processes low-resolution and low-FPS video and a final motion recognizer 308 that processes high-resolution and high-FPS input video.

[0073] Action recognizers 304, 308, and 312 are all trained together using the same videos at different levels of video complexity. Videos of different complexities are input into the corresponding action recognizers 304, 308, and 312, respectively. Low-complexity videos are input into the first action recognizer 304, medium-complexity videos into the second action recognizer 308, and high-complexity videos into the third action recognizer 312. For each action recognizer, a forward propagation algorithm generates an output, and the loss is calculated by comparing this output with a class of real actions in the video. The calculated losses are backpropagated through action recognizers 304, 308, and 312 or traced back to various layers of the relevant neural network. The backpropagation algorithm for the loss is used in conjunction with a stochastic gradient descent algorithm to update the parameters in each neural network. The losses of all action recognizers can be accumulated into a multiple loss function, which can be optimized during training.

[0074] Figure 5A An embodiment of this application is shown for implementing Figure 2The adaptive action recognizer uses a receptive field enhancing ensemble network (RFEEN) 500. RFEEN 500 includes a video decomposer 214 that downsamples the input video to the spatial or temporal domain, or both. In the illustrated embodiment, the video decomposer 214 performs spatial and temporal decomposition, also known as spatiotemporal decomposition, in two stages to generate medium-complexity and low-complexity videos from a high-complexity input video. In the illustrated example, the input video has an image resolution of W (pixels) × H (pixels) and a frame rate of X (FPS). The medium-complexity video has an image resolution of W / 2 pixels × H / 2 pixels and a frame rate of X / 2 FPS. The low-complexity video has an image resolution of W / 4 pixels × H / 4 pixels and a frame rate of X / 4 FPS.

[0075] In some examples, video decomposer 214 performs a downsampling operation to generate a video with both low resolution and low frame rate. The downsampling operation is performed by reducing the input video by a factor of 2 using bilinear interpolation. The downsampling operation is repeated n-1 times, resulting in n video complexity levels. Video generation modules 202 and 204 can be cascaded, where the output of the first video generation module 202 is the input of the second video generation module 204, so that the high-resolution, high-frame-rate input video is progressively downsampled to generate a low-complexity video with both low resolution and low frame rate.

[0076] RFEEN 500 comprises multiple deep CNNs, uniformly designated by reference numeral 504, which implement a hierarchical action recognizer 228. RFEEN 500 includes a deep CNN for each video complexity level processed by RFEEN 500, each deep CNN being connected and receiving a video source of the corresponding video complexity level as input. Each CNN comprises multiple fully connected layers and is trained to perform action recognition on videos of the corresponding video complexity level. The deep CNN can be any deep CNN used to operate on videos of different video complexities, such as densely sampled or sparsely sampled (e.g., stride sampling), and to recognize actions, such as gestures. For example, a suitable CNN architecture is the temporal shift module (TSM) described in "Temporal shift module for efficient video understanding," published by Ji Lin, Chuang Gan, and Song Han in 2018 on the arXiv preprint arXiv:1811.08383.

[0077] In the example shown, there are three deep CNNs: a deep CNN 504-1 for high-complexity input videos (also known as fine-grained videos "F"), a deep CNN 504-2 for medium-complexity input videos (also known as moderately fine or medium-fine-grained videos "M"), and a deep CNN 504-3 for low-complexity input videos (also known as coarse-grained videos "C"). The fine-grained path includes the first deep CNN 504-1. The medium-fine-grained path includes the first video generation module 202 and the second deep CNN 504-2. The coarse-grained path includes the second video generation module 204 and the third deep CNN 504-3.

[0078] Each deep CNN 504 defines multiple convolutional kernels, each including a weight matrix for recognizing actions in the video. Each convolutional kernel has a receptive field, which refers to the input region that influences the kernel, i.e., the video region affecting the kernel. A convolutional kernel can only learn from the portions of the input that influence it. Therefore, giving convolutional kernels larger receptive fields allows them to learn better representations of the input data. However, the effective receptive field of convolutional kernels in deep CNNs is significantly reduced compared to the theoretical receptive field; the ratio of effective receptive field to theoretical receptive field is approximately... Where N is the number of layers between the input and the convolutional kernel. The inventors believe that using the complete image without center cropping can improve accuracy because pixels far from the center in a frame can still provide valuable information in recognizing actions in the video.

[0079] The inventors believe that a large effective receptive field can help improve the accuracy of action recognition. Spatiotemporal scaling (e.g., size reduction) of the input video, performed in the manner described above, compensates for the decrease in effective receptive field compared to the theoretical receptive field. Specifically, in Figure 5A In the described embodiment, the input video is downsampled progressively by a factor of 2 in both the spatial and temporal domains. For an input video of height H and the lowest resolution path reaching a receptive field of depth i and height H, the effective receptive field E of depth i is... i Approximately equal to Since the same receptive field size on the input video downsampled by a factor of 2 covers twice the input video region in each spatiotemporal dimension, the E of the depth i used for the downsampled video path becomes formal, Where n∈Z:n∈[1,N], n=1 represents the original input video without downsampling. Although increasing the receptive field coverage of the convolution kernel will reduce the resolution, this loss can be avoided by processing the original full-resolution input video. Figure 8A The comparison between the theoretical receptive field and the effective receptive field of the video at its original resolution is shown. Figure 8B It showsFigure 8A A comparison between the theoretical receptive field and the effective receptive field of the video after downsampling by a factor of 2.

[0080] The final layer of each deep CNN504 applies a softmax function to the output real-valued vector. The real-valued vector output by each deep CNN504 is a probability vector p of a class (or group) of actions. The probability vector p consists of a set of values ​​ranging from 0 to 1, representing the probability that a corresponding action will appear in the video. The action with the highest probability value in the probability vector p (i.e., a value close to 1) is the best match and most likely action determined by each deep CNN in the video.

[0081] The outputs (e.g., features) of the deep CNN 504 in each path are fused by the feature fusion block 508. The feature fusion block 508 concatenates the pre-softmax outputs of each deep CNN 504 through cascaded blocks 508-1, and fuses these accumulated features through reparameterized fully connected (FC) layers 508-2 to generate intermediate output I. C2F The output O of the reparameterized FC layer 508-2 C2F This is called the C2F ensemble, which receives the output as the input to decision block 302-1.

[0082] The reparameterized FC layer 508-2 is trained along I during training. C2F A more important feature is that it adaptively incentivizes the gradient flow. Intuitively, for a specific output node of FREEN 500 (e.g., an output node representing a "move hand from right to left" gesture), there may be some I... C2F Features are more important than others (e.g., the corresponding nodes for "moving hand," "right to left," and "left to right" in each processing path are more important than other nodes). Adaptively activating gradient flow along these important nodes results in more gradient backpropagation along all learnable deep CNN parameters at nodes with greater influence. This can be achieved by adaptively modifying each weight (w) of the reparameterized FC layer 508-2:

[0083] ω r =0.5*[((2-β)*ω) o1 +(β*ω) o3 (1)

[0084]

[0085] Where 'o' represents the Hadamard power, if the above reparameterization is not applied. Let w represent the backpropagation gradient of w, and β represent the hyperparameters, where... β∈[0,2], Let w represent the backpropagation gradient of w. The larger the magnitude of w, the greater the backpropagation gradient of w if any β > 0. r The larger the magnitude of the gradient, the larger the weight value, and the greater the backpropagation gradient. The larger the value of β, the more it supports this asymmetric amplitude gain, which disappears when β = 0. It is worth noting that since this weight reparameterization only occurs during training, it does not increase computational cost during inference compared to a normal fully connected layer.

[0086] Figure 5B Another embodiment of this application is shown for implementing Figure 2 The RFEEN 550 is an adaptive action recognizer. RFEEN 550 is similar to the RFEEN 500 described above. However, RFEEN 550 includes two feature fusion blocks, 508-1 and 508-2. Feature fusion block 508-1 concatenates the pre-softmax outputs of each deep CNN 504 (i.e., for coarse, intermediate, and fine pathways) through cascade block 508-11, and fuses these accumulated features through a reparameterized FC layer 508-12 to generate an intermediate output I. C2F Receives the output O of the reparameterized FC layer 508-22. C2F As input to decision block 302-1, feature fusion block 508-2 cascades the pre-softmax outputs of deep CNNs 504-2 and 504-3 for coarse and intermediate pathways via cascade block 508-21, and fuses these accumulated features through a reparameterized FC layer 508-22 to generate intermediate output I. C2M The output O of the reparameterized FC layer 508-22 C2M This is called the C2M set, which receives the output as the input to decision block 302-2.

[0087] The action recognizer decision unit 302 applies a decision scheme to generate action recognition outputs (O) for RFEEN 500 and 550 based on a cascaded coarse-exit scheme. This cascaded coarse-exit scheme encourages the recognition of actions output by the coarse path during inference. The action recognizer decision unit 302 includes multiple decision blocks 302-1, 302-2, and 302-3.

[0088] Combining Freen 500, decision block 302-3 determines the highest probability O in the probability set predicted by depth CNN 504-3 on the coarse path. CIs it equal to or greater than the hyperparameter T? If yes, an action has been identified, and the action recognizer decision 302 outputs an indication of the corresponding action or a class of actions. If no, the intermediate path is used to process and analyze the medium-complexity video, and then decision block 302-2 determines the highest probability O in the probability set predicted by the deep CNN 504-2 on the intermediate path. M Is it equal to or greater than the hyperparameter T? If yes, an action has been identified, and the action recognizer decision 302 outputs an indication of the corresponding action or a class of actions. If no, the fine-grained path is used to process and analyze the high-complexity video (fine-grained video), and its result is combined with the results of the coarse-grained and intermediate-grained paths to generate a set output O. C2F Decision block 302-1 determines whether the highest probability in the probability set is equal to or greater than the hyperparameter T. If yes, an action has been identified, and the action recognizer decision 302 outputs an indication of the corresponding action or a class of actions. If no, the action recognizer decision 302 outputs an indication that no action was detected. Optionally, the action recognizer decision 302 may output the highest probability O in the probability set predicted by the depth CNN504-1 on the ensemble path. C2F The corresponding action or a class of actions, regardless of whether the highest probability is equal to or greater than the hyperparameter T.

[0089] Combining FREEN 550, decision block 302-2 determines the output O from the coarse to the medium set. C2M The highest probability in the probability set is equal to or greater than the hyperparameter T. If so, an action has been identified, and the action recognizer decision 302 outputs an indication of the corresponding action or a class of actions. If not, the fine-grained path is used to process and analyze the high-complexity video (fine-grained video), and its result is combined with the results of the coarse-grained and intermediate-grained paths to generate the ensemble output O. C2F As mentioned above.

[0090] Alternatively, in other embodiments, the non-ensemble output O of deep CNNs 504-1, 504-2, and 504-3 C O M and O F It can be used in decision blocks 302-1, 302-2 and 302-3, so that the output of the action recognizer decision 302 is based on coarse path, medium path or fine path, and there is no set.

[0091] The action recognizer decision unit 302 encourages the use of coarse-grained paths that have low average computational cost while maintaining high accuracy. Inference begins with a forward propagation along the coarsest path to achieve fast inference. To ensure accurate action recognition on the coarsest path, action recognition is only performed if the softmax output p NAction recognition is performed when the value is greater than or equal to T, where T represents the hyperparameter controlled externally by the controller 212 from the RFEEN 500. The hyperparameter T ranges from 0 to 1.

[0092] The hyperparameter T is a free parameter that allows the FreeN 500 to operate in a continuously flexible cost-accuracy balance, limited only by the cost-accuracy balance point of the coarsest and finest processing paths. The hyperparameter T is related to computational cost and the probabilities of the outputs of the deep CNN 504 (and FreeN 500). Generally, as the value of the hyperparameter T increases, the probabilities of the outputs of the deep CNN and FreeN 500 also increase (i.e., the confidence of action recognition increases), the computational cost of FreeN 500 increases, and the likelihood of FreeN 500 using finer (or more complex) processing paths increases. There is a complex non-linear relationship between the hyperparameter T and the probabilities of the outputs of the deep CNN 504 (and FreeN 500), influenced by factors such as video quality and complexity (resolution, frame rate, scene characteristics), action complexity, network configuration, and network training.

[0093] To account for the possibility of overconfidence in the softmax output probability and the possibility that the softmax output may not be well calibrated against the uncertainty metric, the softmax output p can be adjusted using the global training accuracy context. N As shown below:

[0094]

[0095] in, p represents the adjusted softmax output of the coarsest path C, p represents the unadjusted softmax output of that path, and e represents the unadjusted softmax output of that path. N Let represent the training misclassification rate of the coarsest path, and e1 represent the training misclassification rate at the end of the C2F set. Similar to the coarsest path example above, the finer paths in RFEEN 500 also have an adjusted softmax score, and are only invoked when the coarse exit in the coarsest path fails, i.e., p N <T。

[0096] The hyperparameter T can be based on the budgeted computing resources B, which can be received as input from the host computing device or scheduler 216. Computational resources are a measure of one or more available computing resources, and can be the budgeted floating-point operations per second (FLOPS). Controller 212 can continuously modify the hyperparameter T as follows:

[0097]

[0098] Where, B∈R:B∈[f C ,f C2F ], f C The coarse path C represents the actual computational cost (e.g., FLOPS) used in a single identification, f C2F f represents the actual computational cost of generating a C2F set in a single identification process. av T represents the average computing resources (e.g., FLOPS) used during operation. av This represents the average of the previous r identifications used for r identifications, with r = 100 by default. Controller 212 attempts to operate RFEEN 500 with the expected computational budget B and modifies the coarse exit decision criterion (i.e., hyperparameter T) so that RFEEN 500 operates with a given computational budget B or a computational budget close to the given computational budget B. Therefore, RFEEN 500 only operates with the expected budget (i.e., B – f) av When r = 0, the hyperparameter T is not updated. The smaller the value of r, the faster the hyperparameter T changes, while the larger the value of r, the slower the hyperparameter T changes. The parameter r can be a hyperparameter controlled externally from RFEEN 500, such as controller 212.

[0099] While increasing the value of r makes RFEEN 500 respond more slowly to modified budgeted computational resources, it also makes the response more stable, meaning that changes in accuracy are also more stable. This design choice depends on the application. For example, applications that require RFEEN 500 to quickly adapt to modified budgeted computational resources need to use a smaller value of r. On the other hand, applications that require RFEEN 500 to smoothly transition between different computational costs (and varying accuracy performance) may benefit from a larger value of r, making the application more application-friendly.

[0100] The optimization of hyperparameter T is achieved by minimizing the computational cost C corresponding to the processing using coarse-path methods. L And the highest computational cost C corresponding to processing using fine-grained paths H The optimal operating point of the adaptive action recognizer 200 is estimated between these parameters. The operation characteristic curve in the computational cost vs. complexity curve can be used to represent the operation characteristics of the processing path of the adaptive action recognizer 200 (e.g., the fine, medium, or coarse processing path of RFEEN 500 or 550), where complexity is represented by the hyperparameter T. An example of an operation characteristic curve is shown below. Figure 6 As shown, each computational cost vs. complexity curve represents the operational characteristics of the adaptive action recognizer 200 when operating along a given processing path. Figure 6 The upper curve in the figure represents the operational characteristics of a high-complexity path. Figure 6 The lower curve in the figure represents the operational characteristics of low-complexity pathways (such as coarse processing pathways).

[0101] The operational feature region is a simple region, limited by the computational cost on the vertical axis (from C). L To C H The hyperparameter T (from 0 to 1) on the horizontal axis is used to derive the optimal value of the hyperparameter T, as described below. Figure 6 The shaded area in the image is an example of the operational feature region of the adaptive motion recognizer 200. For relatively easy-to-process videos (e.g., nighttime surveillance videos) where no activity occurs, the adaptive motion recognizer 200 tends to have a feature region close to C. L With low computational cost, and only for values ​​close to 1, the adaptive action recognizer 200 has a value close to C. H The computational cost (lower curve). Conversely, for relatively difficult-to-process videos, the adaptive action recognizer 200 tends to have a computational cost close to C. H The computational cost is high (as shown on the upper curve). However, in reality, video complexity often varies. For example, when a video shows a driver interacting with gestures while driving a vehicle, the video may not include gestures most of the time, but at certain points in time, it may include many gestures performed in succession.

[0102] Figure 9 An optimization provided by one embodiment of this application is illustrated. Figure 2 Method 900 describes the operation of an adaptive action recognizer. Method 900 proposes an alternative method for optimizing the hyperparameter T.

[0103] In operation 902, controller 212 determines the current operating point (T) on the operating characteristic curve at time t. t C t The current operation point is at... Figure 7 The exemplary operating characteristic curve is shown. In some examples, an average scheme is used based on the above formula (4), where T t It is T av C t It is f av , where r represents the hyperparameter of the external control. (See reference) Figure 7 The operational characteristic curve can be derived as two straight lines: the first straight line connects C. L (0,C L ) and estimated operating point (T) t C t The second straight line connects the estimated operating point (T). t C t ) and C H (1,C H ), where the expected operation point is located at the budget calculation resource B(T) B C B ) place.

[0104] In operation 904, controller 212 determines the current computational cost C of adaptive action recognizer 200 at time t. t Is it equal to, less than, or greater than the budgeted computational resource B? In operation 906, when determining the current computational cost C of the adaptive action recognizer 200... t When calculating resource B equal to the budget, keep the hyperparameter T constant.

[0105] In operation 908, when the current computational cost C of the adaptive action recognizer 200 is determined... t When the computational resource B is less than the budget, increase the hyperparameter T. In some examples, the hyperparameter T is increased according to the following formula:

[0106]

[0107] In operation 912, controller 212 makes T = T new Modify the adaptive motion recognizer 200, such as RFEEN500 or 550.

[0108] In operation 910, when the current computational cost C of the adaptive action recognizer 200 is determined... t When the computational resource B exceeds the budget, decrease the hyperparameter T. In some examples, the hyperparameter T is decreased according to the following formula:

[0109]

[0110] In operation 912, controller 212 makes T = T new Modify the adaptive motion recognizer 200, such as RFEEN500 or 550.

[0111] Alternatively, without using formula (4) above, the current operating point on the operating characteristic curve can be determined using an average scheme based on the following formula:

[0112] C t =β*C (t–1) +(1–β)C t (7)

[0113] T t =β*T (t–1) +(1–β)T t (8)

[0114] Here, parameter β represents the sensitivity to noise in the video, ranging between 0 and 1. A larger value of β indicates less sensitivity to noise and greater stability. A smaller value of β indicates greater sensitivity to noise and less stability. The value of β can be set according to the application used with the adaptive video recognizer 200. The inventors have discovered that a value of β equal to 0.9 can be C.t and T t Generate smooth and reliable estimates.

[0115] Control scheme expansion to multiple devices

[0116] For example, when an adaptive motion recognizer, such as the RFEEN 500 or 550, is shared among multiple devices and / or a group of multiple devices jointly executes a video recognition method, the use of budgeted computational resources to control the hyperparameter T can be scaled to that group of devices. The devices in this group can be identical or different. In other words, the same adaptive motion recognizer (i.e., the same network) can be deployed across multiple devices. This group of devices can include IoT or “smart” devices, including but not limited to smartphones, smart TVs, alarms / sirens, cameras (including camcorders), emergency buttons, smoke detectors, gas detectors, remote controls, motion detectors (e.g., passive infrared (PIR) motion detectors), door / smart detectors, smart sockets, glass break detectors, or environmental sensors. This group of devices can constitute or be part of a smart home system that includes multiple smart devices. Controller 112 can automatically determine the level of complexity required to be performed on different devices. It should be understood that different devices have different computational resources. When an adaptive action recognizer task is to be executed, the scheduler 218 determines which device to execute the task based on data transmission time, computing resources, and other factors, as described in more detail below.

[0117] Figure 10 This application illustrates a workflow for determining budget computing resources B based on the computing status of multiple devices, according to one embodiment of the present application. Figure 11 This illustrates a workflow for multi-device scheduling provided in one embodiment of this application. For example... Figure 10 As shown, the multi-device status is used to determine the budgeted computational resource B. If a device lacks the computational capacity to run the minimum computational portion of the adaptive action recognizer, then computational resource C can be used. i Less than C L If the available computing resources of a device are not available, then the device's computing resources are unavailable and cannot be used to determine budget computing resources. For each device i in a set of N devices, determine the available computing resources C of that device. i Is it equal to or greater than the minimum computational cost C of the adaptive action recognizer? L This refers to the path with the lowest complexity.

[0118] If the available computing resources of the device are C i Equal to or greater than the minimum computational cost C of the adaptive action recognizer L Then the available computing resources of the device will be added to the budgeted computing resources B:

[0119]

[0120] If the available computing resources of the device are C i The minimum computational cost C less than that of an adaptive action recognizer L If so, the available computing resources of that device will not be added to the budget computing resources B.

[0121] In other words, a device is included in a group of devices, and when the available computing resources of the device are equal to or greater than the computing cost of the first action recognizer 304, the available computing resources of the device are included in the budgeted computing resources.

[0122] like Figure 11 As shown, the scheduler 218 has the lowest computational cost C among the devices in the group, equal to or greater than that of the adaptive action recognizer. L Available computing resources C i For each device, determine the corresponding device's delay L. i The latency depends at least on the following factors: data transmission (receive and send) time and inference time. Scheduler 218 will select devices from the group with the lowest computational cost C equal to or greater than that of the adaptive action recognizer. L Computing resources C i The latency of all devices is sorted from lowest to highest. The scheduler 218 then assigns processing tasks to each device in the subgroup. Processing tasks may include multiple video frames for motion recognition. The number of video frames assigned can be determined based on one or both of the latency and available computing resources of the respective device. This process is repeated in a round-robin fashion until all processing tasks are completed. Since the latency and / or available computing resources of each device in the device group may vary over time due to multitasking and changing network conditions that affect data transmission time, the latency and / or available computing resources of each device in the device group can be reassessed periodically, for example, after each round (or cycle) or after a predetermined number of rounds (or cycles), and the budgeted computing resources and latency-based sorting of each respective device can be reassessed.

[0123] The controller 212 and the scheduler 218 can be one of the multiple devices in the group, or they can be external to the group of multiple devices. The controller 212 and the scheduler 218 can be implemented by a single device.

[0124] train

[0125] The spatiotemporal decomposition block (video decomposer 214), receptive field enhancement complementary pathway block (hierarchical action recognizer 228), and feature fusion block 508 in RFEEN 500 and 550 are trained end-to-end using a loss function that uniformly optimizes all depth CNNs 504 through supervised learning (e.g., utilizing supervised learning algorithms and a training dataset including labeled training samples). Each labeled training data sample consists of an RGB video input and a corresponding label representing an action in the RGB video input, which consists of a certain number of frames (16 by default). In one example, the training data sample consists of an RGB video input corresponding to 16 frames of video (or 3.2 seconds at 5 frames per second) and a label representing "walking". The goal of training is for RFEEN 500 and 550 to learn from the labeled training samples and predict labels representing a set of action categories during inference, assuming the input RGB video consists of a certain number of frames. Training can be broadly divided into two steps—forward propagation and backward propagation.

[0126] Forward Propagation: First, the RGB input is spatiotemporally downsampled (i.e., across spatial and temporal axes). Each downsampled input, along with the original input, passes through various convolutional networks, namely deep CNNs 504-1, 504-2, and 504-3. As the input size gradually decreases compared to the convolutional network filter size, the receptive field of the low-resolution pathways increases. Therefore, the low-resolution pathways, which lose detail due to the reduced resolution, provide supplementary information through receptive field enhancement. At the end of each convolutional network pathway is a fully connected (FC) layer, whose output is a feature vector of the same size as the number of action categories the network was trained to classify. For example, if the network was trained to classify 200 categories, the feature vector size would also be 200. This feature vector passes through a softmax layer, producing a probability feature vector of the same size as the number of action categories (i.e., 200 in the example above). These softmax outputs are the predicted outputs of each convolutional network pathway (the three pathways in the diagram above). These softmax outputs are cascaded, and the cascaded features are then passed through the reparameterized fully connected layers described above. The reparameterized fully connected layer is a new extension of the fully connected layer. This layer has a feature vector output with the same size as the number of action categories that the network classifies during training (i.e., 200 in the example above). This output is also passed through a softmax layer.

[0127] Backpropagation: All outputs of the softmax layers (i.e., O in RFEEN 500) C O M O F and O C2F Or O in RFEEN550 C OM O F O C2M and O C2F The values ​​are compared with the labels, and the loss for each output is calculated. Then, all losses are accumulated according to a multiple loss function, which for RFEEN 500 is:

[0128]

[0129] Among them, L n and L C2F These are the ground truth values With O n and O C2F The softmax cross-entropy loss is compared, O n Represents the output of path n, O C2F This represents the output after reparameterizing the FC layer, where α∈R:B∈[0,1]. A larger value of α causes RFEEN 500 or 550 to focus solely on optimizing each pathway, while a smaller value of α causes RFEEN 500 or 550 to focus more on extracting supplementary information to improve O. C2F To improve the performance of the coarse path, α can be set to a large value, such as α = 0.9, to exit the calculation earlier and save computational costs.

[0130] The loss is then backpropagated through the individual convolutional neural networks, namely the deep CNNS 504-1, 504-2, and 504-3. A stochastic gradient descent optimization method is used to update the trainable parameters of the convolutional neural networks.

[0131] Using a joint optimization formula and multiple loss functions, the processing pathways are trained together, so that RFEEN 500 and 550 are trained so that each processing pathway can reliably perform action recognition on its own, and also provide supplementary features for use by the more refined pathways during the fusion process. The multiple loss functions defined above are differentiable, therefore the joint optimization method is end-to-end differentiable, and different processing pathways can be trained together.

[0132] Multi-loss paradigm with pretrained fine-tuned pathways

[0133] The student-teacher learning paradigm involves a large network trained to train a smaller network in a manner analogous to a teacher instructing students. In the classroom learning paradigm, this analogy is extended, presupposing that students can learn from other students, and the classroom as a whole can learn better. In the context of network learning, the classroom learning paradigm uses the most complex CNN as the teacher, namely a deep CNN 504-1 with the finest path F. The deep CNN 504-1 can be trained to identify actions in fine-grained videos, i.e., high-resolution RGB videos. Then, the deep CNN 504-1 can be used to teach coarse paths (i.e., coarse and intermediate paths), and the classroom, in the form of one or more reparameterized fully connected (FC) layers, learns from both the students and the teacher, making it perform better than the finest path F. Similar to student-teacher learning, the deep CNN 504-3 with the finest path F is no longer trained but is used to teach deep CNNs with coarse paths (i.e., students), such as deep CNNs 504-1 and 504-2 with coarse and intermediate paths. To optimize the C and M pathways, as well as one or more reparameterized FC layers, the solution objective of RFEEN500 is to minimize the following multiple loss functions:

[0134]

[0135] Among them, L n,KLD P represents N / τ and p F The Kullback-Leibler divergence between the distributions of / τ, p N p represents the softmax output of the nth path. F This represents the softmax output of F, and τ represents the temperature parameter, which is empirically set to 6.0 to allow p to... F and p N The distribution between them is more gradual, and α represents the hyperparameter, which is empirically set to 0.1. L d The main difference from the original knowledge distillation scheme is that (1–α / 2)L C2F The term, which supports each coarse pathway in providing some supplementary information to F, may help improve O. C2F Overall performance.

[0136] The above multiple loss functions can be modified for use with RFEEN 550 or other networks.

[0137] The adaptive action recognizer 200 of this application, including a hierarchical action recognizer 228, can be evaluated from the perspective of accuracy and reduced computational cost using commonly used gesture recognition datasets, such as the known 20BN-jester V1 dataset, the known Something-Something V1 dataset, and the known Something-Something V2 dataset. Experimental results show that the adaptive action recognizer 200, such as RFEEN 500, improves accuracy and reduces computational cost. Figure 12 The experimental results are summarized in a table, which compares the performance of various implementations of RFEEN 550 with state-of-the-art methods based on the known 20BN-jester dataset. Experiments were also conducted on RFEEN 500, which performs comparably to RFEEN 550, but with faster and better training speed, higher recognition accuracy, and a simpler network structure.

[0138] like Figure 12 As shown in the table, in the validation data of the 20BN-jester dataset, using the default coarse exit threshold, RFEEN 550 identified 42.2% of the videos using the coarse path (first motion generator), 13.7% using the medium path (second motion generator), and the remaining 44.1% using the fine path (third motion generator). Compared to using only the fine path (third motion generator), this equates to a computational cost reduction of approximately 45%.

[0139] In the validation dataset of the 20BN-jester dataset, RFEEN 550 achieved 96.48% accuracy without a coarse output and 96.45% accuracy with a coarse output. Both of these accuracies are higher than the previous state-of-the-art 96.35%. Furthermore, the computational cost of RFEEN 550 with a coarse output is only about 15% of that of the previous state-of-the-art, making RFEEN 550 significantly faster and more efficient.

[0140] When comparing the proposed end-to-end RFEEN 550 with two original coarse-to-fine ensemble schemes, the softmax scores of each neural network in the action recognizer are first summed; then, the product of the softmax scores of each neural network in the action recognizer is used. The performance of the two original ensemble schemes is consistently worse than the end-to-end RFEEN 550 (RFEEN 550: 96.48%, original ensemble scheme using addition: 95.34%, original ensemble scheme using multiplication: 95.27%). In fact, both original ensemble methods outperform the third action recognizer (accuracy 96.35%). This decrease in accuracy may be due to the relatively weak classifier, where the output of the first action recognizer's neural network confuses the output of the finer pathways.

[0141] Training RFEEN 550 in an end-to-end manner can be compared to training each neural network for the action recognizer separately, and then training the ensemble fully connected layers. Here, end-to-end learning appears to outperform individual learning paths and the ensemble (end-to-end learning 96.48%, individual learning 96.32%). Similar to the original ensemble, the accuracy of RFEEN 550 without end-to-end learning is slightly lower than the overall accuracy. This is likely because, without multiple loss functions, the ensemble tends to overfit across different segments of the entire ensemble network.

[0142] Figure 13 A graph showing the budgeted computational cost of controller 212 using hyperparameter T based on the Something-something V2 dataset. The data was obtained using RFEEN 500, and the computational cost of RFEEN 500 was compared to that of a traditional network with hyperparameter T, depending on the number of videos processed.

[0143] While the above description primarily focuses on recognizing actions in videos, the teachings of this invention can be applied to any classification task to be performed on a video, and can be appropriately adapted for the classification task to be performed.

[0144] SUMMARY

[0145] The steps and / or operations in the flowcharts and figures described herein are for illustrative purposes only. These steps and / or operations can be varied in many ways without departing from the teachings of this application. For example, steps can be performed in different orders, or steps can be added, deleted, or modified as needed.

[0146] In consideration of this application, the coding of software for performing the above methods is within the scope of those skilled in the art. Machine-readable code that can be executed by one or more processors of one or more corresponding devices to perform the above methods can be stored in a machine-readable medium such as the memory of a data manager. The terms "software" and "firmware" are interchangeable within this application and include any computer program stored in memory for execution by a processor, including random access memory (RAM), read-only memory (ROM), EPROM, electrically EPROM (EEPROM), and non-volatile RAM (NVRAM). The above memory types are merely examples and therefore do not limit the types of memory that can be used to store computer programs.

[0147] All values ​​and subranges within the scope of the disclosure are also disclosed. Furthermore, while the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements, these systems, devices, and components may be modified to include more or fewer such elements. Although several exemplary embodiments are described herein, modifications, adjustments, and other implementations are possible. For example, elements shown in the figures may be replaced, added, or modified, and the exemplary methods described herein may be modified by replacing, reordering, or adding steps.

[0148] One or more features from the above embodiments can be selected to create alternative embodiments consisting of sub-combinations of features that may not be explicitly described above. Additionally, one or more features from the above embodiments can be selected and combined to create alternative embodiments consisting of combinations of features that may not be explicitly described above. Upon review of this application in its entirety, the features applicable to such combinations and sub-combinations will be apparent to those skilled in the art.

[0149] Furthermore, numerous specific details are set forth to provide a thorough understanding of the exemplary embodiments described herein. However, those skilled in the art will understand that the exemplary embodiments described herein can be implemented without these specific details. Moreover, well-known methods, processes, and elements have not been described in detail so as not to obscure the exemplary embodiments described herein. The subject matter described herein is intended to cover and include all suitable modifications to the technology.

[0150] Although this application has been described at least partially in terms of method, those skilled in the art will understand that this application also relates to various elements, whether hardware, software, or any combination thereof, for performing at least some aspects and features of the described methods. Accordingly, the technical solutions of this application may be embodied in a non-volatile or non-transitory machine-readable medium (e.g., optical disc, flash memory, etc.) that tangibly stores executable instructions that enable a processing device to perform various examples of the methods disclosed herein.

[0151] The term "processor" can include any programmable system, including systems that use microprocessors / controllers or nanoprocessors / controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), reduced instruction set circuits (RISCs), logic circuits, and any other circuits or processors capable of performing the functions described herein. The term "database" can refer to one or both a data subject and a relational database management system (RDBMS). As used herein, "database" can include any collection of data, including hierarchical databases, relational databases, flat file databases, object-relational databases, object-oriented databases, and any other structured collection of records or data stored in a computer system. The above is merely illustrative and is not intended to limit the definition and / or meaning of the terms "processor" or "database" in any way.

[0152] This application may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are to be regarded in all respects as illustrative rather than restrictive. This application is intended to cover and include all suitable modifications to the technology. Therefore, the scope of this application is described by the appended claims rather than by the foregoing description. The scope of the claims should not be limited to the embodiments set forth in the examples, but should be given the broadest interpretation consistent with the overall description.

Claims

1. A method for recognizing actions in a video, characterized in that, The method comprises: processing a high complexity video to generate a medium complexity video and a low complexity video; a first action recognizer processes the low complexity video to recognize actions in the low complexity video; when it is determined that a probability associated with a highest probability action recognized by the first action recognizer is less than a threshold, a second action recognizer processes the medium complexity video to recognize actions in the medium complexity video; when it is determined that a probability associated with a highest probability action recognized by the second action recognizer is less than the threshold, a third action recognizer processes the high complexity video to recognize actions in the high complexity video; wherein the first action recognizer, the second action recognizer and the third action recognizer each outputs a probability set, the method further comprises: when it is determined that a probability associated with a highest probability action recognized by the first action recognizer, the second action recognizer or the third action recognizer is equal to or greater than the threshold, outputting the highest probability action recognized by the first action recognizer, the second action recognizer or the third action recognizer; when it is determined that the probability associated with the highest probability action recognized by the second action recognizer is less than the threshold, combining the probability sets outputted by the first action recognizer, the second action recognizer and the third action recognizer respectively to generate a probability collection; when it is determined that a probability associated with a highest probability action in the probability collection is equal to or greater than the threshold, outputting the highest probability action in the probability collection; wherein the threshold is a hyperparameter T based on a budgeted computing resource B, the hyperparameter T is optimized according to the following equation: where B : B [f C , f C2F ], f C denotes the actual computational cost of generating the set of probabilities by the first action recognizer for a single recognition, f C2F denotes the actual computational cost of generating the set of probabilities for a single recognition, f av denotes the running average of computational resources, T av denotes the average of recognitions before the rth recognition.

2. The method of claim 1, wherein, one probability in the probability set is associated with each action in a class of actions.

3. The method of claim 1, wherein, The first action recognizer, the second action recognizer and the third action recognizer each comprises a deep convolutional neural network (CNN), the deep CNN comprises a plurality of fully connected layers, and the combining the probability sets outputted by the first action recognizer, the second action recognizer and the third action recognizer respectively to generate a probability collection comprises: concatenating the probability sets outputted by the first action recognizer, the second action recognizer and the third action recognizer respectively; a reparameterized fully connected layer generates the probability collection according to the concatenated probability sets.

4. The method according to claim 1 or 2, characterized in that, The first action recognizer, the second action recognizer and the third action recognizer are each configured to recognize a class of actions in a video, and the first action recognizer, the second action recognizer and the third action recognizer are configured to recognize the class of actions in the video at a low video complexity level, a medium video complexity level and a high video complexity level.

5. The method according to claim 1 or 2, characterized in that, The budgeted computing resource is determined based on available computing resources of a group of devices jointly performing the method, an available computing resource of a device is included in the budgeted computing resource when it is determined that the available computing resource of the device is equal to or greater than a computing cost of the first action recognizer.

6. The method of claim 1 or 2, wherein, The first action recognizer, the second action recognizer, and the third action recognizer each comprise a deep convolutional neural network, CNN.

7. The method according to claim 1 or 2, characterized in that, The high complexity video is spatio-temporally decomposed to generate the medium complexity video and the low complexity video.

8. The method of claim 7, wherein, The spatio-temporal decomposition comprises downsampling the high complexity video to generate the medium complexity video and the low complexity video both having low resolution and low frame rate.

9. The method of claim 8, wherein, The downsampling comprises reducing the high complexity video by a factor of 2 using a bilinear interpolation technique to generate the medium complexity video and reducing the medium complexity video by a factor of 2 using the bilinear interpolation technique to generate the low complexity video.

10. The method of claim 7, wherein, The high complexity video has a resolution of W pixels by H pixels and a frame rate of X frames per second, FPS, the medium complexity video has a resolution of W / 2 pixels by H / 2 pixels and a frame rate of X / 2 FPS, and the low complexity video has a resolution of W / 4 pixels by H / 4 pixels and a frame rate of X / 4 FPS.

11. An adaptive motion recognizer, characterized by The adaptive action recognizer comprises: a video decomposer configured to process a high complexity video to generate a medium complexity video and a low complexity video; a first action recognizer configured to process the low complexity video to identify actions in the low complexity video; a second action recognizer configured to process the medium complexity video to identify actions in the medium complexity video when it is determined that a probability associated with a highest probability action identified by the first action recognizer is less than a threshold value; a third action recognizer configured to process the high complexity video to identify actions in the high complexity video when it is determined that a probability associated with a highest probability action identified by the second action recognizer is less than the threshold value; wherein the first action recognizer, the second action recognizer, and the third action recognizer each output a set of probabilities, and the adaptive action recognizer further comprises: a controller configured to output a highest probability action identified by the first action recognizer, the second action recognizer, or the third action recognizer when it is determined that a probability associated with the highest probability action identified by the first action recognizer, the second action recognizer, or the third action recognizer is equal to or greater than the threshold value; combine the sets of probabilities output by the first action recognizer, the second action recognizer, and the third action recognizer to generate a set of probabilities when it is determined that the probability associated with the highest probability action identified by the second action recognizer is less than the threshold value; output the highest probability action in the set of probabilities when it is determined that a probability associated with the highest probability action in the set of probabilities is equal to or greater than the threshold value; wherein the threshold value is a hyperparameter T based on a budgeted computing resource B, and the hyperparameter T is optimized according to the following equation: where B : B [f C , f C2F ], f C denotes the actual computational cost of generating the set of probabilities by the first action recognizer for a single recognition, f C2F denotes the actual computational cost of generating the set of probabilities for a single recognition, f av denotes the running average of computational resources, T av denotes the average of recognitions before the rth recognition.

12. The adaptive motion recognizer of claim 11, wherein, one probability in the set of probabilities is associated with each action in a class of actions.

13. The adaptive action recognizer according to claim 11 or 12, characterized by, The high complexity video is spatio-temporally decomposed to generate the medium complexity video and the low complexity video.

14. The adaptive action recognizer according to claim 11 or 12, characterized by, The first action recognizer, the second action recognizer, and the third action recognizer each comprise a deep convolutional neural network, CNN.

15. A machine-readable medium comprising executable instructions tangibly stored therein for execution by a processing system of a computing device, the machine-readable medium comprising instructions for: The processing system includes at least one processor that, when the processing system executes the executable instructions, causes the processing system to perform the method of any one of claims 1-10.

16. A computer program product comprising instructions, characterized in that, The processing system, when executing the instructions, is caused to perform the method of any one of claims 1-10.

Images