Efficient image recognition method, system, and program

Abstract

An efficient method, system, and program for video recognition includes receiving an input video including a sequence of video segments in a plurality of data modalities, and selecting a data modality for each video segment of the sequence based on data representing the video segment, where each selected data modality is optimal for video recognition of the video segment. Next, for each selected data modality, providing a data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality, and generating a first type prediction representative of the video segment via the machine learning model. Then, determining a second type prediction representative of the entire input video by aggregating all the generated first type predictions.

Description

[Technical Field]

[0001] The field of embodiments of the present invention relates generally to video recognition. [Background technology]

[0002] Video recognition is a machine learning (ML)-based computer vision task that involves acquiring, processing, and analyzing incoming data from visual sources such as video. Multimodal learning is widely used in video recognition. Multimodal learning utilizes diverse data modalities to improve model performance. Models trained through multimodal learning represent a joint representation of various data modalities. Most traditional deep multimodal models focus on how to fuse information from multiple data modalities. However, traditional video recognition solutions using multimodal learning are computationally expensive because they typically process all incoming data from visual sources, including redundant / irrelevant portions of the data. For example, traditional solutions may analyze multiple data streams representing the video in different data modalities, such as RGB streams and audio streams, to recognize activities (i.e., actions) taking place in the video. However, analyzing multiple data streams for every video segment in the video may not be necessary to recognize activities taking place in the video. It is necessary to provide an adaptive, multi-modal learning framework that improves the efficiency of video recognition (i.e., increases computational efficiency), achieves significant savings in computation (i.e., reduces computational cost), and enhances the accuracy of video recognition (i.e., improves prediction / classification accuracy / quality) by on-the-fly selecting the optimal data modality for video recognition for each video segment based on the input (i.e., data-dependent selection of different data modalities for each video segment). Summary of the Invention

[0003] FIELD Embodiments of the present invention relate generally to video recognition, and more particularly to methods and systems for adaptive selection of data modalities for efficient video recognition.

[0004] One embodiment of the present invention provides a method for video recognition. The method includes receiving input video, including a sequence of video segments, in multiple data modalities. The method further includes, for a video segment of the sequence, selecting one or more data modalities from the multiple data modalities based on data representing the video segment. Each selected data modality is optimal for video recognition of the video segment. For each selected data modality, the method further includes providing at least one data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality and generating a first type prediction representative of the video segment via the machine learning model. The method further includes determining a second type prediction representative of the entire input video by aggregating all generated first type predictions. The second type prediction indicates an object or activity captured in the input video. Other embodiments include a system for video recognition and a computer program product for video recognition. These features contribute to an optimal tradeoff between video recognition accuracy and computational efficiency.

[0005] One or more of the following features may be included.

[0006] In some embodiments, each data modality of the plurality of data modalities has a corresponding machine learning model that is trained in conjunction with one or more other machine learning models corresponding to one or more other data modalities of the plurality of data modalities. In some embodiments, each machine learning model corresponding to each data modality of the plurality of data modalities comprises a sub-network. These optional features contribute to the advantage of learning a decision policy that prioritizes the selection of more computationally efficient data modalities in recognizing video.

[0007] These and other aspects, features, and advantages of embodiments of the present invention may be understood with reference to the drawings and detailed description herein and realized by means of the various elements and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following brief description of the drawings and detailed description of embodiments of the present invention are exemplary and explanatory of preferred embodiments of the invention and are not intended to limit the embodiments of the invention as claimed.

[0008] The subject matter which is considered to be embodiments of the present invention is particularly pointed out and distinctly claimed in the claims in the concluding portion of the specification. The foregoing and other objects, features, and advantages of embodiments of the present invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an exemplary computing architecture for implementing adaptive selection of data modalities for efficient video recognition, according to an embodiment of the present invention. [Figure 2] 1 is a diagram of an exemplary adaptive multimodal learning system, according to an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram of exemplary components of a decision policy network and a video recognition network included in a system according to an embodiment of the present invention. [Figure 4]1 is a diagram of an exemplary application scenario involving a system according to an embodiment of the present invention. [Figure 5A] 2A-2C are diagrams of a first sequence of video segments in different modalities according to an embodiment of the present invention; [Figure 5B] FIG. 10 is a diagram of a second sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5C] FIG. 10 is a diagram of a third sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5D] FIG. 10 is a diagram of a fourth sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5E] FIG. 10 is a diagram of a fifth sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5F] FIG. 10 is a diagram of a sixth sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5G] FIG. 10 is a diagram of a seventh sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5H] FIG. 10 is a diagram of an eighth sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 5I] FIG. 10 is a diagram of a ninth sequence of video segments in different modalities according to an embodiment of the present invention. [Figure 6] 1 is a flowchart of an exemplary process for adaptive selection of data modalities for efficient video recognition, according to an embodiment of the present invention. [Figure 7] 1 is a diagram of a cloud computing environment according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram of an abstract model layer according to an embodiment of the present invention. [Figure 9] 1 is a high-level block diagram illustrating an information processing system useful for implementing embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] The detailed description explains preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings.

[0011] Embodiments of the present invention generally relate to video recognition and, more particularly, to methods and systems for adaptive selection of data modalities for efficient video recognition. One embodiment of the present invention provides a method for video recognition. The method includes receiving input video including a sequence of video segments in multiple data modalities. The method further includes, for a video segment of the sequence, selecting one or more data modalities from the multiple data modalities based on data representing the video segment. Each selected data modality is optimal for video recognition of the video segment. For each selected data modality, the method further includes providing at least one data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality and generating a first type prediction representative of the video segment via the machine learning model. The method further includes determining a second type prediction representative of the entire input video by aggregating all generated first type predictions. The second type prediction indicates an object or activity captured in the input video.

[0012] Another embodiment of the present invention provides a system for video recognition. The system includes at least one processor and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include receiving input video including a sequence of video segments in multiple data modalities. The operations further include selecting, for a video segment of the sequence, one or more data modalities from the multiple data modalities based on data representing the video segment. Each selected data modality is optimal for video recognition of the video segment. The instructions further include, for each selected data modality, providing at least one data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality and generating a first type prediction representative of the video segment via the machine learning model. The instructions further include determining a second type prediction representative of the entire input video by aggregating all generated first type predictions. The second type prediction indicates an object or activity captured in the input video.

[0013] One embodiment of the present invention provides a computer program product for video recognition. The computer program product comprises a computer-readable storage medium having program instructions embodied thereon. The program instructions are executable by the processor to cause the processor to receive input video including a sequence of video segments in multiple data modalities. The program instructions are further executable by the processor to cause the processor to select, for a video segment of the sequence, one or more data modalities from the multiple data modalities based on data representing the video segment. Each selected data modality is optimal for video recognition of the video segment. The program instructions are further executable by the processor to cause the processor to provide, for each selected data modality, at least one data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality, and generate a first type prediction representative of the video segment via the machine learning model. The program instructions are further executable by the processor to cause the processor to determine a second type prediction representative of the entire input video by aggregating all generated first type predictions. The second type of prediction indicates objects or activities captured in the input video.

[0014] 1 illustrates an exemplary computing architecture 300 for implementing adaptive data modality selection for efficient video recognition, according to an embodiment of the present invention. In one embodiment, computing architecture 300 is a centralized computing architecture. In another embodiment, computing architecture 300 is a distributed computing architecture.

[0015] In one embodiment, computing architecture 300 includes computational resources such as, but not limited to, one or more processor units 310 and one or more storage units 320. One or more applications may execute / operate on computing architecture 300 utilizing the computational resources of computing architecture 300. In one embodiment, an application on computing architecture 300 includes, but is not limited to, adaptive and multi-modal learning system 330. As described in more detail later herein, system 330 is configured for data-dependent selection of different modalities for efficient video recognition.

[0016] In one embodiment, system 330 is configured to exchange data with one or more electronic devices 350, or with one or more remote server devices 360, or a combination thereof, over a connection (e.g., a wireless connection such as a Wi-Fi connection or a cellular data connection, a wired connection, or a combination of the two).

[0017] In one embodiment, electronic device 350 includes one or more computational resources, such as, but not limited to, one or more processor units 351 and one or more storage units 352. One or more applications may execute / operate on electronic device 350 utilizing the one or more computational resources of electronic device 350, such as, but not limited to, one or more software applications 354 loaded or downloaded onto electronic device 350. Examples of software applications 354 include, but are not limited to, artificial intelligence (AI) applications.

[0018] Examples of electronic device 350 include, but are not limited to, desktop computers, mobile electronic devices (e.g., tablets, smartphones, laptops, etc.), wearable devices (e.g., smart watches, etc.), Internet of Things (IoT) devices, etc.

[0019] In one embodiment, electronic device 350 includes one or more input / output (I / O) units 353, such as a keyboard, keypad, touch interface, display screen, etc., integrated into or coupled to electronic device 350. A user can utilize I / O modules 353 of electronic device 350 to configure one or more user settings, set one or more parameters, provide input, etc.

[0020] In one embodiment, electronic device 350 and / or remote server device 360 ​​may be the source of at least one of the following: input video, training video.

[0021] In one embodiment, system 330 may be accessed or utilized by one or more online services (e.g., AI services) hosted on a remote server device 360, or one or more software applications 354 (e.g., AI applications) running on electronic device 350, or both. For example, in one embodiment, a virtual assistant, search engine, or another type of software application 354 running on electronic device 350 may call system 330 to perform an AI task.

[0022] 2 is a diagram of an exemplary adaptive multi-modal learning system 330 according to an embodiment of the present invention. The input video comprises a sequence of video segments 400 in one or more data modalities. Examples of data modalities include, but are not limited to, RGB modality, audio modality, optical flow modality, etc.

[0023] The sequence of video segments 400 is represented as one or more data streams. Each data stream includes a sequence of data inputs that represent the sequence of video segments 400 in a particular data modality. Examples of data streams include, but are not limited to, an RGB stream including a sequence of RGB inputs (i.e., RGB frames of video segments 400), an audio stream including a sequence of audio inputs (i.e., audio frames of video segments 400), an optical flow stream including a sequence of optical flow inputs (i.e., optical flow frames of video segments 400), or an RGB difference stream including a sequence of RGB difference inputs (i.e., RGB difference frames of video segments 400), or combinations thereof.

[0024] In one embodiment, system 330 includes multimodal decision policy network 410 and video recognition network 430. Video recognition network 430 includes multiple machine learning models trained for video recognition of different video segments 400 in multiple data modalities. In one embodiment, the multiple machine learning models include different subnetworks 435 (FIG. 3). Each subnetwork 435 of the different subnetworks 435 corresponds to a particular data modality that is different from another data modality that another subnetwork 435 of the different subnetworks 435 corresponds to. Each subnetwork 435 is configured to receive and process data input representing video segments 400 in the corresponding data modality.

[0025] In one embodiment, the multimodal decision policy network 410 is configured to (1) receive an input video including a sequence of video segments 400, and (2) for each video segment 400 of the input video, adaptively select, from a plurality of data modalities, one or more data modalities 420 that are optimal for video recognition of the video segment 400 based on data input representing the video segment 400 in the plurality of data modalities. For each video segment 400 of the input video, the multimodal decision policy network 410 selects on the fly one or more data modalities 420 that are optimal for video recognition of the video segment 400, conditional on the data input representing the video segment 400 in the plurality of data modalities. For each video segment 400 of the input video, the multimodal decision policy network 410 implements adaptive selection of data modalities 420 that are optimal for video recognition of the video segment 400, where the selection is data dependent. During video recognition, each selected data modality 420 helps to focus on what to look at in the data stream (if the data stream is an RGB stream or an optical flow stream) or what to listen to in the data stream (if the data stream is an audio stream).

[0026] In one embodiment, for each video segment 400 of the input video, the video recognition network 430 is configured to: (1) receive (e.g., from the decision policy network 410) one or more selected data modalities 420 that are optimal for video recognition of the video segment 400; (2) forward one or more data inputs representing the video segment 400 in the one or more selected data modalities 420 to one or more sub-networks 435 corresponding to the one or more selected data modalities 420; and (3) generate one or more first type predictions (i.e., classifications) representative of the video segment 400 by processing the one or more data inputs through the one or more sub-networks 435. Specifically, for each selected data modality 420, at least one data input representing the video segment 400 in the selected data modality 420 is routed to the sub-network 435 corresponding to the selected data modality 420. During video recognition of a video segment 400, only data inputs representing the video segment 400 in one or more selected data modalities 420 are processed through one or more subnetworks 435, while data inputs representing the video segment 400 in unselected data modalities 421 (FIG. 3) are skipped / bypassed (i.e., not processed). Each selected data modality 420 is relevant for video recognition of the video segment 400, while each unselected data modality 421 is redundant / irrelevant for video recognition. Skipping / bypassing unselected data modalities 421 eliminates the need to process multiple entire data streams representing the input video in different data modalities, thereby increasing computational efficiency, reducing computational cost, and improving video recognition accuracy.

[0027] In one embodiment, the first type of prediction representing the video segment 400 is a segment-level prediction that indicates the objects or activities captured in the video segment 400 that the different sub-networks 435 are trained to recognize.

[0028] In one embodiment, the video recognition network 430 includes a fusion unit 436 (FIG. 3) configured to (1) receive, for each video segment 400 of the input video, one or more first-type predictions (i.e., segment-level predictions) representative of the video segment 400 (e.g., from one or more of the different sub-networks 435), and (2) determine a second-type prediction 440 representative of the entire input video by averaging (i.e., aggregating) all first-type predictions received for all video segments 400 of the input video.

[0029] In one embodiment, the second type of prediction 440, which is representative of the entire input video, is a video-level prediction that indicates objects or activities captured in the input video that the different sub-networks 435 are trained to recognize.

[0030] In one embodiment, the different sub-networks 435 are trained together (eg, via late fusion with learnable weights).

[0031] 3 is a diagram of example components of a decision policy network 410 and a video recognition network 430, according to an embodiment of the present invention. In general, V is a set of K data modalities {M1, M2, ..., M K}, the video segments {s1,s2,…,s T Let V denote an input video containing a sequence of {\displaystyle V, ...

[0032] In one embodiment, the decision policy network 410 includes a unit 415 configured to model causal relationships across different time steps in an input video V. In one embodiment, the unit 415 is configured to: (a) determine corresponding trainable parameters θ Φ and (2) a combined feature extractor with corresponding trainable parameters θ LSTM At each time step t, the combined feature extractor (1) extracts the current video segment s t (1) receiving a data input (e.g., an RGB input, an RGB difference input, or an audio input, or a combination thereof) representing a current video segment s in K data modalities; and (2) obtaining a current video segment s from the data input. t The corresponding binding feature f t and extracting the

[0033] At each time step t, the LSTM computes the current video segment s according to equation (1) given below: t The corresponding binding feature f t (e.g., from a combined feature extractor), the previous hidden state h t-1 and cell output o t-1 and (2) the current hidden state h t and cell output o t and calculating: h t ,o t =LSTM(f t ;h t-1 ;o t-1 ) (1)

[0034] In one embodiment, the decision policy network 410 includes corresponding trainable parameters

number

number

[0035] In one embodiment, at each time step t, for each data modality k of the K data modalities (k∈[1,...,K]), the decision policy network 410: (1) determines the current hidden state h t (2) apply the Gumbel-Softmax operation to the corresponding policy distribution to obtain the corresponding binary decision u t,k (i.e., sampling from a Gumbel-Softmax distribution), and t,k is the current video segment s t For example, a binary decision u indicates whether to select data modality k or skip / bypass it (i.e., the decision policy for data modality k). t,k If is 1, data modality k is the current video segment s t is selected due to the current video segment s t A data input representing u in data modality k is routed to the video recognition network 430 for processing (i.e., data modality k is the selected data modality 420). t,k If is 0, data modality k is the current video segment s t will be skipped / bypassed, thereby t Any data input representing the current video segment s in data modality k is not routed to the video recognition network 430 for processing (i.e., data modality k is a non-selected data modality 421). t the current video segment s t Only data inputs representing the respective selected data modalities 420 are forwarded to the video recognition network 430 for processing.

[0036] For example, in one embodiment, at each time step t, for each data modality k of the K data modalities, the decision policy network 410 (1) calculates the current hidden state h corresponding to the FC layer 416 via the corresponding FC layer 416; t and trainable parameters θ FCk Based on the corresponding policy distribution z containing policy logits k generating a

number

number

[0037] In one embodiment,

number

number

[0038] In one embodiment, the video recognition network 430 includes a corresponding trainable parameter

number

number

[0039] In one embodiment, the system 330 has at least two distinct phases of operation: a training phase in which the K sub-networks 435 are trained together, and an interference phase in which the resulting trained K sub-networks 435 are deployed for video recognition.

[0040] In one embodiment, to increase the accuracy of video recognition (i.e., prediction / classification accuracy / quality) as well as to minimize the selection of data modalities that require further computation for computational efficiency, the training phase involves minimizing the loss expressed according to equation (4) given below:

number

[0041] Assume that an input video V captures a scene of mowing the lawn and includes a sequence of video segments in the RGB modality, the optical flow modality, and the audio modality. Specifically, the sequence of video segments is represented as an RGB stream including a sequence of RGB inputs (i.e., RGB frames of the video segments) representing the video segments in the RGB modality, an optical flow stream including a sequence of optical flow inputs (i.e., optical flow frames of the video segments) representing the video segments in the optical flow modality, and an audio stream including a sequence of audio inputs (i.e., audio frames of the video segments) representing the video segments in the audio modality. In one embodiment, when determining whether to select an optical flow modality for each video segment, the decision policy network 410 utilizes an RGB difference stream (i.e., RGB difference frames of the video segments) instead of the optical flow stream D to reduce computational cost.

[0042] As shown in FIG. 3, at time step t, the current video segment s tFor example, the decision policy network 410 determines the current video segment s from the RGB stream. t RGB input s expressed in RGB modality t,rgb (2) receiving the current video segment s from the RGB difference stream; t RGB difference input s representing in optical flow modality t,rgbd (3) receiving the current video segment s from the audio stream; t Speech input s expressed in speech modality t,aud and (4) generating a corresponding binary decision policy vector based in part on the received input. Each element of the binary decision policy vector is a binary value (e.g., a binary decision) corresponding to one of RGB, optical flow, and audio modalities, and the element is a binary value corresponding to the current video segment s. t For example, if the element of the binary decision policy vector corresponding to the RGB modality is 1, it means that the RGB modality is the modality of the current video segment s. t If the element is 0, then RGB modality is the selected data modality 420; otherwise, if the element is 0, then RGB modality is the unselected data modality 421.

[0043] Assume that the video recognition network 430 includes a first sub-network 435 ("subnet 1") corresponding to the RGB modality, a second sub-network 435 ("subnet 2") corresponding to the optical flow modality, and a third sub-network 435 ("subnet 3") corresponding to the audio modality. t The binary decision policy vector corresponding to the current video segment s t Assume that for the current video segment s, RGB, optical flow, and audio modalities are all selected data modalities 420. t Then, the image recognition network 430 (1) receives RGB input st,rgb (2) generating a first segment-level prediction via subnet 1 by forwarding the current video segment s t The optical flow input s is expressed in the optical flow modality. t,of (3) generating a second segment-level prediction via subnet 2 by forwarding (from the optical flow stream) s to subnet 2 for processing; and t,aud to subnetwork 3 for processing, thereby generating a third segment-level prediction via subnetwork 3.

[0044] The fusion unit 436 (1) combines all video segments of the input video V (including the current video segment s t 3 , the video level prediction determined for the entire input video V may be "mowing the lawn."

[0045] 4 is a diagram of an example application scenario involving a system 330, according to an embodiment of the present invention. An input video V provided to the system 330 captures a dog barking, and the input video V is a sequence of video segments s1, s2, ..., s in RGB and audio modalities. C Specifically, a sequence of video segments is assumed to include (1) an RGB input s representing the video segments in RGB modality; 1,rgb ,s 2,rgb ,…,s C,rgb and (2) an audio input s representing the video segments in audio modality. 1,aud ,s 2,aud,…,s C,aud It is represented as an audio stream containing a sequence of

[0046] At each time step t, the decision policy network 410 determines: (1) the current video segment s t RGB input s (from an RGB stream) expressed in RGB modality t,rgb Based on the policy distribution z containing policy logits corresponding to the RGB modalities rgb and (2) generating the current video segment s t Speech input s (from an audio stream) expressed in audio modality t,aud Based on the policy distribution z containing policy logits corresponding to the speech modality, aud and (3) generating the current video segment s t A binary decision u indicating whether to select RGB modality or skip / bypass it. t,rgb We apply the Gumbel-Softmax operation to sample the policy distribution z rgb and (4) applying the current video segment s t A binary decision u indicating whether to select an audio modality for or skip / bypass it. t,aud We apply the Gumbel-Softmax operation to sample the policy distribution z aud and applying the

[0047] As shown in Figure 4, the binary decision u 2,rgb , u 5,rgb , and u C,rgb If is 1, the RGB modality is C is the selected data modality 420, thereby C RGB input s (from an RGB stream) expressed in RGB modality 2,rgb , s 5,rgb , and s C,rgbOnly the input s (highlighted by a highlight box for clarity) is forwarded to the first sub-network 435 corresponding to the RGB modality. The first sub-network 435 receives the RGB input s 2,rgb , s 5,rgb , and s C,rgb are processed to obtain video segments s2, s5, and s C As further shown in Figure 4, we generate segment-level predictions for the binary decision u 2,aud , u 3,aud , and u C,aud If is 1, the audio modality is video segments s2, s3, and s C is the selected data modality 420, thereby C Speech input s (from an audio stream) expressed in audio modality 2,aud , s 3,aud , and s C,aud Only the speech input s (highlighted by a highlight box for clarity) is forwarded to the second sub-network 435 corresponding to the speech modality. 2,aud , s 3,aud , and s C,aud are processed to obtain video segments s2, s3, and s C Generate segment-level predictions for

[0048] 4, the fusion unit 436 receives all the segment-level predictions generated by the first and second sub-networks 435 and determines a video-level prediction for the entire input video V by averaging (i.e., aggregating) all the received segment-level predictions. As shown in FIG. 4, the input video V captures a dog barking, and therefore the video-level prediction determined for the entire input video V is "barking".

[0049] 5A is a diagram of a first sequence of video segments 500 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures a fencing scene and includes first sequence of video segments 500 in RGB and audio modalities. For example, as shown in FIG. 5A, first sequence of video segments 500 is represented as (1) an RGB stream 501 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an audio stream 502 including a sequence of audio inputs representing the video segments in the audio modality.

[0050] As shown in Figure 5A, system 330 selects RGB modality only for the second and third video segments of input video V, such that only the RGB input representing the second and third video segments is processed by system 330 for video recognition. As further shown in Figure 5A, system 330 skips / bypasses audio modality for all video segments of input video V, such that the entire audio stream 502 is not processed for video recognition. Because input video V captures fencing, the video level prediction determined by system 330 for the entire input video V may be "fencing."

[0051] 5B is a diagram of a second sequence of video segments 510 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures a piano being played, and that input video V includes second sequence of video segments 510 in RGB and audio modalities. For example, as shown in FIG. 5B, second sequence of video segments 510 is represented as (1) an RGB stream 511 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an audio stream 512 including a sequence of audio inputs representing the video segments in the audio modality.

[0052] As shown in Figure 5B, system 330 selects the RGB modality for only the second video segment of input video V, such that only the RGB input representing the second video segment is processed by system 330 for video recognition. As further shown in Figure 5B, system 330 selects the audio modality for all video segments of input video V, such that the entire audio stream 512 is processed by system 330 for video recognition. Because input video V captures the playing of a piano, the video level prediction determined by system 330 for the entire input video V may be "playing piano."

[0053] 5C is a diagram of a third sequence of video segments 520 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures wood splitting and includes third sequence of video segments 520 in RGB and optical flow modalities. For example, as shown in FIG. 5C, third sequence of video segments 520 may be represented as (1) an RGB stream 521 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an optical flow stream 522 including a sequence of optical flow inputs representing the video segments in the optical flow modality.

[0054] As shown in Figure 5C, system 330 selects RGB modality only for the first and third video segments of input video V, such that only RGB inputs representing the first and third video segments are processed by system 330 for video recognition. As further shown in Figure 5C, system 330 selects optical flow modality only for the second and third video segments of input video V, such that only optical flow inputs representing the second and third video segments are processed by system 330 for video recognition. Because input video V captures the act of chopping wood, the video level prediction determined by system 330 for the entire input video V may be "chopping wood."

[0055] 5D is a diagram of a fourth sequence of video segments 530 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures a paper tear and includes a fourth sequence of video segments 530 in an RGB modality and an optical flow modality. For example, as shown in FIG. 5D, fourth sequence of video segments 530 may be represented as (1) an RGB stream 531 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an optical flow stream 532 including a sequence of optical flow inputs representing the video segments in the optical flow modality.

[0056] As shown in Figure 5D, system 330 selects the RGB modality only for the first, second, and third video segments of input video V, such that only RGB inputs representing the first, second, and third video segments are processed by system 330 for video recognition. As further shown in Figure 5D, system 330 selects the optical flow modality only for the second video segment of input video V, such that only optical flow inputs representing the second video segment are processed by system 330 for video recognition. Because input video V captures the tearing of paper, the video level prediction determined by system 330 for the entire input video V may be "tearing paper."

[0057] 5E is a diagram of a fifth sequence of video segments 540 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures an accordion being played and includes a fifth sequence of video segments 540 in an RGB modality, an optical flow modality, and an audio modality. For example, as shown in FIG. 5E, fifth sequence of video segments 540 is represented as (1) an RGB stream 541 including a sequence of RGB inputs representing the video segments in the RGB modality, (2) an optical flow stream 542 including a sequence of optical flow inputs representing the video segments in the optical flow modality, and (3) an audio stream 543 including a sequence of audio inputs representing the video segments in the audio modality.

[0058] As shown in FIG. 5E, system 330 selects the RGB modality only for the first and fourth video segments of input video V, such that only the RGB input representing the first and fourth video segments is processed by system 330 for video recognition. As further shown in FIG. 5E, system 330 selects the optical flow modality only for the sixth video segment of input video V, such that only the optical flow input representing the sixth video segment is processed by system 330 for video recognition. As further shown in FIG. 5E, system 330 selects the audio modality only for the first through sixth video segments of input video V, such that only the audio input representing the first through sixth video segments is processed by system 330 for video recognition. Because input video V captures the playing of an accordion, the video level prediction determined by system 330 for the entire input video V may be "playing the accordion."

[0059] 5F is a diagram of a sixth sequence of video segments 550 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures cheerleading and includes sixth sequence of video segments 550 in RGB and audio modalities. For example, as shown in FIG. 5F, sixth sequence of video segments 550 is represented as (1) an RGB stream 551 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an audio stream 552 including a sequence of audio inputs representing the video segments in the audio modality.

[0060] As shown in Figure 5F, system 330 selects RGB modality only for the first and second video segments of input video V, such that only the RGB input representing the first and second video segments is processed by system 330 for video recognition. As further shown in Figure 5F, system 330 selects audio modality only for the first and second video segments of input video V, such that only the audio input representing the first and second video segments is processed by system 330 for video recognition. Because input video V captures cheerleading, the video level prediction determined by system 330 for the entire input video V may be "cheerleading."

[0061] 5G is a diagram of a seventh sequence of video segments 560 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures playing a harmonica, and that input video V includes a seventh sequence of video segments 560 in an RGB modality and an audio modality. For example, as shown in FIG. 5G, seventh sequence of video segments 560 is represented as (1) an RGB stream 561 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an audio stream 562 including a sequence of audio inputs representing the video segments in the audio modality.

[0062] As shown in Figure 5G, system 330 selects RGB modality for only the third video segment of input video V, such that only the RGB input representing the third video segment is processed by system 330 for video recognition. As further shown in Figure 5G, system 330 selects audio modality for all video segments of input video V, such that the entire audio stream 562 is processed by system 330 for video recognition. Because input video V captures the playing of a harmonica, the video level prediction determined by system 330 for the entire input video V may be "playing a harmonica."

[0063] 5H is a diagram of an eighth sequence of video segments 570 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures wood splitting and includes eighth sequence of video segments 570 in RGB and optical flow modalities. For example, as shown in FIG. 5H, eighth sequence of video segments 570 is represented as (1) an RGB stream 571 including a sequence of RGB inputs representing the video segments in the RGB modality, and (2) an optical flow stream 572 including a sequence of optical flow inputs representing the video segments in the optical flow modality.

[0064] As shown in Figure 5H, system 330 selects the RGB modality only for the second video segment of input video V, such that only the RGB input representing the second video segment is processed by system 330 for video recognition. As further shown in Figure 5H, system 330 selects the optical flow modality only for the second and third video segments of input video V, such that only the optical flow input representing the second and third video segments is processed by system 330 for video recognition. Because input video V captures the chopping of wood, the video level prediction determined by system 330 for the entire input video V may be "chopping wood."

[0065] FIG. 5I is a diagram of a ninth sequence of video segments 580 in different modalities, according to an embodiment of the present invention. Assume that input video V provided to system 330 captures a tap dance performance and includes a ninth sequence of video segments 580 in an RGB modality, an optical flow modality, and an audio modality. For example, as shown in FIG. 5I, the ninth sequence of video segments 580 is represented as (1) an RGB stream 581 including a sequence of RGB inputs representing the video segments in the RGB modality, (2) an optical flow stream 582 including a sequence of optical flow inputs representing the video segments in the optical flow modality, and (3) an audio stream 583 including a sequence of audio inputs representing the video segments in the audio modality.

[0066] As shown in FIG. 5I, system 330 selects the RGB modality for only the first video segment of input video V, such that only the RGB input representing the first video segment is processed by system 330 for video recognition. As further shown in FIG. 5I, system 330 selects the optical flow modality for only the second video segment of input video V, such that only the optical flow input representing the second video segment is processed by system 330 for video recognition. As further shown in FIG. 5I, system 330 selects the audio modality for all video segments of input video V, such that the entire audio stream 583 is processed by system 330 for video recognition. Because input video V captures tap dancing, the video level prediction determined by system 330 for the entire input video V may be "tap dancing."

[0067] FIG. 6 is a flowchart of an example process 600 for adaptively selecting data modalities for efficient video recognition, according to an embodiment of the present invention. Process block 601 includes receiving input video including a sequence of video segments in multiple data modalities. Process block 602 includes selecting, for a video segment of the sequence, one or more data modalities from the multiple data modalities based on data representing the video segment, each selected data modality being optimal for video recognition of the video segment. Process block 603 includes, for each selected data modality, providing at least one data input representing the video segment in the selected data modality to a machine learning model (e.g., a subnetwork) corresponding to the selected data modality and generating a first type of prediction (e.g., a segment-level prediction) representative of the video segment via the machine learning model. Process block 604 includes determining a second type prediction (e.g., a video-level prediction) that is representative of the entire input video by aggregating all of the generated first type predictions, where the second type prediction is indicative of objects or activity captured in the input video.

[0068] In one embodiment, process blocks 601-604 are performed by one or more components of system 330.

[0069] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0070] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. In one embodiment, this cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

[0071] Its features are as follows:

[0072] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed without requiring human interaction with the service provider.

[0073] Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and personal digital assistants).

[0074] Pooling of Resources: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. In one embodiment, there is a sense of location independence in that consumers generally have no control or information about the exact location of the resources provided, although location can be specified at a higher level of abstraction (e.g., country, state, or data center).

[0075] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, quickly scaled out, and quickly released and quickly scaled in. To the consumer, the capabilities available for provisioning often appear unlimited, and any amount can be purchased at any time.

[0076] Service Metering: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services used.

[0077] The service model is as follows:

[0078] Software as a Service (SaaS): The functionality offered to the consumer is the use of a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the exception of limited user-specific application configuration settings.

[0079] Platform as a Service (PaaS): The capability offered to consumers is the deployment of consumer-created or off-the-shelf applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0080] Infrastructure as a Service (IaaS): The capability offered to the consumer is the provisioning of processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating system, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0081] The deployment model is as follows:

[0082] Private Cloud: Cloud infrastructure is operated exclusively for an organization and, in one embodiment, is managed by that organization or a third party and can exist on-premise or off-premise.

[0083] Community Cloud: Cloud infrastructure is shared by several organizations and supports a specific community with a shared set of objectives (e.g., mission, security requirements, policies, and compliance considerations). In one embodiment, it is managed by the organization or a third party and resides on-premises or off-premises.

[0084] Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0085] Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a unique entity, but are joined by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that allow for data and application portability.

[0086] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0087] FIG. 7 is a diagram of a cloud computing environment 50 according to an embodiment of the present invention. As shown, in one embodiment, the cloud computing environment 50 includes one or more cloud computing nodes 10 that communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or a combination thereof. In one embodiment, the nodes 10 communicate with each other. In one embodiment, they are grouped, physically or virtually, in one or more networks (not shown), such as a private, community, public, or hybrid cloud, or a combination thereof, as described herein above. This enables the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, eliminating the need for cloud consumers to maintain resources on their local computing devices. It will be understood that the types of computing devices 54A-N shown in FIG. 5 are intended to be exemplary only, and that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0088] 8 is a diagram illustrating a set of functional abstraction layers provided by a cloud computing environment 50, in accordance with an embodiment of the present invention. It should be understood in advance that the components, layers, and functions illustrated in FIG. 8 are intended to be merely exemplary, and that embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0089] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computer) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0090] In one embodiment, the virtualization layer 70 provides an abstraction layer at which the following examples of virtual entities are provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0091] In one embodiment, the management layer 80 provides the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and billing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one embodiment, these resources include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance agreement on and procurement of cloud computing resources in anticipation of future demand according to SLAs.

[0092] In one embodiment, the workload layer 90 provides examples of functionality for which a cloud computing environment is utilized. In one embodiment, example workloads and functions provided by this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and AI 96 (e.g., adaptive and multimodal learning systems 330 (FIG. 1)).

[0093] 9 is a high-level block diagram illustrating an information processing system 700 useful for implementing one embodiment of the present invention. The computer system includes one or more processors, such as processor 702. Processor 702 is connected to a communications infrastructure 704 (e.g., a communications bus, crossover bar, or network).

[0094] The computer system may include a display interface 706 that transfers graphics, text, and other data from the audio communications infrastructure 704 (or from a frame buffer, not shown) for display on a display unit 708. In one embodiment, the computer system also includes a main memory 710, preferably random access memory (RAM), and a secondary memory 712. In one embodiment, the secondary memory 712 includes, for example, a hard disk drive 714 and / or a removable storage drive 716, representing, for example, a floppy disk drive, a magnetic tape drive, or an optical disk drive. The removable storage drive 716 reads and / or writes from / to a removable storage unit 718, in a manner well known to those skilled in the art. The removable storage unit 718 represents, for example, a floppy disk, a compact disk, a magnetic tape, or an optical disk that may be read by and written to the removable storage drive 716. As will be appreciated, removable storage unit 718 includes a computer-readable medium having computer software and / or data stored thereon.

[0095] In alternative embodiments, secondary memory 712 includes other similar means for allowing computer programs or other instructions to be loaded into the computer system, such as removable storage units 720 and interfaces 722. Examples of such means include program packages and package interfaces (such as those found in video game consoles), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units 720 and interfaces 722 that allow software and data to be moved from removable storage units 720 to the computer system.

[0096] In one embodiment, the computer system also includes a communications interface 724. The communications interface 724 allows software and data to be transferred between the computer system and external devices. In one embodiment, examples of the communications interface 724 include a modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card. In one embodiment, the software and data transferred through the communications interface 724 are in the form of, for example, electronic, electromagnetic, optical, or other signals capable of being received by the communications interface 724. These signals are provided to the communications interface 724 via a communications path (i.e., channel) 726. In one embodiment, the communications path 726 carries signals and is implemented using electrical wire or cable, fiber optics, a telephone line, a cellular phone link, an RF link, or other communications channels, or a combination thereof.

[0097] Embodiments of the present invention may be systems, methods, and / or computer program products at any possible level of technical detail of integration. The computer program product may include computer-readable storage medium(s) having computer-readable program instructions for causing a processor to perform aspects of embodiments of the present invention.

[0098] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as instruction-recorded punch cards or grooved structures, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as ephemeral signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted over electrical wires.

[0099] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a respective computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0100] Computer-readable program instructions for carrying out operations of embodiments of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state configuration data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to implement aspects of embodiments of the present invention.

[0101] Aspects of embodiments of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0102] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0103] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to create a computer-implemented process and cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0104] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as a single step, or may be executed concurrently, substantially concurrently, partially, or fully in a time-overlapping manner, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, are implemented by special-purpose hardware-based systems that perform the specified functions or actions or execute a combination of special-purpose hardware and computer instructions.

[0105] From the above description, it can be seen that embodiments of the present invention provide systems, computer program products, and methods for implementing the embodiments of the present invention. Embodiments of the present invention further provide a non-transitory computer-usable storage medium for implementing the embodiments of the present invention. The non-transitory computer-usable storage medium has a computer-readable program that, when executed on a computer, causes the computer to perform the steps of the embodiments of the present invention described herein. References to elements in the singular in the claims are not intended to mean "one" or "only one" unless expressly so stated, but rather "one or more." All structural and functional equivalents to the elements of the exemplary embodiments described above, now known or later known to those skilled in the art, are intended to be encompassed by the claims. No claim element herein shall be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase "means for" or "step for."

[0106] The terminology used herein is for the purpose of describing particular embodiments of the invention only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprise" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups or combinations thereof.

[0107] Corresponding structure, material, acts, and equivalents of all means or step-plus-function elements in the following claims are intended to include any structure, material, or acts for performing the function in combination with other claimed elements as specifically claimed.

[0108] While descriptions of various embodiments of the present invention have been presented for illustrative purposes, they are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications or technical improvements over technologies found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for image recognition using computer information processing, the method comprising: receiving input video in a plurality of data modalities, the input video comprising a sequence of video segments; selecting, for a video segment of the sequence, one or more data modalities from the plurality of data modalities based on data representing the video segment, wherein each selected data modality is best suited for video recognition of the video segment; for each selected data modality, providing at least one data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality, and generating a first type of prediction representative of the video segment via the machine learning model; determining a second type prediction representative of the entire input video by aggregating all generated first type predictions, wherein the second type prediction is indicative of an object or activity captured in the input video; Including, extracting, via a joint feature extractor, joint features from the data input representing the video segment in the multiple data modalities; computing a hidden state for the video segment via a long short-term memory (LSTM) based in part on the extracted combined features; For each data modality of the plurality of data modalities, estimating a corresponding policy distribution based on the hidden state for the video segment; and applying a Gumbela-Softmax operation to the corresponding policy distribution to sample a corresponding binary decision indicating whether to select the data modality for the video recognition of the video segment; The method further comprises:

2. The method of claim 1 , wherein the plurality of data modalities includes at least one of an RGB modality, an optical flow modality, and an audio modality.

3. The method of claim 1 , wherein the data representing the video segment includes at least one of one or more RGB frames, one or more RGB difference frames, and one or more audio frames.

4. 10. The method of claim 1, wherein each data modality of the plurality of data modalities has a corresponding machine learning model that is trained in conjunction with one or more other machine learning models corresponding to one or more other data modalities of the plurality of data modalities.

5. The method of claim 4 , wherein each machine learning model corresponding to each data modality of the plurality of data modalities comprises a sub-network.

6. The method of claim 1 , wherein the selected one or more data modalities provide an optimal trade-off between video recognition accuracy and computational efficiency.

7. The method of claim 1 , wherein each data modality of the plurality of data modalities that is not selected is redundant for the video recognition of the video segment.

8. 1. A system for video recognition, comprising: at least one processor; a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, receiving input video in a plurality of data modalities, the input video comprising a sequence of video segments; selecting, for a video segment of the sequence, one or more data modalities from the plurality of data modalities based on data representing the video segment, wherein each selected data modality is best suited for video recognition of the video segment; for each selected data modality, providing at least one data input representing the video segment in the selected data modality to a machine learning model corresponding to the selected data modality, and generating a first type of prediction representative of the video segment via the machine learning model; determining a second type prediction representative of the entire input video by aggregating all generated first type predictions, wherein the second type prediction is indicative of an object or activity captured in the input video; the non-transitory processor-readable memory device performing operations including Equipped with The instruction: extracting, via a joint feature extractor, joint features from the data input representing the video segment in the multiple data modalities; computing a hidden state for the video segment via a long short-term memory (LSTM) based in part on the extracted combined features; For each data modality of the plurality of data modalities, estimating a corresponding policy distribution based on the hidden state for the video segment; and applying a Gumbela-Softmax operation to the corresponding policy distribution to sample a corresponding binary decision indicating whether to select the data modality for the video recognition of the video segment; The system further includes:

9. The system of claim 8 , wherein the plurality of data modalities includes at least one of an RGB modality, an optical flow modality, and an audio modality.

10. The system of claim 8 , wherein the data representing the video segment includes at least one of one or more RGB frames, one or more RGB difference frames, and one or more audio frames.

11. 10. The system of claim 8, wherein each data modality of the plurality of data modalities has a corresponding machine learning model that is trained in conjunction with one or more other machine learning models corresponding to one or more other data modalities of the plurality of data modalities.

12. 12. The system of claim 11, wherein each machine learning model corresponding to each data modality of the plurality of data modalities comprises a sub-network.

13. The system of claim 8 , wherein the selected one or more data modalities provide an optimal trade-off between video recognition accuracy and computational efficiency.

14. The system of claim 8 , wherein each data modality of the plurality of data modalities that is not selected is redundant for the video recognition of the video segment.

15. A computer program causing a computer to execute the method according to any one of claims 1 to 7.

16. A computer-readable storage medium storing the computer program according to claim 15.

Images