Co-action positioning

Abstract

Aspects of the present disclosure relate to an apparatus configured to perform co-action positioning. In certain aspects, the apparatus may receive a query video including a plurality of frames, where a first query proposal is determined based on a subset of frames of the plurality of frames, the first query proposal indicating an action depicted on the subset of frames. In certain aspects, the apparatus may determine a first degree of interest for a first support video of the plurality of support videos. In certain aspects, the apparatus may determine a second degree of interest for a second support video of the plurality of support videos after calculating the first degree of interest.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. non-provisional application serial number 18 / 360,741, filed on July 27, 2023, entitled “COMMMON ACTION LOCALIZATION,” and U.S. patent application serial number 63 / 450,924, filed on March 8, 2023, entitled “COMMMON ACTION LOCALIZATION,” the disclosures of which are expressly incorporated herein by reference in their entireties. Background Art Technical Field

[0003] The present disclosure relates generally to machine learning and, more particularly, to systems and methods for improving action recognition and localization.

[0004] introduction

[0005] An artificial neural network, which may include a set of interconnected artificial neurons (eg, a neuron model), is a computing device or represents a method to be performed by a computing device.

[0006] A convolutional neural network is a type of feedforward artificial neural network. A convolutional neural network can include a collection of neurons, each with a receptive field that collectively map out an input space. Convolutional neural networks (CNNs) have numerous applications. In particular, CNNs have been widely used in the fields of pattern recognition and classification.

[0007] Deep learning architectures such as deep belief networks and deep convolutional networks are layered neural network architectures in which the outputs of neurons in the first layer become the inputs to neurons in the second layer, the outputs of neurons in the second layer become the inputs to neurons in the third layer, and so on. Deep neural networks can be trained to recognize hierarchical structures of features, and as a result, they are increasingly used in object recognition applications. Like convolutional neural networks, computation in these deep learning architectures can be distributed across a set of processing nodes, which can be configured in one or more computation chains. These multi-layer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.

[0008] Other models can also be used for object recognition. For example, support vector machines (SVMs) are learning tools that can be applied to classification. Support vector machines include a separating hyperplane (e.g., a decision boundary) that classifies data. The hyperplane is defined by supervised learning. The desired hyperplane increases the margin of the training data. In other words, the hyperplane should have the maximum minimum distance to the training examples.

[0009] Computational networks such as recurrent neural networks can also be used to recognize sequences and other temporal data. However, such networks are computationally complex and consume a lot of computing resources. Summary of the Invention

[0010] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects and is neither intended to identify key or critical elements of all aspects nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be presented later.

[0011] Certain aspects relate to an apparatus for performing common action localization. The apparatus may include one or more memories having instructions, either individually or in combination, and one or more processors, either individually or in combination, configured to execute the instructions. In some examples, the one or more processors may be configured to cause the apparatus to receive a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, the first query proposal indicating an action depicted on the subset of frames. In some examples, the one or more processors may be configured to cause the apparatus to determine a first attendance for a first supporting video in a plurality of supporting videos. In some examples, the one or more processors may be configured to cause the apparatus to determine a second attendance for a second supporting video in the plurality of supporting videos after calculating the first attendance. In some examples, the one or more processors may be configured to cause the apparatus to output a classification for the subset of frames based at least in part on the first attendance and the second attendance.

[0012] Certain aspects relate to a method for performing common action localization. In some examples, the method includes receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, the first query proposal indicating an action depicted on the subset of frames. In some examples, the method includes determining a first attention level for a first supporting video in a plurality of supporting videos. In some examples, the method includes, after calculating the first attention level, determining a second attention level for a second supporting video in the plurality of supporting videos. In some examples, the method includes outputting a classification for the subset of frames based at least in part on the first attention level and the second attention level.

[0013] Certain aspects relate to an apparatus for performing common action localization. In some examples, the apparatus includes components for receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, the first query proposal indicating an action depicted on the subset of frames. In some examples, the apparatus includes components for determining a first attention level for a first supporting video in a plurality of supporting videos. In some examples, the apparatus includes components for determining a second attention level for a second supporting video in the plurality of supporting videos after calculating the first attention level. In some examples, the apparatus includes components for outputting a classification of the subset of frames based at least in part on the first attention level and the second attention level.

[0014] Certain aspects relate to a non-transitory computer-readable medium comprising computer-executable code that, when executed by one or more processors, causes the one or more processors to perform operations, individually or in combination. In some examples, the operations include: receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, the first query proposal indicating an action depicted on the subset of frames. In some examples, the operations include: determining a first attention level for a first supporting video in a plurality of supporting videos. In some examples, the operations include: after calculating the first attention level, determining a second attention level for a second supporting video in the plurality of supporting videos. In some examples, the operations include: outputting a classification of the subset of frames based at least in part on the first attention level and the second attention level.

[0015] To accomplish the foregoing and related ends, one or more aspects include the features fully described below and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail certain illustrative features of one or more aspects. However, these features are indicative of but a few of the various ways in which the principles of the various aspects may be employed, and this description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An example implementation of designing a neural network using a system on a chip (SoC) including a general-purpose processor according to certain aspects of the present disclosure is illustrated.

[0017] Figure 2 Example implementations of systems according to aspects of the present disclosure are illustrated.

[0018] Figure 3 is a block diagram illustrating an example process for common action localization.

[0019] Figure 4 is a block diagram illustrating an example process flow for common action localization.

[0020] Figure 5 This is an example Figure 4 A block diagram of an example flow process of the first process.

[0021] Figure 6 This is an example Figure 4 A block diagram of an example flow process of the second process.

[0022] Figure 7 This is an example Figure 4 A block diagram of an example flow process of the third process.

[0023] Figure 8 is a block diagram illustrating an example of a relational classifier.

[0024] Figure 9 is a flowchart illustrating a method of performing common motion localization. DETAILED DESCRIPTION

[0025] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some cases, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0026] Various aspects of the present disclosure relate to a single cross-attention for a query proposal for each support video. Typically, there is relatively large intra-class variation between support videos. For example, support videos may have different backgrounds, different lighting, camera angles, etc. Therefore, finding one or more frames of a support video that match the action of the query video requires a more sophisticated approach than conventionally used methods. Conventionally, the attention of a query proposal is determined using frames from multiple support videos simultaneously. Therefore, a frame from one support video that matches the action of the query video may not be attended to by the query proposal.

[0027] Thus, aspects of the present disclosure relate to techniques and methods for determining or calculating attention for one frame of a supporting video at a time rather than for multiple frames of the supporting video simultaneously, thereby appropriately identifying frames of each supporting video that match the action of the query video in the context of the query.

[0028] In certain aspects, a neural network device (e.g., a system-on-chip) may receive a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, and wherein the first query proposal indicates an action depicted in the subset of frames. For example, the query video may comprise a plurality of frames, wherein a subset of one or more frames in the plurality of frames depicts an action, such as a deadlift weight as part of a weight training regimen. The device may then determine a first level of interest for a first supporting video in a plurality of supporting videos. That is, the device may determine which one or more frames of the first supporting video are likely to depict a deadlift.

[0029] Once the device has completed the first attention level for the first supporting video, the device may determine the second attention level for the second supporting video. Here, the device may determine which one or more frames of the second supporting video are likely to depict a hard pull. Thus, the device calculates the attention level for each supporting video individually or serially rather than in parallel.

[0030] Several aspects of telecommunication systems will now be presented with reference to various apparatuses and methods. These apparatuses and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively, "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0031] As an example, an element, or any part of an element, or any combination of elements can be implemented as a "processing system", which includes one or more processors. The example of a processor includes a microprocessor, a microcontroller, a graphics processing unit (GPU), a central processing unit (CPU), an application processor, a digital signal processor (DSP), a reduced instruction set computing (RISC) processor, a system on a chip (SoC), a baseband processor, a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gating logic, a discrete hardware circuit and other suitable hardware configured to perform various functionalities described throughout this disclosure. One or more processors in a processing system can execute software. Whether it is referred to as software, firmware, middleware, microcode, hardware description language or other names, software should be broadly interpreted as meaning instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, processes, functions, etc.

[0032] Therefore, in one or more example embodiments, the described functions can be implemented with hardware, software or any combination thereof. If implemented in software, the function can be stored or encoded on a computer-readable medium as one or more instructions or codes. Computer-readable media includes computer storage media. Storage media can be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media can include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), an optical disk storage device, a magnetic disk storage device, other magnetic storage devices, a combination of computer-readable media of the above type or any other medium that can be used to store computer-executable code in the form of an instruction or data structure that can be accessed by a computer. In some respects, computer-readable media can include non-transient computer-readable media (e.g., tangible media). In addition, for other aspects, computer-readable media can include transient computer-readable media (e.g., signal). The above combination should also be included in the scope of computer-readable media.

[0033] Figure 1 An example implementation of the aforementioned action localization using a system-on-chip (SoC) 100 according to certain aspects of the present disclosure is illustrated, which may include a general-purpose processor (CPU) or a multi-core general-purpose processor (CPU) 102. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a dedicated memory block 118, or may be distributed across multiple blocks. Instructions executed at the general-purpose processor 102 may be loaded from a program memory associated with the CPU 102 or may be loaded from the dedicated memory block 118.

[0034] SoC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fourth generation long term evolution (4G LTE) connectivity, unlicensed Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one embodiment, the NPU is implemented in the CPU, DSP, and / or GPU. SoC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or navigation 120 (which may include a global positioning system).

[0035] SoC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving a sequence of frames. The instructions loaded into the general-purpose processor 102 may also include code for generating an attention feature map for each frame in the sequence of frames. The instructions loaded into the general-purpose processor 102 may also include code for upsampling each attention feature map to determine the attention saliency of the pixels in each frame. The instructions loaded into the general-purpose processor 102 may further include code for generating a bounding box within each frame based on the attention saliency. In addition, the instructions loaded into the general-purpose processor 102 may also include code for generating a bounding box within each frame based on the attention saliency. Additionally, the instructions loaded into the general-purpose processor 102 may also include code for temporally smoothing multiple bounding boxes along the sequence of frames to obtain a smoothed sequence of bounding boxes. Finally, the instructions loaded into the general-purpose processor 102 may also include code for locating the action position within each frame based on the smoothed sequence of bounding boxes.

[0036] Figure 2 An example implementation of a system 200 according to certain aspects of the present disclosure is illustrated. Figure 2 As illustrated, the system 200 may have a plurality of local processing units 202 that may perform various operations of the methods described herein. Each local processing unit 202 may include a local state memory 204 and a local parameter memory 206 that may store parameters of a neural network. In addition, the local processing unit 202 may have a local (neuron) model program (LMP) memory 208 for storing a local model program, a local learning program (LLP) memory 210 for storing a local learning program, and a local connection memory 212. In addition, as shown in FIG. Figure 2 As illustrated, each local processing unit 202 may interface with a configuration processor unit 214 for providing configuration for the local memory of the local processing unit and with a routing connection processor unit 216 for providing routing between the local processing units 202 .

[0037] In one configuration, a processing model is configured to receive a frame sequence, generate an attention feature map for each frame in the frame sequence; upsample each attention feature map to determine the attention saliency of pixels in each frame; generate a bounding box within each frame in the frame sequence based on the attention saliency; temporally smooth a plurality of bounding boxes along the frame sequence to obtain a smoothed sequence of bounding boxes; and locate an action position within each frame in the frame sequence based on the smoothed sequence of bounding boxes. The model includes a generation component, an upsampling component, a localization component, and / or a temporal smoothing component. In one configuration, the generation component, the localization component, and / or the temporal smoothing component can be a general-purpose processor 102, a program memory associated with the general-purpose processor 102, a memory block 118, a local processing unit 202, and / or a routing connection processing unit 216 configured to perform the functions described. In another configuration, the aforementioned components can be any module or any device configured to perform the functions described by the aforementioned components.

[0038] Neural networks can be designed to have a variety of connectivity patterns. In a feedforward network, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be conveyed to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks delivered to the neural network in sequence. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections can be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input.

[0039] Figure 3 is a block diagram illustrating an example process 300 for common action localization. At a first block 302, at least one processor may receive an untrimmed query video. That is, the video may be an unedited video (e.g., a complete set of frames) that includes content related to multiple different actions. For example, the video may include frames depicting content related to an individual performing a deadlift weight training exercise, while other frames may depict other actions (e.g., different weight training exercises, rest periods, etc.).

[0040] At a second block 304, at least one processor may generate a plurality of query proposals, wherein each query proposal indicates a proposed action classifier. For example, the plurality of query proposals may be determined based on a plurality of different actions depicted in the query video. In some examples, a neural network may determine the plurality of query proposals.

[0041] At a third block 306, at least one processor may receive a plurality of supporting videos. In some examples, the supporting videos do not include any action labels or identifiers for the actions depicted therein. The supporting videos may be pruned videos, meaning that the supporting videos may depict only a single action, or fewer actions than the query video.

[0042] At a fourth block 308, at least one processor may determine an alignment between one or more of the query suggestions and the supporting videos. For example, if the first query suggestion is "hard pull" and the at least one processor determines that the first supporting video depicts a hard pull, the at least one processor may determine that the first supporting video is aligned with (e.g., is common to) the first query suggestion.

[0043] At a fifth block 310, at least one processor may determine one or more query proposals that are most common for the support videos. For example, at least one processor may classify each query proposal as being common across the support videos or not common across the support videos. If a query proposal is classified as common, the query proposal is detected as an action that is common with the query video and / or is common between the query video and the support videos. The at least one processor may also define each start time and end time of the common action within the query video. For example, the at least one processor may determine one or more time windows of the query video that depict the common action.

[0044] In some examples, the fourth block 308 may include query-to-support (QtoS) contextual cross-attention and support-to-support temporal dynamic cross-attention, as described in more detail below.

[0045] Figure 4 is a block diagram illustrating an example process flow 400 for common action location. The process flow 400 is illustrated as including three separate processes: a first process 402, a second process 404, and a third process 406. The first process 402 and the second process 404 may be used with Figure 3 The third process 406 may be performed with Figure 3 Corresponding to the fifth frame 310.

[0046] At a first process 402, a set of query proposals (X Q ) and the first supporting video is the input. Each query proposal is given in the untrimmed query video (V Q ) is used to represent possible action instances depicted in the query proposal and each support video by determining the row-wise soft maximum of the cross-correlation matrix. (from query to support)) is applied to each pair of query proposal and support video. For example, the cross-correlation can be performed using the following equation:

[0047]

[0048] Where W Q→S is a learnable matrix shared by all L support videos. The result is the cross-fading frame-level feature of the l-th support video in The definition is as follows:

[0049]

[0050] therefore, is obtained for all L supporting videos, and in a single case it is

[0051] Figure 5 This is an example Figure 4 The block diagram is a block diagram of an example process flow of the first process 402. The block diagram includes a set of query proposals (X Q )502, the first video is supported 504, the learnable matrix (W Q→S )506, cross attention weight 508 and the resulting cross-fading frame-level features 510.

[0052] In some respects, Figure 4 The first process 402 can calculate the attention (e.g., the relevance of the query proposal to the supporting videos) for one frame of the supporting videos at a time. That is, instead of only the dominant segment of the first supporting video being focused on while the other supporting videos are suppressed, all frames of all supporting videos are focused on. As a result, the important frames of each supporting video are appropriately transformed into the context of the query.

[0053] Return Reference Figure 4 , the second process 404 can apply a one-dimensional temporal convolution at the frame level. For example, tuple-level features (e.g., all supporting videos) and frame-level features (e.g., a single supporting video) can together convey a more robust video representation. That is, cross-attention can provide a collaborative use of features of different temporal granularities. Moreover, for compatibility between different supporting videos (e.g., for commonality extraction), the temporal dynamics from all supporting videos can be propagated to the supporting frame-level features including the resulting decay. Therefore, It is obtained individually for all supporting videos.

[0054] Figure 6 This is an example Figure 4 The block diagram includes a cross-fading frame-level feature for all L supported videos. 602. Cross-fading frame-level features 604、Learnable Matrix (W S′→S )606, Cross Attention Weight 608 and the resulting attenuated support frame-level features 610.

[0055] Return Reference Figure 4 , the third process 406 can use all supporting videos from the first process 402 and the second process 404 to focus on all query proposals simultaneously. Thus, cross-fading is used to generate enhanced query proposals Figure 7 This is an example Figure 4 Here, the set of query proposals (X Q )702 and attenuated support frame level features 704 is input to the cross attention weight (A S→Q )708's learnable matrix (W S→Q )706, and get enhanced query suggestions 710. Enhanced Query Proposal 710 may be output to the relationship classifier 408 .

[0056] Figure 8 This is an example Figure 4 8. The relationship classifier 408 may include an auxiliary relationship module 804 and an action classifier 806.

[0057] The auxiliary relationship module 804 may concatenate the average value of all supporting video features (e.g., Figure 6 of 610) and enhanced query proposals (e.g., Figure 7 The i-th enhanced query proposal 710 ) as input. Even during training, pseudo motion classification may be unrelated to real motion classification. Alternatively, pseudo motion classification can be obtained by k-means clustering of enhanced query proposals. For example, the mapped k-means clustering may be pseudo motion classification of query proposals. In some examples, the pseudo motion classification indicator can be 1 or 0, where 1 is for pseudo motion classification and 0 in other cases. The auxiliary relationship module 804 can output a prediction of whether the support video and the query proposal are similar or dissimilar.

[0058] The action classifier 806 can propose the i-th enhanced query As input, the proposal is weighted by the distance between the supporting video and the proposed query. The action classifier 806 can output: (1) the probability of action or non-action, and (2) an offset (a refinement of the start and end time of the query proposal for correct localization). The offset can be used when the action probability is above 0.5.

[0059] As a post-processing, final localization is performed on the predicted and refined action proposals. For overlapping proposals, the one with the highest action probability is used and the rest are removed.

[0060] Figure 9 900 is a flowchart illustrating a method for performing common action positioning. The method may be performed by one or more processors (e.g., Figure 1 CPU 102 in Figure 1 GPU 104 in Figure 1 DSP 106 in Figure 1 NPU 108, etc.) is executed.

[0061] At 902, one or more processors may receive a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, the first query proposal indicating an action depicted in the subset of frames. For example, a device (e.g., one or more processors) may receive a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames. The first query proposal may indicate an action depicted in the subset of frames.

[0062] At 904, one or more processors may determine a first attention level for a first supporting video from among the plurality of supporting videos. In some examples, the attention level may be related to the importance of one or more frames of the supporting video. For example, the frame attention level may indicate an action probability of finding a proposed action in a particular frame of the supporting video.

[0063] At 906 , the one or more processors may determine a second attention level for a second supporting video in the plurality of supporting videos after calculating the first attention level.

[0064] At 908, the one or more processors may optionally apply a one-dimensional temporal convolution to one or more frames from each of the first supporting video and the second supporting video that matches the first query proposal. At 910, the one or more processors may also optionally generate a third supporting video composed of one or more frames from each of the first supporting video and the second supporting video that matches the first query proposal.

[0065] At 912 , the one or more processors may determine that a first query proposal of the plurality of query proposals has a highest probability of sharing a common action among the plurality of supporting videos relative to other query proposals of the plurality of query proposals.

[0066] At 914 , the one or more processors may classify each of the first supporting video and the second supporting video based on the pseudo motion classification.

[0067] At 916 , the one or more processors may output a classification of the subset of frames based at least in part on the first level of interest and the second level of interest.

[0068] In some aspects, the second level of interest is determined independently of the first level of interest.

[0069] In certain aspects, the first interestingness indicates whether the first supporting video includes one or more frames that match the first query proposal, and wherein the second interestingness indicates whether the second supporting video includes one or more frames that match the first query proposal.

[0070] In some aspects, pseudo-motion classification is mapped to k-means clustering.

[0071] Additional Notes

[0072] The means for receiving and the means for outputting may include connectivity blocks (e.g., Figure 3 The components for determining, the components for applying, the components for generating, and the components for classifying may include a CPU (e.g., Figure 1 CPU 102) and digital storage devices (e.g., Figure 1 One or more of the memory blocks 118).

[0073] It should be understood that the specific order or hierarchy of blocks in the disclosed process / flowchart is merely illustrative of exemplary methods. It should be understood that the specific order or hierarchy of blocks in the process / flowchart may be rearranged based on design preferences. Furthermore, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0074] As used herein, a processor, at least one processor, and / or one or more processors configured to perform or operable to perform a plurality of actions (individually or in combination) is intended to include at least two different processors capable of performing different subsets, overlapping subsets, or non-overlapping subsets of the plurality of actions, or a single processor capable of performing all of the plurality of actions. In a non-limiting example of a plurality of processors capable of performing different combinations of actions in the plurality of actions, a description of a processor, at least one processor, and / or one or more processors configured to perform or operable to perform actions X, Y, and Z may include at least a first processor configured to perform or operable to perform a first subset of X, Y, and Z (e.g., perform X) and at least a second processor configured to perform or operable to perform a second subset of X, Y, and Z (e.g., perform Y and Z). Alternatively, the first processor, the second processor, and the third processor may be configured to perform or operable to perform corresponding actions in actions X, Y, and Z, respectively. It should be understood that any combination of one or more processors may each be configured to perform or operable to perform any one of the plurality of actions or any combination of the plurality of actions.

[0075] As used herein, a memory, at least one memory, and / or one or more memories configured to store or having stored thereon instructions for performing a plurality of actions that are executable by one or more processors are intended to include at least two different memories capable of storing different subsets, overlapping subsets, or non-overlapping subsets of instructions for performing the plurality of actions, or a single memory capable of storing instructions for performing all of the plurality of actions. In one non-limiting example of one or more memories (alone or in combination) capable of storing different subsets of instructions for performing different ones of the plurality of actions, a description of a memory, at least one memory, and / or one or more memories configured or operable to store or having stored thereon instructions for performing actions X, Y, and Z may include at least a first memory configured or operable to store or having stored thereon a first subset of instructions for performing a first subset of X, Y, and Z (e.g., instructions for performing X), and at least a second memory configured or operable to store or having stored thereon a second subset of instructions for performing a second subset of X, Y, and Z (e.g., instructions for performing Y and Z). Alternatively, the first memory, the second memory, and the third memory may each be configured to store or have stored thereon a respective one of the first subset of instructions for performing X, the second subset of instructions for performing Y, and the third subset of instructions for performing Z. It should be understood that any combination of one or more memories may each be configured or operable to store or have stored thereon any one or any combination of instructions that can be executed by one or more processors to perform any one or any combination of a plurality of actions. Furthermore, one or more processors may each be coupled to at least one of the one or more memories and configured or operable to execute the instructions to perform the plurality of actions. For example, in the above non-limiting example of different instruction subsets for performing actions X, Y, and Z, a first processor may be coupled to a first memory storing instructions for performing action X, at least a second processor may be coupled to at least a second memory storing instructions for performing actions Y and Z, and the first and second processors may, in combination, execute the respective instruction subsets to perform actions X, Y, and Z. Alternatively, three processors may access one of three different memories, each storing instructions for performing action X, Y, or Z, and the three processors may, in combination, execute the respective instruction subsets to perform actions X, Y, and Z. Alternatively, a single processor may execute instructions stored in a single memory or distributed across multiple memories to perform actions X, Y, and Z.

[0076] The foregoing description is provided to enable anyone skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Accordingly, the claims are not intended to be limited to the aspects shown herein, but rather to be consistent with the full scope of the claims, whereby elements referred to in the singular are not intended to mean "one and only one," unless specifically so indicated, but rather "one or more." Terms such as "if," "when," and "while" should be interpreted as meaning "under the conditions of," rather than implying an immediate temporal relationship or reaction. That is, these phrases, such as "when," do not imply immediate action in response to or during the occurrence of an action, but simply imply that if a condition is met, the action will occur, without requiring a specific or immediate time limit for the action to occur. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects. Unless otherwise specified, the term "some" refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” include any combination of A, B, and / or C and may include multiple As, multiple Bs, or multiple Cs. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” may be A only, B only, C only, A and B, A and C, B and C, or A, B, and C, where any such combination may include one or more members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recited in the claims. Words such as "module," "mechanism," "element," and "device" are not intended to replace the word "component." Thus, no claim element is to be construed as part-plus-function unless the element is explicitly recited using the phrase "means for..."

[0077] Example aspects

[0078] The following examples are merely illustrative and may be combined with aspects of other embodiments or teachings described herein, but are not limited thereto.

[0079] Embodiment 1 is a method for performing common action localization, the method comprising: receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames in the plurality of frames, the first query proposal indicating an action depicted on the subset of frames; determining a first attention level for a first support video in a plurality of support videos; after calculating the first attention level, determining a second attention level for a second support video in the plurality of support videos; and outputting a classification of the subset of frames based at least in part on the first attention level and the second attention level.

[0080] Embodiment 2 is a method according to embodiment 1, wherein the second level of attention is determined independently of the first level of attention.

[0081] Embodiment 3 is a method according to any one of embodiments 1 and 2, wherein the first attention level indicates whether the first supporting video includes one or more frames that match the first query proposal, and wherein the second attention level indicates whether the second supporting video includes one or more frames that match the first query proposal.

[0082] Embodiment 4 is the method of embodiment 3, wherein the method further comprises applying a one-dimensional temporal convolution to the one or more frames from each of the first supporting video and the second supporting video that match the first query proposal.

[0083] Embodiment 5 is a method according to any one of embodiments 3 and 4, wherein the method further comprises: generating a third supporting video, wherein the third supporting video is composed of the one or more frames from each of the first supporting video and the second supporting video that match the first query proposal.

[0084] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the first query proposal is one of multiple query proposals determined based on the multiple frames, and wherein the method further includes: determining that the first query proposal among the multiple query proposals has the highest probability of sharing a common action among the multiple supporting videos relative to other query proposals among the multiple query proposals.

[0085] Embodiment 7 is the method of any one of embodiments 1 to 6, wherein the method further comprises classifying each of the first supporting video and the second supporting video based on a pseudo motion classification.

[0086] Embodiment 8 is a method according to embodiment 7, wherein the pseudo motion classification is mapped to k-means clustering.

[0087] Embodiment 9 is a device comprising: one or more memories, the one or more memories comprising instructions individually or in combination; and one or more processors, the one or more processors being configured individually or in combination to execute the instructions so that the device performs a method according to any one of embodiments 1 to 8.

[0088] Embodiment 10 is an apparatus comprising components for performing the method according to any one of embodiments 1 to 8.

[0089] Embodiment 11 is a non-transitory computer-readable medium comprising instructions that, when executed by a device, cause the device to perform the method according to any one of embodiments 1 to 8.

Claims

1. A device for performing common action positioning, the device comprising: one or more memories, the one or more memories individually or in combination having instructions; and one or more processors, the one or more processors being configured, individually or in combination, to execute the instructions and cause the apparatus to: receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames of the plurality of frames, the first query proposal indicating an action depicted on the subset of frames; determining a first attention level of a first supporting video among a plurality of supporting videos; After calculating the first attention level, determining a second attention level of a second supporting video among the plurality of supporting videos; as well as A classification of the subset of frames is output based at least in part on the first level of interest and the second level of interest. The apparatus according to claim 1 , wherein the second level of attention is determined independently of the first level of attention.

3. An apparatus according to claim 1, wherein the first attention level indicates whether the first supporting video includes one or more frames that match the first query proposal, and wherein the second attention level indicates whether the second supporting video includes one or more frames that match the first query proposal.

4. The apparatus of claim 3, wherein the one or more processors are further configured to cause the apparatus to: A one-dimensional temporal convolution is applied to the one or more frames from each of the first supporting video and the second supporting video that match the first query proposal.

5. The apparatus of claim 3 , wherein the one or more processors are further configured to cause the apparatus to: A third supporting video is generated, the third supporting video consisting of the one or more frames from each of the first supporting video and the second supporting video that matched the first query proposal.

6. The apparatus of claim 1 , wherein the first query proposal is one of a plurality of query proposals determined based on the plurality of frames, and wherein the one or more processors are further configured to cause the apparatus to: The first of the plurality of query suggestions is determined to have a highest probability of sharing a common action among the plurality of supporting videos relative to other query suggestions of the plurality of query suggestions.

7. The apparatus of claim 1 , wherein the one or more processors are further configured to cause the apparatus to: Each of the first support video and the second support video is classified based on a pseudo motion classification. The apparatus of claim 7 , wherein the pseudo motion classification is mapped to k-means clustering.

9. A method for performing common action positioning, the method comprising: receiving, by one or more processors, a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames of the plurality of frames, the first query proposal indicating an action depicted on the subset of frames; determining, by the one or more processors, a first attention level of a first supporting video among a plurality of supporting videos; After calculating the first attention level, the one or more processors determine a second attention level of a second supporting video among the plurality of supporting videos; as well as A classification of the subset of frames is outputted by the one or more processors based at least in part on the first level of interest and the second level of interest.

10. The method of claim 9, wherein the second level of interest is determined independently of the first level of interest.

11. The method of claim 9, wherein the first attention level indicates whether the first supporting video includes one or more frames that match the first query proposal, and wherein the second attention level indicates whether the second supporting video includes one or more frames that match the first query proposal.

12. The method according to claim 11, wherein the method further comprises: A one-dimensional temporal convolution is applied to the one or more frames from each of the first supporting video and the second supporting video that match the first query proposal.

13. The method according to claim 11, further comprising: A third supporting video is generated, the third supporting video consisting of the one or more frames from each of the first supporting video and the second supporting video that matched the first query proposal.

14. The method of claim 9, wherein the first query proposal is one of a plurality of query proposals determined based on the plurality of frames, and wherein the method further comprises: The first of the plurality of query suggestions is determined to have a highest probability of sharing a common action among the plurality of supporting videos relative to other query suggestions of the plurality of query suggestions.

15. The method according to claim 9, further comprising: Each of the first support video and the second support video is classified based on a pseudo motion classification. The method of claim 15 , wherein the pseudo motion classification is mapped to k-means clustering.

17. An apparatus for performing common action positioning, the apparatus comprising: means for receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames of the plurality of frames, the first query proposal indicating an action depicted on the subset of frames; means for determining a first attention level for a first supporting video among a plurality of supporting videos; a component for determining a second attention level of a second supporting video among the plurality of supporting videos after calculating the first attention level; and Means for outputting a classification of the subset of frames based at least in part on the first level of interest and the second level of interest. The apparatus of claim 17 , wherein the second level of attention is determined independently of the first level of attention.

19. The apparatus of claim 17, wherein the first attention level indicates whether the first supporting video includes one or more frames that match the first query proposal, and wherein the second attention level indicates whether the second supporting video includes one or more frames that match the first query proposal.

20. The apparatus according to claim 19, wherein the apparatus further comprises: Means for applying a one-dimensional temporal convolution to the one or more frames from each of the first supporting video and the second supporting video that match the first query proposal.

21. The apparatus according to claim 19, wherein the apparatus further comprises: Means for generating a third supporting video comprised of the one or more frames from each of the first supporting video and the second supporting video that matched the first query proposal.

22. The apparatus of claim 17, wherein the first query proposal is one of a plurality of query proposals determined based on the plurality of frames, and wherein the apparatus further comprises: Means for determining that the first query proposal of the plurality of query proposals has a highest probability of sharing a common action among the plurality of supporting videos relative to other query proposals of the plurality of query proposals.

23. The apparatus of claim 17, wherein the apparatus further comprises: Means for classifying each of the first supporting video and the second supporting video based on a pseudo motion classification.

24. The apparatus of claim 23, wherein the pseudo-motion classes are mapped to k-means clustering.

25. A non-transitory computer-readable medium comprising computer-executable code that, when executed by one or more processors, causes the one or more processors to perform operations comprising, individually or in combination: receiving a query video comprising a plurality of frames, wherein a first query proposal is determined based on a subset of frames of the plurality of frames, the first query proposal indicating an action depicted on the subset of frames; determining a first attention level of a first supporting video among a plurality of supporting videos; After calculating the first attention level, determining a second attention level of a second supporting video among the plurality of supporting videos; as well as A classification of the subset of frames is output based at least in part on the first level of interest and the second level of interest.

26. The non-transitory computer-readable medium of claim 25, wherein the second level of interest is determined independently of the first level of interest.

27. A non-transitory computer-readable medium according to claim 25, wherein the first attention level indicates whether the first supporting video includes one or more frames that match the first query proposal, and wherein the second attention level indicates whether the second supporting video includes one or more frames that match the first query proposal.

28. The non-transitory computer readable medium of claim 27, wherein the operations further comprise: A one-dimensional temporal convolution is applied to the one or more frames from each of the first supporting video and the second supporting video that match the first query proposal.

29. The non-transitory computer readable medium of claim 27, wherein the operations further comprise: A third supporting video is generated, the third supporting video consisting of the one or more frames from each of the first supporting video and the second supporting video that matched the first query proposal.

30. The non-transitory computer-readable medium of claim 25, wherein the first query proposal is one of a plurality of query proposals determined based on the plurality of frames, and wherein the operations further comprise: The first of the plurality of query suggestions is determined to have a highest probability of sharing a common action among the plurality of supporting videos relative to other query suggestions of the plurality of query suggestions.

Images