Representation learning using joint semantic vectors

Abstract

Embodiments of the present disclosure relate to representation learning using joint semantic vectors. Disclosed herein are techniques for learning motion in a video. In one implementation, an artificial neural network extracts features from a video. A correspondence proposal (CP) module performs a search for corresponding features in the video based on a semantic similarity of a given feature to other features for at least some of the features. The CP module then generates a joint semantic vector for each feature based on at least the semantic similarity of the given feature to one or more corresponding features and a spatiotemporal distance of the given feature to the one or more corresponding features. The artificial neural network is capable of identifying motion in the video using the joint semantic vectors generated for the features extracted from the video.

Description

Technical Field

[0001] Aspects of the present disclosure relate to the fields of machine learning and artificial intelligence, and in particular to representation learning in artificial neural networks. Background Art

[0002] Machine learning has advanced to the point where artificial neural networks routinely perform learning (or recognition) tasks with the same or even superior confidence as humans. For example, machine learning is now capable of highly accurate image classification, object detection, semantic segmentation, and image parsing. However, learning motion in videos remains a difficult area.

[0003] Videos can be thought of as sequences of images that change over time. To successfully model the representation of motion in videos, machine learning models identify both features in static images of the video and correspondences between features across multiple frames. Previous solutions involved a first step of extracting features (such as objects, edges, etc.) from the images that make up the video. In a second step, correspondences between features are identified. For example, it can be determined that an object in one frame corresponds to one or more objects in other frames. This relationship can be represented in several ways, such as the temporal distance between objects or the distance in feature space between objects. The extracted data (features and their correspondences) are then fed into an artificial neural network that is able to recognize motion (or other patterns) in the data.

[0004] To identify patterns in data, an artificial neural network is configured during a training phase using data with known outcomes. In a highly simplified example, an artificial neural network consists of layers of neurons. These layers are connected to each other in various ways, and the neurons fire in response to various inputs. The input to the first layer is the data being analyzed, such as the amount of data describing a video. The output of the last layer represents what the neural network has learned or recognized. In between are intermediate or "hidden" layers that perform additional processing.

[0005] Each neuron has a rule that tells it to fire (or not fire) based on its inputs. The inputs are weighted, and the weights can change based on a feedback process. In the case of video data, the input to the first layer of the artificial neural network is a vector filled with numerical or symbolic descriptions of the video data's attributes. The input to the hidden layers is the output of the neurons in the first layer. Within the hidden layers, the output from one layer is the input to the next layer. The final layer of the artificial neural network takes as input the output of the last hidden layer, and the value of its own output will indicate what has been learned as the signal passes through the layers of neurons.

[0006] During the training phase of a neural network, the weights associated with the neurons are changed based on the discrepancy between the results produced by the neural network and the correct classification of the neural network input. Over time, after multiple feedback cycles, the weights are configured so that the neural network can accurately recognize new data. The resulting configuration of weights and neurons is considered a model of the subject matter being examined, such as a facial recognition model, a speech recognition model, or, in the case of video motion, a motion representation model.

[0007] Various methods for identifying motion in videos model the correspondence between features in different ways. For example, one method describes the correspondence of features across frames by the temporal relationship between them, while another method models the flow of pixels across images, and yet another method models the distance in feature space between pairs of features. Unfortunately, these methods struggle due to the challenging behavior of features (e.g., objects) in videos when presented with fast motion or at low frame rates. For example, features may not appear in every frame, and even if they do appear, they may be found in frames that are far apart. In another example, features may deform or change their pose between frames, further inhibiting the performance of the model.

[0008] In the first approach, convolutional neural networks are used to extract features from videos, each of which can be described by a semantic vector. Convolutional neural networks work by concentrating neurons in hidden layers on a limited receptive field. While in a fully connected neural network, every neuron in one layer is connected to every neuron in another layer, in a convolutional neural network, each neuron is only connected to a limited number of neurons in other layers. This allows convolutional neural networks to identify specific features regardless of their location in the image, making them well-suited for feature extraction.

[0009] Returning to the first approach, a convolutional neural network identifies objects, edges, or other such elements that appear in each frame of a video. Their corresponding semantic vectors describe the numerical or symbolic nature of the features. A recurrent neural network is then used to detect temporal relationships between the features. However, due to the exploding and vanishing gradient problems, recurrent neural networks perform poorly on benchmarks and are difficult to train.

[0010] The second method also uses a convolutional neural network to identify features in the video and couples it with an optical flow branch to model correspondences across frames. The results of the convolutional neural network and optical flow analysis are then fed into a deep neural network. Because the optical flow branch captures pixel-level flows rather than feature-level correspondences, this technique performs poorly on benchmark data.

[0011] Finally, a 3D convolutional neural network has been employed to jointly and implicitly learn dynamic changes with static appearance using only RGB input. This approach is able to recognize short-range motion of features in videos. However, the convolutional function operates locally and is therefore unable to learn fast-moving objects or long-range motion occurring at low frame rates.

[0012] Non-local neural networks have been proposed to overcome this problem, where the identified features are globally reweighted by their pairwise feature distances. Coupled with 3D convolutional neural networks, non-local neural networks can learn long-range motion, but not very effectively. Conversely, the reweighted features produce excessive noise, leading to error-prone motion classification.

[0013] It's understandable, then, that while machine learning techniques have achieved impressive results for many types of recognition tasks, identifying motion in videos remains a daunting challenge. More generally, identifying behavior in many types of datasets (similar to videos) remains a challenge. For example, some dynamic datasets can be visualized as a series of changing images. Identifying dynamic behavior in such datasets can be as problematic and difficult as identifying motion in videos. Summary of the Invention

[0014] Disclosed herein are enhancements to correspondence proposals that improve the learning of motion in videos and the learning of behavior in dynamic data more generally. In one implementation, a correspondence proposal module (CP module) focuses on semantically similar feature pairs in a video and takes into account positional information of the feature pairs in the video. The CP module produces a joint semantic vector that has both semantic and positional information for the feature pairs implicitly encoded therein. The joint semantic vector is fed to an artificial neural network, which can then use the joint semantic vector as input to recognize motion in the video. The focus on semantically similar feature pairs and the addition of positional information represent an advancement over previous solutions and result in improved kinematic learning in videos relative to the performance of other methods on popular benchmarks.

[0015] In one implementation, an artificial neural network extracts features from a video. A CP module searches for corresponding features in the video based on the semantic similarity between a given feature and other features. The CP module then generates a joint semantic vector for each feature based on at least the semantic similarity between the given feature and one or more corresponding features and the spatiotemporal distance between the given feature and the one or more corresponding features. The artificial neural network uses the joint semantic vector generated by the CP module to identify motion in the video.

[0016] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. It should be understood that this summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Many aspects of the present disclosure may be better understood with reference to the following drawings. The components in the drawings are not necessarily drawn to scale, emphasis instead being placed on clearly illustrating the principles of the present disclosure. Furthermore, in the drawings, like reference numerals refer to corresponding parts throughout the several views. Although several embodiments have been described in conjunction with these drawings, the present disclosure is not limited to the embodiments disclosed herein. Rather, it is intended to encompass all alternatives, modifications, and equivalents.

[0018] Figure 1 Diagram illustrating the learning architecture in an implementation.

[0019] Figure 2 Diagram illustrating the learning process in an implementation.

[0020] Figure 3 Illustration of a correspondence proposal module in implementation.

[0021] Figure 4 Another learning process in one implementation is illustrated.

[0022] Figure 5 The diagram shows an operation scenario in an implementation.

[0023] Figure 6 Diagram illustrating the learning architecture in an implementation.

[0024] Figure 7 Another learning architecture in one implementation is illustrated.

[0025] Figure 8 A computing system suitable for implementing the various operating environments, modules, architectures, processes, scenarios, and sequences discussed below with respect to the figures is illustrated. DETAILED DESCRIPTION

[0026] This article discloses a correspondence proposal (CP) module that can be inserted into an artificial neural network to enhance the neural network's use of data, image data, etc. to classify motion, action, or other such dynamic behaviors in a video. The CP module enhances motion learning in a video by generating a joint semantic vector that has both semantic and spatiotemporal information about the extracted features encoded therein. In addition, the CP module selects corresponding features to consider when generating the joint semantic vector based on the semantic similarity of the corresponding features to a given feature. The artificial neural network takes the joint semantic vector as input and outputs one or more learned motions represented in the video.

[0027] Including spatiotemporal information in the joint semantic vector represents an advance over previous approaches to the problem of learning motion in videos. Taking semantic similarity into account when identifying corresponding features represents another advance. The combination of the two strategies allows an artificial neural network to identify distant motion in videos more accurately than previous solutions. This is because the spatiotemporal information allows the artificial neural network to consider the location of each feature of a correspondence pair in its corresponding frame. Additionally, corresponding features (which form a correspondence pair with a given feature) are selected based on their semantic similarity to the given feature, narrowing the field of possible features in other frames that may be identical to the given feature. The semantic (appearance) and spatiotemporal (motion) information implicitly encoded in the joint semantic vector represents a rich dataset that, when fed into the artificial neural network during both the training and learning phases of operation, improves the neural network's ability to recognize motion in videos.

[0028] In one implementation, a main artificial neural network extracts features from a video and provides semantic and positional information for each feature. Examples of features include objects, edges, pixels, or other such elements that can be extracted from a video. Semantic information describes one or more properties of a given feature in a numerical or symbolic manner, such as, for example, its color components, length, area, roundness, gradient magnitude, gradient direction, texture, or grayscale intensity. Positional information describes the feature by its spatial location in the frame and its temporal location in the video. Thus, the positional information describes the spatiotemporal location of a given feature.

[0029] The CP module performs two main functions: feature grouping and motion embedding. Specifically, for each feature extracted from the video, the CP module finds and groups a corresponding set of features. It then generates a joint semantic vector for each feature, encoding both the feature's appearance and motion. The CP module searches for corresponding features based on the semantic similarity between a given feature and other features. In some cases, this search is a nearest neighbor search of the top k most similar features.

[0030] Each corresponding feature is described by its semantic information and spatiotemporal information. The semantic information can be represented by a semantic vector, which includes numbers or symbols that describe the characteristics of the feature.

[0031] Spatiotemporal information can be the location of a feature in a video described in terms of height (H), width (W), and frame (T) in the video where the feature can be found. In some cases, semantic and spatiotemporal information can be combined in a video representation tensor, which is viewed by the CP module as a point cloud of features with T×H×W points.

[0032] The CP module generates a joint semantic vector for each given feature using the semantic information and spatiotemporal information for the corresponding feature. Therefore, the resulting joint semantic vector has the appearance and motion of its corresponding feature encoded in it.

[0033] To generate a joint semantic vector, the CP module identifies pairs of a given feature with its corresponding feature found through similarity search. The CP module generates a concatenated vector for each pairing, which consists of the semantic vector of each feature in the pair and the spatiotemporal displacement between the two. In other words, each concatenated vector includes two semantic vectors and spatiotemporal displacement information. In an alternative, each concatenated vector can include the distance between the two semantic vectors instead of the two semantic vectors themselves. Therefore, the CP module produces a set of concatenated vectors corresponding to feature pairs.

[0034] The concatenated vectors can be fed into one or more internal neural networks that are different from the main neural network that extracted features from the video. The main neural network extracted features from the video, while the internal neural networks are trained to approximate the collective function. In some cases, the internal neural network is a multi-layer perceptron (MLP) neural network. The internal neural network takes the concatenated vectors as input, performs the collective function on the vectors, and outputs a potential joint semantic vector.

[0035] Each latent joint semantic vector outputted corresponds to one of the concatenated vectors provided as input and includes a numerical or symbolic semantic representation of the motion learned from the concatenated vector. Thus, a plurality of latent joint semantic vectors are generated corresponding to the plurality of concatenated vectors provided as input to the internal neural network.

[0036] The potential joint semantic vectors are then aggregated through a pooling operation, examples of which include max pooling and average pooling. The output of the pooling operation is a joint semantic vector that represents the motion of the features in the video. In this way, the CP module produces a joint semantic vector for each feature in the video.

[0037] The final output of the CP module is therefore a set of joint semantic vectors corresponding to the features extracted from the video by the main artificial neural network. These joint semantic vectors are fed as input to the main artificial neural network. An artificial neural network trained on such data can respond to the joint semantic vectors and identify and classify motion in the video based on the signals generated by the neurons in the neural network. For example, the artificial neural network can identify the dribbling of a basketball, the rolling of a can, or the gestures of a person from the joint semantic vectors generated by the CP module.

[0038] Figure 1 The learning architecture 100 in one implementation is illustrated. The learning architecture 100 includes an artificial neural network 101 (referred to herein as neural network 101). The neural network 101 includes various layers represented by layers 103 and 105. The neural network 101 also includes a correspondence proposal (CP) module 104 inserted between layers 103 and 105. The learning architecture 100 can be implemented in hardware, software, or firmware and on one or more physical or virtual computing systems. Figure 8 The computing system 801 in FIG. 8 is a representative thereof.

[0039] Neural network 101 represents any type of artificial neural network capable of learning motion in video data. Examples of neural network 101 include, but are not limited to, convolutional neural networks (CNNs), 3D CNNs, and any other type of deep learning artificial neural network.

[0040] Neural network 101 takes video data 107 as input and provides as output an indication 109 of motion learned from video data 107. For example, video data 107 may include representations of objects moving in a scene, the postures of people, etc., which neural network 101 is able to recognize and classify (i.e., learn). Video data 107 may include red-green-blue (RGB) values for pixels in the video, or alternative representations of RGB values such as hue-saturation-value (HSV) and hue-saturation-lightness (HSL) values. In some cases, video data 107 primarily includes RGB values. For example, at least half of video data 107 may be RGB values. In some cases, video data 107 may only include RGB values. It will also be appreciated that neural network 101 may also be able to take any type of dynamic data as input and recognize various dynamic behaviors represented in the data.

[0041] Layer 103 of neural network 101 represents one or more layers of neurons capable of taking video data as input and extracting features from the video. Layer 105 of neural network 101 represents one or more layers of neurons capable of taking a joint semantic vector as input, extracting (or learning) motion from the joint semantic vector, and outputting an indication of the learned motion.

[0042] CP module 104 takes the features extracted by layer 103 as input and generates a joint semantic vector to be output to layer 105. In some implementations, the features extracted by layer 103 are presented to CP module 104 in a video representation tensor. In such instances, CP module 104 adds the resulting joint semantic vector back to the video representation tensor and feeds the modified video representation tensor to layer 105. In some scenarios, the joint semantic vector replaces the original semantic vector associated with the feature corresponding to the joint semantic vector.

[0043] Video data 107 represents one or more segments of video from which features may be extracted by layer 103. Video data 107 includes frames 108 having features that may be discovered by layer 103. The location of each feature in the video may be defined in terms of the height (H), width (W), and frame (T) at which a given feature is found in the video.

[0044] Indications 109 represent the end results of learning architecture 100, such as a description of the motion of an object or entity in video data 107. That is, neural network 101 may be able to describe what certain features are and the nature of the motion in the video. Indications 109 may be in the form of reports, messages, database entries, inter-process messages, intra-process messages, displayed information, logged data, or any other type of indication.

[0045] The CP module 104 represents any (multiple) hardware and / or software elements capable of generating joint semantic vectors from features extracted from a video as described herein. The CP module 104 employs a learning process 200 to extract joint semantic vectors from which other parts of the neural network 101 can learn motion. The learning process 200 can be implemented in program instructions in the context of any hardware, software application, module, component, or other such programming element that can include the CP module 104. The program instructions instruct the underlying physical or virtual computing system(s) to perform the learning process 200 (incidentally, reference is made to the program instructions in the context of the learning process 200). Figure 2 Follow the steps described in .

[0046] In operation, the CP module 104 receives a video representation tensor from the layer 103 (step 201). Both the input and output of the CP module 104 are video representation tensors of shape THW×C, where T represents the number of frames, H×W represents the spatial dimension, and C represents the number of channels.

[0047] The CP module 104 treats the video representation tensor as a point cloud, the data points of which correspond to the features extracted by layer 103 from the video data 107. Each point in the tensor is defined according to the spatiotemporal position of one of the features extracted from the video. In this example, the spatiotemporal position of the feature is represented by its THW coordinates in the video. In addition, a semantic vector representing the describable dimension of the feature is provided for each point. Many dimensions are possible and depend on the configuration of layer 103. In some cases, the semantic vector may include information derived primarily from the red, green, and blue (RGB) values input to the artificial neural network 101.

[0048] For each feature represented in the tensor, the CP module 104 performs a vector-based similarity search of other features in the tensor (step 203). The similarity search considers the semantic vectors of other features in the video, but may exclude those features in the same frame as the given feature.

[0049] The result of a similarity search for each feature is a set of corresponding features that can be associated with the given feature. Various feature pairs can be formed between a given feature and its corresponding feature. As an example, a similarity search can identify the top k features with semantic vectors that are most similar to the semantic vector of the given feature. Thus, k feature pairs can be formed between the given feature and each corresponding feature.

[0050] The CP module 104 generates a concatenation vector for each feature pair formed between a given feature and its corresponding feature (step 205). The concatenation vector includes the semantic vector of the given feature, the semantic vector of the corresponding (or paired) feature, and the spatiotemporal displacement between the two. In some implementations, the concatenation vector may include the distance (or difference) between the two semantic vectors, rather than the vectors themselves.

[0051] The concatenated vectors are fed into one or more internal neural networks that are trained to approximate the collective function. An internal neural network, distinct from the main artificial neural network represented by neural network 101, takes the concatenated vectors as input and generates a potential joint semantic vector (step 207). In some cases, there is a one-to-one correspondence between the number of concatenated vectors generated by CP module 104 and the number of internal neural networks present. In this way, the concatenated vectors can be fed into the internal neural networks in parallel.

[0052] The outputs of the internal neural network are aggregated through a pooling operation (e.g., a max pooling operation), and the output of the pooling operation becomes a joint semantic vector for a given feature (step 209). The CP module 104 adds the joint semantic vector to the original video representation tensor or a new video representation tensor (step 211). The CP module 104 then feeds the video representation tensor (including the joint semantic vector) to the layer 105 of the neural network 101 (step 213). One or more layers of the neural network 101 (including layer 105) continue to process the joint semantic vector and the signal it produces to arrive at an indication 109 of motion identified in the video data 107.

[0053] Figure 3 A more detailed view of the CP module 300 and its operation in one implementation is shown. The CP module 300 includes a grouping layer 301 and an embedding layer 303. The grouping layer 301 represents any one or more hardware, software, or firmware elements of the CP module 300 that are capable of identifying a corresponding feature set for each feature extracted from a video. The embedding layer 303 represents any one or more hardware, software, or firmware elements of the CP module 300 that are capable of generating a concatenated vector and deriving a joint semantic vector from the concatenated vector.

[0054] The grouping layer 301 takes the video representation tensor 311 as input and selects the k most likely corresponding features in the tensor for each feature. The video representation tensor 311 is treated by the grouping layer 301 as a three-dimensional point cloud, in which its points (defined by their THW coordinates in the point cloud) represent features extracted from the video by a neural network. The grouping layer 301 performs a semantic similarity search on each feature, which results in a feature group associated with a given feature. The search can be based solely on semantic similarity to ensure that correspondences can be identified across arbitrarily long spatiotemporal ranges. Features within the same frame can be excluded to ensure temporal consistency between frames. For example, the correspondence set 313 includes one feature / point at its center and 8 different features / points around it. Each of the other 8 points represents a feature in different frames of the video with the same or similar semantic vector relative to the feature / point at the center of the set.

[0055] Each feature / point is located at a distance from the central feature / point of the correspondence set 313, which can be calculated by subtracting their THW coordinates from the THW coordinates of the central feature. The displacement is then a vector representing the distance and direction from the central feature to each feature in the group.

[0056] The embedding layer 303 takes the identified correspondence groups as input and produces a joint semantic vector 315 for each feature as output. In the process, the embedding layer 303 generates a concatenated vector from the semantic vectors associated with the features in the correspondence group. Additionally, the embedding layer 303 generates a latent joint semantic vector from the concatenated vectors and a joint semantic vector 315 ("g") from the latent joint semantic vector. The resulting joint semantic vector for the features can be blended back into the original or a different video representation tensor and fed into one or more neural networks.

[0057] Figure 4 The learning process 400 in one implementation is illustrated, which is an example of another learning process that can be employed by elements of the CP module 300. The learning process 400 can also be implemented in program instructions in any context of hardware, software applications, modules, components, or other such programming elements deployed in the various elements of the CP module 300. The program instructions instruct the underlying physical or virtual computing system(s) to follow the learning process 400 (see also Figure 4 Follow the steps described in .

[0058] In operation, the grouping layer 301 receives the video representation tensor 311 from the artificial neural network (step 401). For each feature (or point) in the tensor, the grouping layer 301 performs a k-nearest neighbor search to identify the top k elements with semantic vectors that are most similar to the semantic vector of the given feature (step 403).

[0059] In some implementations, step 403 includes computing a pairwise negative L2 distance matrix for a tensor of shape THW×THW in feature space (step 403A), which can be efficiently accomplished by matrix multiplication. The negative L2 distance can be used as a metric for comparing a given feature pair with other features in the tensor.

[0060] Next, the grouping layer 301 sets the eigenvalues in the T diagonal block matrix of shape HW×HW to negative ∞ (step 403B). This step is used to exclude features in the same frame as the given feature that are selected as potential corresponding features through similarity search. Finally, the grouping layer 301 applies the argument (arg) top-k operation along the row dimension of the matrix (step 403C), which outputs a tensor of shape THW×k, where the i-th row is the index of the k nearest neighbors of the i-th feature.

[0061] After finding the nearest neighbors of a feature, the embedding layer 303 generates a concatenated vector for each pairing of a given feature with the corresponding feature of its corresponding feature (step 405). Each pairing consists of the given feature that generates the pairing and a different feature from the top k features that are most similar to the feature. Each concatenated vector includes two semantic vectors for the two features in the corresponding pair, as well as the spatiotemporal displacement of one feature relative to the other. The displacement can be calculated by subtracting the corresponding coordinate values (e.g., tt, hh, and ww) from each other.

[0062] Next, the embedding layer 303 feeds the concatenated vectors into one or more internal neural networks (step 407). In some implementations, the internal neural networks can be multilayer perceptrons with shared weights that are trained to approximate collective functions. The internal neural networks output a potential joint semantic vector based on the signals generated by the concatenated vectors in the internal neural networks.

[0063] After the concatenated vectors are fed into the internal neural network, the embedding layer 303 performs a max pooling operation on the aggregated output of the internal neural network (step 409). The max pooling operation is performed on the potential joint semantic vectors to select the signal that best provides the corresponding relationship for a given feature. Due to the entangled representation that occurs by mixing the semantic vectors and displacements in the concatenated vectors fed into the internal neural network, the resulting joint semantic vector has both appearance and motion implicitly encoded in it.

[0064] The embedding layer 303 adds the resulting joint semantic vector of features back to the video representation tensor (step 411) and feeds the modified video representation tensor into the main neural network (step 413). The main neural network takes the modified video representation tensor as input, which generates signals in the remaining layers of the main neural network. The signals ultimately generate outputs from the main neural network, which classify the motion found in the video segment from which the features were extracted.

[0065] Figure 5 An operational scenario 500 is illustrated in another implementation of enhanced motion learning. The operational scenario 500 illustrates the process involved in generating a joint semantic vector for a single feature extracted from a video. The operational scenario 500 is repeated for many or all features extracted from the video to generate many joint semantic vectors, which are then provided as input to a convolutional neural network. The convolutional neural network is then able to identify motion occurring in the video.

[0066] In operation, the grouping layer identifies correspondence groups 501 of features based on a k-nearest neighbor (k-NN) search of other features extracted from the video. The correspondence groups 501 are represented by i 0 The feature 503 represented is the center, and the feature 503 has coordinates [t i0 ,hi0 ,w i0 ] and semantic vector f i0 .

[0067] The correspondence group 501 includes five other features in the THW space, which are found to be the top k features most similar to feature 503, represented by their corresponding coordinates and semantic vectors as follows: 1) Feature i 1 , with coordinates [t i1 ,h i1 ,w i1 ], and the semantic vector f i1 2) Feature i 2 , with coordinates [t i2 ,h i2 ,w i2 ] and semantic vector f i2 ;3) Feature i 3 , with coordinates [t i3 ,h i3 ,w i3 ] and semantic vector f i3 4) Feature i 4 , with coordinates [t i4 ,h i4 ,w i4 ] and semantic vector f 4 ; and 5) Feature i 5 , with coordinates [t i5 ,h i5 ,w i5 ] and semantic vector f 5 .

[0068] The embedding layer takes the correspondence set 501 as input and produces concatenated vectors 511-515 corresponding to the pairing of feature 503 and each of its corresponding features. The concatenated vectors each include two semantic vectors and spatiotemporal displacement information. For example, the concatenated vector 511 includes the semantic vector f i0 and f i1 And by [t i0 ,h i0 ,w i0 ]-[t i1 ,h i1 ,w i1 ] generated displacement; the cascade vector 512 includes the semantic vector f i0 and f i2 And by [t i0 ,h i0 ,w i0 ]-[t i2 ,h i2 ,w i2] generated displacement; and so on, until the cascade vector 515, which includes the semantic vector f i0 and f i5 And by [t i0 ,h i0 ,w i0 ]-[t i5 ,h i5 ,w i5 ]The displacement generated.

[0069] The embedding layer feeds the concatenated vectors 511-515 into an MLP-type neural network represented by MLPs 521-525. For example, concatenated vector 511 is fed into MLP 521; concatenated vector 512 is fed into MLP 522; and concatenated vector 515 is fed into MLP 525. MLPs 521-525 are trained to approximate the collective function. Thus, they each apply the collective function to their corresponding concatenated vectors to output a potential joint semantic vector.

[0070] The embedding layer aggregates the potential joint semantic vector in the max pooling operation 530. The max pooling operation 530 is used to extract the most important signal from the potential joint semantic vector generated by MLP 521-525. The result of the max pooling operation is represented by g i0 The joint semantic vector 531 represented.

[0071] Figure 6 The diagram shows a learning architecture 600 in one implementation. The learning architecture 600 includes a convolutional neural network 601, a CP module 602, and a convolutional neural network 607. The convolutional neural network 601 and the convolutional neural network 607 can represent separate neural networks or different layers of the same neural network. The CP module 602 includes a grouping layer 603 and an embedding layer 605. The learning architecture 600 can be implemented in hardware, software, or firmware and can be implemented by Figure 8 The computing system 801 represents one or more physical or virtual computing systems.

[0072] In operation, the convolutional neural network 601 extracts features from the video and provides semantic and spatiotemporal information of each feature to the CP module 602 in the form of a video representation tensor (VRT). Examples of features include objects, edges, pixels, or other such elements that can be extracted from the video. Semantic information describes one or more attributes of a given feature in a numerical or symbolic manner, such as, for example, its color component, length, area, roundness, gradient magnitude, gradient direction, texture, or grayscale intensity. Spatiotemporal information describes the feature by its spatial position in the frame and its temporal position in the video.

[0073] The grouping layer 603 in the CP module 602 searches for a corresponding feature set for each feature extracted from the video, and passes the k features to the embedding layer 605 .

[0074] The embedding layer 605 generates a concatenated vector for a feature pair formed by a given feature and its corresponding feature. The concatenated vector includes semantic information and spatiotemporal information (or the difference / displacement between them) of the two features in the pair included in the VRT.

[0075] The embedding layer 605 then feeds the concatenated vectors into an MLP neural network trained to approximate a collective function. The output of the MLP neural network is aggregated through a max pooling function, which produces a joint semantic vector denoted by "g". The joint semantic vector can be added to the original video representation tensor to obtain a modified video representation tensor (VRT'). The VRT' can then be fed into a convolutional neural network 607. The convolutional neural network 607 responds to the signal generated by the VRT' to output motion indications in the video recognized by its neuron layer.

[0076] Figure 7 Another learning architecture 700 in one implementation is illustrated. Learning architecture 700 includes multiple convolutional neural networks (or different layers of the same convolutional neural network) represented by CNN 701, CNN 705, and CNN 709. CP module 703 is inserted between CNN 701 and CNN 705. CP module 707 is inserted between CNN 705 and CNN 709. Finally, CP module 711 is inserted after CNN 709.

[0077] In operation, CNN 701 extracts features from a video and provides semantic and spatiotemporal information of each feature to CP module 703 in a video representation tensor VRT(f). As described above, examples of features include objects, edges, pixels, or other such elements that can be extracted from a video. Semantic information describes one or more attributes of a given feature in a numerical or symbolic manner, such as its color component, length, area, roundness, gradient magnitude, gradient direction, texture, or grayscale intensity. Spatiotemporal information describes a feature by its spatial location in a frame and its temporal location in a video.

[0078] The CP module 703 operates as described above with respect to the CP module 602. That is, the CP module 703 identifies a k-NN set of corresponding features for each feature in the VRT (f). The CP module 703 then generates a concatenated vector from the corresponding relationship pair, feeds the concatenated vector into the MLP, and aggregates the latent vectors output by the MLP in the max pooling function. The output of the max pooling function is a joint semantic vector denoted by "g". The resulting joint semantic vector generated for the features in the VRT (f) is added back to its VRT - replacing the original semantic vectors associated with these features - thereby generating the VRT (g).

[0079] CNN 705 takes VRT(g) as input and extracts features from it that have a different focus than CNN 701. The result is VRT(f'), which is provided by CNN 705 to CP module 707. In other words, because the focus of CNN 705 is different from that of CNN 701, and because their inputs are different, at least some of the features represented in VRT(f') are different from the features represented in VRT(f).

[0080] The CP module 707 operates as described above with respect to the CP module 602. That is, the CP module 707 identifies a k-NN set of corresponding features for each feature in the VRT(f'). The CP module 707 then generates a concatenated vector from the corresponding relationship pair, feeds the concatenated vector into the MLP, and aggregates the latent vector output by the MLP in a max pooling function. The output of the max pooling function is a joint semantic vector denoted as "g'". The resulting joint semantic vector generated for the features in the VRT(f') is added back to the VRT(f'), thereby obtaining the VRT(g').

[0081] CNN 709 takes VRT(g') as input and extracts features from it that have a different focus than CNN 705. The result is VRT(f"), which CNN 709 provides to CP module 711. Because the focus of CNN 709 is different relative to CNN 705, and because their inputs are different, at least some of the features represented in VRT(f") are different from the features represented in VRT(f').

[0082] The CP module 711 operates as described above with respect to the CP module 602. That is, the CP module 711 identifies, for each feature in the VRT(f”), a k-NN set of corresponding features. The CP module 711 then generates a concatenated vector from the corresponding pairs, feeds the concatenated vector into the MLP, and aggregates the latent vectors output by the MLP in a max pooling function. The output of the max pooling function is a joint semantic vector denoted by “g”. The resulting joint semantic vector generated for the features in the VRT(f”) is added back to the VRT(f”) to obtain the VRT(g”).

[0083] The process may terminate after the CP module 711, where the VRT(g") is sent to the remaining layers of the CNN. Optionally, the process may continue for one or more iterations until the video representation tensor is sent to the remaining layers. The remaining layers of the CNN respond to the signals generated by the video representation tensor to output an indication of the motion that the layers identified in the video.

[0084] In some implementations, the correspondence module disclosed herein can be inserted into a CNN architecture and interleaved with the convolutional layers. Such an architecture allows for the mixing and joint learning of static image features from the convolutional layers and motion signals extracted by the correspondence module.

[0085] In a specific example, the correspondence module can be inserted into the C2D ResNet architecture, where the module is located just after the residual block. The time dimension can be completely abandoned, which keeps the time size T unchanged during the forward feed. However, the correspondence module can be relied upon to capture temporal information. Although the discussion in this article focuses on feature space, the nearest neighbors found by the correspondence module can be determined on any metric space (such as time space only, spatiotemporal space, feature space, or spatiotemporal feature space).

[0086] The C2D portion of the architecture can be initialized using a pre-trained ImageNet model. The gamma parameters of the final batch normalization layers are initialized to all zeros, and the MLP in the correspondence module is randomly initialized using MSRA initialization. This ensures that the mapping is identified at the start of training, allowing the use of pre-trained image feature extractors.

[0087] As can be understood from the foregoing, the correspondence proposal module discussed in this paper computes correspondence pairs for each feature in a tensor and feeds the proposed pairs into a shared neural network, followed by max pooling to learn a new feature tensor. In this way, the module can effectively capture motion correspondence information in videos. The proposed correspondence module can be integrated with existing frame-based or segment-based architectures. In addition, the proposed correspondence module has been tested on standard video recognition benchmarks and achieved excellent performance.

[0088] Figure 8 The diagram shows a computing system 801 that represents any system or collection of systems in which the various modules, processes, procedures, and scenarios disclosed herein can be implemented. Examples of computing system 801 include, but are not limited to, server computers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machines, containers, and any variations or combinations thereof. Other examples include desktop computers, laptop computers, tablet computers, Internet of Things (IoT) devices, wearable devices, and any other physical or virtual combinations or variations thereof.

[0089] The computing system 801 can be implemented as a single device, system, or apparatus or can be implemented as multiple devices, systems, or apparatuses in a distributed manner. The computing system 801 includes, but is not limited to, a processing system 802, a storage system 803, software 805, a communication interface system 807, and a user interface system 809 (optional). The processing system 802 is operatively coupled to the storage system 803, the communication interface system 807, and the user interface system 809.

[0090] Processing system 802 loads and executes software 805 from storage system 803. Software 805 includes and implements a learning process 806, which is representative of the learning process discussed with respect to the previous figures. When executed by processing system 802 to provide enhanced representation learning, software 805 instructs processing system 802 to operate as described herein for the various processes, operational scenarios, and sequences discussed in at least the aforementioned implementations. Computing system 801 may optionally include additional devices, features, or functionality not discussed for the sake of brevity.

[0091] Still refer to Figure 8 , processing system 802 may include a microprocessor and other circuitry that retrieves and executes software 805 from a storage system 803. Processing system 802 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate to execute program instructions. Examples of processing system 802 include general-purpose central processing units, graphics processing units, special-purpose processors and logic devices, and any other type of processing device, combination, or variation thereof.

[0092] The storage system 803 may include any computer-readable storage medium that can be read by the processing system 802 and capable of storing the software 805. The storage system 803 may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage medium. In any case, a computer-readable storage medium is not a propagated signal.

[0093] In addition to computer-readable storage media, in some implementations, the storage system 803 may also include a computer-readable communication medium on which at least some of the software 805 may communicate internally or externally. The storage system 803 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems that are co-located or distributed relative to each other. The storage system 803 may include additional elements (such as a controller) that can communicate with the processing system 802 or possibly other systems.

[0094] The software 805 (including the learning process 806) may be implemented in program instructions and, when executed by the processing system 802, may, among other things, instruct the processing system 802 to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, the software 805 may include program instructions for implementing a learning process to learn motion in a video as described herein.

[0095] In some embodiments, the software 805 may include software components or modules that can be used to perform various processes and operating scenarios described herein, such as, for example, operating system software, virtualization software, or other application software. The software 805 may also include firmware or other forms of machine-readable processing instructions that can be executed by the processing system 802.

[0096] In general, when software 805 is loaded into processing system 802 and executed, software 805 can convert a suitable device, system, or apparatus (represented by computing system 801) generally from a general-purpose computing system into a specialized system designed to provide motor learning. In practice, the encoded software 805 on storage system 803 can convert the physical structure of storage system 803. The specific conversion of the physical structure can depend on various factors in different implementations described herein. Examples of such factors can include, but are not limited to, the technology of the storage medium used to implement storage system 803 and whether the computer storage medium is characterized as a primary storage device or a secondary storage device, as well as other factors.

[0097] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, then when program instructions are encoded therein, the software 805 may transform the physical state of the semiconductor memory, such as by transforming the states of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of the physical medium are possible without departing from the scope of this specification, and the foregoing examples are provided merely to facilitate this discussion.

[0098] The communication interface system 807 may include communication connections and devices that allow communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that together allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuit devices, transceivers, and other communication circuit devices. The connections and devices may communicate over a communication medium (such as metal, glass, air, or any other suitable communication medium) to exchange communications with other computing systems or system networks. The aforementioned media, connections, and devices are well known and need not be discussed in detail herein.

[0099] Communication between computing system 801 and other computing systems (not shown) can occur over one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, the Internet, the internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The aforementioned communication networks and protocols are well known and do not require extensive discussion here.

[0100] The included description and drawings depict specific embodiments to teach those skilled in the art how to make and use the best mode. In order to teach the principles of the invention, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these embodiments that fall within the scope of the present disclosure. Those skilled in the art will also appreciate that the above-described features can be combined in various ways to form multiple embodiments. Consequently, the present invention is not limited to the specific embodiments described above, but is limited only by the claims and their equivalents.

Claims

1. A machine learning method in an artificial neural network, the method comprising: In an artificial neural network, features and semantic vectors associated with the features are extracted from the video; For at least one of the features extracted from the video: performing a similarity search for corresponding features in the video based at least on the semantic vector associated with the feature; as well as generating a joint semantic vector based at least on a semantic vector associated with the one feature, a semantic vector associated with at least one of the corresponding features, and a spatiotemporal distance between the one feature and the at least one of the corresponding features, wherein generating the joint semantic vector at least for the one feature comprises: generating a concatenation vector for each pairing of the one feature and a corresponding feature of the corresponding features of the one feature; generating a latent vector based on the concatenated vector; and performing a maximum pooling operation on the latent vector to generate the joint semantic vector; and In the artificial neural network, motion in the video is identified based at least on the joint semantic vector identified for the one feature. 2 . The method of claim 1 , wherein performing the similarity search for the corresponding features in the video comprises performing a nearest neighbor search based on the semantic vector.

3. The method of claim 2, wherein performing the nearest neighbor search comprises: Those features in the same frame as a feature in the video are excluded. The method of claim 3 , wherein the video comprises red, green, and blue (RGB) values for pixels in the video.

5. The method of claim 4, wherein performing the nearest neighbor search comprises: The semantic vector associated with the one feature is compared with other semantic vectors in the semantic vectors to identify top k features in the features that are most similar to the one feature.

6. The method according to claim 5, wherein generating a concatenated vector for each pairing of the one feature and a corresponding feature of the corresponding feature of the one feature comprises: identifying a pairing of the one feature with each of the corresponding features, wherein each of the pairings includes the one feature and the corresponding feature that is different relative to a corresponding feature in each other of the pairings; and Generate the concatenated vectors, wherein each of the concatenated vectors corresponds to a different one of the pairs, wherein each of the concatenated vectors includes the semantic vector associated with the one feature, the semantic vector associated with the corresponding feature in the corresponding pair, and the spatiotemporal distance between the one feature and the corresponding feature.

7. The method of claim 6, wherein the artificial neural network comprises a convolutional neural network, and wherein generating the latent vector based on the concatenated vector comprises feeding the concatenated vector into a multilayer perceptron (MLP) neural network.

8. A method for enhancing machine learning in an artificial neural network, the method comprising: generating a feature tensor in an artificial neural network, the feature tensor comprising features extracted from the image data and positions of the features in the image data; For at least one feature in the feature tensor: performing a search for corresponding features in the image data based on semantic similarity between the one feature and other features in the feature tensor; generating a joint semantic vector based at least on the semantic similarity between the one feature and one or more corresponding features of the corresponding features and the spatiotemporal distance between the one feature and the one or more corresponding features of the corresponding features, wherein generating the joint semantic vector at least for the one feature includes: generating a concatenation vector for each pairing of the one feature and a corresponding feature of the corresponding features of the one feature; generating a latent vector based on the concatenated vector; and performing a maximum pooling operation on the latent vector to generate the joint semantic vector; adding the joint semantic vector to the feature tensor; and The feature tensor including the joint semantic vector is fed into the artificial neural network.

9. The method of claim 8, wherein performing the search for the corresponding features in the image data comprises: A nearest neighbor search is performed based on the semantic similarity of the one feature to the other features, and features in the image data that are identical to the one feature are excluded.

10. The method according to claim 9: wherein the feature tensor further includes a semantic vector associated with the feature; and The performing of the nearest neighbor search includes performing a comparison of the semantic vector associated with the one feature with the semantic vectors associated with the other features to identify top k features that are most similar to the one feature.

11. The method according to claim 10, wherein generating a concatenated vector for each pairing of the one feature and a corresponding feature of the corresponding feature of the one feature comprises: identifying a pairing of the one feature with each of the corresponding features, wherein each of the pairings includes the one feature and the corresponding feature that is different relative to a corresponding feature in each other of the pairings; and Generate a concatenated vector, wherein each of the concatenated vectors corresponds to a different one of the pairs, wherein each of the concatenated vectors includes the semantic vector associated with the one feature, the semantic vector associated with the corresponding feature in the corresponding pair, and the spatiotemporal distance between the one feature and the corresponding feature.

12. The method according to claim 11, further comprising: The feature tensor is processed in the artificial neural network to extract motion from the image data.

13. The method according to claim 12, wherein the artificial neural network comprises a convolutional neural network; and wherein generating the latent vector based on the concatenated vector comprises feeding the concatenated vector into a multi-layer perceptron (MLP) neural network. The method of claim 13 , wherein the image data comprises a video, and wherein the features comprise objects in the video.

15. A computing device comprising: one or more computer-readable storage media; as well as Program instructions stored on the one or more computer-readable storage media, which, when executed by a processing system, instruct the computing device to at least: generating a video representation tensor comprising features extracted from a video and positions of the features in the video; For at least one of the features extracted from the video: Based on the semantic similarity between the one feature and other features, performing a search for corresponding features in the video; generating a joint semantic vector based at least on the semantic similarity between the one feature and one or more of the corresponding features and the spatiotemporal distance between the one feature and the one or more of the corresponding features, wherein to generate the joint semantic vector for the one feature, the program instructions instruct the computing device to: generating an input for each pairing of the one feature with a corresponding feature of the one feature's corresponding features; Feeding the input to a multi-layer perceptron (MLP) neural network to generate an output; as well as Performing a maximum pooling operation on the output of the MLP neural network to generate the joint semantic vector; as well as The joint semantic vector is added to the video representation tensor.

16. The computing device of claim 15, wherein the search for the corresponding feature in the video comprises a nearest neighbor search, and wherein the nearest neighbor search excludes those features in the same frame as a feature in the video.

17. The computing device according to claim 16, wherein the video representation tensor further includes a semantic vector associated with the feature; and The nearest neighbor search includes comparing the semantic vector associated with the one feature with the semantic vectors associated with the other features to identify top k features that are most similar to the one feature.

18. The computing device of claim 17, wherein to generate an input for each pairing of the one feature with a corresponding feature of the one feature's corresponding features, the program instructions instruct the computing device to: identifying a pairing of the one feature with each of the corresponding features, wherein each of the pairings includes the one feature and the corresponding feature that is different relative to a corresponding feature in each other of the pairings; and Generating inputs, wherein each of the inputs corresponds to a different one of the pairings, wherein each of the inputs includes the semantic vector associated with the one feature, the semantic vector associated with the corresponding feature in the corresponding pairing, and the spatiotemporal distance between the one feature and the corresponding feature.

19. The computing device of claim 18, wherein: The video includes red, green, and blue (RGB) values for pixels in the video; and The top k features most similar to the one feature include top 8 features most similar to the one feature.

Images