Machine learning of spatiotemporal manifolds for source-free video domain adaptation

Abstract

A method and system for training a model includes performing spatial augmentation on unlabeled input video (204) to generate spatially augmented video. Performing temporal augmentation on the input video (206) to generate temporally augmented video. Using a model pre-trained on a labeled dataset, predictions are generated for the unlabeled input video, the spatially augmented video, and the temporally augmented video (208). Parameters of the model are adapted using the predictions (404), enforcing spatial consistency, temporal consistency, and historical consistency. This model can be used for activity recognition in healthcare settings, where the recognition results are used to determine whether a patient is performing rehabilitation exercises correctly.

Description

[Technical Field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 63 / 424,155, filed November 10, 2022, U.S. Patent Application No. 63 / 425,327, filed November 15, 2022, and U.S. Patent Application No. 18 / 504,469, filed November 8, 2023, each of which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates to action recognition, and more particularly to manifold learning for video domain adaptation. 2. Description of Related Art

[0003] Action recognition is a task used in automatic video understanding: identifying actions taking place in a video sequence. However, action recognition models trained on annotated data drawn from one statistical distribution can perform poorly when tested on data outside that distribution. Summary of the Invention

[0004] The method for training a model includes training a model that performs spatial augmentation on an unlabeled input video to generate a spatially augmented video. Performs temporal augmentation on the input video to generate a temporally augmented video. Using a model pre-trained on a labeled dataset, predictions are generated for the unlabeled input video, the spatially augmented video, and the temporally augmented video. Parameters of the model are adapted using the predictions while enforcing temporal consistency, temporal consistency, and historical consistency.

[0005] A system for training a model includes a hardware processor and a memory that stores a computer program that, when executed by the hardware processor, causes the hardware processor to perform spatial augmentation on unlabeled input video to generate spatially augmented video, perform temporal augmentation on the input video to generate temporally augmented video, generate predictions for the unlabeled input video, the spatially augmented video, and the temporally augmented video using a model pre-trained on a labeled dataset, and adapt parameters of the model using the predictions while enforcing temporal consistency, temporal consistency, and historical consistency.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0007] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0008] [Figure 1] 1 is a diagram of a video including a sequence of frames depicting an action, according to one embodiment of the present invention;

[0009] [Figure 2] FIG. 1 is a diagram of a learning framework for an activity recognition model including spatial, temporal, and historical consistency constraints, according to one embodiment of the present invention.

[0010] [Figure 3] 1 is a pseudocode for updating parameters of a pre-trained activity recognition model based on spatial, temporal, and historical consistency, according to one embodiment of the present invention.

[0011] [Figure 4] FIG. 1 is a block / flow diagram of a method for training and using an activity recognition model, according to one embodiment of the present invention.

[0012] [Figure 5] FIG. 1 is a block diagram of an exemplary application of an activity recognition model in a healthcare facility, according to an embodiment of the present invention.

[0013] [Figure 6] FIG. 2 is a block / flow diagram illustrating stages in an activity recognition model, according to one embodiment of the present invention.

[0014] [Figure 7] FIG. 1 is a block diagram of a computing device capable of training and using an activity recognition model, according to one embodiment of the present invention.

[0015] [Figure 8] FIG. 1 is a diagram of a neural network architecture that can be used to implement portions of an activity recognition model, according to one embodiment of the present invention.

[0016] [Figure 9] FIG. 1 is a diagram of a deep neural network architecture that can be used to implement portions of an activity recognition model, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] Unsupervised Video Domain Adaptation (UVDA) uses labeled video data from a source domain and unlabeled video data from a target domain to train an adaptive model. UVDA takes input video from both domains, minimizes the classification risk of the labeled source video, and trains a model by aligning the video from both domains in a class-independent manner. As described herein, a spatiotemporal manifold (STEM) may be learned, and actions of the same category may be drawn from the same low-dimensional manifold regardless of their spatiotemporal variance in the high-dimensional space. As used herein, the term "manifold" refers to a space of a given dimension that projects high-dimensional video data into a low-dimensional space, such that videos of the same class are located near each other in the low-dimensional space. This allows a space-time-history-consistent (STHC) model trained in the source domain to adapt to handle spatiotemporal variance by learning such a low-dimensional manifold.

[0018] Learning reliable manifold structure without supervised labels is challenging. To mitigate this, spatial and temporal data augmentation can be performed. For each unlabeled target video, spatial and temporal augmentation can be applied in a probabilistic manner to generate an augmented video that outlines the neighborhood structure of the manifold in which the video resides. Encouraging consistent classification predictions for a video and its augmented version ensures they are drawn from the same spatiotemporal manifold. These learned manifolds enable the model to generalize to unlabeled target domains and make consistent predictions for videos drawn from the same manifold.

[0019] The models described herein can be used, for example, to implement activity recognition in video streams. Such activity recognition can be used in a variety of applications, including sports analysis and training, industrial training, and health and rehabilitation. For example, activity recognition can be used to identify a patient's form and whether they are performing the exercise correctly while videotaping a patient undergoing physical therapy. This information can then be used to aid in medical decision-making, for example, by medical professionals modifying the patient's physical therapy or rehabilitation regimen.

[0020] 1, an exemplary video stream 100 is shown, which is comprised of a series of consecutive video frames 101. Each video frame 101 may include an image capturing a camera's view at a particular time. The video stream 100 includes a series of behavioral frames 102 that show particular behaviors being performed. The behavioral frames 102 have a start time 104 and an end time 106.

[0021] The video stream 100 may include labels that explicitly indicate the start time 104 and end time 106 of the activity, and may further include labels that identify the activity occurring in the activity frame 102. In an unannotated video stream, the video stream 100 may lack these time markers and class labels.

[0022] Each frame 101 includes multiple types of information. For example, each frame 101 may include appearance information including RGB values ​​for each pixel in the frame 101, or other suitable encoding of static visual information. Additionally, multiple frames 101 may be considered together to identify motion information, for example, by identifying optical flow information that captures dynamic and motion information from one frame 101 to the next.

[0023] In one example of a video stream 100, the video stream 100 may capture a sporting event such as a long jump. Action frame 102 shows the actual jump being taken, and frames 101 before and after action frame 102 may show related or unrelated actions, such as the jumper running leading up to the jump, or the jumper landing and walking away.

[0024] Source-free video domain adaptation (SFVDA) is a method for solving the problem of applying source-free video domain adaptation to M labeled videos with each label y.

number

number

number

[0025] Model H can be separated as follows:

number

number

number

number

number

number

number

[0026] Given a model H with pre-trained weights learned from labeled source data S, the STHC model adapts the weights to the target domain using unlabeled data T. This is achieved by learning a space-time manifold onto which videos of the same category are mapped, despite their spatiotemporal variance. This allows model H to overcome the domain gap between the source and target domains and achieve better generalization performance. To learn such a manifold, stochastic augmentation is used to generate numerous spatially or temporally augmented versions of each target video. Consistent predictions are made for the video and its augmented versions in terms of space, time, and history.

[0027] Referring now to Figure 2, a block diagram of the STHC model framework is shown.

number

number

[0028]

number

number

[0029] In some embodiments, spatial dilation 204 may include a random combination of image transformations that may be selected from a pool including color inversion, distortion, and adjustment. In some embodiments, spatial dilation 204 may include removing image information, for example, by replacing random square patches of the frame with gray pixels.

[0030] Spatially Extended Video Sequences

number

number

number

[0031] In block 210, spatial proximity constraints can be enforced to ensure spatial consistency. For example,

number

number

number

[0032] A temporal proximity constraint may also be implemented to ensure temporal consistency in block 212. Sampled Frame Sequence

number

number

number

number

number

number

number

number

number

number

[0033] Similar to spatial consistency 210, in block 212, classification prediction

number

number

number

[0034] History consistency 214 is used to enforce temporal consistency 212 .

number

number

[0035] In each training round, past sequences can be viewed as other versions of the current sequence, and consistency between past and current predictions can be enforced. A memory bank M can be used to store the past Q predictions for each video. M is used to calculate the history consistency loss as follows:

number

number

number

[0036] The learning goal during training can be formulated as follows:

number

number

number

number

number

[0037] Referring now to FIG. 3, pseudocode for adapting the parameters of the STHC model according to the framework shown in FIG. 2 is shown.

[0038] Referring now to Figure 4, a method for training an STHC model is shown. Block 400 involves training the model in a two-stage process. Block 402 begins with pre-training the model according to a labeled training dataset in the source domain. This training can be performed, for example, using supervised training of an action classifier that identifies actions performed in input video.

[0039] Next, block 404 adapts the model according to an unlabeled training dataset in a target domain. For example, the target domain may include videos taken under different lighting conditions or from different viewpoints. Enforcing spatial, temporal, and historical consistency across different augmented versions of the videos in the unlabeled training dataset can improve the generalizability of the model while maintaining its effectiveness in the source domain.

[0040] After the model is trained in block 400, the model may be deployed 410 for use, for example, in a healthcare facility. New video is recorded 420 and actions in the new video related to the recorded movements of people in the video are classified 430. Block 440 generates a report related to the recognized actions.

[0041] The STHC model can be easily extended to other video domain adaptation problems. In partial domain adaptation, the classes in the target domain can be a subset of the classes in the source domain. Under the source-free constraint, this means that only samples from a subset of all classes in the source domain are used to adapt the model. The STHC model can be directly applied to such settings because the consistency term described above makes no assumptions about the class distribution in the target domain. L im The class balancing term in may be removed in the setting of partial domain adaptation.

[0042] In open-set domain adaptation, the source classes may be a subset of the target classes. The entropy of the predictions can be used as an uncertainty measure to separate the target samples into groups, for example using K-means clustering. Groups with higher average uncertainty are considered to be samples from unknown classes and may be rejected for adaptation.

[0043] In black-box model adaptation, the source model may not be available for adaptation, but instead may act as a black box that generates outputs for given inputs. The STHC model can be extended to this scenario with a two-stage solution. In the first stage, we treat the black-box model as the teacher model and train a student model with randomly initialized weights by knowledge distillation on the target data. In the second stage, the student model can be adapted as described above.

[0044] 5, there is shown a diagram of activity recognition in the context of a healthcare facility 500. A patient may perform rehabilitation exercises 506 in the healthcare facility 500, and the activities may be recorded by one or more video cameras. The rehabilitation exercises may utilize a therapy system 504, such as exercise equipment or vital signs monitoring devices.

[0045] The health care facility may include one or more medical professionals 502 who provide information related to the patient's rehabilitation progress and information provided by the treatment system 504. When behavioral recognition 508 is performed on the recorded rehabilitation exercises 506, information regarding the patient's form may be automatically generated. Ineffective or risky movements may be identified, and a report may be generated for the medical professional 502 to use in decision-making. For example, based on the report, the medical professional 502 may modify the patient's movements to aid in further rehabilitation of the patient.

[0046] The different elements of the healthcare facility 500 may communicate with each other via a network 510, for example, using any suitable wired or wireless communication protocol and medium, thereby enabling the healthcare professional 502 to receive information related to the rehabilitation exercise 506 from the activity recognition 508.

[0047] 6, details of the STFC model 208 are shown. The model 208 includes a frame feature extractor 602 that generates features for each frame of the input video. The frame feature extractor 602 may include any suitable image feature coding model, such as a residual neural network (ResNet).

[0048] The temporal information encoder 604 captures information about the temporal movement of elements in the video, and the classifier 606 uses this information to, for example, identify actions performed in the video. The temporal information encoder 604 accepts input as a sequence of frame features representing the entire video, extracted by the frame feature extractor 602. The temporal information encoder 604 can be implemented as a multi-layer perceptron (MLP) that models the interaction of temporal information between all frames using an average pooling operator over all frame features.

[0049] The classifier 606 takes the features of a video as input and outputs a predicted score of the category that the video belongs to. The classifier 606 can be implemented as a single fully connected layer.

[0050] 7, an exemplary computing device 700 is shown, in accordance with an embodiment of the present invention. The computing device 700 is configured to perform activity recognition.

[0051] Computing device 700 may be embodied as any type of computational or computing device capable of performing the functions described herein, such as, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a user electronic device. Additionally or alternatively, computing device 700 may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically separated computing device.

[0052] 7, computing device 700 illustratively includes a processor 710, an input / output subsystem 720, a memory 730, a data storage device 740, and a communications subsystem 750, and / or other components and devices typically found in a server or similar computing device. In other embodiments, computing device 700 may include other or additional components (e.g., various input / output devices) such as those typically found in a server computer. Furthermore, in some embodiments, one or more of the illustrative components may be incorporated into or otherwise form part of another component. For example, memory 730, or portions thereof, may be incorporated into processor 710 in some embodiments.

[0053] The processor 710 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0054] Memory 730 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 730 may store various data and software used during operation of computing device 700, such as an operating system, applications, programs, libraries, and drivers. Memory 730 is communicatively coupled to processor 710 via I / O subsystem 720 and may be embodied as circuits and / or components for facilitating input / output operations with processor 710, memory 730, and other components of computing device 700. For example, I / O subsystem 720 may be embodied as or otherwise include a memory controller hub, an input / output control hub, a platform controller hub, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations. In some embodiments, I / O subsystem 720 may form part of a system-on-chip (SOC) and be integrated with processor 710, memory 730, and other components of computing device 700 on a single integrated circuit chip.

[0055] The data storage 740 may be embodied as any type of device or apparatus configured for short-term or long-term storage of data, such as, for example, a memory device and circuitry, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage 740 may store program code 740A for training a model, program code 740B for recognizing actions in video, and / or program code 740C for generating reports on the recognized actions. The communications subsystem 750 of the computing device 700 may be embodied as any network interface controller or other communications circuitry, device, or collection thereof that can enable communications between the computing device 700 and other remote devices over a network. The communications subsystem 750 may be configured to achieve such communications using any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0056] As shown, computing device 700 may also include one or more peripheral devices 760. Peripheral devices 760 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 760 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0057] Of course, computing device 700 may include other elements (not shown) or omit certain elements, as would readily occur to one skilled in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 700, depending on the particular implementation of the same, as would be readily understood by one skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc. may be utilized in a variety of configurations. These and other variations of processing system 700 will be readily contemplated by one skilled in the art in view of the teachings of the present invention provided herein.

[0058] 8 and 9, exemplary neural network architectures are shown that can be used to implement portions of the present models, such as STHC model 208. Neural networks are generalized systems whose functionality and accuracy improve with exposure to additional empirical data. Neural networks learn by exposure to empirical data. During training, the neural network stores and adjusts weights that are applied to input empirical data. Applying the adjusted weights to the data can identify that data belongs to a specific predefined class from a set of classes and output the probability that the input data belongs to each class.

[0059] Empirical data (also called training data) from a set of examples is formatted as a string of values ​​and fed to the input of a neural network. Each example is associated with a known outcome or output. Each column is represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can be of various data types and contain multiple different values. The network can have one input node for each value that makes up the example input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network being built and trained.

[0060] Neural networks "learn" by comparing neural network outputs generated from input data with known values ​​from examples and adjusting stored weights to minimize the difference between the output value and the known value. Adjustments can be made to the stored weights through backpropagation, where the weight's influence on the output value is determined by calculating a mathematical gradient and adjusting the weight in a way that shifts the output toward the minimum difference. This optimization, called gradient descent, is a non-limiting example of how training occurs. A subset of examples with known values ​​that were not used in training can be used to test and validate the accuracy of the neural network.

[0061] During operation, a trained neural network can be used on new data not previously used for training or validation through generalization. The adjusted neural network weights can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0062] In a layered neural network, nodes are arranged in layers. An exemplary simple neural network has an input layer 820 of source nodes 822 and a single computational layer 830 with one or more computational nodes 832 that also function as output nodes, with a single computational node 832 for each possible category into which an input example may be classified. The input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The data values ​​812 in the input data 810 may be represented as a column vector. Each computational node 832 in the computational layer 830 generates a weighted linear combination of values ​​from the input data 810 provided to the input node 820 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0063] A deep neural network, such as a multilayer perceptron, may have an input layer 820 of source nodes 822, one or more computational layers 830 with one or more computational nodes 832, and an output layer 840 with one output node 842 for each possible category into which an input example may fall. The input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The computational nodes 832 in the computational layer 830 are also referred to as hidden layers because they are located between the source nodes 822 and the output nodes 842 and are not directly observed. Each node 832, 842 in the computational layer generates a weighted linear combination of values ​​from the values ​​output by the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values ​​from each previous node may be, for example, w1, w2, ... w n-i ,w n The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in a computational layer is connected to every node in the previous layer, or the connections between layers can have other configurations. If links between nodes are missing, the network is said to be partially connected.

[0064] Training a deep neural network involves two phases: a forward phase, in which the weights of each node are fixed and inputs are propagated through the network, and a backward phase, in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0065] Computational nodes 832 in one or more computational (hidden) layers 830 perform nonlinear transformations on the input data 812 to generate a feature space. Classes or categories may be more easily separated in the feature space than in the original data space.

[0066] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0067] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or propagation medium. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0068] Each computer program can be tangibly stored on a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0069] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0070] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0071] As employed herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that may be on-board or off-board or dedicated for use by the hardware processor subsystem.

[0072] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0073] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0074] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0075] References herein to "one embodiment" or "one embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment," as well as any other variations thereof, in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of one or more embodiments may be combined given the teachings of the present invention provided herein.

[0076] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one," such as "A and / or B" or "at least one of A and B" will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0077] The foregoing is understood in all respects to be illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A method of training a model, comprising: performing spatial dilation on the unlabeled input video to generate a spatially dilated video (204); performing temporal dilation on the input video to generate a temporally dilated video (206); generating predictions for the unlabeled input video, the spatially augmented video, and the temporally augmented video using a model pre-trained on a labeled dataset (208); and adapting (404) parameters of a model using the prediction while enforcing temporal consistency, time consistency, and historical consistency.

2. 10. The method of claim 1, The method, wherein performing spatial consistency includes comparing the spatially augmented video with the unlabeled input video.

3. 10. The method of claim 1, The method, wherein performing temporal consistency includes comparing the temporally extended video with the unlabeled input video.

4. 10. The method of claim 1, The method wherein performing historical consistency includes comparing predictions for the unlabeled input video with predictions for a second unlabeled video used in a previous training iteration.

5. 10. The method of claim 1, The method wherein generating a prediction includes performing action recognition.

6. 6. The method of claim 5, The method further includes identifying actions in the therapy footage that are related to the patient's rehabilitation activities and generating a report for use by a medical professional in making rehabilitation care decisions for the patient.

7. 10. The method of claim 1, The method further comprising pre-training the model using supervised learning with a labeled dataset.

8. 8. The method of claim 7, The domain of the labeled dataset is different from the domain of the unlabeled dataset.

9. 10. The method of claim 1, The method wherein fitting the parameters of the model further comprises an information maximization loss.

10. 10. The method of claim 9, The information maximization loss includes a first term that minimizes the entropy of the probabilities and a second term that maximizes the average entropy of predictions across images in the target domain.

11. A system for training a model, a hardware processor (710); and a memory (740) for storing a computer program, the computer program, when executed by a hardware processor, causing the hardware processor to: performing spatial dilation on the unlabeled input video to generate a spatially dilated video (204); performing temporal dilation on the input video to generate a temporally dilated video (206); generating predictions for the unlabeled input video, the spatially augmented video, and the temporally augmented video using a model pre-trained on a labeled dataset (208); and adapting (404) the parameters of a model using the predictions while enforcing temporal consistency, time consistency, and historical consistency.

12. 12. The system of claim 11, The computer program further causes the hardware processor to compare the spatially extended image with the unlabeled input image.

13. 12. The system of claim 11, The computer program further causes the hardware processor to compare the temporally extended video with the unlabeled input video.

14. 12. The system of claim 11, The computer program further causes the hardware processor to compare the prediction for the unlabeled input image with a prediction for a second unlabeled image used in a previous training iteration.

15. 12. The system of claim 11, The system wherein the generated predictions include activity recognition.

16. 16. The system of claim 15, The computer program further causes the hardware processor to identify activities in the therapy footage that are related to the patient's rehabilitation activities and generate a report for use by medical professionals in making decisions about the patient's rehabilitation care.

17. 12. The system of claim 11, The computer program further causes the hardware processor to pre-train a model using supervised learning with a labeled dataset.

18. 18. The system of claim 17, The domain of the labeled dataset is different from the domain of the unlabeled dataset.

19. 12. The system of claim 11, The computer program further causes the hardware processor to adapt the parameters with an information maximization loss.

20. 20. The system of claim 19, The information maximization loss includes a first term that minimizes the entropy of the probabilities and a second term that maximizes the average entropy of predictions across images in the target domain.

Images