Cut-paste training extension for machine learning models

Abstract

A method and system for training a machine learning model includes identifying objects or people associated with an action in a first video. The objects or people are copied from the first video to a second video to generate a third video. The machine learning model is trained using the first video and the third video.

Description

[Technical Field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 63 / 445,049, filed February 13, 2023, and U.S. Patent Application No. 18 / 439,242, filed February 12, 2024, each of which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates to machine learning systems, and more particularly to expanding training datasets for machine learning systems. 2. Description of Related Art

[0003] Machine learning models that perform action recognition on video streams utilize training datasets containing videos showing a variety of actions. However, training data for rare events in various environments may not be readily available. For example, videos of a person walking may be commonly available in a training dataset, but videos of a person falling may be relatively rare. Videos showing such rare actions may not be available in all environments.

[0004] Models trained on a given dataset may experience reduced accuracy when deployed in various environments, and the number of false positives may increase due to scene and object biases inherent in the training data, which may be due, for example, to large differences between the distribution of training dataset images and images of the target scene in terms of scene complexity, lighting, camera angle, field of view, etc. Summary of the Invention

[0005] A method for training a machine learning model includes identifying objects or people associated with an action in a first video. The objects or people are copied from the first video to a second video to generate a third video. The machine learning model is trained using the first video and the third video.

[0006] A system for training a machine learning model includes a hardware processor and a memory that stores a computer program that, when executed by the hardware processor, causes the hardware processor to identify objects or people associated with an action in a first video, copy the objects or people from the first video to a second video to generate a third video, and train a machine learning model using the first video and the third video.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0008] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0009] [Figure 1] FIG. 2 illustrates a set of videos in a training dataset for an action recognition model, according to one embodiment of the present invention.

[0010] [Figure 2] FIG. 1 illustrates cut-paste expansion of a training dataset to include rare behaviors in a new environment, according to one embodiment of the present invention.

[0011] [Figure 3] FIG. 1 illustrates cut-paste augmentation of a training dataset to combat object bias in a trained model, according to one embodiment of the present invention.

[0012] [Figure 4] FIG. 2 is a block / flow diagram of a method for expanding a training dataset according to one embodiment of the present invention.

[0013] [Figure 5] FIG. 2 is a block / flow diagram of a method for training and using an activity recognition model according to one embodiment of the present invention.

[0014] [Figure 6] 1 is a block diagram illustrating a healthcare facility in which activity recognition is used to inform patient treatment, according to one embodiment of the present invention.

[0015] [Figure 7] FIG. 1 is a block diagram of a computing device capable of performing dataset augmentation, model training, and activity recognition, according to one embodiment of the present invention.

[0016] [Figure 8] FIG. 1 illustrates a neural network architecture that may be used as part of an activity recognition model, according to one embodiment of the present invention.

[0017] [Figure 9] FIG. 1 illustrates a neural network architecture that may be used as part of an activity recognition model, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] The training dataset can be expanded to increase the representation of rare occurrences in new environments. For example, rare behavioral examples from a first video set in a first environment can be copied to a second video set in a second environment. This generates new videos in the second environment for rare behaviors and events, not only reducing data collection costs but also improving overall action recognition performance by addressing the problem of scene bias. Video examples can also be generated by adding specific artifacts or objects to address the problem of object bias. By fine-tuning or training the model on the expanded dataset, the model learns the important biases necessary to detect behaviors in the training data and avoid other shortcuts.

[0019] Referring now to Figure 1, a set of different images is shown. A first image 100 shows a relatively rare event 104 occurring, in this case a person falling. A second image 102 shows a relatively common event 106 occurring, in this case a person walking normally. The first image 100 and the second image 102 show the same first environment and include one or more objects 108. The objects 108 may be involved in the event, or may be separate objects or background objects.

[0020] A third image 110 of a second environment is shown, which differs in at least one respect from the first environment. In this example, the second environment shows a completely different scene from the first environment, although in some cases the two environments may share characteristics, such as overlapping fields of view. A fourth image 112 shows the same second environment as the third image 110 and includes a relatively general event 114.

[0021] These videos can be used to train a machine learning classifier, for example, to recognize actions or events occurring in the input video. The resulting machine learning classifier can be used in either the first environment or the second environment, but may have difficulty identifying events that are represented in the training dataset only in a different environment. For example, the trained machine learning classifier may have difficulty identifying a fall event in the second environment because that event is not represented in the second environment in the training dataset.

[0022] 2, a combination of video elements for expanding a training dataset is shown. A clipping event 202 is identified in a source video 200 showing a first environment. The clipping event 202 is extracted from the source video 200, for example, by identifying an object or figure performing an action in a series of frames of the source video and copying the object or figure as a respective set of sub-frames.

[0023] The cropped event 202 is then added to the target video 210. The target video 210 may depict a second environment different from the first environment, may contain a different event, or may not contain any event at all. The cropped event 202 is added to the target video 210 by pasting a subframe of the cropped event 202 into each frame of the target video 210. This combination results in an augmented video 220 showing the cropped event 202 in the second environment. Adding the augmented video 220 to the training dataset results in a trained model that exhibits superior performance when attempting to identify the cropped event 202 in the second environment.

[0024] 3, a combination of video elements for augmenting a training dataset is shown. An object clip 302 is identified in a source video 300 depicting a first environment. The object clip 302 is extracted from the source video 300, for example, by identifying an object or shape based on semantic segmentation within a series of frames of the source video and copying the object or shape into a respective set of sub-frames.

[0025] The object clip 302 is then added to the target video 310. The target video 310 may depict a second environment different from the first environment, may contain different events, or may not contain any events at all. The object clip 302 can be added to the target video 310 by pasting subframes of the object clip 302 onto each frame of the target video 310. The result of this combination is an augmented video 320 showing the object clip 302 in the second environment. Adding the augmented video 320 to the training dataset may mitigate bias in the trained model, which can lead to identifying objects performing certain actions and resulting in false positives when such objects are detected in the scene. This augmentation results in a video in which the object of interest is present in the scene even when there are no associated actions or events.

[0026] Objects associated with model bias can be identified by evaluating false positive predictions (predictions where the classifier outputs a particular behavior but it does not). In some cases, this is evaluated during operation by a human operator. In other cases, they are detected by creating an augmented video containing a given object without the associated behavior. A gradient map of the model on the augmented video can be used to determine whether the model is focused before making a prediction. The gradient map can be calculated using a backward pass of the model.

[0027] Referring now to Figure 4, a method for performing dataset augmentation is shown. Given a set of labeled training videos, block 402 performs video frame segmentation. Video frame segmentation 402 extracts semantic segmentation information from the labeled videos, e.g., providing a pixel-by-pixel classification of all objects in a given scene. The segmentation information can include person or object masks that identify the locations of people or objects detected in a given video frame.

[0028] Block 404 performs object filtering, which removes segmentation data for each frame based on known model biases and the system's target task. Semantic information from the video frame segmentation 402 is used to generate relevant object and person cutouts based on the needs of the training data. In some cases, object filtering can remove all non-human object masks, leaving only human cutouts from the video, to generate videos with actions that include only people or person-human interactions. Such actions include, for example, running, walking, jumping, dancing, etc. In some embodiments, object filtering can maintain certain object and person masks within specified size and scale ranges, for example, by considering the field of view of the target scene video. This allows object filtering to ignore masks that do not meet the constraints.

[0029] Block 406 performs augmentation of the training dataset by adding actions to videos in new environments. For example, augmentation can take video frames of the target scene and add previously extracted object clippings, e.g., derived from object filtering 404, to generate new videos. These custom object clippings can be selected based on the type of bias of the action recognition model. For example, if there is a laptop or computer in the scene, and the model predicts that it will always identify the action "working on a computer," new videos can be generated by including the associated objects instead of including the actions.

[0030] Following this example, we can add an object clip featuring a laptop or desktop computer without adding a corresponding person clip showing the activity "working." Adding such footage to the training dataset can counteract existing biases where the presence of a particular object causes the model to generate false positives for the associated activity.

[0031] In some embodiments, the cut-paste extension 406 can scale object or person masks according to the height and width of a target frame of a new video and overlay the clippings on the target frame to generate a new video with rare actions or events in the new target scene. In some embodiments, the cut-paste extension 406 can be performed using deep learning models based on generative adversarial networks or diffusion models. Such models can be trained to generate perceptually realistic videos by combining the clippings with the target video. In particular, diffusion probability models can be trained conditionally. Custom object clippings or filtered person or object clippings can be used to condition the diffusion model to generate temporally and spatially consistent frames.

[0032] The cut-paste extension 406 can add appropriate labels to the augmented footage. For example, a person clip showing a person performing a given action can have a label obtained from the clip's source video. Thus, for a source video labeled as showing the action "falling," the person clip may be assigned the same label. If the clipped person is added to a new video, the new video may similarly be labeled as showing the action "falling." When using object clipping to mitigate model object bias, the labels of associated actions may be omitted or excluded from the labels of the new video. This helps to establish that the model should not necessarily associate the presence of an object with an action.

[0033] 5, a method for training and using an action recognition model is shown. As described above, dataset expansion 400 augments an existing training dataset by, for example, adding rate actions or events to footage from environments where they are not present, or by adding objects to footage where association of the objects with particular actions would bias the model. The result of dataset expansion 400 may be a combined training dataset that includes the original and expanded training examples.

[0034] Block 502 performs training of a machine learning model using the enriched training dataset. In particular, it is contemplated that supervised training is performed using labels from the enriched training dataset. In this example, the model may be trained to perform action recognition on input video. By training the model using the enriched training dataset, the model becomes more robust. As a result, the model provides more accurate results for rare actions in various environments and is less susceptible to bias when an object is present without performing the relevant action.

[0035] In some cases, blocks 400 and 502 may be repeated before deployment 504. For example, after model training 502, the trained model may be tested for biases associated with particular objects. Videos containing such objects may be provided as input to the model, and the output may be evaluated to determine whether the model generates false positives for actions due to the presence of the objects. If such biases are detected, dataset expansion 400 may be repeated to generate new videos that contain the objects but do not contain the associated actions.

[0036] The model is deployed to an operational environment 504. For example, the model may be deployed in a security system or hospital where it is used to monitor actions performed by individuals. Action recognition 506 is performed in the operational environment as new footage is collected and processed by the trained model. Action recognition can be used for any appropriate purpose, such as identifying risky patient behaviors or adverse health events. A corresponding action 508 can then be taken, for example, automatically calling a security guard or automatically administering treatment to the patient.

[0037] Referring now to FIG. 6, an illustration of activity recognition in the context of a healthcare facility 600 is shown. Activity recognition may be used to identify actions performed by a patient in the context of the healthcare facility 600, which may help devise a treatment for the patient. For example, video monitoring of a patient and subsequent activity recognition may indicate that the patient is in distress or experiencing an adverse health event, such as a fall or stroke. Treatment may be automatically initiated or implemented to help the patient.

[0038] The healthcare facility may include one or more medical professionals 602 that provide information related to events and system status measurements to the activity recognition system 608. The treatment system 604 may further be designed to monitor the patient's condition, generate a medical record 606, and automatically manage and adjust treatment as needed. In some cases, the medical record 606 may include video surveillance of the patient.

[0039] Based on information drawn from at least the medical professional 602, the treatment system 604, and the medical record 606, the behavior recognition system 608 identifies behaviors and events associated with the patient, and in particular, behaviors associated with abnormalities in the patient's health condition. Information regarding the recognized behaviors may be forwarded to the medical professional 602 for diagnosing and treating the patient's condition.

[0040] The different elements of the healthcare facility 600 can communicate with each other over the network 610, for example, using any suitable wired or wireless communication protocol and medium. Thus, the activity recognition system 608 can access the remotely stored medical records 606, communicate with the treatment system 604, receive instructions for the medical professional 602, and send reports. In particular, the activity recognition system 608 can automatically initiate changes in the patient's treatment in response to new information obtained from the medical records 606 by sending instructions to the treatment system 604. For example, the treatment system can automatically administer medication or discontinue treatment in response to an adverse health event.

[0041] In some cases, the behavior recognition system 608 can initiate a treatment for the patient in response to a particular recognized behavior or event. Using the example of detecting that a patient is having a stroke, the treatment system 604 can automatically adjust the dosage of an anti-epileptic drug. Thus, the output of the behavior recognition system 608 can include one or a combination of the above automatic treatments and notifications to the medical professional 602. In some cases, the treatment plan can be used by the medical professional to support decision-making for patient management. For example, upon being notified of a change in the patient's condition, the medical professional 602 can visit the patient to check in.

[0042] 7, an exemplary computing device 700 is shown, in accordance with an embodiment of the present invention. The computing device 700 is configured to perform activity recognition.

[0043] Computing device 700 may be embodied as any type of computational or computing device capable of performing the functions described herein, such as, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a user electronic device. Additionally or alternatively, computing device 700 may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically separated computing device.

[0044] 7, computing device 700 illustratively includes a processor 710, an input / output subsystem 720, a memory 730, a data storage device 740, and a communications subsystem 750, and / or other components and devices typically found in a server or similar computing device. In other embodiments, computing device 700 may include other or additional components (e.g., various input / output devices) such as those typically found in a server computer. Furthermore, in some embodiments, one or more of the illustrative components may be incorporated into or otherwise form part of another component. For example, memory 730, or portions thereof, may be incorporated into processor 710 in some embodiments.

[0045] The processor 710 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0046] Memory 730 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 730 may store various data and software used during operation of computing device 700, such as an operating system, applications, programs, libraries, and drivers. Memory 730 is communicatively coupled to processor 710 via I / O subsystem 720 and may be embodied as circuits and / or components for facilitating input / output operations with processor 710, memory 730, and other components of computing device 700. For example, I / O subsystem 720 may be embodied as or otherwise include a memory controller hub, an input / output control hub, a platform controller hub, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations. In some embodiments, I / O subsystem 720 may form part of a system-on-chip (SOC) and be integrated with processor 710, memory 730, and other components of computing device 700 on a single integrated circuit chip.

[0047] The data storage device 740 may be embodied as any type of device or device configured for short-term or long-term storage of data, such as, for example, a memory device and circuitry, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage device 740 may store program code 740A for dataset expansion, program code 740B for model training, program code 740C for behavior recognition, and / or program code 740D for performing corrective actions responsive to recognized behaviors. The communications subsystem 750 of the computing device 700 may be embodied as any network interface controller or other communications circuitry, device, or collection thereof that can enable communications between the computing device 700 and other remote devices over a network. The communications subsystem 750 may be configured to achieve such communications using any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0048] As shown, computing device 700 may also include one or more peripheral devices 760. Peripheral devices 760 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 760 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0049] Of course, computing device 700 may include other elements (not shown) or omit certain elements, as would readily occur to one skilled in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 700, depending on the particular implementation of the same, as would be readily understood by one skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc. may be utilized in a variety of configurations. These and other variations of processing system 700 will be readily contemplated by one skilled in the art in view of the teachings of the present invention provided herein.

[0050] Referring now to Figures 8 and 9, exemplary neural network architectures are shown that can be used to implement portions of the present models, such as action recognition models 800 and 900. Neural networks are generalized systems whose capabilities and accuracy improve with exposure to additional empirical data. Neural networks learn by being exposed to empirical data. During training, the neural network stores and adjusts weights that are applied to input empirical data. By applying the adjusted weights to the data, the neural network can identify that data belongs to a specific predefined class from a set of classes and output the probability that the input data belongs to each class.

[0051] Empirical data (also called training data) from a set of examples is formatted as a string of values ​​and fed to the input of a neural network. Each example is associated with a known outcome or output. Each column is represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can be of various data types and contain multiple different values. The network can have one input node for each value that makes up the example input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network being built and trained.

[0052] Neural networks "learn" by comparing neural network outputs generated from input data with known values ​​from examples and adjusting stored weights to minimize the difference between the output value and the known value. Adjustments can be made to the stored weights through backpropagation, where the weight's influence on the output value is determined by calculating a mathematical gradient and adjusting the weight in a way that shifts the output toward the minimum difference. This optimization, called gradient descent, is a non-limiting example of how training occurs. A subset of examples with known values ​​that were not used in training can be used to test and validate the accuracy of the neural network.

[0053] During operation, a trained neural network can be used on new data not previously used for training or validation through generalization. The adjusted neural network weights can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0054] In a layered neural network, nodes are arranged in layers. An exemplary simple neural network has an input layer 820 of source nodes 822 and a single computational layer 830 with one or more computational nodes 832 that also function as output nodes, with a single computational node 832 for each possible category into which an input example may be classified. The input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The data values ​​812 in the input data 810 may be represented as a column vector. Each computational node 832 in the computational layer 830 generates a weighted linear combination of values ​​from the input data 810 provided to the input node 820 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0055] A deep neural network, such as a multilayer perceptron, may have an input layer 820 of source nodes 822, one or more computational layers 830 with one or more computational nodes 832, and an output layer 840 with one output node 842 for each possible category into which an input example may fall. The input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The computational nodes 832 in the computational layer 830 are also referred to as hidden layers because they are located between the source nodes 822 and the output nodes 842 and are not directly observed. Each node 832, 842 in the computational layer generates a weighted linear combination of values ​​from the values ​​output by the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values ​​from each previous node may be, for example, w1, w2, ... w n-1 ,w n The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in a computational layer is connected to every node in the previous layer, or the connections between layers can have other configurations. If links between nodes are missing, the network is said to be partially connected.

[0056] Training a deep neural network involves two phases: a forward phase, in which the weights of each node are fixed and inputs are propagated through the network, and a backward phase, in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0057] Computational nodes 832 in one or more computational (hidden) layers 830 perform nonlinear transformations on the input data 812 to generate a feature space. Classes or categories may be more easily separated in the feature space than in the original data space.

[0058] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0059] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or propagation medium. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0060] Each computer program can be tangibly stored on a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0061] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0062] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0063] As employed herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that may be on-board or off-board or dedicated for use by the hardware processor subsystem.

[0064] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0065] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0066] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0067] References herein to "one embodiment" or "one embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment," as well as any other variations thereof, in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of one or more embodiments may be combined given the teachings of the present invention provided herein.

[0068] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one," such as "A and / or B" or "at least one of A and B" will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0069] The foregoing is understood in all respects to be illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A computer-implemented method for training a machine learning model, comprising: Identifying (404) an object or person associated with an action in the first video; copying (406) the object or person from the first image into a second image to generate a third image; training (502) a machine learning model using the first video and the third video.

2. 10. The method of claim 1, The method, wherein identifying the object or person is identifying a person performing the action, and the second image includes an environment different from the environment of the first image.

3. 3. The method of claim 2, The method further comprising labeling the third video with an activity-related label.

4. 3. The method of claim 2, The method, wherein copying the object or person includes copying a portion of a frame of the first video showing the activity and pasting the portion of the frame into a respective frame of the second video.

5. 10. The method of claim 1, A method for identifying an object or person, wherein identifying an object or person is related to a bias in the machine learning model.

6. 6. The method of claim 5, The method further comprising labeling the third video with one or more labels that exclude labels associated with the activity.

7. 6. The method of claim 5, training the machine learning model using the first video; detecting the bias of the machine learning model as a false positive associated with an input including the object; The identifying and copying is performed in response to detecting the bias in the machine learning model.

8. 10. The method of claim 1, The method, wherein the behavior is related to an abnormality in a patient's health status in a medical setting.

9. 9. The method of claim 8, processing the patient video using the machine learning model; and determining a behavior associated with the patient's health status; and and informing a medical professional of the patient's health status to assist the medical professional in making patient management decisions.

10. 10. The method of claim 9, The method further comprising performing a therapeutic action responsive to the patient's health condition, including instructions to a therapeutic system to automatically administer a therapy to the patient.

11. 1. A system for training a machine learning model, comprising: a hardware processor (710); When executed by a hardware processor, the hardware processor: Identifying (404) an object or person associated with the action in the first video; copying (406) the object or person from the first image into a second image to generate a third image; and a memory (740) storing a computer program for training (502) a machine learning model using the first video and the third video.

12. 12. The system of claim 11, The computer program causes the hardware processor to identify a person performing the action, and the second image includes an environment different from the environment of the first image.

13. 13. The system of claim 12, The computer program further causes the hardware processor to label the third video with an action-related label.

14. 13. The system of claim 12, The computer program further causes the hardware processor to copy a portion of a frame of the first video that indicates the behavior and paste the portion of the frame into each frame of the second video.

15. 12. The system of claim 11, The computer program further causes the hardware processor to identify objects associated with bias in the machine learning model.

16. 16. The system of claim 15, The computer program further causes the hardware processor to label the third video with one or more labels that exclude a label associated with the behavior.

17. 16. The system of claim 15, The computer program further causes the hardware processor to: training the machine learning model using the first video; detecting the bias of the machine learning model as a false positive associated with an input including the object; The system wherein the identifying and copying is performed in response to detecting the bias in the machine learning model.

18. 12. The system of claim 11, The system wherein the behavior is related to an abnormality in a patient's health condition in a medical setting.

19. 20. The system of claim 18, The computer program further causes the hardware processor to: processing an image of the patient using the machine learning model; determining a behavior associated with the patient's health condition; A system for informing a medical professional of a patient's health status to assist the medical professional in making decisions for patient management.

20. 20. The system of claim 19, The computer program further causes the hardware processor to perform a therapeutic action responsive to the patient's health condition, including instructions to a therapeutic system to automatically administer a therapy to the patient.

Images