Infant action generative modeling and classification

Abstract

Provided herein are methods and systems for generating infant action data and training an infant action recognition model. Video recordings for training an infant action recognition model are difficult to acquire, thus the methods and systems described may generate synthetic infant action data for training infant action recognition models with greater accuracy. The methods and systems described include a process of using real infant action recordings and corresponding infant action classifications to generate synthetic infant action data. Using an infant action recognition model, the synthetic infant action data is evaluated for accuracy. The synthetic infant action data is scored based on diversity. The synthetic infant action data may be filtered based on the accuracy and the diversity such that the filtered data represents a diverse distribution across infant action classifications. The filtered synthetic infant action data may be added to the training data to further refine the action recognition model.

Description

[0001] TITLE

[0002] INFANT ACTION GENERATIVE MODELING AND CLASSIFICATION

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the priority of U.S. Provisional Application No. 63 / 676,323 filed July 26, 2024 and titled “Infant Action Generative Modeling and Classification”, the whole of which is hereby incorporated by reference.

[0005] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0006] This invention was made with government support under Grant Number 1915065 awarded by the National Science Foundation. The Government has certain rights in the invention.

[0007] BACKGROUND

[0008] Despite the significant advancements in current vision-based human action recognition (HAR) models, which leverage extensive datasets like Nanyang Technological University’s Red Blue Green and Depth (NTU RGB+D) 120

[0021] , Human3.6M

[0017] , and Northwestern- University of California, Los Angeles (N-UCLA)

[0036] with abundant samples per class, infant activity recognition is still in its infancy. High costs associated with collecting and labeling infant data, coupled with concerns regarding security and privacy, have resulted in a significantly limited pool of infant data available for model training. Recent research

[0012] highlights a challenge in transferring knowledge from adult-based HAR models to infant datasets, resulting in poor performance. This discrepancy stems from differences in action types and settings between adults and infants. Adult datasets [19, 21, 31, 33] often feature actions like taking selfies or shaking hands, absent in infant behavior. Conversely, infant actions like crawling are not typically found in adult datasets. Variations in how common actions are performed further hinder generalization. Unlike controlled lab setups for adults, obtaining precise infant datasets is challenging due to infants’ uncooperative nature. Consequently, most current accessible infant datasets like InfAct

[0016] and InfActPrimitive

[0012] lack three-dimensional (3D) skeleton ground truth and are sourced from diverse camera angles without control, often collected through sources such as YouTube.

[0009] Recent advances in generative models [10, 11, 35, 40] have facilitated the creation of synthetic motion data to augment datasets. However, their effectiveness is hindered by a reliance on large datasets for training, posing a challenge when applied to smaller datasets. Models such as generative adversarial networks (GANs), variational autoencoders (VAEs) struggle to generate sufficient data for limited datasets due to their need for extensive training data. This issue is notably pronounced in specialized areas such as infant motion recognition, where accurately capturing domain-specific interactions and knowledge from sparse real data is exceptionally challenging [12, 15],

[0010] SUMMARY OF THE INVENTION

[0011] The present technology provides computer implemented systems and methods for generating infant action data and training an infant action recognition model. The present technology also provides for training and re-training (or fine-tuning) of an infant action generation model and the infant action recognition model. The generated synthetic infant action data may be filtered based on infant action classification accuracy and confidence based on infant action classification certainty provided by the infant action recognition model. The synthetic infant action data may also be filtered on diversity of actions to create a robust training set, where the diversity of samples within the synthetic infant action date represents a statistical measure of sample variability' relative to a training data distribution. Generating synthetic infant action data and iterative refinement of the synthetic infant action data may provide a more comprehensive training set resulting in highly accurate infant action recognition models.

[0012] An aspect of the technology is a computer implemented method for generating infant action data and training an infant action recognition model. The computer-implemented method includes the following operations: receiving real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; generating synthetic infant action data using an infant action generation model trained on the real training data, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification; evaluating the synthetic infant action data by processing the set of video samples with an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification; determining a filtered set of video samples by selecting a first subset of video samples from the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; combining the filtered set of video samples with the real training data to form modified training data; and training the infant action recognition model using the modified training data.

[0013] The aspect of technology for the computer-implemented method for generating infant action data and training an infant action recognition model further includes the following operations for evaluating the synthetic infant action data using the infant action recognition model includes assigning a confidence score to each video sample, the confidence score indicating infant action classification certainty provided by the infant action recognition model, and wherein determining the filtered set of video samples, further comprises: selecting a second subset of video samples, from the first subset of video samples, wherein each video sample in the second subset has a corresponding confidence score that exceeds a predetermined confidence threshold; and storing the second subset of video samples as the filtered set of video samples.

[0014] Another aspect of the computer-implemented method for generating infant action data and training an infant action recognition model further includes the following operations for determining the filtered set of video samples, further comprises: computing, for each video sample in the set of video samples, a diversity score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; and determining, the filtered set of video samples from the first subset of video samples by selecting a predetermined percentage of the video samples based on ranked diversity scores, wherein the video samples that are selected represent a diverse distribution across infant action classifications.

[0015] Another aspect of the computer-implemented method for generating infant action data and training an infant action recognition model, where the diversity score is defined as a ratio of within-class distances to between-class distances; wherein a within-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data that share the same infant action classification; and wherein a between-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data with different infant action classifications.

[0016] The technology7can be further summarized with the following list of features.

[0017] 1. A computer-implemented method for generating infant action data and training an infant action recognition model, comprising: receiving real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; generating synthetic infant action data using an infant action generation model trained on the real training data, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification; evaluating the synthetic infant action data by processing the set of video samples with an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification; determining a filtered set of video samples by selecting a first subset of video samples from the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; combining the filtered set of video samples with the real training data to form modified training data; and training the infant action recognition model using the modified training data.

[0018] 2. The computer-implemented method of feature 1, wherein the infant action generation model is a transformer-based variational autoencoder (VAE) trained on pose-based representations extracted from the real training data.

[0019] 3. The computer-implemented method of feature 1 or feature 2, wherein the infant action recognition model is a spatial-temporal graph convolutional network (ST-GCN), and the method further includes training the ST-GCN using the modified training data.

[0020] 4. The computer-implemented method of any of the preceding features, wherein each video sample from the synthetic infant action data comprises a sequence of infant poses sampled from continuous temporal trajectories.

[0021] 5. The computer-implemented method of any of the preceding features, wherein evaluating the synthetic infant action data using the infant action recognition model includes assigning a confidence score to each video sample, the confidence score indicating infant action classification certainty provided by the infant action recognition model, and wherein determining the filtered set of video samples, further comprises: selecting a second subset of video samples, from the first subset of video samples, wherein each video sample in the second subset has a corresponding confidence score that exceeds a predetermined confidence threshold; and storing the second subset of video samples as the filtered set of video samples.

[0022] 6. The computer-implemented method of any of the preceding features, wherein determining the filtered set of video samples, further comprises: computing, for each video sample in the set of video samples, a diversity score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; and determining, the fdtered set of video samples from the first subset of video samples by selecting a predetermined percentage of the video samples based on ranked diversity scores, wherein the video samples that are selected represent a diverse distribution across infant action classifications.

[0023] 7. The computer-implemented method of feature 6, wherein the diversity score is defined as a ratio of within-class distances to between-class distances; wherein a within-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data that share the same infant action classification; and wherein a between-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data with different infant action classifications.

[0024] 8. The computer-implemented method of any of the preceding features, wherein the infant action classifications include sitting, standing, rolling, and crawling.

[0025] 9. The computer-implemented method of any of the preceding features, further comprising, after training the infant action recognition model using the modified training data: training the infant action generation model using the modified training data; generating second synthetic infant action data using the infant action generation model, wherein the second synthetic infant action data comprises a second set of video samples depicting at least one infant in motion and the associated infant action classification; evaluating the synthetic infant action data by processing the second set of video samples with the infant action recognition model to obtain the accuracy score of the associated infant action classification of each sample in the second set of video samples; determining a second filtered set of video samples by selecting a second subset of video samples from the second synthetic infant action data based on the accuracy score exceeding the predetermined accuracy threshold; adding the second filtered set of video samples to the modified training data; determining each accuracy score of the second set of video samples exceeds a predetermined model accuracy threshold; and in response to determining each accuracy score of the second set of video samples exceeds the predetermined model accuracy threshold, ending training of the infant action recognition model.

[0026] 10. The computer-implemented method of feature 9, wherein the predetermined model accuracy threshold is 80%, 85%, 90%, or 95%.

[0027] 11. The computer-implemented method of feature 9, further comprising: receiving real sample data representing unclassified real-world video of a real infant in mot on; determining at least one infant action classification for the real sample data using the infant action recognition model; and outputting the at least one infant action classification for the real sample data.

[0028] 12. A computer-implemented method for generating an infant action training dataset, comprising: receiving real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; generating synthetic infant action data using an infant action generation model trained on the real training data, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification; evaluating the synthetic infant action data by processing the set of video samples with an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification and wherein the infant action recognition model assigns a confidence score, the confidence score indicating classification infant action classification certainty provided by the infant action recognition model, to each video sample in the set of video samples; determining a first subset of video samples from the set of video samples in the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; determining a second subset of video samples, from the first subset of video samples, based on video samples from the first subset of video samples with corresponding confidence scores exceeding a predetermined confidence threshold; determining, for each video sample from the second subset of video samples, a diversity score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; determining, a filtered set of video samples from the second subset of video samples by selecting a predetermined percentage of the video samples in the second subset of video samples based on ranked diversity scores, wherein the video samples that are selected represent a diverse distribution across infant action classification; and generating the infant action training dataset by combining the filtered set of video samples and the real training data.

[0029] 13. The computer-implemented method of feature 12. wherein infant motion classifications include sitting, standing, rolling, and crawling.

[0030] 14. A system for an infant action sample generation model and recognition model pipeline, comprising: at least one processor; at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; tram an infant action generation model using the real training data; generate synthetic infant action data using the trained infant action generation model, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification; evaluate the synthetic infant action data by processing the set of video samples using an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification; determine a filtered set of video samples by selecting a first subset of video samples from the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; combine the filtered set of video samples with the real training data to form modified training data; train the infant action recognition model using the modified training data; and train the infant action generation model using the modified training data.

[0031] 15. The system of feature 14, wherein each video sample from the synthetic infant action data comprises a sequence of infant poses sampled from continuous temporal trajectories.

[0032] 16. The system of feature 14 or feature 15. wherein the infant action classifications include sitting, standing, rolling, and crawling. 17. The system of any of features 14-16, wherein evaluating the synthetic infant action data using the infant action recognition model includes assigning a confidence score to each video sample, the confidence score indicating infant action classification certainty provided by the infant action recognition model, and wherein determining the filtered set of video samples, further includes instructions to: select a second subset of video samples, from the first subset of video samples, wherein each video sample in the second subset has a corresponding confidence score that exceeds a predetermined confidence threshold; and store the second subset of video samples as the filtered set of video samples.

[0033] 18. The system of any of features 14-17, wherein determining the filtered set of video samples, further includes instructions to: compute, for each video sample in the set of video samples, a diversity score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; and determine, the filtered set of video samples from the first subset of video samples by selecting a predetermined percentage of the video samples based on ranked diversity scores, the video samples that are selected represent a diverse distribution across infant action classifications.

[0034] 19. The system of feature 18, wherein the diversity' score is defined as a ratio of within- class distances to between-class distances; wherein a within-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data that share the same infant action classification; and wherein a between-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data with different infant action classifications.

[0035] 20. The system of any of features 14-19, wherein the infant action generation model is a transformer-based variational autoencoder (VAE) trained on pose-based representations extracted from the real training data.

[0036] BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG. 1A illustrates an overview of the Infant Action Generation and Classification (InfAGenC) Pipeline. FIG. 1A depicts the cyclic pipeline, which combines a transformerbased variational autoencoder (VAE) with a spatial -temporal graph convolutional network (ST-GCN) to create synthetic infant action samples. By iterative refinement of the action generative model with diverse and accurate data, the realism of synthetic data is improved, leading to more precise infant action recognition models.

[0038] FIG. IB illustrates the action generative model of the InfAGenC Pipeline that uses the transformer-based VAE to generate the synthetic infant action samples.

[0039] FIG. 1C illustrates the action recognition model of the InfAGenC Pipeline using the ST-GCN to evaluate the classification of the synthetic infant action samples.

[0040] FIG. 1 D illustrates the synthetic data recycling of the InfAGenC Pipeline that filters the synthetic infant action data based on accuracy and diversity.

[0041] FIG. 2 illustrates an operational process of the of InfAGenC Pipeline, in accordance with some embodiments.

[0042] FIG. 3 illustrates a timeline for monitoring a baby’s motor actions while in their natural environments which may provide for early detection of problems and administering timely interventions.

[0043] FIG. 4 illustrates snapshots of the generated samples. The illustrated images are organized by action class, displayed in the following order from left to right: Supine. Prone. Sitting, Standing, and All-fours. The first row is the real action samples, the other two rows present distinct motions depicted as skeletons per action class.

[0044] FIG. 5 illustrates sample sequences, or frames, of generated actions. Each sample’s frames are extracted from a generated action sequence spanning 3 seconds. The actions, displayed sequentially from top to bottom, are: Sitting, Crawling, Standing.

[0045] FIG. 6 illustrates snapshots of generated adult action samples. The generated samples are produced by a trained action generative model from the InfAGenC Pipeline using a prepared small NTU dataset with four action classes: Falling. Jumping, Sitting, and Standing. Each row shows one action class samples.

[0046] FIGS. 7A-C illustrate a t-distributed Stochastic Neighbor Embedding (T-SNE) feature visualization of real and synthetic data. The data points are color-coded to distinguish between various infant actions. The data sources are shape-coded: • for real part of training data, ▲ for real test data, and + for selected synthetic samples used during training. FIG. 7A compares the features of real training data, real test data, and synthetic data (generated during training of the InfAGenC model) for an experimental InfactPrimitive dataset. FIG. 7B compares the features of real training data, real test data, and synthetic data (generated during training of the InfAGenC model) for an experimental InfantAction dataset. FIG. 7C compares the features of real training data, real test data, and synthetic data (generated during training of the InfAGenC model) for an experimental NTU dataset.

[0047] DETAILED DESCRIPTION OF THE INVENTION

[0048] Given the limited infant action data and domain gap between real and synthetic datasets, high-quality skeleton-based infant motions may be successfully generated by introducing the methods and techniques described here for infant action generation and classification (InfAGenC) pipeline that is depicted in FIG. 1 A. InfAGenC pipeline harnesses the synergistic capabilities of a transformer-based VAE

[0027] for data generation, combined with a spatial- temporal graph convolutional network (ST-GCN)

[0038] for action recognition. Drawing inspiration from active learning’s pool-based sampling method

[0039] , our approach selectively enriches the dataset. Every ten iterations, high-quality synthetic samples are identified and incorporated using between- and within-class distance measurements. The high-quality synthetic samples may be evaluated on accuracy and diversity7. These samples may be carefully chosen, alongside real data, and are reintroduced to the system, creating a more robust and diverse dataset for model training.

[0049] Compared to GANs

[0037] , VAEs are known for their more stable training dynamics and the ability7to generate interpretable latent spaces, which facilitate a more accurate assessment of diversity scores. Recognizing the variability in infants’ movement speeds across different actions, this approach enhances motion consistency between real and synthetic data by integrating velocity7as a factor in the loss function through the use of regularization metrics.

[0050] The experimental outcomes indicate a marked improvement in the performance of action recognition models test on “in-the wild’' videos of InfActPrimitive dataset, achieving increasing accuracy to more than 15%. In summary7, this introduces the following significant contributions. The development of a novel pipeline, called InfAGenC that combines a transformer-based VAE for generation with a spatial-temporal graph convolutional network (ST-GCN) for recognition, which aims to enhance infant action recognition with limited data and bridges the gap between real and synthetic data. The implementation of a pool-based sampling approach to enrich the dataset, selecting samples that offer an optimal balance between accuracy, such as determining accuracy using predicted confidence scores, and diversity, such as assessing through between- and within-class distances. The creation of an infant action dataset (InfantAction) with more complex action movements compared to the existing InfActPrimitive data. InfantAction may include daily activity data from infant subjects, covering broader action classes, such as Sitting, Standing, Rolling, and Crawling. Skeleton-based methods for Human Activity Recognition (HAR) [5, 8, 23. 28, 38] have emerged as a prominent choice due to their ability to efficiently represent human movements using joint coordinates, effectively minimizing potential disruptions caused by red-green-blue (RGB) appearance variations. ST-GCN

[0038] introduces inter-frame edges, connecting corresponding joints across consecutive frames, enhancing inter-frame relationship modeling and improving temporal dynamics understanding within skeletal data. Multi-Scale Graph Convolutional 3D (MS-G3D) model

[0023] , a spatial -temporal graph convolutional network designed for skeleton-based action recognition, combines multi-scale graph convolutions into a unified G3D module, enhancing long-range modeling by prioritizing nodes in different neighborhoods. It utilizes dense cross-spacetime edges as skip connections for direct information propagation across the spatial-temporal graph. InfoGCN [5] merges a learning objective with an encoding method utilizing attention-based graph convolution, capturing discriminative information regarding human actions.

[0051] However, these previous noted models have been trained on the enormous data backbones, such as NTU RGB+D 120

[0021] containing 120 kinds of actions, a total of 114,480 samples, in the form of depth. 3D skeleton. RGB and infrared sequence, Kinetics-700 [4], a video dataset of 650,000 clips that covers 700 human action classes, and Bodies, Action, and Behavior with English Labels (BABEL)

[0029] , a large dataset with language labels describing the actions being performed in mocap sequences. There are over 28,000 sequence labels, and 63,000 frame labels in BABEL, which belong to over 250 unique action categories. In relation to infant studies, several infant-specific image and video datasets have been released, where each are tailored for specific applications [1, 2, 41, 42], Synthetic Real Infant Pose (SyRIP)

[0015] and Moving Infants in RGB-D (MINI-RGBD)

[0013] respectively with 17 and 24 annotated joints were created to benchmark for a standardized evaluation of pose estimation algorithms in infants. AggPose [3] was proposed to train a deep aggregation transformer for infant pose detection. AggPose adopted general movements assessment (GMA) devices to record infant movement videos in supine position. More than 216 hours of videos and 15 million frames were extracted to generate AggPose. The AggPose dataset randomly sampled 20,748 frames from the videos and used professional clinicians to annotate infant 21 keypoints locations. Both MINI-RGBD and AggPose have considerable amounts of data. However, all of these datasets contains image-only data and are not suited for the task of infant action recognition. .Baby- Pose

[0026] contains over 1000 videos of preterm infants captured using a depth-sensing camera along with annotations of 12 limb-joint positions for each frame. However, it only contains the data of newborns with limited motions on only supine body pose with one-fold background. The InfActPrimitive, for infant action recognition, includes a small set of 975 video clips from 5 classes of action collected from two different sources: YouTube and recruited subjects.

[0052] Domain adaptation has been extensively used to bridge the gap between source (real) and target (synthetic) data distributions encountered during the application of Deep Neural Networks

[0030]

[0032] , Hatamimajoumerd et al.

[0012] fine-tuned state-of-the-art skeleton-based action recognition models pre-trained on adult skeleton datasets on InfActPrimitive. The results indicate a gap between the action recognition results on infant and adult datasets.

[0053] Recent advancements in human pose estimation have led to the development of motion generation models using skeleton data. These models may enrich the training set and tackle the small data problem, specifically in the infant action recognition domain, where data collection is difficult [6, 10, 18, 20, 25], Tevet et al.

[0035] developed Motion Diffusion Model (MDM) a transformer-based generative model for human motion, prioritizing sample prediction over noise in diffusion steps while employing geometric losses. Degardin et al. [7] introduced Kinetic-GAN, an architecture blending Generative Adversarial Networks and Graph Convolutional Networks, capable of conditioning up to 120 actions. Compared to GAN- based models, VAEs may offer more stable training and interpretable latent spaces, aiding in accurate diversity score assessment. Lucas et al.

[0024] introduced PoseGPT, an auto-regressive transformer based approach which internally compresses human motion into quantized latent sequences. Feng et al. [9] leveraged Large Language Models (LLMs) to directly generate 3D human body poses from images or text by embedding SMPL poses within a multi-modal LLM. However, they have not been explored in small data domains like infant motion, where unpredictability and lack of 3D skeleton ground truth present significant challenges.

[0054] The methods and techniques described herein introduce infant action generation and classification (InfAGenC) pipeline, which may be crafted to improve action recognition by creating a variety of accurate samples, especially within the context of small data domains, with a particular emphasis on infant action recognition.

[0055] The problem formulation considers a skeleton dataset X = {x}, x2, . . . , xn], where each sample xtencapsulates a sequence of joint locations or axis angle values xte UkR xR xT. Here, K denotes the number of infant joints, R denotes the dimensionality of the joint representation, and T denotes the sequence length. Each sample xtis associated with an action class atbelonging to a set of m actions A = {alta2, .. . , um}. The directive is to devise a generative model G(X; 0) adept at mirroring the original data distribution Pdata(%). The purpose of G is to fabricate synthetic action sequences s, such that s ~ pmOdeiM, with pmodei(%; 0) closely emulating Pdata(x) . This includes that the synthetic data should exhibit diverse pose variations and preserve the intrinsic motion styles and statistical characteristics inherent to the original dataset. Additionally, an action recognition model, denoted as M(S). is defined to dynamically evaluate each synthetic sample stwithin the set of generated action sequences S = to determine if the sample is good enough to be assigned into the correct action class, which indicates predicted action equals to the given class, noted as cp = at. To highlight the capabilities of the generative model, the refined, high-quality synthetic action sequences S are amalgamated with the infant real data A, forming a comprehensive training dataset Xtrain= {s1, s2, - - - , -sp} U {x1, x2, . . . , xn}, which is employed to further augment the training of generative model G and recognition model M.

[0056] Action generation, action recognition, and sample selection / filtering are major components of the InfAGenC pipeline. As depicted in FIG. 1A, the process begins by feeding real data from the training data set 120, which is a set of body keypoints / pose angles, into a pretrained generative model (e.g., an action generative model 105). Subsequently, the resultant samples undergo evaluation by an action recognition model 110. Utilizing predicted confidence scores and diversity scores as part of a synthetic data recycling 115 process, selected samples are transferred to the synthetic data pool for integration into the training data set 120 for subsequent iterations.

[0057] The steps depicted in FIG. 1A of the cyclic pipeline 100, begin with Step 1, where real and synthetic data form the initial training set 120 for the transformer-based VAE pose generative model (e.g., the action generative model 105). In Step 2, the action recognition model 110, an ST-GCN, evaluates the generated samples to identify those with the highest Classification Accuracy Score (CAS). Step 3 applies a sample filtering technique to retain high-confidence samples, while Step 4 involves a sample selection process based on weighted within-class and between-class distance metrics. The central portion of these samples is then selected in Step 5 for synthetic data recycling 115. In Step 6, the filtered high-quality samples are concatenated back to the training set 120. Finally, Step 7 closes the loop by updating the action recognition model 110 with the augmented training set 120, preparing it for the next cycle of evaluation for the cyclic pipeline 100.

[0058] As shown in a detailed depiction in FIG. IB, transformer-based conditional VAE model G is trained using training data that initially comprises only real infant data V which serves as the baseline action generative model 105. Following this, the action generative model 105 is integrative trained using the training set XtrainXtrain. This phase involves iterative training on the set, focusing on the optimization of losses including reconstruction loss £rec. motion / velocity loss £vet. and Kullback-Leibler (KL) divergence loss £KL. The equation representing the loss function of the generative model may be formalized as follows, withKL, rec, andvetrepresenting the weights assigned to each respective loss term:

[0059] For action recognition, and as shown in a detailed depiction in FIG. 1C. the action recognition model 110 training plays a pivotal role in the methodology for the InfAGenC pipeline. A well-trained action recognition model M 110, such as the ST-GCN, is utilized for action recognition. The action recognition model 110 is trained on the original real infant motion data X. which is used for evaluating the generated samples' quality and performing action classification tasks effectively. Cross-entropy loss is the loss function to minimize. Following each action generative model G 105 training iteration, it produces samples {sftk} _fcand Nkis the number of samples generated at epoch tfe, that are subsequently evaluated by the ST-GCN model to assess their classification accuracy score (CAS). In this example, after every 10 epochs, the samples achieving the highest CAS are selected as candidates for the next synthetic data filtering (e.g.. synthetic data recycling 115). ensunng a continuous improvement in sample quality. This may be formalized as:

[0060] Subsequently, the set of generated samples from the best performing epoch, tbest, is selected as a candidate:

[0061] Notably, with the training set 120 Xtrainaugmented by synthetic data, the action recognition model M 1 10 undergoes 3 epochs of updates to adapt to the new input data, in this scenario embodiment. The updated action recognition model 110 is then utilized for evaluating the next cycle of generated samples. This update ensures that the action recognition model 110 remains attuned to the nuances of both original and newly generated synthetic samples, thereby maintaining high accuracy and efficiency in action classification tasks.

[0062] As shown in a detailed depiction in FIG. ID, sample filtering and selection (e.g., synthetic data recycling 115) occurs before transferring generated samples to the next training iterations. In some embodiments, low quality samples may be discarded in the set 5candidatebased on their classification accuracy and confidence scores. Only samples that are correctly classified and exceed a predefined confidence threshold, such as defined in Equation 7, are retained. This may ensure high-quality7motion samples in the synthetic data recycling 115 process. This filtering maintains integrity and quality of the training data.

[0063] In Equation 7, Sfilteredis the set of samples retained after filtering, ConftsJ represents the confidence score of sample sh6 is the predefined confidence threshold, and Accfs J is a Boolean indicating whether sample siwas correctly classified by the action recognition model 110. Given the inherent limitation of VAE models to sample data closely aligned with the distribution of their training dataset, a strategic sample selection method aimed at enhancing the training dataset’s variance is employed. In the context of minimal real training data, this restricts the diversity7of generated samples. This method is quantitatively defined by two essential metrics for each sample siin filtered samples 5filtered: Within-Class Distance, ^within , the average of Euclidean distances between the feature vector of synthetic samples and the feature vectors of all real samples within the same class: Between-Class Distance, ^between, is the average of the Euclidean distances between the feature vector of the synthetic sample and the feature vectors of all real samples from different classes:

[0064] In Equation 9, the NOsis the number of real data points Xj with the same action class label of the synthetic sample st, while Na^Osis the number of real data points Xj with the different action class label of the synthetic sample s,. / (■) is the encoder of action recognition model 110, which extracts the feature vector of motion sample. To reconcile these objectives, the ratio of weighted within-class to between class distances is evaluated for each sample, prioritizing samples based on this ratio within each class and ultimately retaining the central 50% of samples to balance diversity and distinction. These selected samples SSeiectedare then used as part of training data Xtrain= {SselectedU X] for further action generative model 105 training and contribute to building our initial infant synthetic motion dataset.

[0065] FIG. 2 illustrates a flow chart of the InfAGenC pipeline process 200. The InfAGenC pipeline process 200 begins at operation 205, by receiving real-world infant motion video data (i.e., video recordings of human infants performing various movements). The real -world infant motion video data is training data 120 for the first iteration of the InfAGenC pipeline process 200. The real-world infant motion video data, or real data, is provided for model training 210. At operation 215 as part of the model training 210, the action generation model 105 is trained with the training data 120, or real-world infant motion video data for the first iteration. Similarly, at operation 220 as part of the model training 210, the action recognition model 110 is trained with the training data 120, or real-world infant motion video data for the first iteration.

[0066] At operation 225, the action generation model 105 generates synthetic data comprising video samples of infants in motion and corresponding infant action classification labels for the infant action of a particular video sample. At operation 230, the action recognition model 110 evaluates the synthetic data and assigns to each video sample an accuracy score (e.g., CAS) representing recognition confidence for the corresponding infant action classification label. Additionally, at operation 230. the action recognition model 110 assigns to each video sample a confidence score indicating infant action classification certainty. The InfAGenC pipeline process 200 then filters the synthetic data using the synthetic data recycling 235. At operation 240, the synthetic data is filtered based on the CAS, or accuracy score. As described above, the synthetic data may be filtered by selecting the video samples with the highest CAS values. For example, the video samples may be selected based on CAS values that are above an accuracy score threshold or the video samples may be selected based on certain percentage of the ranked video samples (e.g., the video samples in the top 25% of the synthetic data). At operation 245, the synthetic data may be filtered by selecting the video samples with the highest confidence scores. Similarly, the selection may be based on those confidence scores that are above a confidence score threshold value or a percentage of the video samples ranked by confidence score.

[0067] At operation 250 of the synthetic data recycling 235, the synthetic data may be filtered based on diversity so that the video samples that are selected represent a diverse distribution across infant action classifications. As described in above, the diversity determination is based on weighted within-class and between-class distance metrics.

[0068] At operation 255, the resulting filtered synthetic data is added to the training data 120, such that the training data 120 now comprises both real and synthetic data for additional iterations of the InfAGenC pipeline process 200. At decision 260, an operation is performed to determine if the total sample accuracy meets or exceeds a model accuracy threshold value. For example, this may be based on determining if all, or a certain percentage, of the pre-filtered synthetic data video samples have an accuracy score that is above the model accuracy threshold value. At decision 260, if the total sample accuracy does not meet the model accuracy threshold value, then the InfAGenC pipeline process 200 returns to the model training 210 for a new iteration with training data 120 that comprises both real and synthetic data. The InfAGenC pipeline process 200 then continues from operations 215 and 220 as previously described, where the action generative model 105 and action recognition model 110 are fine tuned by being further trained using the training data 120 that now includes synthetic data.

[0069] At decision 260, if the total sample accuracy meets or exceeds the model accuracy threshold value, then at operation 270, the action recognition model 110 is refined and may be considered a highly accurate action recognition model 110 that may be used for real world applications to identify infant actions. Additionally, at operation 265, the training data 120 comprises a set of real and synthetic video samples (and corresponding infant action labels) that may be used for training other infant action recognition models.

[0070] FIG. 3 illustrates a timeline for monitoring a baby's motor actions while in their natural environments which may provide for early detection of problems and administering timely interventions. The disruption of motor development in infancy is a risk indicator for a host of developmental delays and disabilities, like Cerebral Palsy and Autism Spectrum Disorders. Tracking a baby's motor activities within their familiar environments, such as their crib or playroom, provides for the potential early detection of problems and making timely interventions.

[0071] The InfAGenC pipeline was evaluated using two Infant datasets including InfactPrimitive

[0012] and this collect infant action dataset. A small portion of NTU

[0021] has been used to illustrate the InfAGenC pipeline’s performance with limited data. With 3D ground truth provided, sourced from a standard skeleton representation, potential noise was effectively controlled in estimation, underscoring the robustness of this approach.

[0072] The following describes the implementation of the InfAGenC pipeline and the experiments performed for the InfActPrimitive dataset. The InfantAction dataset, and the NTU dataset.

[0073] The InfAGenC pipeline framework was deployed on a transformer-based VAE model integrated with a ST-GCN. This model leverages the 6D rotations of SMIL model

[0014] 24 joints’ as the joints representation, offering a detailed and comprehensive depiction of the dynamic interactions between joints. Each video clip was processed to consist of 60 frames. During the training phase, the Adam optimizer was utilized with a learning rate set to 0.0001 and a batch size established at 16 for epochs. Initially, to ensure accurate performance evaluation, our action recognition component underwent training for 15 epochs using the InfActPrimitive dataset.

[0074] To ensure effective evaluation of generated samples, an action recognition model was pre-trained. However, it's essential to strike a balance in training this action recognition model. Over-training can reduce synthetic data diversity due to overfitting, while undertraining may lead to inaccurate action classification. To address this, the pre-trained action recognition model’s training was halted once it achieves 85% accuracy, ensuring both model performance and synthetic data quality.

[0075] For the action generation component, the loss term weights — AKL,rec. andvei— were adjusted to 1.0. 1.0, and 0.001. respectively. This component was pre-trained for 1100 epochs on the training set of InfActPrimitive, aiming to enrich the generated samples with temporal details beyond mere static poses. Subsequently, the synthetic data recycling phase was initiated for an additional 200 epochs. In this strategy' for filtering and selecting generated samples, the confidence threshold (0) was set at 0.75 and the weights for within-class distance (wwithin) and between-class distance wbetweeni) were set at 0.6 and 0.4, respectively.

[0076] Upon completing the training of our infant action generative model, 1275 synthetic samples were successfully generated, which were then incorporated into the training set for the action recognition models. This synthetic data was further incorporated into the training set for training the action recognition models up to 100 epochs but ceasing upon model convergence, specifically targeting the infant action recognition task.

[0077] For the experiments conducted on the InfantAction dataset, a similar configuration to that of the InfActPrimitive dataset was followed, with the main difference being the adjustment of the number of frames per video clip to 90 instead of 60. This adjustment was made to accommodate the longer duration required to capture complex actions accurately. The remaining settings, including the model architecture, optimizer, loss term weights, training duration, and synthetic data recycling strategy, remained consistent. Following the training process, 816 synthetic samples were successfully generated, which were seamlessly integrated into the training set for action recognition models.

[0078] For the NTU data, a different data representation was adopted due to the availability of relatively accurate j oint location annotations of NTU data. Instead of relying on 24 joints’ 6D rotations, the 25 joints’ 3D coordinates were directly utilized as the data representation for both the action generative and action recognition models. The duration of videos in this dataset was set to 90 frames as well. Similar to the previous experiments, synthetic samples were generated during the training process. Upon completion, 1288 synthetic samples were successfully generated, which were then incorporated into the training set for further analysis and evaluation of the action recognition models.

[0079] All the experiments utilize a robust compute environment featuring the NVIDIACg) vlOO-pcie graphics processing unit (GPU) from the Volta generation. This GPU comes equipped with 32GB of memory, enabling substantial data processing capabilities.

[0080] NTU Dataset The NTU RGB+D 120

[0021] dataset is a widely-used dataset for the human activity recognition task. Data was sampled from the NTU dataset to create a smaller set for training the InfAGenC pipeline. Four classes were selected: '‘Sitting Down". “Standing Up”, “Jumping”, and “Falling”, which are similar to infant action classes. This training set includes 5 samples per class from 5 subjects, totaling 100 data points. For testing, two sets were created: (1) a small test set with 10 samples per class from another 5 subjects (200 samples in total), comparable to the size of the infant datasets, and (2) a large test set that includes all samples from the remaining subjects for these four classes, totaling 3081 samples, to provide more robust evaluations.

[0081] InfActPrimitive Dataset The InfActPrimitive

[0012] dataset comprises video clips from YouTube and real-life scenarios, involving 127 infant subjects. The YouTube portion initially had 400 clips, but after removing unreliable and outlier samples, 310 clips across five classes remained: 46 Supine, 40 Prone, 99 Sitting, 67 Standing, and 58 All-fours. One-third of these clips (103 clips) were designated as a validation set to maintain class balance and avoid subject overlap during training for action recognition and generation. The ‘'in-the-wild” segment of InfActPrimitive is set as the test set, featuring home-based clips of infants aged 3 to 12 months engaged in various activities. The older infants often perform Sitting, Standing, and All-fours actions, while the younger ones primarily showed Supine and Prone actions. This led to a naturally imbalanced distribution of action classes, reported in Table 1.

[0082] Table 1. The distribution of action classes within the “in-the-wild” segment of InfActPrimitive varies for each infant participant.

[0083] InfantAction Dataset The InfActPrimitive dataset contains infant basic actions with limited body movements. To include more complex actions, a dataset was created through recruitment of five infants. This dataset contains four action classes: “Crawling”, “Sitting”, “Standing”, and “Rolling”. However, similar to InfAct- Primitive

[0012] due to age variations among the infants, the distribution of action classes remained imbalanced, as detailed in Table. 2. Inf04 and Inf05 are included in this test set which cover the whole action classes, and used the remaining 3 subjects in this training resulting a total of 145 and 118 samples in training and test sets. The collection and usage of infant data under Institutional Review Board (IRB #22- 11 -32) strictly adhere to the highest ethical standards. All data handling and sharing are in full compliance with applicable data protection regulations, ensuring that the data remains secure and accessible only to authorized personnel for research purposes.

[0084] Table 2. The distribution of action classes of action classes of the created InfantAction dataset.

[0085] For the creation of the InfantAction dataset, infant subjects were recruited and clips collected from home-based monitoring sessions, capturing moments when the infants were either playing or sleeping. This process was conducted with IRB approval and parental permission. These infant participants ranged in age from 3 to 12 months, which introduced significant variation in their motion capabilities. To create the final InfantAction dataset, the following steps were performed. Video Clips Cropping: lengthy video recordings were reviewed to extract short clips (each approximately 4-5 seconds) that showcased predefined actions such as “Sitting”, “Standing”, “Crawling”, and “Rolling”. Video Selection: clips were selected that were of high quality and displayed clear action movements, ensuring a variety of movements for each subject. Each clip was manually assigned an action class label. Object Detection: using Y0L0v8

[0034] , bounding boxes were automatically detected for each subject. Object tracking algorithms were employed to maintain consistency in the bounding boxes, manually correcting any inaccuracies. 3D Pose Estimation: As these videos were solely RGB with no motion capture data, the HW-HuP

[0022] infant 3D pose estimation model was applied to determine joint locations in each frame. Error Filtering: after pose estimation, the predicted 3D poses were visualized and any clips with incorrect estimations were removed.

[0086] Following these processing steps, a dataset of 273 video clips was complied. The class distribution of these clips is detailed in Table 2.

[0087] Table 3 reports the performance comparison of the InfAGenC pipeline and the VAE baseline among all three datasets in terms of evaluating Fr' echet Inception Distance (FID), action recognition accuracy, overall diversity, as well as diversity and multimodality on a peraction basis. Existing generative models, like MDM and Action2Motion, are primarily designed for adult datasets with a large number of samples. Also adapted and fine-tuned are MDM model on the InfActPrimitive training set to benchmark its performance against this model in scenarios with limited data availability in Table 3. The results show substantial improvements in both accuracy and diversity when the InfAGenC model is trained on any of these three datasets. This model also outperforms the MDM model, confirming the effectiveness of this approach in handling limited and diverse data contexts. Even though it demonstrates higher FID on both InfActPrimitive and NTU datasets compared to the counterparts, this underscores the capability of the generated samples to span a broader distribution, affirming their sufficient variance. Given the constraints of dealing with sparse data and lacking 3D ground truth, the goal is to prevent overfitting to outliers and noise in the dataset while ensuring the diversity of samples remains accurate. This trade-off in reducing fidelity compared to other baseline models may be explained by the inclusion sampling and selection approach considering both between and within class distance. Additionally, the models were qualitatively evaluated by visualizing the generated samples. Fig. 4 displays snapshots of the skeletal representations of the generated samples based on InfactPrimitive. These visuals demonstrate the model’s capability to generate a diverse array of motion samples for each targeted class, significantly enriching the dataset and supporting advancements in research on infant motion. More visualizations of generated motions based on InfantAction and NTU data are presented FIGS. 5 & 6.

[0088] Table 3. Comparative Evaluation of the InfAGenC Model with the Baseline Conditional VAE Model. This table presents the performance metrics of the InfAGenC model versus the baseline model across the NTU. InfActPrimitive, and InfantAction datasets, and also makes the contrast with MDM model on InfActPrimitvie dataset. The best results are bold.

[0089] Employed are two distinct action recognition models. The first action recognition model is the ST-GCN model, identical to the one utilized in this methodology', which uses 6D rotations for joint representations. The second action recognition model, MS-G3D

[0023] . adopts xyz coordinates for joints representation and features a disentangled aggregation approach coupled with a unified spatial-temporal graph convolution (G3D) operator to enhance action feature learning. Each action recognition model trained under two data configurations: (a) Real, using only the real training set and (2) Real+Syn, including both real training set and synthetic data samples generated by the InfAGenC model. The overall performance outcomes of these configurations are compared in Table 4. Either the ST-GCN model or the MS-G3D model, the accuracy improves significantly when synthetic data is included, with the notable increase seen in both InfActPrimitive dataset and NTU dataset, indicating that the synthetic data aids in generalizing the models’ predictive capabilities. However, for the InfantAction dataset, the performance of both models trained with only the real dataset outperforms those trained with the hybrid dataset. This inconsistency may be attributed to two main factors: (1) Compared to other datasets, the InfantAction dataset is relatively small, with only 145 samples, which may not be sufficient to generate high-quality synthetic data. (2) The imbalance in class distribution could introduce a bias in evaluations, favoring classes with more samples. The addition of balanced synthetic data diminishes the model’s preference for specific classes, thereby enhancing performance across all classes, not just those that are overrepresented. To further support our hypotheses, detailed subject-wise and action- wise evaluations will be presented in the upcoming ablation studies. Also visualized are each experimental dataset’s feature distnbution by using T-SNE algorithm in FIGS. 7A-C. Most classes are clustered very well and those generated samples are largely scattered around the real training data. This expansion of each class’s distribution effectively broadens the variety of actions covered within each class, thereby enhancing the performance of the action recognition model.

[0090] Table 4. Performance of Infant Action Recognition Models Using Different Data Configurations. Respectively trained ST-GCN model and MS-G3D model on InfActPrimitive, InfantAction, and NTU dataset under two conditions: (1) Real - utilizing the training set of each dataset and (2) Real+Syn - a hybrid approach combining both real and synthetic data samples. The highest accuracy for each model and dataset are distinguished in bold.

[0091] To ensure an unbiased evaluation of these methods and provide an ablation analysis, experiments were conducted on the subset of NTU data, which features a balanced class distribution, 3D ground truth, and sufficient test data, allowing assessment of these methods effectively. Synthetic Data Recycling In Table 3, the performance of the action generative model was evaluated in comparison with a baseline VAE model, specifically examining the quality of synthetic samples generated by both. The effectiveness of the synthetic data recycling approach was assessed by comparing the performance of ST-GCN action recognition models trained on equivalent amounts of synthetic samples produced by (1) the baseline VAE model and (2) the InfAGenC model. These models were tested on the NTU dataset's small test set of 200 samples and a large test set of 3081 samples, with results detailed in Table 5. The action recognition model trained with fusion data, combining synthetic samples from the InfAGenC model and real data, shows a performance increase of 4% on the small test set and 1.35% on the large test set compared to models trained only with real data. In contrast, synthetic data from the baseline model, which lacks the recycling component, only improves performance by 2% on the small test set and actually decreases performance by 3.65% on the large test set. These results highlight the significant benefits of integrating the synthetic data recycling module into the training process.

[0092] Table 5. Ablation Analysis of ST-GCN Action Recognition Models on NTU Dataset Using Various Synthetic Training Sets. Evaluated are the quality of synthetic data produced by three different configurations of action generative model training: (1) a baseline conditional transform-based VAE, (2) the InfAGenC network with random sample selection, and (3) the InfAGenC network employing a distance-based sample selection strategy. The action recognition models were trained using different combinations of datasets: (a) Real, which solely utilizes the prepared NTU real training set; (b) Syn, which uses synthetic infant data generated from the corresponding generative model configuration; and (c) Real+Syn, a hybrid approach that combines both real and synthetic data. The highest accuracy for each class is underlined, and the highest overall accuracy is highlighted in bold.

[0093] Distance-based Sample Selection A comparative experiment was conducted to assess the efficacy of the distance-based sample selection strategy for synthetic data. In contrast to selectively using high-quality, diverse synthetic samples during InfAGenC model training, experiments were conducted with random synthetic sample selection to augment training data. The performance of ST-GCN action recognition models trained with two configurations was evaluated: (1) using the real NTU training set combined with synthetic samples generated by our InfAGenC model employing a random selection strategy, and (2) using the real NTU training set alongside synthetic samples generated by our InfAGenC model wi th a distancebased selection strategy. The findings reveal that the ST-GCN models trained with synthetic data produced via the distance-based selection consistently achieved the highest accuracy, not just on the NTU’s small test set but also on the large test set, reaching accuracies of 98% and 97.08%, respectively. This underscores the advantage of the distance-based selection approach in enhancing the effectiveness of training data.

[0094] Subject-wise and Action- wise Evaluation Due to the rapid motor development in infants, the range of action shifts over time. Additionally, there is a huge within-class discrepancy in infant actions, as illustrated by

[0012] , even for the same class of action. For better interpretability, the performance outcomes of infant action recognition models were compared for each infant subject and action class across both the InfActPrimitive and InfantAction datasets, as presented in Table 6 and Table 7. respectively. As shown in Table 6, across both ST-GCN and MS-G3D, the highest accuracy for most subjects and action classes is achieved with the hybrid training set. Specifically, older subjects D02 and D03, who perform more advanced actions but exhibit fewer elementary motor skills (their data class distribution is reported in Table 1), show significant performance improvements when trained on hybrid data compared to just real data. Similarly, in Table 7, although the overall accuracy of models trained solely with real data appears higher, the accuracy distribution across different action classes is more uneven when compared to models trained on hybrid data. This evidence justifies the value of integrating balanced synthetic data to enhance model performance across diverse action classes.

[0095] Table 6. Detailed Performance of Infant Action Recognition Models on the InfActPrimitive Dataset. This table displays the accuracy for each subject and action class, comparing models trained across different data configurations: (1) Real - utilizing the InfActPrimitive dataset and (2) Real+Syn - a hybrid approach that merges both real and synthetic data generated by the InfAGenC model. In the analysis, entries showcasing the highest accuracy for each class and subject are distinguished in bold.

[0096] SWCN MSABD A«r'

[0097] Table 7. Detailed Performance of Infant Action Recognition Models on the InfantAction Dataset. This table displays the accuracy for each subject and action class, comparing models trained across different data configurations: (1) Real - utilizing the InfantAction dataset and (2) Real+Syn - a hybrid approach that merges both real and synthetic data generated by the InfAGenC model. The highest accuracy for each class and subject are distinguished in bold.

[0098] The methods and techniques described herein developed an infant action generation pipeline (e.g., the InfAGenC pipeline) and introduced a unique dataset with a more complex set of actions. Due to the uncooperativeness of infants, data gathered in home settings without assumptions about camera angles and views vary from infant to infant, resulting in limited samples but substantially different positions. Additionally, due to age variations, even the same action, such as standing, differs among infants (some use support, while others stand by themselves). Another limitation is that acquiring 3D pose ground truth using motion capture is not possible for infants, as they are often surrounded and occluded by toys and other objects, posing significant challenges for 3D pose estimation and tracking in videos. However, the InfAGenC pipeline’s results on the NTU dataset, with less noise and reliable ground truth, demonstrate its capability to handle small data effectively.

[0099] As used herein, "consisting essentially of' allows the inclusion of materials or steps that do not materially affect the basic and novel characteristics of the claim. Any recitation herein of the term "comprising", particularly in a description of components of a composition or in a description of elements of a device, can be exchanged with "consisting essentially of' or "consisting of'.

[0100] While the present invention has been described in conjunction with certain preferred embodiments, one of ordinary skill, after reading the foregoing specification, will be able to effect various changes, substitutions of equivalents, and other alterations to the compositions and methods set forth herein.

[0101] REFERENCES

[0102] [1] Ghada Alsebayel, Mahsa Nasri, Caleb P Myers, Giovanni M Troiano, Elaheh Hatamimajoumerd, Sarah Ostadabbas, Kristen Allison, and Casper Harteveld. Articumotion: Towards assessing motor speech disorders via gamification. In Proceedings of the 23rd Annual ACM Interaction Design and Children Conference, pages 232-247, 2024.

[0103] [2] Somaieh Amraee, Bishoy Galoaa, Matthew Goodwin. Elaheh Hatamimajoumerd. and Sarah Ostadabbas. Multiple toddler tracking in indoor videos. In Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pages 11-20, 2024.

[0104] [3] Xu Cao, Xiaoye Li, Liya Ma, Yi Huang, Xuan Feng, Zening Chen, Hongwu Zeng, and Jianguo Cao. Aggpose: Deep aggregation vision transformer for infant pose estimation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22) Special Track on Al for Good, 2022. 3

[0105] [4] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv: 1907.06987, 2019. 3

[0106] [5] Hyung-gun Chi, Myoung Hoon Ha, Seunggeun Chi. Sang Wan Lee, Qixing Huang, and Karthik Ramani. Infogen: Representation learning for human skeleton-based action recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 20186-20196, 2022. 2, 3

[0107] [6] Qiongjie Cui. Huaijiang Sun. and Fei Yang. Learning dynamic relationships for 3d human motion prediction. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 6519-6527, 2020. 3

[0108] [7] Bruno Degardin, Joao Neves, Vasco Lopes, Joao Brito, Ehsan Yaghoubi, and Hugo Proenc.a. Generative adversarial graph convolutional networks for human action synthesis. In Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pages 1150-1159, 2022. 3

[0109] [8] Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 2969-2978, 2022. 2

[0110] [9] Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Posegpt: Chatting about 3d human pose. arXiv preprint arXiv: 2311.18836, 2023. 3

[0111]

[0010] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021-2029, 2020. 1, 3

[0112]

[0011] F' ehx G Han ey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions on Graphics (TOG), 39(4):60-l, 2020. 1

[0113]

[0012] Elaheh Hatamimaj oumerd, Pooria Daneshvar Kakhaki, Xiaofei Huang, Lingfei Luan, Somaieh Amraee, and Sarah Ostadabbas. Challenges in video-based infant action recognition: A critical examination of the state of the art. Pages 21-30, 2024. 1, 3, 5. 6, 8

[0114]

[0013] Nikolas Hesse, Christoph Bodensteiner, Michael Arens, Ulrich G Hofmann, Raphael Weinberger, and A Sebastian Schroeder. Computer vision for medical infant motion analysis: State of the art and rgb-d data set. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0. 2018. 3

[0115]

[0014] Nikolas Hesse, Sergi Pujades, Javier Romero, Michael J Black, Christoph Bodensteiner, Michael Arens, Ulrich G Hofmann, Uta Tacke, Mijna Hadders-Algra, Raphael Weinberger, et al. Learning an infant body model from rgbd data for accurate full body motion analysis. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain. September 16-20, 2 18, Proceedings, Part 1. pages 792-800. Springer, 2018. 11

[0116]

[0015] Xiaofei Huang, Nihang Fu, Shuangjun Liu, and Sarah Ostadabbas. Invariant representation learning for infant pose estimation with small data. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1-8. IEEE, 2021. 1, 3

[0117]

[0016] Xiaofei Huang, Lingfei Luan, Elaheh Hatamimaj oumerd, Michael Wan, Pooria Daneshvar Kakhaki, Rita Obeid, and Sarah Ostadabbas. Posture-based infant action recognition in the wild with very limited data. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 4911-4920, 2023. 1

[0118]

[0017] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence. 36(7): 1325-1339, jul 2014.

[0119]

[0018] Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges. Convolutional autoencoders for human motion infilling. In 2020 International Conference on 3D Vision (3DV), pages 918-927. IEEE, 2020. 3

[0120]

[0019] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back. Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1705.06950, 2017. 1

[0020] Taeryung Lee, Gyeongsik Moon, and Kyoung Mu Lee. Multiact: Long-term 3d human motion generation from multiple action labels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1231-1239, 2023. 3

[0121]

[0021] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A largescale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, 42(10):2684-2701, 2019. 1, 3, 5, 6

[0122]

[0022] Shuangjun Liu, Michael Wan, and Sarah Ostadabbas. Heuristic weakly supervised 3d human pose estimation. arXiv preprint arXiv:2105. 10996, 2021. 11

[0123]

[0023] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 143-152, 2020. 2, 6

[0124]

[0024] Thomas Lucas, Fabien Baradei, Philippe Weinzaepfel, and Gregory Rogez. Posegpt: Quantization-based 3d human motion generation and forecasting. In European Conference on Computer Vision, pages 417-435. Springer, 2022. 3

[0125]

[0025] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2891-2900, 2017. 3

[0126]

[0026] Lucia Migliorelli, Sara Moccia, Rocco Pietrini, Virgilio Paolo Camielli, and Emanuele Frontoni. The baby-pose dataset. Data in brief, 33: 106329, 2020. 3

[0127]

[0027] Mathis Petrovich, Michael J Black, and G" ul Varol. Action-conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 10985-10995, 2021. 1

[0128]

[0028] Trevor Powers, Elaheh Hatamimajoumerd, William Chu, Vishakk Rajendran, Rishi Shah, Frank Diabour, Marc Vaillant, Richard Fletcher, and Sarah Ostadabbas. Vision-based treatment localization with limited data: Automated documentation of military emergency medical procedures. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 1819-1828, 2023. 2

[0129]

[0029] Abhinanda R Punnakkal, Arjun Chandrasekaran. Nikos Athanasiou, Alejandra Quiros- Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english labels.

[0130] In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 722-731, 2021. 3

[0131]

[0030] Arun V Reddy, Ketul Shah. William Paul, Rohita Mocharla, Judy Hoffman, Kapil D Katyal, Dinesh Manocha, Celso M de Melo, and Rama Chellappa. Synthetic-to-real domain adaptation for action recognition: A dataset and baseline performances. arXiv preprint arXiv:2303.10280. 2023. 3

[0132]

[0031] Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: a local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pages 32-36. IEEE, 2004. 1

[0133]

[0032] Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio-temporal contrastive domain adaptation for action recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 9787-9795, 2021. 3

[0134]

[0033] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UcflOl: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402, 2012. 1

[0135]

[0034] Fatma M Talaat and Hanaa ZainEldin. An improved fire detection approach based on yolo-v8 for smart cities. Neural Computing and Applications, 35(28):20939-20954, 2023.

[0136]

[0035] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.

[0137]

[0036] Jiang Wang. Xiaohan Nie, Yin Xia. Ying Wu, and Song-Chun Zhu. Cross-view action modeling, learning and recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2649-2656, 2014.

[0138]

[0037] Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan. Yichao Yan, Xin Jin, Xiaokang Yang, et al. Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 2228-2238, 2023. 2

[0139]

[0038] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 1, 2

[0140]

[0039] Xueying Zhan, Huan Liu, Qing Li, and Antoni B Chan. A comparative survey: Benchmarking for pool-based active learning. In IJCAI, pages 4679-4686, 2021. 1

[0141]

[0040] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 1, 6

[0142]

[0041] Shaotong Zhu, Amal Mathew, Elaheh Hatamimajoumerd, Michael Wan, Briana Taylor, Rajagopal Venkatesaramani, and Sarah Ostadabbas. Cribnet: Enhancing infant safety in cribs through vision-based hazard detection. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pages 01-08. IEEE, 2024. 3

[0042] Shaotong Zhu, Michael Wan, Sai Kumar Reddy Manne, Elaheh Hatamimajoumerd, Marie J Hayes. Emily Zimmerman, and Sarah Ostadabbas. Subtle signals: Video-based detection of infant non-nutritive sucking as a neurodevelopmental cue. Computer Vision and Image Understanding, 247: 104081, 2024. 3

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for generating infant action data and training an infant action recognition model, comprising: receiving real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; generating synthetic infant action data using an infant action generation model trained on the real training data, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification; evaluating the synthetic infant action data by processing the set of video samples with an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification; determining a filtered set of video samples by selecting a first subset of video samples from the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; combining the filtered set of video samples with the real training data to form modified training data; and training the infant action recognition model using the modified training data.

2. The computer-implemented method of claim 1, wherein the infant action generation model is a transformer-based variational autoencoder (VAE) trained on pose-based representations extracted from the real training data.

3. The computer-implemented method of claim 1, wherein the infant action recognition model is a spatial-temporal graph convolutional network (ST-GCN), and the method further includes training the ST-GCN using the modified training data.

4. The computer-implemented method of claim 1, wherein each video sample from the synthetic infant action data comprises a sequence of infant poses sampled from continuous temporal trajectories.

5. The computer-implemented method of claim 1, wherein evaluating the synthetic infant action data using the infant action recognition model includes assigning a confidence score to each video sample, the confidence score indicating infant action classification certainty provided by the infant action recognition model, and wherein determining the filtered set of video samples, further comprises: selecting a second subset of video samples, from the first subset of video samples, wherein each video sample in the second subset has a corresponding confidence score that exceeds a predetermined confidence threshold; and storing the second subset of video samples as the filtered set of video samples.

6. The computer-implemented method of claim 1, wherein determining the filtered set of video samples further comprises: computing, for each video sample in the set of video samples, a diversity7score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; and determining, the filtered set of video samples from the first subset of video samples by selecting a predetermined percentage of the video samples based on ranked diversity scores, wherein the video samples that are selected represent a diverse distribution across infant action classifications.

7. The computer-implemented method of claim 6, wherein the diversity score is defined as a ratio of within-class distances to between-class distances; wherein a within-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data that share the same infant action classification; and wherein a between-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data with different infant action classifications.

8. The computer-implemented method of claim 1, wherein the infant action classifications include sitting, standing, rolling, and crawling.

9. The computer-implemented method of claim 1, further comprising, after training the infant action recognition model using the modified training data:training the infant action generation model using the modified training data; generating second synthetic infant action data using the infant action generation model, wherein the second synthetic infant action data comprises a second set of video samples depicting at least one infant in motion and the associated infant action classification; evaluating the synthetic infant action data by processing the second set of video samples with the infant action recognition model to obtain the accuracy score of the associated infant action classification of each sample in the second set of video samples; determining a second filtered set of video samples by selecting a second subset of video samples from the second synthetic infant action data based on the accuracy score exceeding the predetermined accuracy threshold; adding the second filtered set of video samples to the modified training data; determining each accuracy score of the second set of video samples exceeds a predetermined model accuracy threshold; and in response to determining each accuracy score of the second set of video samples exceeds the predetermined model accuracy threshold, ending training of the infant action recognition model.

10. The computer-implemented method of claim 9, wherein the predetermined model accuracy threshold is 85%.

11. The computer-implemented method of claim 9, further comprising: receiving real sample data representing unclassified real-world video of a real infant in motion; determining at least one infant action classification for the real sample data using the infant action recognition model; and outputting the at least one infant action classification for the real sample data.

12. A computer-implemented method for generating an infant action training dataset, comprising: receiving real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; generating synthetic infant action data using an infant action generation model trained on the real training data, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification;evaluating the synthetic infant action data by processing the set of video samples with an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification and wherein the infant action recognition model assigns a confidence score, the confidence score indicating classification infant action classification certainty provided by the infant action recognition model, to each video sample in the set of video samples; determining a first subset of video samples from the set of video samples in the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; determining a second subset of video samples, from the first subset of video samples, based on video samples from the first subset of video samples with corresponding confidence scores exceeding a predetermined confidence threshold; determining, for each video sample from the second subset of video samples, a diversity score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; determining, a filtered set of video samples from the second subset of video samples by selecting a predetermined percentage of the video samples in the second subset of video samples based on ranked diversity scores, wherein the video samples that are selected represent a diverse distribution across infant action classification; and generating the infant action training dataset by combining the filtered set of video samples and the real training data.

13. The computer-implemented method of claim 12. wherein infant motion classifications include sitting, standing, rolling, and crawling.

14. A system for an infant action sample generation model and recognition model pipeline, comprising: at least one processor; at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive real training data comprising at least one video of a real-world infant in motion, annotated with an infant action label; train an infant action generation model using the real training data;generate synthetic infant action data using the trained infant action generation model, wherein the synthetic infant action data comprises a set of video samples depicting at least one infant in motion and an associated infant action classification; evaluate the synthetic infant action data by processing the set of video samples using an infant action recognition model to obtain an accuracy score representing recognition confidence for the associated infant action classification; determine a filtered set of video samples by selecting a first subset of video samples from the synthetic infant action data based on the accuracy score exceeding a predetermined accuracy threshold; combine the filtered set of video samples with the real training data to form modified training data; train the infant action recognition model using the modified training data; and train the infant action generation model using the modified training data.

15. The system of claim 14, wherein each video sample from the synthetic infant action data comprises a sequence of infant poses sampled from continuous temporal trajectories.

16. The system of claim 14, wherein the infant action classifications include sitting, standing, rolling, and crawling.

17. The system of claim 14, wherein evaluating the synthetic infant action data using the infant action recognition model includes assigning a confidence score to each video sample, the confidence score indicating infant action classification certainty provided by the infant action recognition model, and wherein determining the filtered set of video samples, further includes instructions to: select a second subset of video samples, from the first subset of video samples, wherein each video sample in the second subset has a corresponding confidence score that exceeds a predetermined confidence threshold; and store the second subset of video samples as the filtered set of video samples.

18. The system of claim 14, wherein determining the filtered set of video samples further includes instructions to:compute, for each video sample in the set of video samples, a diversity score, the diversity score representing a statistical measure of sample variability relative to a training data distribution; and determine, the filtered set of video samples from the first subset of video samples by selecting a predetermined percentage of the video samples based on ranked diversity7scores, the video samples that are selected represent a diverse distribution across infant action classifications.

19. The system of claim 18, wherein the diversity score is defined as a ratio of w ithin- class distances to between-class distances; wherein a within-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data that share the same infant action classification; and wherein a between-class distance is computed as an average of Euclidean distances between a feature vector of the video sample from the synthetic infant action data and feature vectors of the real training data with different infant action classifications.

20. The system of claim 14, wherein the infant action generation model is a transformerbased variational autoencoder (VAE) trained on pose-based representations extracted from the real training data.

Images