Machine learning methods and systems for processing dyadic parameter data

Abstract

Methods and systems for processing dyadic parameter data to analyze development of a child subject and / or maternal mood are disclosed A machine learning method of assessing and promoting the development of a child subject comprises: receiving input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for the child subject and a time series of sensed data indicative of the parameter for an adult collected during interactions between the child subject and the adult; synchronizing the dyadic parameter data; generating a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult, wherein the set of features comprises a combination of non-linear features and linear features; classifying the development of the child subject by inputting the set of features into a machine learning classifier trained to classify development of a child based on the set of features; and outputting a prediction indicating the classification of the development of the child subject.

Description

[0001] MACHINE LEARNING METHODS AND SYSTEMS FOR PROCESSING DYADIC PARAMETER DATA

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to machine learning systems and methods for assessment of child development and maternal mood. In particular, the present invention relates to machine learning systems and methods which process dyadic parameter data relating to interactions between a child subject and an adult subject.

[0004] BACKGROUND

[0005] Parent-child interactions are crucial not only for an infant’s mental wellbeing, but also for the development of Executive Function skills that form part of healthy brain development. These skills allow a child to hold and manipulate information in their memory, control their impulses, plan ahead, and manage multiple tasks at once - all core skills that are needed to navigate life as an adult. However, while we routinely measure a child’s height and weight to monitor developmental trajectories, there are currently no early, scalable measures of a child’s brain and socioemotional health. Being able to routinely check an infant’s social and brain development would provide crucial information to clinicians making decisions about whether to refer a child for further investigations, and to parents seeking reassurance and guidance on their parenting skills and approaches.

[0006] Executive functions (EFs) are core mental skills that allow children to sustain attention and resist distraction (inhibitory control), hold and manipulate information in mind (memory updating) and shift attention and strategies to adapt to changing demands (cognitive flexibility). These skills are highly predictive of future life success, whether measured classically in terms of academic achievement in language and mathematical skills, or more broadly to include socioemotional skills such as theory of mind and creative problem solving. Well-developed EF underpins greater productivity and wellbeing. Indeed, children with underdeveloped EF at age 3 represent about 20 percent of the population but make up nearly 80 percent of adults who are likely to require some form of societal or economic assistance. EF skills can be differentiated from intelligence. For example working memory is strongly correlated with intelligence, whereas inhibition and shifting are only weakly or not correlated. Individual differences in EF components are relatively stable over childhood and adolescence, and population variance in EF is highly heritable suggesting a strong (poly)genetic influence. However, some EF skills, particularly shifting / flexibility, show significant environmental variance - suggesting that the early social environment plays a particularly important role in generating individual differences in this EF skill. This also suggests that cognitive flexibility may be a particularly malleable target for intervention. Accordingly, ensuring that parents and clinicians have the tools to measure the quality of a child’s social relationships is of fundamental importance for childhood development and wellbeing.

[0007] The development of executive functions begins during infancy, forming the foundation for later developing higher-order cognitive processes such as reasoning, problemsolving, and planning. During the early years, cognitive precursors of EF (such as the ability to overcome perseveration and reversal learning), can already be measured even in infants as young as 6-12 months of age. Therefore, the collection of EF data during infancy could inform the construction of computational models that forecast the trajectory of a child’s developing EF abilities, allowing stratification of infants by EF risk.

[0008] Cognitive development occurs within the context of positive social interactions and relationships. Specifically, variations in the quality of parent-child interaction (“caregiver scaffolding” and sensitivity / responsiveness) are known to impact EF development. These early social influences trigger functional alterations in infants’ neural circuits that are foundational for the development of individual cognitive and emotional traits, including individual differences in EF development. Although multiple forms of social influence impact upon early neurodevelopment, one important and well-established factor is the mother-infant affective relationship. Parent and infant function as a mutually coordinated unit whose behavior, emotion and physiology are intimately connected and synchronized in a "dyadic dance”. Reciprocal, warm and contingent social interactions promote the development of executive function skills and socioemotional development in children, whereas unresponsive, intrusive or unpredictable interactions (which may occur in parental mental health disorders) are associated with increased risk for disrupted cognitive and psychosocial development and mental health issues in later life. The health of this shared parent-infant socioemotional ecosystem is also evidenced in (and quantifiable as) high levels of biobehavioral synchrony and co-ordination within parent-infant dyads, with strong temporal contingencies between parent and child long noted for gaze, vocalizations, affect, autonomic arousal, hormone levels, and neural activity.

[0009] Social co-ordination between parents and their offspring engenders early (social) learning and scaffolds development across social and cognitive domains. For example, during episodes of parent-infant joint play and social interaction, maternal responsivity has been shown to have a direct positive impact on her infant’s ability to sustain attention, thereby potentiating longer and more productive opportunities for learning. Specifically, maternal neural responsivity (bursts in maternal Theta 4-6 Hz EEG power following onsets of infant-led looks to the object) scaffolded and prolonged infants’ subsequent period of visual sustained attention on play objects. That is, greater maternal neural responsivity during dynamic moment-to-moment social interactions improved her child’s sustained attention. Importantly, maternal “neural scaffolding” only occurred during social interactive (joint) play and did not occur when the infant was playing alone - underscoring the importance of caregiver social interactions in providing real-time scaffolding for infants’ nascent EF abilities.

[0010] Social interaction involves multiple sensory modalities, types of behaviors, and patterns of coordination across different timescales. Accordingly, social interactive behavior can be analyzed at multiple levels through diverse lenses, from microanalytic approaches that track detailed social behaviors at a timescale of milliseconds to ethnographic approaches that seek to understand broad sociocultural influences on children’s social lives (e.g. Kidd’s 1906 study on Kafir children). Unsurprisingly therefore, there are a large number of measures of parent-infant interaction currently in use by the research community. These include the Ainsworth Maternal Sensitivity Scales (AMSS) which is often viewed as a gold standard for measurement of maternal sensitivity. The AMSS involves observation of mother-infant interactions across a variety of settings (including play, feeding and teaching) and can be used with children aged 3 to 24 months, taking between 25 min to 2 h to complete, generating a score between 1 to 9. The Maternal Behavior Q-Sort (MBQS) and CARE-lndex are other popular options. In fact, a recent review identified 906 observational tools for measuring parent-infant interaction across 2,554 articles. Out of these 906 candidates, only 24 tools met criteria of having been used by more than one study and being usable before 12 months of infant age. Studies using these 24 selected tools were primarily conducted in North America and Europe (75% and 19% respectively), with only 6% of papers from Asia, Australia and South America. Notably, only 2 studies included populations from more than one country, suggesting that we know little to nothing about the cross-cultural validity of most of these measures - therefore local data is urgently needed to validate these tools. This concern is compounded by the highly subjective nature of video analysis on which most of these tools are based, which requires hours of hand coding by trained staff. Therefore, despite decades of active psychological research, there is still no objective, scalable and culture-fair tool to assess early psychosocial health and parent-infant social interaction quality.

[0011] The caregiver-infant relationship is defined by high levels of bio-behavioral synchrony — reflected in the coordination of gaze, vocalizations, emotional expressions, physiological arousal, hormonal activity, and neural responses. Sensitive caregiving, marked by appropriately tuned responses to an infant’s cues, promotes stronger neural synchrony compared to more intrusive parenting styles. This synchrony, underpinned by activity in the prefrontal cortex, enhances the infant’s capacity to track and engage with communicative signals, thereby supporting early cognitive and social development.

[0012] Caregiver depression or anxiety disrupts social ostensive cues across modalities. For instance, depressed mothers show reduced pitch modulation and behavioral contingency in infant-directed speech, as well as atypical facial expressions, gaze, and touch. Such disruptions reduce caregiver-infant synchrony, impairing the child’s ability to predict and entrain to communicative signals. Increased unpredictability in caregiver sensory signals (measured via entropy rate) correlates with poorer cognitive outcomes, including memory and effortful control. Infants exposed to low-quality IDS show reduced responsiveness even to non-depressed speakers over. Thus, assessing caregiver social cues offers insights into both maternal mood and infant developmental trajectories. SUMMARY

[0013] According to a first aspect of the present disclosure a machine learning method of assessing and promoting the development of a child subject is provided. The method comprises: receiving input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for the child subject and a time series of sensed data indicative of the parameter for an adult subject collected during interactions between the child subject and the adult subject; synchronizing the dyadic parameter data; generating a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classifying the development of the child subject by inputting the set of features into a machine learning classifier trained to classify development of a child based on the set of features; and outputting a prediction indicating the classification or description of the development of the child subject. In an embodiment, the dyadic parameter data comprises one or more domains of behavioral parameter data and / or a physiological parameter data.

[0014] The machine learning classifier may be further trained to classify a maternal mood of the adult subject based on the set of features and the method may further comprise: classifying the maternal mood of the adult subject by inputting the set of features into the machine learning classifier, and the prediction indication further indicates the classification of the maternal mood of the adult subject.

[0015] According to a second aspect of the present disclosure a machine learning method of assessing maternal mood of an adult subject is provided. The method comprises: receiving input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for a child subject and a time series of sensed data indicative of the parameter for the adult subject collected during interactions between the child subject and the adult subject; synchronizing the dyadic parameter data; generating a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classifying the maternal mood of the adult subject byinputting the set of features into a machine learning classifier trained to classify the maternal mood of the adult subject based on the set of features; and outputting a prediction indicating the classification or description of the maternal mood of the adult subject.

[0016] In an embodiment, wherein the dyadic parameter data comprises child and adult electroencephalogram data and wherein the combination of non-linear and linear features are determined from non-oscillatory and oscillatory features of the signal respectively.

[0017] In an embodiment, the non-oscillatory and oscillatory metrics comprise topographical graph network, features assessed via weighted Symbolic Mutual Information and weighted Phase Lag Index respectively.

[0018] In an embodiment, the dyadic parameter data comprises electroencephalogram data and video data collected from child and adult, and wherein the video data is used to identify portions of interest in the electroencephalogram data.

[0019] In an embodiment, the dyadic parameter data comprises child and adult electrocardiogram data, and the method further comprises pre-processing the electrocardiogram data to re-sample electrocardiogram data for the child subject and electrocardiogram data for the adult such that re-sampled electrocardiogram data for the child subject and re-sampled electrocardiogram data for the adult subject are equidistantly sampled in time.

[0020] In an embodiment, the dyadic parameter data comprises video data and the method further comprises detecting pose data of the child subject and pose data of the adult and analyzing linear and non-linear contingencies between the pose data of the child subject and the pose data of the adult subject.

[0021] In an embodiment, analyzing non-linear contingencies between the pose data of the child subject and the pose data of the adult subject comprises applying multidimensional recurrence quantification analysis. In an embodiment, the synchronizing the dyadic parameter data comprises aligning synchronization signals in the dyadic parameter data.

[0022] According to a third aspect of the present disclosure, a computer readable medium carrying processor executable instructions which when executed on a processor cause the processor to cany out a method as described above is provided.

[0023] According to a fourth aspect of the present disclosure a system tor assessing and promoting the development of a child subject is provided. The system comprises: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for the child subject and a time series of sensed data indicative of the parameter for an adult subject collected during interactions between the child subject and the adult subject; synchronize the dyadic parameter data; generate a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classify the development of the child subject by inputting the set of features into a machine learning classifier trained to classify development of a child based on the set of features; and output a prediction indicating the classification of the development of the child subject.

[0024] The machine learning classifier may be further trained to classify a maternal mood of the adult subject based on the set of features, and the data storage device may be further stores computer program instructions operable to cause the processor to classify the maternal mood of the adult subject by inputting the set of features into the machine learning classifier, and the prediction indication may further indicate the classification of the maternal mood of the adult subject.

[0025] According to a fifth aspect of the present disclosure, a system for assessing maternal mood of an adult subject is provided. The system comprises: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for a child subject and a time series ot sensed data indicative of the parameter for the adult subject collected during interactions between the child subject and the adult subject; synchronize the dyadic parameter data: generate a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features: classify the maternal mood of the adult subject by inputting the set of features into a machine learning classifier trained to classify the maternal mood of the adult subject based on the set ot features: and output a prediction indicating the classification or description of the maternal mood of the adult subject.

[0026] In an embodiment, the system further comprises a plurality of sensors configured to capture the dyadic parameter data.

[0027] In an embodiment, the system further comprises a trigger box configured to generate a synchronization signal on each sensor of the plurality of sensors and wherein the data storage device further stores computer program instructions operable to cause the processor to: synchronize the dyadic parameter data using the synchronization signals.

[0028] In an embodiment, the dyadic parameter data comprises one or more domains of behavioral parameter data and / or a physiological parameter data.

[0029] In an embodiment, the dyadic parameter data comprises child and adult electroencephalogram data and wherein the combination of non-linear and linear features are determined from non-oscillatory and oscillatory metrics respectively. in an embodiment, the non-oscillatory and oscillatory metrics comprise topographical graph network features assessed via weighted Symbolic Mutual Information and weighted Phase Lag Index respectively. In an embodiment, the dyadic parameter data comprises child and adult electroencephalogram data and video data and wherein the video data is used to identify portions of interest in the electroencephalogram data.

[0030] In an embodiment, the dyadic parameter data comprises child and adult electrocardiogram data, and the data storage device further stores computer program instructions operable to cause the processor to pre-process the electrocardiogram data to re-sample electrocardiogram data for the child subject and electrocardiogram data for the adult subject such that re-sampled electrocardiogram data for the child subject and re-sampled electrocardiogram data for the adult subject are equidistantly sampled in time.

[0031] In an embodiment, the dyadic parameter data comprises video data and the data storage device further stores computer program instructions operable to cause the processor to detect pose data of the child subject and pose data of the adult subject and analyze linear and non-linear contingencies between the pose data of the child subject and the pose data of the adult subject.

[0032] In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to analyze non-linear contingencies between the pose data of the child subject and the pose data of the adult subject by applying multi-dimensional recurrence quantification analysis.

[0033] BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In the following, embodiments of the present invention will be described as non-limiting examples with reference to the accompanying drawings in which:

[0035] FIG.1 is a block diagram showing a dyadic parameter processing system according to an embodiment of the present invention;

[0036] FIG.2 is block diagram showing a dyadic parameter processing system according to an embodiment of the present invention; FIG.3A is a flow chart showing a machine learning method of assessing development of a child subject according to an embodiment of the present invention;

[0037] FIG.3B is a flow chart showing a machine learning method of assessing maternal mood according to an embodiment of the present invention;

[0038] FIG.4 is a flowchart showing a workflow for EEG analysis according to an embodiment of the present invention;

[0039] FIG.5A is a flow chart showing pre-processing of EEG data according to an embodiment of the present invention;

[0040] FIG.5B is a flow chart showing denoising of EEG data according to an embodiment of the present invention;

[0041] FIG.6 illustrates the general concept of weighted phase lag index (wPLI) calculations;

[0042] FIG.7 illustrates a Representative workflow for within and between brains weighted phase lag index (wPLI) calculations;

[0043] FIG.8 illustrates a representative workflow for within and between brains wSMI calculations;

[0044] FIG.9 shows creation of surrogate data and standardization (z-score) steps prior calculating graph theory based metrics;

[0045] FIG.10 is a flowchart showing a workflow for ECG analysis according to an embodiment of the present invention;

[0046] FIG.11 shows an example two-electrode setup;

[0047] FIG.12 is a table showing an example of re-sampling the HRV data in real-time to produce evenly-sampled heart periods / R-R intervals in time units; FIG.13 is a flowchart showing a workflow of a pose algorithm used in embodiments of the present invention;

[0048] FIG.14 is a flowchart showing a workflow for audio analysis according to an embodiment of the present invention;

[0049] FIG.15 is a flowchart showing logical steps in conversational turn-taking analysis;

[0050] FIG.16 shows validation of dependent variable across tasks;

[0051] FIG.17 shows EF score histograms for Brazil and Singapore;

[0052] FIG.18 shows variance of a dataset described against the number of PCA features for EEG and ECG;

[0053] FIG.19 shows variance of a dataset described against the number of PCA features for Gaze, Audio, and Emotion;

[0054] FIG.20 is a graph showing accuracy vs ratio of Brazil test used in training, based on behavioral only domains as inputs;

[0055] FIG.21 shows a comparison of the number of times each classifier model is picked by the algorithm as the best performing;

[0056] FIG.22 is a graph showing feature availability vs Accuracy and Number of Combinations; and

[0057] FIG.23 is a graph showing the combination of domains and accuracy values.

[0058] DETAILED DESCRIPTION

[0059] During social interaction, interpersonal neural dependencies arise between partners and increasing evidence suggests that these neural dependencies are crucial for communication and social interaction success. Human perception relies on neural oscillatory processes in the cortex that shape our conscious experience. The oscillatory phase of neural activity at the time of stimulus presentation relates systematically to the excitability of cortical neural populations and to the magnitude of event-related responses elicited by stimuli. Accordingly, stimuli arriving during a high- excitability oscillatory phase are more likely to be detected than stimuli that arrive at a low-excitability phase. Extending this framework to the social (dyadic) domain, it has been proposed that during social interaction, parents use social cues (like gaze or touch) to reset the phase of on-going oscillations in their child’s brain, effectively ‘nudging’ their child into a transient state of neural synchronization. For example, parent-initiated mutual gaze might trigger a brief increase in neural synchrony. During a high synchrony state, parents’ and infants’ high excitability oscillatory phases are well-aligned, ensuring that information offered by the parent (words) are presented at optimal times for learning by the infant.

[0060] Consistent with this hypothesis, it has been demonstrated that eye contact increases synchronization between adult and infant EEG oscillations at Theta and Alpha frequencies (3-5 Hz and 6-9 Hz respectively). Further, using Granger-causal methods to assess directional influences, it has been shown that interpersonal neural synchronization was reinforced by infants’ own communicative efforts, with longer infant vocalizations producing (causing) stronger synchrony effects on the adult. The parent-infant neural network is also exquisitely sensitive to the emotional tone of social interactions. In a recent dyadic-EEG study with 47 mother-infant dyads (10.9 months), infants observed their mothers express positive or negative facial emotions. The mother-infant interpersonal Alpha (6-9 Hz) network increased in strength and integration efficiency during the expression of positive relative to negative emotion, suggesting that positive emotions create a ‘super-networked’ neural state during which information flows efficiently between mother and infant. Notably, emotion-related changes in network connectivity were much more evident in the interpersonal neural network than in mothers’ or infants’ individual brain networks, suggesting that dyadic neural connectivity is a more sensitive marker of the emotional tone of social interactions than measurements taken from individual brains. Finally, recent work suggests that parent-infant neural connectivity supports infants’ social learning: stronger mother-infant Alpha neural connectivity predicted a higher likelihood of successful infant social learning from mothers on individual trials. Collectively, these data strongly suggest that interpersonal sociometrics (such as neural synchrony) are exquisitely sensitive to parent-infant social interaction quality and can potentially predict early EF-related behavior (sustained attention, vocalizations, and social learning). Accordingly, sociometrics provide an important and novel way to measure the health and quality of early social interactions and may permit earlier identification of disordered social development.

[0061] The present disclosure provides systems and methods for analyzing development such as cognitive development of a child subject. The system gathers data during social interactions between the child subject and an adult subject. The interactions may be termed naturalistic play-based social interactions. The interactions may be between a child and a parent (i.e. the child’s mother or father) or a non-related adult. In the present disclosure, references to a parent correspond to the adult subject with whom the child subject is interacting. Further, in the present disclosure, the term child subject is used to indicate a child (for example a child 12 years old or younger). References to child or infant are intended to indicate the child subject. It is noted that many of the methods described herein are particularly aimed at young children or infants, for example having an age of 3 years or less, or 12 months or less respectively. The analysis of younger children is particularly challenging as they generally have short attention spans and therefore the time periods over which data can be gathered are short. Further, the neurophysiological parameters such as brain activity or heart rate of young children differ from those of adults which presents additional challenges when analyzing interactive patterns.

[0062] FIG.1 is a block diagram showing a dyadic parameter processing system according to an embodiment of the present invention. The dyadic parameter processing system 100 is a computer system which implements methods for processing dyadic parameters to assess development of a child subject and / or to assess a maternal mood. FIG.1 shows examples of the sensors and input data to the dyadic parameter processing system 100.

[0063] The dyadic parameter processing system 100 processes data captured during interactions between a child subject and an adult subject. As shown in FIG.1 , a video camera 30 captures child subject video data 50, an EEG / ECG cap captures child subject EEG data 52 and child subject ECG data 53, and a microphone 34 captures child subject audio data 54. A video camera 40 captures adult subject video data 60, an EEG / ECG cap captures adult subject EEG data 62 and adult subject ECG data 63, a microphone 44 captures adult subject audio data 64 and set of eye tracking glasses captures adult subject eye-tracking data 66.

[0064] It will be appreciated that different combinations of sensors may be used to capture data for the dyadic parameter processing system 100 and the specific sensors shown in FIG.1 are intended to be examples. For example, multiple video cameras may be used to capture the child subject video data 50 and the adult subject video data 60. Alternatively a single video camera may be used to capture both the child subject video data 50 and the adult subject video data 60.

[0065] In some implementations, the sensors shown in FIG.1 may be wirelessly connected to the dyadic parameter processing system 100. The data collected by the sensors may be provided in real-time or near real time to the dyadic parameter processing system 100.

[0066] The processing carried out by the dyadic parameter processing system 100 uses features between the child subject parameter data and the adult subject parameter data. In the present disclosure, the pairs of data such as child subject EEG data and adult subject EEG data are referred to as dyadic parameter data.

[0067] In order to process the dyadic parameter, it is important that the captured data is synchronized in time. In order to allow the dyadic parameter processing system 100 to synchronize the data, a trigger box 70 is provided. The trigger box 70 may be provided with a trigger button and in response to actuation of the trigger button, the trigger box 70 provides a trigger signal to each of the sensors attached to the dyadic parameter processing system 100. The trigger signal may be provided as a light signal to the video camera 30 and the video camera 40. The trigger signal may be provided as an electronical signal to the EEG / ECG caps, microphones and eye-tracking glasses.

[0068] The Dyadic Sociometrics approach permits real-time estimation of social interaction effects on brain function. During social interaction, interpersonal neural dependencies arise between partners and increasing evidence suggests that these neural dependencies are crucial for communication and social interaction success. Dyadic sociometric measures refer to indices that capture bi-directional temporal contingencies in behavioral, neural and physiological patterns between two interacting individuals, including visual gaze (shared eye contact), vocalizations, movement, touch, physiology and neural activation patterns. These indices capture conditional probabilities, temporal patterns and synchronicities between each partner’s social behavior and neurophysiology that provide unique insights into social interaction dynamics and quality which cannot be gleaned from observing individual actions alone. Dyadic sociometrics are collected at high temporal precision (millisecond accuracy) whereas existing parent-child scoring schemes typically only assign a single global score with no time resolution. Given their high temporal precision, the data structure of sociometric indices well affords advanced time-series signal processing approaches such as autoregressive modelling and Granger-causality analyses, yielding precise estimates of synchrony and temporal pacing in gaze, vocalizations, posture / gestures, neural and physiological activation (EEG & ECG), as well as in parental Reaction Time (RT) to infant-initiated actions and attentional bids. These objective and precise indices complement traditional subjective ratings of parental sensitivity, warmth and verbal scaffolding that are commonly used to assess parent-child interactions. To relate these sociometric measures to EF development, it is possible to perform deep phenotyping (the precise and fine-grained analysis and description of individual phenotypic patterns, described in a computationally accessible manner) of the social interaction characteristics of each parent-infant dyad. These sociometric profiles can be fed into machine learning-based computational models to stratify infants according to their EF developmental status and risk - these risk profiles may in future be used in commercial applications for screening tools.

[0069] FIG.2 is block diagram showing a dyadic parameter processing system according to an embodiment of the present invention. The dyadic parameter processing system 100 is a computer system with memory that stores computer program modules which implement methods assessing development of a child subject and / or of assessing a maternal mood according to embodiments of the present invention. The dyadic parameter processing system 100 comprises a processor 110, a working memory 112, an input interface 114, a user interface 116, program storage 1 0, and data storage 140. The processor 110 may be implemented as one or more central processing unit (CPU) and / or graphics processing unit (GPU) chips. The program storage 120 is a non-volatile storage device such as a hard disk drive which stores computer program modules. The data storage 140 is a non-volatile storage device such as a hard disk drive which stores data, in this case a machine learning classifier which is used during execution of the computer program modules. The computer program modules are loaded into the working memory 112 for execution by the processor 110. The input interface 114 is an interface which allows data, including the output from the various sensors shown in FIG.1 to be received by the dyadic parameter processing system 100. The user interface 116 allows results of assessing development of a child subject to be displayed to a user and may also allow the user to make input or selections relating to the assessment.

[0070] The program storage 120 stores a pre-processing module 122, a synchronization module 124, a feature generation module 126, a classification module 128 and a training module 130. The computer program modules cause the processor 110 to execute various child development assessment and maternal mood assessment processing which is described in more detail below. The program storage 120 may be referred to in some contexts as computer readable storage media and / or non- transitory computer readable media. As depicted in FIG.2, the computer program modules are distinct modules which perform respective functions implemented by the dyadic parameter processing system 100. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into sub-modules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or sub-module. It will also be appreciated that, while a software implementation of the computer program modules is described herein, these may alternatively be implemented as one or more hardware modules (such as field- programmable gate array(s) or application-specific integrated circuit(s)) comprising circuitry which implements equivalent functionality to that implemented in software. Although the dyadic parameter processing system 100 is described with reference to a computer, it should be appreciated that the dyadic parameter processing system 100 may be formed by two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and / or parallel processing of the instructions of the application. Alternatively, the data processed by the application may be partitioned in such a way as to permit concurrent and / or parallel processing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the dyadic parameter processing system 100 to provide the functionality of a number of servers that is not directly bound to the number of computers in the dyadic parameter processing system 100. In an embodiment, the functionality disclosed above may be provided by executing the application and / or applications in a cloud computing environment. Cloud computing may comprise providing computing services via a network connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and / or may be hired on an as-needed basis from a third- party provider.

[0071] The data storage 140 stores a machine learning classifier 142 which is trained to classify child development and / or maternal mood using features generated by the feature generation module 126. The machine learning classifier 142 may be implemented as a k-nearest neighbors classifier, a linear discriminant analysis classifier, a quadratic discriminant analysis classifier, a random forest classifier, a logistic regression classifier, a support vector machine classifier, an adaptive boosting classifier or other type of machine learning classifier.

[0072] FIG.3A is a flow chart showing a machine learning method of assessing development of a child subject according to an embodiment of the present invention. The method 300 shown in FIG.3A is carried out by the dyadic parameter processing system 100 shown in FIG.2.

[0073] In step 302, the input interface 114 of the dyadic parameter processing system 100 receives dyadic parameter input data. The dyadic parameter data comprises time series of sensed data for the child subject and the adult subject which is captured during interactions between the child subject and the adult subject. The dyadic parameter data may be captured while the child subject and the adult subject perform a specifically defined task.

[0074] Following step 302, the dyadic parameter input data may be pre-processed by the preprocessing module 122

[0075] In step 304, the dyadic parameter data is synchronized by the synchronization module 124.

[0076] In step 306, the feature generation module 126 is executed by the processor 110 to generate a set of features indicative of temporal contingencies between the child subject and the adult subject. Specific examples of features which may be generated are described in more detail below.

[0077] In step 308, the classification module 128 is executed by the processor 110 to classify the development of the child subject. In step 306, the features generated in step 306 are input into the machine learning classifier 142 to classify the development of the child subject.

[0078] In step 310, the user interface 116 of the dyadic parameter processing system 100 outputs a prediction indication indicating the classification of the child subject. The output may be an indication that represents the cognitive development or socioemotional development of the child subject. The indication may be a summary score, a categorical indicator (e.g. 'traffic light’) or a profile report.

[0079] FIG.3B is a flow chart showing a machine learning method of assessing maternal mood according to an embodiment of the present invention. The method 350 shown in FIG.3B is carried out by the dyadic parameter processing system 100 shown in FIG.2. The method 350 shown in FIG.3B may be carried out concurrently with the method 300 shown in FIG.3A, for example the set of features input may be used in both the assessment of the development of the child subject and the assessment of the maternal mood of the adult subject. Alternatively, the methods may be carried out individually.

[0080] In step 352, the input interface 114 of the dyadic parameter processing system 100 receives dyadic parameter input data. The dyadic parameter data comprises time series of sensed data for the child subject and the adult subject which is captured during interactions between the child subject and the adult subject. The dyadic parameter data may be captured while the child subject and the adult subject perform a specifically defined task.

[0081] Following step 352, the dyadic parameter input data may be pre-processed by the preprocessing module 122

[0082] In step 354, the dyadic parameter data is synchronized by the synchronization module 124.

[0083] In step 356, the feature generation module 126 is executed by the processor 110 to generate a set of features indicative of temporal contingencies between the child subject and the adult subject. Specific examples of features which may be generated are described in more detail below.

[0084] In step 358, the classification module 128 is executed by the processor 110 to classify the maternal mood of the adult subject. In step 306, the features generated in step 356 are input into the machine learning classifier 142 to classify the maternal mood of the adult subject.

[0085] In step 360, the user interface 116 of the dyadic parameter processing system 100 outputs a prediction indication indicating the maternal mood of the adult subject. The prediction indication may indicate the maternal mood as a 2-class prediction of low or high risk of depression or anxiety.

[0086] In the following, detailed examples of processing dyadic parameters to generate features will be described. FIG.4 is a flowchart showing a workflow for EEG analysis according to an embodiment of the present invention. The method 400 shown in FIG.4 may be carried out by the dyadic parameter processing system 100 shown in FIG.2.

[0087] In step 402, EEG data for the child and adult is received. Dyadic EEG in the current protocol is collected as two separate data streams, recorded concurrently from the parent and the child.

[0088] The EEG data may be synchronized offline following a synchronization protocol. Concurrent video and audio data streams are recorded and synchronized following the same protocol as well. A battery-powered input / output box is configured to receive a signal from a hand-button device and to output an electrical pulse to the two EEG streams at the same time as to an LED on-switch. The triggers are concurrently marked on both EEGs within their intrinsic timelines, and the LED is visible on all video recordings. In offline post-processing, the intrinsic time of each LED event is offset to the concurrent EEG marker, and the two EEGs are offset to each other. The video offsets are then used to trim or pad the videos to a starting point coinciding with the start of the infant EEG recording. As the sampling rates of the different modalities differ, a series of frames-to-time conversions are implemented to calculate their respective offsets, resulting in a common timeline.

[0089] The protocol allows use of wireless technology for recording EEG, which enables freedom of movement and natural interactions by the parent-child dyad. In the majority of studies involving infant EEG, or infant-adult EEG hyper-scanning, either wireless technology is used with low-density EEG, or higher-density montages are employed in a wired setup whereby the child is constrained within a cable’s reach to the EEG recording device. This allows the collection of high-density and high-quality individual EEG data streams while still accommodating for a naturalistic environment. Additionally, the high-density EEG montage is customized to a focal coverage of EEG topographies that previous work has demonstrated as least likely to be affected by motion-related artifacts introduced when the dyad is engaged in social interactive activities such as play and communication. In step 404, pre-processing of the EEG data is performed. As extracting temporal contingencies between the infant and adult in the EEG domain is crucial, all signal preprocessing and noise removal is done with the aim of retaining the temporal structure of the two time series.

[0090] The pre-processing is shown in more detail in FIG.5A. FIG.5A is a flow chart showing pre-processing of EEG data according to an embodiment of the present invention. The method 500 shown in FIG.5A corresponds to step 404 of the method 400 shown in FIG.4.

[0091] In step 502, temporal offsets between all recording streams (video, audio, electrophysiology) are calculated in common units (seconds) time-locked to the infant EEG stream (i.e. the start of the infant EEG’s timeline becomes time 0).

[0092] In step 504, the EEG segments of interest are identified based on the video timings of events of interest. These are task-long segments, with an average duration of several minutes, and differ between individual parent-infant dyads.

[0093] In step 506, denoising is carried out. The denoising may be carried out by independently processing the adult and infant data streams. The processing making up step 506 is described below in more detail with reference to FIG.5B.

[0094] FIG.5B is a flow chart showing denoising of EEG data according to an embodiment of the present invention.

[0095] In step 510, each segment is first high-pass filtered at 0.5Hz, then low-pass filtered at 40Hz.

[0096] In step 512, channel noise is detected. Step 512 may comprise automatic detection of noisy or bridged channels, followed by visual inspection. Channel noise present in over 80% of the segment of interest and with high voltage deflections are removed from the EEG recording. In step 514, offline average re-reference is calculated, recovering an additional centralpoint channel (Cz) equivalent to the common average of all remaining channels.

[0097] In step 516, first-pass visual noise detection is carried out. Large noise segments (those spanning several seconds, where the majority of channels have high voltage power) are manually rejected. Stereotypical artifacts and smaller noise segments are retained. This step results in individual time series that are not matched in time between the adult and the infant in the dyad, due to different portions of the data being rejected.

[0098] In step 518, automatic artifact rejection using a technique like independent component analysis (ICA) is performed using software such as EEGLAB (Matlab, runica). . In ICA, time series are decomposed to orthogonal independent contributors using a series of temporal and spatial filters and a demixing function. The weights of each independent component (IC) are then plotted in time and frequency space, and on a topographic scalp map, as a per standard labelling protocol. Each component is inspected and assigned to either signal or noise category.

[0099] In step 520, second-pass visual noise detection is carried out following noise ICs removal. The time series are inspected for noise that could not be segregated and removed by ICA. Again, noise segments are deleted at individual dataset bases.

[0100] Returning now to FIG.5A, in step 508, the timeline of the EEG time series is reconstructed based on the remaining available clean data and diary entries of noise segments. NaNs (Not a Number labels) are inserted in place of the noise deletions. The reconstructed infant and adult time series are checked against each other and against the visual data stream to ensure they are matched in time and represent the proportion of recording of interest. This step is not implemented for single EEG application but was designed here to allow perfect alignment to a single data-point accuracy (2ms) between the two EEG data streams. It is crucial for enabling us to analyze the time-locked dyadic EEG data.

[0101] Returning now to FIG.4, in step 406, EEG functional connectivity is performed. Thus prepared, the EEG time series time-locked to the task events of interest can be analyzed to extract linear (wPLI) and nonlinear (wSMI) estimates of functional connectivity within and across brains.

[0102] EEG recording is a popular technique for estimation of brain functional connectivity, mainly due to its high temporal resolution and low cost. However, an issue to consider when using EEG is volume conduction problem, that is the propagation of electric fields generated by a primary current source in the brain to all (or part) of the rest of the on-scalp sensors. Several metrics have been proposed to overcome this problem, in particular weighted phase lag index (wPLI) and weighted symbolic mutual information (wSMI) are two of the most commonly used linear and non-linear methodologies respectively. Both of them minimize the contribution of ‘almost-zero’ lag interactions, allowing one to identify true time-lagged coupling.

[0103] EEG recordings at scalp level are a combination of oscillatory (linear) and non- oscillatory (non-linear) dynamics. While the oscillatory dynamics are characterized by distinctive peaks in the power spectrum, non-oscillatory dynamics are not linked to a specific rhythm or temporal pulse. Both forms of neural activity, linear and nonlinear, are known to support neural connectivity between brain regions. The high complexity of the brain may be better described through a combination of different measures, linear and nonlinear, rather than by each of them in isolation.

[0104] The present invention combines the use of wPLI and wSMI (oscillatory and non- oscillatory) metrics which allows us to obtain complementary information about linear and nonlinear functional connectivity patterns at different developmental stages. Furthermore, the use of synchronized dual EEG recording allows us to gain unique insights into the neural mechanisms by which interpersonal and social factors influence infant behavior and cognition.

[0105] FIG.6 illustrates the general concept of weighted phase lag index (wPLI) calculations. The phase lag index (PLI) is a linear method measuring the extent to which instantaneous phase angle differences between two time series x(t) and y(t) are distributed towards positive or negative parts of the imaginary axis. This asymmetry implies the presence of time delay (lag) between the two time series (Figure A3). The fundamental idea is to disregard phase values centred around 0 (or IT, 2TT and so on) as they will be mainly related to volume conduction effects. While a flat distribution around 0 reflects no coupling, a weighted version of the lag index (wPLI) was introduced to make those points reflecting instantaneous mixing less influential due to noise signals. This was implemented by weighting using the imaginary component of the cross-spectrum. where Imag is the imaginary part of the cross-spectral density (Pxy) between two signals x(t) and y(t), sng indicates the sign. The wPLI as the PLI ranges between 0 and 1 , the former indicates no coupling and the latter meaning perfect phase locking.

[0106] As both within-brain and between-brain connectivity are being calculated, infant and adult EEG recordings are concatenated into one matrix of channels x samples for further analysis. The first step to calculate the wPLI is to obtain the instantaneous phase series from each EEG channel. This was done using the analytic signal based on the Hilbert transform after filtering the signal in the frequency band of interest. The obtained analytical signal is divided into a series of non-overlapping windows before calculating the cross-spectral density and the final wPLI value. The result of this process is a 3 dimensional matrix whose dimensions are as follows: channels x channels x windows, where the number of windows will depend on the length of the signal (can be trial based or continuous recording) and the size of the window.

[0107] The method employs 2s-long windows for the mother and 1 s-long windows for the infant and for dyadic connectivity as previous research showed this was the optimal configuration instead of using the same window length for all the cases. The resulting matrix contains the within (first and last quadrant) and between (2nd and 3rd quadrants) brain connectivity as indicated in FIG.7. wPLI connectivity is a non-directed metric, hence the connectivity from mother to infant will be identical to the connectivity from infant to mother.

[0108] FIG.7 illustrates a Representative workflow for within and between brain weighted phase lag index (wPLI) calculations. FIG.8 illustrates a representative workflow for within and between brain wSMI calculations. As shown in FIG.8, EEG signals are first transformed into a series of discrete symbols defined by the ordering of k time samples separated by a temporal separation T. The symbols are coded according to the trends in amplitudes of a specific predefined number of consecutive time points. The approach adopted kernel k to be 3, implying that the symbols are constituted of three elements, leading to 3! = 6 different potential symbols in total. To capture the infant theta and alpha frequency range, the temporal separation of elements that constitute a symbol can be set to be T=1 6 or12 frames, as the definition of TT also indicates the maximum resolved frequency (e.g. / max = fs / kt = 500 / 3*10 = 16.6Hz). Signals are low pass filtered at the corresponding frequencies to avoid aliasing artifacts. A windowing procedure was also followed, in line with the wPLI calculations, using 1 s window length for adults, 2s for infants and 1 s for the dyad. Note that adult and infant window lengths are opposite for wPLI and wSMI. This procedure enabled the acquisition of the most stable outcome for each functional connectivity metric.

[0109] Afterwards, the joint probability matrix of each pair of symbols co-occurring in two different time series is computed to estimate the SMI between the two signals. Finally, to address volume conduction issues, the joint probability matrix is multiplied by binary weights. The weights are set to zeros for pairs of identical symbols, which could mean they originated from a common source, and for opposed sign symbols, which could reflect the two sides of a single electric dipole. The formal mathematical equation of wSMI is as follows:

[0110] The wSMI can lead to negative values after the weighting procedure, thus all negative values were set to 0 as they did not represent true entropy.

[0111] FIG.9 shows creation of surrogate data and standardization (z-score) steps prior calculating graph theory based metrics.

[0112] In order to ensure that the functional connectivity values obtained are not spurious connections but true coupling, surrogate datasets are computed for each participant’s EEG recording as shown in FIG.9. The surrogates are created for both wPLI and wSMI through time-point shuffling of the preprocessed EEG dataset. For each window size, we standardize (z-score) the original connectivity matrix with its corresponding surrogates: std (surrogates)

[0113] After this step, all negative values are set to 0, indicating that they are spurious (random) connections hence they should not be included in further analysis.

[0114] In step 408 shown in FIG.4, EEG graph metrics are performed. A graph consists of a series of nodes, in this case EEG electrodes, and a set of edges showing the relationships between the nodes. Therefore, the first step to construct a graph is to obtain a matrix, named an adjacency matrix, containing the connection information between the nodes. An adjacency matrix is constructed by comparing the link between each pair of nodes in the connectivity matrix against a corresponding threshold. Edges whose values are larger than the threshold (for example, the strongest 15%) remained in the adjacency matrix, whilst those with values under the limit were set to 0. Different thresholds were used: no-threshold (all links after the standardization were included), 10%, 15%, 20% and 25% of the strongest connections. Afterwards, a series of graph theoretic indices were extracted:

[0115] Individual metrics:

[0116] Strength (S) is typically defined as the sum of neighboring link weights. In this case, we report the highest value of neighboring link weights. This is likely to be more informative than mean strength as network density was fixed to retain only the most strongly connected links.

[0117] Community structure detection metrics:

[0118] Transitivity (T) is the overall probability for the network to contain interconnected adjacent nodes, revealing the existence of tightly connected communities. In simple terms of network topology, this index represents the mean probability that two vertices that are network neighbors of the same other vertex will themselves be neighbors.

[0119] Modularity (Q): quantifies the degree to which the network may be subdivided into non-overlapping groups of nodes in a way that maximizes the number of within-group edges and minimizes the number of between-group edges. Modules or distinct communities are useful to partition larger networks into more basic “building blocks” that can separate functionally related neural elements.

[0120] Paths, distances and efficiency metrics:

[0121] Characteristic path length (CPL):is the average shortest path length in the network. A path is a sequence of linked nodes that never visit a single node more than once. A lower value, indicates shortest path or distance between two nodes, hence a quicker transfer of information and consequently, lower cost.

[0122] Diameter: after calculating the shortest path length from every node to all other nodes, the longest of all them is the diameter of the network. It shows the shortest path between the two most distant nodes in the network, providing inference about the path it needs to travel to get to all sides of the network.

[0123] Global Efficiency (GE) is inversely related to the topological distance between nodes and is typically interpreted as a measure of the capacity for parallel transfer and integrated processing. It is based on the inverse of the shortest path length, which is an indicator of the ease with which each node can reach other nodes within the network using a path that is composed of only a few edges. Hence, the global efficiency is an indicator of the degree to which a network can share information between distributed regions.

[0124] Centrality metrics:

[0125] Betweenness centrality (BC) is a measure of centrality. These measures identify central nodes that connect various brain regions. The betweenness centrality of a node is defined as the fraction of all shortest paths in the network that pass through the given node. Nodes with a larger betweenness centrality value will participate in a higher number of shortest paths.

[0126] Edge betweenness centrality: It is the sum of the fraction of all-pairs shortest path that passes through that particular edge (instead of a node as the above). A higher score represents a bridge-like connector between two parts of a network, meaning that if it is removed, it will likely affect the communication between several pairs of nodes.

[0127] All these previously listed measures may be calculated for both within and between brain data.

[0128] Metrics for between brain data only:

[0129] Divisibility: is a measure of how well the entire connectivity network (including intra- and inter-brain connections) can be divided into two sets of nodes, corresponding to the brain of each member of the dyad. where W is the total weight of the network (including within and inter-brain subnetworks), Ci and Cj indicate the community (which brain) the nodes i and j belong to respectively. The function 6 is binary with values 0 or 1 (1 if vertices i and j are in the same community and 0 otherwise). The resulting values of D (divisibility) range between [0,1 ].

[0130] An analysis performed revealed that dyadic (mother-infant) graph metrics were amongst the most important predictors of infant executive function such as infant’s attention shifting, and of infant’s temperament. Dyadic measures of parent-infant neural connectivity may provide robust and generalizable indices for the prediction of developing cognition, particularly emerging executive function skills. This empirical validation is an important first step toward developing reliable screening tools for early assessment of EF and its disorders. FIG.10 is a flowchart showing a workflow for ECG analysis according to an embodiment of the present invention. The method 1000 shown in FIG.10 may be carried out by the dyadic parameter processing system 100 shown in FIG.2.

[0131] During early social interactions, infants and mothers are known to display physiological synchrony in addition to behavioral synchrony, which has the potential to shape infant development through entrainment of the infant’s biological rhythms, providing an external regulation to the consolidation of cyclic physiological processes. This regulation of the human autonomic nervous system as reflected by heart-rate variability (HRV) is known to be linked to early socio-emotional development and cognitive functioning such as executive function (EF). While most ECG analyses have focused on time-domain and frequency-domain measures derived from the HRV data, little is known about the underlying nonlinear physiological dynamics at play. Recent research has emphasized the need to explore non-linear metrics over the traditional linear counterpart, owing to the complex nature of the cardiovascular regulation system arising from nonlinear interplay of multiple physiological control loops, which exhibit complex patterns of variability that can be described by mathematical chaos theory. Furthermore, in naturalistic settings or experimental tasks where subjects can move freely, nonlinear measures can be more robust to artifacts and non-stationarity present in the data.

[0132] Previous research on mother-infant physiological (ECG) synchrony has predominantly used linear methods such as cross-correlation and coherence to study the similarity between the two signals in the time and frequency domains. While this has led to key insights, little is known about the complexity or stability of the synchrony at play, since real-life systems often go in and out of sync in a chaotic manner. In this implementation, a complex systems approach, Recurrence Quantification Analysis (RQA), is used to describe the nonlinear physiological dynamics of mother-infant dyads during tasks. Specifically, Multidimensional Recurrence Quantification Analysis (MdRQA) is used to describe the state of the dyadic system as a whole, and Cross Recurrence Quantification Analysis (CRQA) to extract relations between the mother’s and infant’s HRV. HRV refers to the changes in the time intervals between consecutive heartbeats, denoted by the “R peaks.” In the current study, HRV data is extracted from the Electrocardiography (ECG) signals in the form of R-R intervals or inter-beat intervals. Described next is the pipeline employed for the ECG data collection, pre-processing, and analysis of dyadic physiological synchrony.

[0133] In step 1002, ECG data for the adult and the child is received. ECG may be collected concurrently with EEG from the infant and the adult using two extended electrodes from the EEG electrode branch to record a single lead ECG.

[0134] FIG.11 shows an example two-electrode setup. The two-electrode setup follows a modified Lead II configuration, with one electrode placed below the right clavicle bone, and the other placed on the lower edge of the left ribcage of the subjects. The reference and the ground electrodes are part of the scalp EEG electrodes on the head, whereby Cz is the reference electrode.

[0135] A trigger solution is applied to time-synchronize the dyadic EEG-ECG recording with the audio-visual data acquired during the experimental tasks. The accepted error tolerance / lag between the video / audio and EEG / ECG data streams is 1 video frame (here 40ms, with a video frame rate of 25FPS) and 2ms (1 sample here, with a sampling rate of 500Hz) between the 2 EEG / ECG recordings. This ensures tight timesynchronization between the mother’s and the infant’s neurophysiological data with high temporal precision, encouraging the study of task-related dyadic coupling or synchrony.

[0136] In step 1004, the received data is pre-processed and R-R intervals are extracted. Prior to processing the signals to minimize the influence of artifacts, task-related segments of the ECG recording are extracted from the raw signal by calculating the temporal offsets between the synchronized video recordings (which mark the events of interest) and the ECG recording. The adult and the infant ECG data are then pre-processed independently as described in the following section and subsequently matched in time to study temporal contingencies. The ECG data is processed using HRV software such as Kubios (Kubios HRV Scientific Ver 4.0.3 used here). Kubios uses a built-in QRS detection algorithm based on the Pan-Tompkins method to extract the R-R interval time series. The software also comes with automatic noise-detection (set to “strong” here) and beat-correction algorithms (accepted threshold set to 5%), which identify and mark noisy segments in the data and correct incorrectly detected / missed beats after the first beat detection step.

[0137] Each dataset is then manually scanned in 10-second windows to ensure accurate detection of R peaks and marking of noisy segments (segments where R peaks are not detectable due to noisy sources such as muscle movement / electrode displacement, or segments with ectopic or arrhythmic beats). The final output is then a time series or R-R intervals.

[0138] In step 1006, the data is re-sampled and R-R time series are matched. For the purpose of studying dyadic synchrony, it is important for the mother’s and infant’s HRV time series to be matched in time. By default, an R-R time series is not evenly sampled (each data-point corresponds to the duration between two consecutive beats), and since infants typically have a much higher heart rate than adults, the resulting R-R time-series after preprocessing tends to be longer for infants, for the same task segment analyzed. Therefore, the most important pre-processing step involves resampling the adult and infant R-R series to be equidistantly sampled and of the same data length using the means-epoch method shown in FIG.12. Using this method, the R-R time series is first resampled at 10 Hz, and epochs of 1000ms are created. For each epoch, the weighted mean R-R interval is computed, which results in a time series of epoch means for each participant.

[0139] FIG.12 is a table showing an example of re-sampling the HRV data in real-time to produce evenly-sampled heart periods / R-R intervals in time units, instead of “beat” units. Here, the epoch length is 2000ms.

[0140] In step 1008, parameters for time delayed embedding of R-R times series are optimized in phase space. Missing values (which correspond to the noisy segments marked in Kubios) as indicated by “NaN”s (Not a Number) in the data are later filled using cubic spline interpolation. This procedure is performed for both the adult and the infant R-R time series to retrieve the final time-locked dyadic HRV data. The data is retained if it has a minimum of 40 seconds of usable and continuous R-R intervals for both the mother and the infant for short tasks that are roughly 1 minute long. For the long tasks that are several minutes long, the inclusion criteria is for at least 30% of the data to be usable, with a minimum of 40 seconds of continuous R-R interval data available. This step is crucial because the RQA analysis performed subsequently does not tolerate NaNs as part of the input data.

[0141] In step 1010, Recurrence Quantification Analysis (RQA) is performed. Recurrencebased methods are particularly suited for time-series that are non-stationary or have complicated dynamics, such as continuous physiological data. They are robust against extreme outliers and do not make any assumptions about particular distributions or particular relationships between the time-series of interest. Additionally, RQA is able to extract meaningful information from short data segments (as low as ~ 30 - 40 data points), making it attractive for assessing real-life data prone to noise and missing data. Accordingly, this technique is useful for applications in naturalistic settings. The following sections describe the RQA-derived techniques used in this study.

[0142] Time-delayed embedding (Takens theorem) forms the basis of the RQA approach, wherein, a phase-space of a system is constructed by embedding a single measured observable of the system “m” times, to represent the true dimensionality and dynamics of the system. A phase-space is a space that charts out all the possible states that the system can take. According to Takens theorem, if a system of interest comprises multiple interdependent variables that drive its dynamics but one has access only to a single observable dimension “x” of the system, then the multidimensional dynamics of that system can be reconstructed from the single measured dimension by plotting the observable “x” against itself a certain number of times at a certain time delay, “T”, obtaining a system vector “V.” This creates an “m” dimensional phase-space that maps out the trajectory of the system over time. Essentially, RQA uses a Recurrence Plot (RP) to display the dynamics of the phase-space of V by defining repetitions or “revisitations” of the system over time. A distance plot is first created by calculating the distance (here, Euclidean) in phase-space between every 2 possible states in the system. A point RPji in the RP is then considered recurrent if the distance between the point Vi(x) at time ti and the point Vj(x) at time tj falls within the user-defined threshold or radius in phase-space “T.” This can be represented as: where 00 is the Heaviside step function, and is 0 when x < 0 and 1 when x > 0. T is typically denoted relative to the maximum Euclidean distance norm in the corresponding phase-space.

[0143] A crucial step in this embedding process is to determine the parameters T, m, and T. Firstly, T, or the embedding delay can be approximated as the local minimum of an Average Mutual Information (AMI) function plotted between the original time series and the shifted time series at every lag. Secondly, m, or the embedding dimension refers to the number of surrogate copies of the data to be created at the estimated T and this can be approximated using a False Nearest Neighbours approach (FNN), to ensure we are not introducing “spurious” dimensions into the system. Lastly, a radius “T” in phase-space for distance-thresholding is typically chosen in a way that results in a sparse recurrence matrix with few recurrent points.

[0144] MdRQA is a multivariate extension of a simple RQA and it subjects the entire multidimensional system of interest (here, 2 dimensions - maternal and infant HRV over time) to a phase-space reconstruction through Takens time-delayed embedding. This facilitates exploring the revisitations in phase-space of the dyadic system as a whole, and provides key insights regarding the stability, chaoticity and complexity of the system as a single joint unit over time. The quantification of the resulting multidimensional RP provides descriptive measures, some of which are described below and used for predictive modelling:

[0145] In the context of the present implementation, the REC% of the system indicates how often the dyadic system revists the same state in the HRV phase-space together, the DET% tells us how periodic / regular the recurring patterns in the system are. Lastly maxL offers insights about the stability of the system - the longer the maxL, the longer the sequence of states that the system revisits at a particular point in time.

[0146] As opposed to MdRQA that looks at the dyadic HRV as a single unit in a single unified phase space, CRQA treats the two signals of interest independently, in their respective phase-spaces profiles. This becomes evident when comparing the equation below for the cross-recurrence plot (CRP) with equation (i) that represents the RP

[0147] Here, as in equation (i), T is the threshold parameter that controls the points that are counted as recurrent. However, the formula here ||Vt(x) - V / (y)|| represents the distance between a point Vi(x) in the phase space reconstructed with points from time series x, and a point Vi(y) reconstructed with the points from time series y.

[0148] A key feature of the CRQA is its potential to describe leader-follower relationships, since it looks at the signals of interest independently. Therefore, in addition to the features in the table above that can be extracted from CRPs as well, we are able to extract directional synchrony-related information. This can be done by summing across the upper triangular and lower triangular matrices of the RP and comparing them in magnitude. Since in each of the triangles, one signal consistently comes before the other in time, the triangle with the higher magnitude of recurrences can be used to derive conclusions regarding the person who “led” the synchrony.

[0149] While the above-mentioned procedures for computing the recurrence metrics seem rather straightforward, specific adaptations have been introduced to overcome challenges in dyadic datasets. Specifically, 1 ) different tasks have different durations (some task segments of interest are less than 1 minute long while some last for several minutes), thereby requiring tailored approaches. Furthermore, 2) for a given task, the duration can still differ between individual parent-infant dyads, simply due to the naturalistic nature of the task administration by the mother to the infant. Given these intricacies, applying the RQA technique and ensuring consistency and standardization across dyads for statistical comparisons, proves challenging. Some of these challenges are highlighted below, along with the mitigation approaches taken. Parameter selection and optimisation. Several studies in the HRV field have fixed the values for T and m, typically as T = 1 , m = 10 based on literature. Other studies have computed the embedding parameters using the AMI and FNN methods mentioned in above for each individual dataset, and then used the average / maximum values across all datasets for comparison. This falls within the understanding that moderately over-embedding a system does not affect the recurrence measures strongly. While this works for large datasets with many data points over time (hundreds or thousands), and where the length of the data is equal across all the subjects under comparison, it tends to fail in dyadic datasets, where neither of the above holds true. This is because the T and m values that can be used for a given data, depend highly on the length of the data. For instance, if the maximum or average T and m across all dyads are 5 and 10 respectively, for a dyad that only has 40 data points, this would mean that there is not enough information in order to create the 10-dimensional data required, thereby creating errors. Additionally, this also introduces errors during the grid search in batch-processing because the same upper limit for m cannot be used for all the datasets.

[0150] To address the above problem, the following approach is taken:

[0151] A. A calculation is carried out to determine the m for each dyad using the FNN method, while having a unique upper limit for the maximum embedding dimension possible for each dyad at the estimated T from AMI. For example, this can be done by following a simple equation below: where n is the length of the data, T is the delay estimated from AMI, and n’ is the user input number which corresponds to the number of data points still remaining in the original data, ensuring sufficient data for calculations.

[0152] B. The values of m and T are subsequently used for each dyad separately to reconstruct the phase-space as opposed to using a common average value. This could be explained based on the level of heterogeneity present across dyads in the parameter space. Additionally, each m is tied to a corresponding T value by theory, thereby justifying using these parameters together as opposed to using separately averaged values and potentially introducing errors in the embedding.

[0153] Optimal threshold / radius in phase-space. It is typically observed in literature that a radius is fixed across all subjects to produce a REC% within 1 -5%, in order to keep the recurrence matrix sparse. However, in this case, this produces a diverse range of REC% where some dyads have values lower than 1% while others have values as high as 20% for the same radius used. This heterogeneity in the sample could be present due to the naturalistic nature of the data and the level of noise present, in addition to individual differences. Therefore, in this study, the REC% is fixed at 5% and the radius across dyads changed to achieve this common REC%. This, of course, means that REC% was excluded from statistical comparisons since it does not vary across dyads.

[0154] Data length. The output recurrence measures are sensitive to the length of the data. Therefore, in this pipeline, all the output recurrence measures is normalised based on the length of the data for each dyad. While this is straightforward for the line-based measures (such as maxL) - whereby a dyad having 40 data points cannot possibly have a maxL equal to 50 as opposed to a dyad with 60 data points, it can also be argued for the rate-based measures (such as DET%) that when there is more data available, there could be more opportunities for recurring patterns to form in the phasespace, thereby inflating the results, if not dealt with properly.

[0155] In an implementation, the pipeline utilize the following open-source libraries and frameworks to perform the desired calculations, as listed below:

[0156] Python. Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. It has been used as a programming language where Machine Learning and Computer Vision algorithms are involved.

[0157] Anaconda. Anaconda is a free, open-source platform created for writing and executing code in Python. It has been used to create different developing environments. CUDA. CUDA is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs). With CUDA, developers can dramatically speed up computing applications by harnessing the power of GPUs.

[0158] PyTorch PyTorch is a deep learning framework that provides flexible tensor computation and automatic differentiation, allowing SMPLer-X to efficiently implement, train, and scale its Vision Transformer-based architecture for expressive pose and shape estimation. mmhuman3d(0penMMLab) mmhuman3d is a specialized 3D human modeling toolbox built on OpenMMLab, used by SMPLer-X for standardized dataset handling, 3D keypoint management, SMPL / SMPL-X model operations, and streamlined data formatting (HumanData format) critical for large-scale training.

[0159] Timm (PyTorch Image Models)

[0160] Timm is a popular PyTorch library offering a wide range of pre-implemented vision backbones, including Vision Transformers (ViTs), which are utilized in SMPLer-X to achieve high-accuracy feature extraction and to leverage state-of-the-art pretraining strategies for efficient model scaling.

[0161] The pipeline takes as its input, a video of Adult-Infant interaction. The considered recording typically involves a PCI (Parent-Child Interaction) task that is around 10 minutes long, with a .mp4 format and recorded at 25 / 29 frames per second. However, the video input may also feature other types of interactions, and have a different format, resolution, and fps than the above-mentioned. It should be noted that the higher the quality of the video (as in both technical and content terms), the greater the probability of accurate synchrony estimation. The method is shown in 1300, FIG.13.

[0162] The purpose of this section is to explain how the algorithm pipeline structure works and how it has been developed, providing details on how the various modules work together to calculate metrics of Pose synchrony. FIG.13 is a flowchart showing a workflow of a pose algorithm used in embodiment of the present invention.

[0163] The pose estimation pipeline leverages the SMPLer-X algorithm to analyze video recordings of parent-child interactions, 1302, enabling detailed estimates of movement synchrony and coordination. The system consists of two primary modules: the Key Points Extraction Module, 1304, and the Synchrony Metrics Computation Module, 1306. Each module plays a critical role in transforming raw video data into interpretable synchrony metrics.

[0164] The Key point Extraction Module, 1304, is built around the SMPLer-X model, a foundation model for expressive human pose and shape estimation that unifies the body, hands, and face capture in a single architecture. SMPLer-X uses Vision Transformer (ViT) backbones, including variants such as ViT-Small, Base, Large, and Huge, pretrained by ViTPose, to enhance its ability to model complex human motion data. The model is trained on a massive, diverse dataset composed of over 4.5 million instances collected from 32 datasets, ensuring robustness across unseen environments. SMPLer-X utilizes libraries such as PyTorch, OpenMMLab's mmhuman3d, and timm (PyTorch image models) to implement its core functionalities. The pipeline benefits from a minimalistic yet powerful architecture composed of a Vision Transformer backbone, a neck module for predicting bounding boxes and cropping regions of interest (for hands and face), and independent regression heads for the body, hands, and face parameters. This design avoids heavy decoders, external detectors, or multi-stage processing, emphasizing simplicity and scalability.

[0165] In the present context, SMPLer-X is utilized to accurately distinguish and track adults and infants in video recordings, process 1373D body key points per frame, and ensure robust tracking even under partial occlusion by inferring hidden joint positions. This capability enables a fine-grained understanding of adult-infant interaction dynamics.

[0166] The Synchrony Metrics Computation Module, 1306, analyzes time-series data from the extracted key points to compute metrics reflecting the dynamic interaction between parents and children. The primary analytical technique is windowed cross-correlation, complemented by non-linear synchrony indices. Windowed cross-correlation allows the examination of rapidly changing interactions by focusing on specific time segments, detecting synchrony patterns and leadership shifts, and providing insights into social and developmental dynamics such as mimicry and leadership in learning.

[0167] The processing workflow starts with the preparation of inputs, including pose detection key points from SMPLer-X, validation of sequence length, specification of window size, and frame rate standardization. Data loading and preparation involve identifying and merging continuous sequences containing both adult and infant detections to maintain interaction flow. Specific key points are then extracted to capture relevant features: pelvis and neck for posture, shoulders for gestural communication, wrists for expressive hand movements, and nose for head orientation. Windowed crosscorrelation is performed on these features to calculate summary indices such as the initiation of interactions (identifying the leader), changes of leadership, and synchrony stability across time. The computed synchrony features are compiled into CSV reports, including normalized metrics, 1308.

[0168] The integration of SMPLer-X for high-fidelity key point extraction with the Synchrony Metrics Computation Module for dynamic interaction analysis provides a comprehensive, scalable solution for studying parent-child interactions through video. The pipeline’s precision, scalability, and robustness enable nuanced insights into early human developmental and social dynamics, significantly advancing observational research in this field.

[0169] FIG.14 is a flowchart showing a workflow for audio analysis according to an embodiment of the present invention. The method 1500 shown in FIG.14 may be carried out by the dyadic parameter processing system 100 shown in FIG.2.

[0170] This section describes the estimation of conversational turn-taking features during parent-child dyadic interaction. This focuses on temporal (timing based) features such as verbal reaction times, number of vocalizations, number of conversations, and number of conversational turns.

[0171] In step 1502 audio data is extracted. In step 1504, speaker diarization is carried out. Speaker diarization models automatically identify “who said what,” and their annotations provide enough temporal data to analyze dyadic conversational turn-taking. Doing so allows for the calculation of the number of infant vocalizations and the calculation of infants' verbal response times to maternal vocalizations, which are useful metrics since slower and more varied response times can indicate future developmental outcomes, such as autism spectrum disorder. However, most general speaker diarization models do not fare well with infant vocalizations, such as crying or mumbling, by design, which requires extensive fine-tuning and retraining. The adopted fine-tuning process involved a set of 308 laboratory recordings, each ranging from 1 to 10 minutes. The recordings were sourced from two tasks: the Sequential Touching Task, where mothers demonstrated the compressibility of toy stimuli to their children, and Piaget’s classical A-not-B task. Maternal-infant dyads from Brazil (127 recordings) and Singapore (181 recordings) contributed a total of 22.5 hours of training data (Brazil: 11.5 hours; Singapore: 11 hours).

[0172] Following the speaker diarization, in step 1506, the annotated data is post processed. Prior to analyzing annotations, post-processing of speaker diarization outputs is crucial due to the presence of conversational gaps and pauses. Conversational gaps are brief silences (typically under 200ms) that occur when speakers change. Pauses occur within a speaker's turn, typically as they consider their next words. Their durations, which can range from 200ms to an extreme of 2 seconds, are subject to ongoing debate as pauses vary depending on speaker types and conversation context. In maternal-infant conversations, we limited pauses to one second because children (who are still learning to converse) may struggle to find the right words or how to vocalize them. Though often confused with conversational gaps, research shows that conversations with children tend to be more structured and formal. Adults or caregivers clearly signal turn-taking and indicate when the child should speak. As a result, gaps are longer because children follow these cues, listening without overlapping like adults do.

[0173] In step 1508, conversational turn analysis is carried out. MATLAB and R software were used to develop scripts that analyze dyadic mother-infant conversational turn-taking, focusing on deriving verbal reaction times, number of vocalizations, number of conversations, and conversational turns.

[0174] The logical framework used in this analysis proceeds in the following steps, summarized in FIG.15.

[0175] FIG.15 is a flowchart showing logical steps in conversational turn-taking analysis.

[0176] First, in this context, the mother is indicated as initiating the conversation (this may not always be the case). The scripted mother vocalizations 1602 establish a consistent starting point for each interaction, simplifying the process of identifying subsequent turns.

[0177] Once the conversation has started, the script determines when a conversational turn has ended by monitoring for pauses or gaps in speech. A conversational turn is considered to have ended if there is a silence from the speaker lasting 1 second or more.

[0178] Then a determination 1604 is made whether infant vocalizations occur within the response window. Vocalizations that occur within this 1 -second pause are viewed as a continuation of the original vocalization in our speaker diarization post-processing procedure. If there is no infant response, it is determined that the conversation initialization failed in step 1606 and the method returns to step 1602.

[0179] The analysis pipeline then searches for verbal responses from the listener in step 1608. This response window extends from 100ms after the start of the speaker’s vocalization to 2000ms after vocalization has ended. If no response occurs, then in step 1610 it is determined that the conversation has terminated and the method returns to step 1602. If a response is identified, the process then repeats, swapping the listener for the speaker and vice versa. However, if no vocalizations occur within 2 seconds of the last instance of vocalization, any subsequent vocalization from either the speaker or the listener will be considered the start of a new conversational turn. The temporal detail derived from identifying conversations and conversational turns were then used to compute the number of vocalizations and verbal response times of the replies of both infant and mother.

[0180] Intraclass Correlation Coefficient (ICC) reliability analyses were conducted on vocalization count and verbal response times of both mothers and their infants, between manually-annotated vocalizations (gold standard) and the model’s output. In cases where the model performed well, ICC scores exceeded 0.85 for both infants and mothers in terms of number of vocalizations, as well as infant’s verbal response times. Maternal verbal response times were slightly lower, with an ICC score of 0.67.

[0181] In the following description an example of the training of the machine learning classifier 142 by the training module 130 of the development assessment system will be described.

[0182] Examples of tests used with the dyadic parameter processing system are as follows:

[0183] The Sequential Touching Object Categorization (STOC) task assesses infant attention-set shifting (a component of cognitive flexibility) using toys that differ in shape or material. The specific innovation is that the parent provides a demonstration to the child of a toy’s compressibility (soft or hard material) using a range of gaze, speech, postural and emotional cues. The task is used to assess how differences in these social cues affect infant’s later demonstration of attention shifting skills.

[0184] The A-not-B task is a measure of infant memory, inhibitory control and shifting. In this hide-and-seek game, the mother repeatedly hides a toy object at location A until the child learns to search for the toy successfully at this location. Then, she switches the hiding location to B instead of A. Infants’ ability to inhibit searching at the previous (A) location and switch to the new (B) location is a measure of emerging EF. The task is adapted to be delivered by the parent (instead of an experimenter), and social cues (gaze, speech, emotion, posture) are measured during different phases of the task delivery as predictors of infant performance. Data were collected from three locations: Singapore, Brazil, and Cambridge. Each child performed two tasks: the STOC (Sequential Touching Object Categorization Test) and the A-not-B task. A methodology was developed to create a classifier capable of predicting infants' performance on these EF tasks with high accuracy, based on the gathered dyadic sociometrics. This proposed methodology utilizes a combination of standard statistical methods, machine learning algorithms, and novel mathematical modeling techniques, resulting in high accuracies for the classifier. Lastly, in addition to the STT and AnB metrics, the methodology was also applied to standardized tests such as the Bayley and IBQ (Infant Behavior Questionnaire).

[0185] Maternal mood data were collected using standardized and validated self-report questionnaires to assess symptoms of depression and anxiety during and after pregnancy. For maternal depression, the Edinburgh Postnatal Depression Scale (EPDS) was administered across both the Singapore and Brazil cohorts. To evaluate maternal anxiety, different tools were used based on cohort: the State-Trait Anxiety Inventory (STAI) for the Singapore cohort, and the Generalized Anxiety Disorder scale (GAD) for the Brazil cohort. In the merged dataset, responses from each cohort were analyzed independently to ensure that high-anxiety classifications accurately reflected elevated anxiety scores within the context of each specific instrument. These maternal mood questionnaires are widely recognized in both clinical and research settings as screening tools for monitoring psychological well-being and identifying mood disturbances. They provide valuable insights into the emotional state of mothers during the perinatal period. Here we leverage the data obtained from these instruments to predict the risk class of each individual. The objective is to identify those at higher risk of experiencing adverse mental health outcomes. By analyzing response patterns, our predictive models stratify participants into risk categories, thereby enabling more targeted intervention efforts and efficient allocation of mental health resources and support.

[0186] The available metrics from each location, as well as the features and the number of data points collected, are summarized in the table below.

[0187] Overall, the available data is largely sufficient for the classifier to achieve the needed accuracy across all scenarios.

[0188] The data acquired can be split into three categories:

[0189] 1 ) Demographics: Age and Sex

[0190] 2) Behavioral Measures: Gaze, Pose, Emotion, Audio

[0191] 3) Neurophysiological Measures: EEG, ECG

[0192] Each of the above datasets (such as Gaze, or ECG) are to be referred to as domains. Different combinations of the domains are to be used in order to acquire the best classifiers for the prediction of EF performance.

[0193] When considering the EF tasks, different performance measures are employed in the AnB and STOC datasets, though investigations have demonstrated these measures to be statistically similar.

[0194] For the STOC dataset, the Shift measure was selected as the metric for assessing the EF abilities of the infants. For the AnB dataset, a comparable metric was used. However, due to differences in the experiments, the ranges of these metrics varied, even though they are expected to quantify the same EF measure. To ensure consistency across different tasks, the Shift metric was normalized per task. This normalization allows us to define classes based on the percentages within which an infant's EF score falls — the lower the normalized number, the better the EF score of the subject.

[0195] FIG.16 shows validation of dependent variables across tasks. FIG.16 demonstrates that the two metrics are highly correlated, with a Pearson coefficient of 0.452. When plotted together, the outcomes for AnB and STOC tasks, measured by the Shift variable, show a correlation between each other. This indicates that the outcome variables are psychologically consistent across the different tasks of the experiment.

[0196] The objective is to categorize subjects into three distinct classes based on their EF metrics. Given that the data distributions closely approximate a normal distribution, it would be impractical to simply divide the dataset into three equal parts. Consequently, it was necessary to determine how the division should be made to ensure it is representative and that all classes contain sufficient data for the effective training of classifiers.

[0197] FIG.17 displays histograms of the EF scores from two locations, illustrating their resemblance to a normal distribution. Notably, the data from Brazil show a limited number of subjects in the high EF performance category (represented by lower numbers on the histogram). If the class boundaries were to be set at 0-15-85-100 percentiles, this would result in only four subjects in the high-performance class of the Brazil dataset. Statistically, four data points in a dataset of 128 can be considered outliers, making it challenging to construct a classifier that accurately represents these data. This situation underlines the insufficiency of a simplistic approach to class division for classifier training. The table below details the number of data points in each group based on the proposed class split for each location. Given the above, it can be seen that a realistic split for the classes is the 0-30%, 30- 85%, 85-100% class option.

[0198] After the classes have been defined, then the data from different domains need to be merged in order to address any missing values per participant. Ideally, all the participants have values for all their domains, but there are times when some values may be missing. There are two options to address this issue. Firstly, by domain check, meaning that if any one domain has more than a certain threshold percentage of its values missing, then the participant is eliminated. And the second option is a crossdomain check, where a participant is eliminated only if missing values exceed threshold after all the domains have been concatenated. In the second option, crossdomain imputation can be performed. If a smaller (in number of measurements) domain -like the pose- is missing, then these values can be populated by imputation through the rest of the domains. This leads to about 30% higher retention rates for the participants and is especially helpful in the overall performance of the model. The imputation is performed after the data is split and the final training set for its iteration is determined.

[0199] Next, it is necessary to define the features to be used for the classifiers.

[0200] In the table below and example of the possible number of features in each domain is presented. It may be observed that the number of features can be very high. For smaller datasets, the number of features needs to be reduced in order for the classification process to be viable.

[0201] Different options are available for feature reduction. In this exemplary embodiment the option chosen was Principal Component Analysis (PCA).

[0202] Briefly explained, the PCA method consists of the following mathematical steps:

[0203] Step 1 - Data standardization Xl} with p being the mean value and o the standard deviation.

[0204] Step 2 - Calculation of the convolution matrix: cov

[0205] Step 3 - Eigenvector and Eigenvalue extraction: cov(X) v - Av with v being the eigenvector and A being the eigenvalue of the matrix.

[0206] Step 4 - Choice of number of principal components

[0207] Having concluded the PCA process we investigate how much of the original information of the system is retained for each domain, given a number of components.

[0208] The information retained is shown in the table below and in FIG.18 and FIG.19.

[0209] FIG.18 shows variance of a dataset described against the number of PCA features for EEG and ECG. FIG.19 shows variance of a dataset described against the number of PCA features for Gaze, Audio, and Emotion.

[0210] Through this method it is observed that we can reduce 200+ features to 3 or less features per domain while keeping a significant amount of the information. This is an important step of the process as now we are able to describe the bulk of our data with only 14 features.

[0211] For the Pose domain PCA was not considered to be the ideal choice due to the low number of features. In addition, it was noticed that the variance of all 4 of the Pose features is similar. As such it was decided that taking the L2 norm of the separate features would be sufficient to describe their overall behavior. This has been later verified given the fact the Pose is one of the more often picked domains for the classifier based on its performance.

[0212] The aim behind the algorithm is the creation of a model that will be trained on a part of the dataset and predict another part. In this exemplary embodiment, the Singapore dataset is used for training to predict the Brazil dataset. Where available, the Cambridge is used together with the Singapore dataset for training, but a different permutation is also established where the Brazil and Singapore datasets are used to predict the Cambridge dataset. This cross-country prediction exercise is conducted to demonstrate the generalizability of the predictive models.

[0213] Whilst all the datasets are made to be as cross compatible as possible with one another, some differences still exist which may arise from technical or cultural factors.

[0214] To mitigate this issue, it was decided to incorporate a portion of the target dataset into the training set. This inclusion is carefully managed to exclude this portion from the testing set, ensuring the model is tested on completely unseen data.

[0215] Class population.

[0216] As seen through the table above showing class splits, even with allocating 30% of the range to the high class still the numbers of high class subject in the Brazil dataset are very limited. Also, the fact that part of the Brazil dataset is utilized for training leads to the testing dataset to at some points being left without any high class subjects. This is very problematic as the classifiers degenerates to a 2-class model, an outcome that is undesirable. To circumvent this, a "safety set" is implemented in the testing dataset to guarantee the presence of high-class subjects by allocating more than half of the subjects from all three classes to the testing set.

[0217] The proportion of Brazil data that are utilized in the training dataset is a problem that is dealt with an unsupervised approach, with the code picking the optimal ratio. Limits are set for the lowest and highest usage of the Brazil dataset in the training. The lowest limit is set to 30% and the highest to 90%. These amounts do not include the safety set data. Consequently, even in scenarios where 90% of the Brazil dataset is utilized for training, the testing set is ensured to have a minimum of 13 subjects in the worstcase scenario. This scenario includes the usage of all domains and only 10% plus the safety set to be included in the final testing set. The average numbers of the testing dataset are around 20 with all the domains used and 25-30 on average when less domains are utilized.

[0218] The amount of the Brazil dataset used in the training significantly affects the overall accuracy of the model as it can be seen through FIG.20.

[0219] FIG.20 is a graph showing accuracy vs ratio of Brazil data used in training, based on behavioral only domains as inputs. Overall, while a higher percentage of the Brazil dataset used in training significantly improves the accuracy of the classifier, it is important especially when dealing with high ratios for the safety set to exist in order for the classifier not to regress to a two-class model.

[0220] Given the disproportional number of subjects of each class it is more difficult for the classifier to predict successfully the low and especially the high class. For this reason, the Synthetic Minority Oversampling Technique (SMOTE) process can be used to create a dataset that has equal numbers of all the classes. The way that SMOTE operates is that a multivariate distribution is fitted on the data and based on this distribution additional data is generated in order for all classes to have an equinumerous population. When implementing SMOTE, two critical factors must be considered:

[0221] 1 . Representativeness of the Distribution: If a class has very few data points, the multivariate distribution based on this limited data may not be representative. Consequently, the generated synthetic data may not accurately reflect real- world observations. This issue is mitigated by ensuring a minimum 30% split for high-class subjects, providing a more stable foundation for distribution fitting and synthetic data generation.

[0222] 2. Application to Training Data Only: While SMOTE enhances the training process by balancing class distribution, applying it to the testing set could result in misleading outcomes. To avoid this, SMOTE is exclusively applied to the training data, with the model evaluated against real, unaltered test data to ensure the validity of its predictive performance.

[0223] Overall, as it will be seen through the results section SMOTE is a method that when utilized leads to a noticeable improvement in the outputs of the model.

[0224] The table below shows an example of SMOTE utilization.

[0225] Having now established the dataset that is to be used for the classifiers, given the different allocation of subjects to training and testing sets the classification part of the model can be applied. The classification algorithms that may be used include:

[0226] • Random Forest

[0227] • Linear Discriminant Analysis

[0228] • Quadratic Discriminant Analysis

[0229] • Support Vector Machine

[0230] • Adaboost

[0231] • K-Nearest Neighbors

[0232] • Logistic Regression

[0233] In every run all of the algorithms are tried on the dataset and the best performing on (based on accuracy) is kept. Therefore, by running all the algorithms we can find which algorithm is the most effective for the dataset.

[0234] FIG.21 shows a comparison of the number of times each classifier model is picked by the algorithm as the best performing. Analysis of FIG.21 reveals that the Random Forest classifier emerges as the preferred choice, being selected more frequently than any other model in our evaluations. However, it is important to also acknowledge the performance of the Adaboost model. Despite its lower overall success rate, Adaboost consistently ranks as the second most effective model following Random Forest. This observation can be attributed to the fact that both Random Forest and Adaboost are ensemble models. Their comparable behaviors are expected due to their underlying mechanisms, which aggregate multiple models' predictions to form a more accurate and robust prediction.

[0235] The classifiers generally have access to all of the features available from the different domains. That said increasing the number of features the code has access to increases the computational time and also the number of coefficients the model must find. This leads to the fact that when all the features are made available the performance of the model is poor, as the model has to find more coefficients than the data it has. Therefore, the number of features to be input to the code needed to be investigated. To address this, experiments were conducted with varying domains and feature counts to identify the most effective combinations for model training. Initially, the algorithm's performance was evaluated across all domains with varying numbers of features.

[0236] FIG.22 is a graph showing feature availability vs Accuracy and Number of Combinations. From FIG.22 it can be seen that the accuracy increases with the increase in the available features plateauing after 4 features. Also, with the increase in the available features the combinatorics lead to a much greater number of runs needed for checking all the possible combinations and therefore the amount of calculation time significantly increases. Based on the graph, it is observed that the plateau in the accuracies as well as the almost exponential increase in the amount of combinations, is broadly analogous to the computational time. Based on FIG.22 it is understood that in order to achieve sufficient accuracy and keep the computational time low, 4 available features are the ideal number in this exemplary embodiment. Therefore, in the given model 4 features are available to be used out of the 14 in total.

[0237] FIG.23 is a graph showing the combination of domains and accuracy values.

[0238] In addition to the number of features available it is important to understand and quantify the effect of each domain. For this reason, the accuracy of each domain and combinations of domains are checked. In FIG.23 the accuracy vs domains is presented, with 4 available features to be picked.

[0239] Based on FIG.23 it is seen that the number of available domains has a significant effect on the behavior of the classifier outcome. The best result is when all domains are available and therefore the model is left unsupervised to pick up the best features from all the domains to achieve the best accuracy.

[0240] In this section the results of the model are to be presented. The results are presented based on the different available domains, given the available measurements.

[0241] Behavioral Data In this run of the model the Singapore data and parts of Brazil data were utilized to train the model and then test against the rest of the Brazil dataset.

[0242] The safety set was used, SMOTE was used in the SG data, SMOTE was not used in the Brazil part of the training set.

[0243] Accuracy class F1 -score Precision Recall high 0 647 0 92 0.53

[0244] 0.84 medium 0.904 0.83 0.99 low 0.596 0.94 0.61

[0245] Based on the results of the above table it can be seen that even with only the behavioral measures the model is able to achieve an accuracy of above 80%. In this model the minimum number of available subjects for testing is 25 (10% plus safety set) with the average being 32 (26% + safety set).

[0246] Behavioural + ECG measures

[0247] Similarly, the Singapore data and parts of Brazil data were utilized to train the model and then test against the rest of the Brazil dataset.

[0248] The safety set was used, SMOTE was used in the SG data, SMOTE was not used in the Brazil part of the training set.

[0249] As it can be seen from the classification report in the above table the behavior of the classifier is augmented with the addition of the ECG dataset. We observe an improvement in both the accuracy and the homogeneity of the precision and recall scores. Behavioral + ECG + EEG measures.

[0250] Again, the Singapore data and parts of the Brazil dataset were utilized to train the model and then test against the rest of the Brazil dataset.

[0251] The safety set was used, SMOTE was used in the SG data, SMOTE was not used in the Brazil part of the training set.

[0252] From the above table it can be seen that the behavior of the classifier when both ECG and EEG are used is much improved offering greater accuracy and almost completely homogenous F1 , Recall and Precision. In this case the minimum number of test subjects is 13 (10% plus safety set) and the average is 19 (18% plus safety test).

[0253] In addition to the datasets from Brazil and Singapore, we also have measurements for the Cambridge dataset. This dataset was investigated both as a target set and as a training set. When using the Cambridge dataset only the Behavioral measures can be used as ECG and EEG measurements are not available for this dataset.

[0254] Singapore + Cambridge + (part)Brazil Brazil

[0255] In this case the same method as the Brazil and Singapore analysis was used for the classifier. The results are very comparable and within the margin of error. The minimum possible number of test subjects is 25 (10% + safety set) and the average is 34 (28% + safety set).

[0256] Results - 2 class classifiers LDESS

[0257] In addition to the 3 class model the behavior of the system using a 2-class classifier was also investigated. This was done in order to be able to capture better prediction of the lower class. Again, different runs are done for different domains available.

[0258] As per the class definition the original split for the lower class is retained, being 0-85% for the high class and 85-100% for the low class.

[0259] In this case, similar to the 3 class classifier, we are training on Singapore data and part of the Brazil data and testing in the rest of the Brazil dataset.

[0260] Behavioral only

[0261] From the results of the above table it is seen that the behavior of the 2 class classifier is a significant improvement compared to the 3 class model. This is something that is to be expected and even with only the behavioral measures we are achieving accuracies higher than 90%.

[0262] Behavioral + ECG

[0263] In this run the algorithm has access to all the domains. With the addition of ECG metrics it is seen that the outcomes of the model are significantly improved. The accuracy is 97% and all the scores are homogeneous.

[0264] Analysis on Bayley and IBQ data

[0265] In addition to the prediction of EF performance, the described method can also be utilized for the prediction of the Bayley and IBQ scores. Since not all subjects have measurements for IBQ and Bayley the number of used subjects is smaller in size.

[0266] Bayley dataset. The Bayley dataset contains 4 target variables (developmental domains):

[0267] • Cognition

[0268] • Language

[0269] • Motor

[0270] • Socioemotional

[0271] Total number of subjects in Bayley dataset: 95

[0272] IBQ dataset. The IBQ dataset contains 3 target variables (temperament constructs):

[0273] • Surgency

[0274] • Negative

[0275] • Regulation

[0276] Total number of subjects in IBQ dataset: 97

[0277] The usable number of datapoints is smaller because not all of the subjects have measurements for all the sociometric domains. Due to the reduced number of subjects a 2 class classifier is applied.

[0278] Class definitions. Given the reduced number of subjects the 2 classes that are defined need to have sufficient subjects. Therefore the split used is:

[0279] 1 . Low class: performance <30%

[0280] 2. High class: performance >30%

[0281] Using only the behavioral features of the AnB dataset we apply the previously defined classification model with minor modifications. The changes in the classifier are that 1 ) SMOTE is not used, as SMOTE is not helpful in cases with low datapoints. 2) the code is modified to work as a 2 class classifier.

[0282] Results using the AnB task

[0283] I BQ results.

[0284] Behavioral only measures

[0285] First, the classifier is used with behavioral metrics only, it is seen that the accuracy is generally high even using behavioral measures only. Behavioral + ECG

[0286] Given the behavioral and the ECG variables it is seen that the accuracy is higher with the precision remaining very high, while the recall values in the cases of the Surgency and the Regulation could be improved.

[0287] Behavioral + EEG + ECG

[0288] Through the above tables it can be seen that when predicting IBQ scores, the behavior of the classifier is very good for surgency and negative affective but lower for regulation.

[0289] Bayley results

[0290] Behavioral only measures As it can be observed from the above tables the results in the Socioemotional domain leave little room for improvement.

[0291] Behavioral + ECG

[0292] Through the above tables is seen that the inclusion of ECG metrics significantly improves prediction of Motor and Language scores. Behavioral + ECG + EEG

[0293] With the addition of EEG metrics in the classifier, perfect or near perfect prediction is achieved in the Socioemotional, Motor and Language domains, though performance for Cognition prediction are less robust.

[0294] Results using the STOC task

[0295] I BQ results Behavioral only measures

[0296] With Behavioral only measures, prediction of IBQ scores is generally high. However, whilst surgency was better predicted by the performance was slightly better for the AnB dataset, regulation and negative affect were better predicted by STOC data.

[0297] Behavioral + ECG measures

[0298] The same behavior as with the behavioral only values is observed when ECG metrics are added. Similarly through the tables below the performance of the classifiers is generally improved by the addition of the EEG metrics.

[0299] Behavioral + ECG + EEG measures

[0300] Bayley dataset results

[0301] Behavioral-only Measures

[0302] By observing the above tables it is seen that the classification reports based on STOC data are similar to those for AnB data, with prediction of Socioemotional and the Motor scores being particularly good.

[0303] Behavioral + ECG

[0304] Behavioral + ECG + EEG

[0305] Viewing the above tables, an increase in the performance of the classifier is observed with the addition of the ECG and EEG measures respectively. This increase though is incremental, suggesting that the complexity of gathering ECG and EEG data must be carefully weighed against expected gains in prediction performance.

[0306] Maternal Mood Results Maternal Anxiety The maternal anxiety metrics are based on the STAI questionnaire for Singapore and the GAD questionnaire for Brazil. The high anxiety class for the Brazil cohort corresponds to GAD scores greater or equal to 15. For the Singapore cohort the high anxiety class corresponds to STAI scores greater or equal to 51 . These values arise by splitting the individual datasets to a 0-70% and 70-100% ranges, but are also in line with the clinical interpretation of the questionnaire reports.

[0307] The available dataset for maternal anxiety metrics from the STOC task is:

[0308] For the AnB dataset the anxiety metrics are:

[0309] Maternal Depression

[0310] The maternal depression metrics are based on the EPDS questionnaire for both Singapore and Brazil. The high depression class corresponds to EPDS scores greater or equal to 14.

[0311] The available dataset for maternal depression metrics from the STOC task is:

[0312] For the AnB dataset the depression metrics are: Based on the same classifier now adjusted for 2 class prediction the results are:

[0313] Predictive Modelling Results : Anxiety

[0314] The results of the model are presented based on the different available domains.

[0315] ECG + Pose + Emotion

[0316] AnB data

[0317] STOC

[0318] ECG + Pose + Emotion + Audio + Gaze

[0319] AnB data

[0320] STOC

[0321] ECG + Pose + Emotion + Audio + Gaze +EEG AnB data

[0322] STOC Predictive Modelling Results: Depression

[0323] The results of the model are presented based on the different available domains.

[0324] ECG + Pose + Emotion

[0325] AnB data

[0326] STOC ECG + Pose + Emotion + Audio + Gaze

[0327] AnB data

[0328] STOC

[0329] ECG + Pose + Emotion + Audio + Gaze +EEG

[0330] AnB data

[0331] STOC In these runs of the model, the Singapore data and parts of Brazil data were utilized to train the model and then the model was tested on the rest of the Brazil dataset. The safety set was used, and SMOTE was used on the training set.

[0332] The models achieve up to 96% accuracy in predicting child executive function outcomes, as well as maternal anxiety and depression, surpassing the reliability of self-reported assessments. Given the close relationship between maternal mental health and child development, timely identification and support of maternal mood disorders can positively impact both mother and child. Together, these innovations provide a comprehensive, data-driven framework for early intervention in both child development and maternal mental well-being.

[0333] Whilst the foregoing description has described exemplary embodiments, it will be understood by those skilled in the art that many variations of the embodiments can be made within the scope and spirit of the present invention.

Claims

1. CLAIMS1. A machine learning method of assessing development of a child subject, the method comprising: receiving input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for the child subject and a time series of sensed data indicative of the parameter for an adult subject collected during interactions between the child subject and the adult subject; synchronizing the dyadic parameter data; generating a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classifying the development of the child subject by inputting the set of features into a machine learning classifier trained to classify development of a child based on the set of features; and outputting a prediction indicating the classification or description of the development of the child subject.

2. The method according to claim 1 , wherein the machine learning classifier is further trained to classify a maternal mood of the adult subject based on the set of features, the method further comprises: classifying the maternal mood of the adult subject by inputting the set of features into the machine learning classifier, and the prediction indication further indicates the classification of the maternal mood of the adult subject.

3. A machine learning method of assessing maternal mood of an adult subject, method comprising: receiving input data comprising dyadic parameter data, the dyadic, parameter data comprising a time series of sensed data indicative of a parameter for a child subject and a time series of sensed data indicative of the parameter for the adult subject collected during interactions between the child subject and the adult subject; synchronizing the dyadic parameter data;generating a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classifying the maternal mood of the adult subject by inputting the set of features into a machine learning classifier trained to classify the maternal mood of the adult subject based on the set of features; and outputting a prediction indicating the classification or description of the maternal mood of the adult subject.

4. The method according to any preceding claim, wherein the dyadic parameter data comprises one or more domains of behavioral parameter data and / or a physiological parameter data.

5. The method according to any preceding claim, wherein the dyadic parameter data comprises electroencephalogram data and wherein the combination of non-linear and linear features are determined from non-oscillatory and oscillatory features of the signal respectively.

6. The method according to claim 5, wherein the non-oscillatory and oscillatory metrics comprise weighted Symbolic Mutual Information and weighted Phase Lag Index respectively.

7. The method according to any preceding claim, wherein the dyadic parameter data comprises electroencephalogram data and video data and wherein the video data is used to identify portions of interest in the electroencephalogram data.

8. The method according to any preceding claim, wherein the dyadic parameter data comprises electrocardiogram data, and the method further comprises preprocessing the electrocardiogram data to re-sample electrocardiogram data for the child subject and electrocardiogram data for the adult subject such that re-sampled electrocardiogram data for the child subject and re-sampled electrocardiogram data for the adult subject are equidistantly sampled in time.

9. The method according to any preceding claim, wherein the dyadic parameter data comprises video data and the method further comprises detecting pose data of the child subject and pose data of the adult subject and analyzing linear or non-linear contingencies between the pose data of the child subject and the pose data of the adult subject.

10. The method according to claim 9, wherein analyzing non-linear contingencies between the pose data of the child subject and the pose data of the adult subject comprises applying multi-dimensional recurrence quantification analysis.

11. The method according to any preceding claim, wherein synchronizing the dyadic parameter data comprises aligning synchronization signals in the dyadic parameter data.

12. A computer readable medium carrying processor executable instructions which when executed on a processor cause the processor to carry out a method according to any one of claim 1 to 11 .

13. A system for assessing development of a child subject, the system comprising: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for the child subject and a time series of sensed data indicative of the parameter for an adult subject collected during interactions between the child subject and the adult subject; synchronize the dyadic parameter data; generate a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classify the development of the child subject by inputting the set of features into a machine learning classifier trained to classify development of a child based on the set of features; and output a prediction indicating the classification or description of the development of the child subject.

14. The system according to claim 13, wherein the machine learning classifier is further trained to classify a maternal mood of the adult subject based on the set of features, and the data storage device further stores computer program instructions operable to cause the processor to classify the maternal mood of the adult subject by inputting the set of features into the machine learning classifier, and the prediction indication further indicates the classification of the maternal mood of the adult subject.

15. A system for assessing maternal mood of an adult subject, the system comprising: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive input data comprising dyadic parameter data, the dyadic parameter data comprising a time series of sensed data indicative of a parameter for a child subject and a time series of sensed data indicative of the parameter for the adult subject collected during interactions between the child subject and the adult subject; synchronize the dyadic parameter data; generate a set of features indicative of temporal contingencies between the parameter of the child subject and the parameter of the adult subject, wherein the set of features comprises a combination of non-linear features and linear features; classify the maternal mood of the adult subject by inputting the set of features into a machine learning classifier trained to classify the maternal mood of the adult subject based on the set of features; and output a prediction indicating the classification or description of the maternal mood of the adult subject.

16. The system according to any one of claims 13 to 15, further comprising a plurality of sensors configured to capture the dyadic parameter data.

17. The system according to claim 16, further comprising a trigger box configured to generate a synchronization signal on each sensor of the plurality of sensors and wherein the data storage device further stores computer program instructions operableto cause the processor to: synchronize the dyadic parameter data using the synchronization signals.

18. The system according any one of claims 13 to 17, wherein the dyadic parameter data comprises one or more domains of behavioral parameter data and / or a physiological parameter data.

19. The system according any one of claims 13 to 18, wherein the dyadic parameter data comprises electroencephalogram data and wherein the combination of non-linear and linear features are determined from non-oscillatory and oscillatory metrics respectively.

20. The system according to claim 19, wherein the non-oscillatory and oscillatory metrics comprise weighted Symbolic Mutual Information and weighted Phase Lag Index respectively.21 . The system according any one of claims 13 to 20, wherein the dyadic parameter data comprises electroencephalogram data and video data and wherein the video data is used to identify portions of interest in the electroencephalogram data.

22. The system according any one of claims 13 to 21 , wherein the dyadic parameter data comprises electrocardiogram data, and the data storage device further stores computer program instructions operable to cause the processor to pre-process the electrocardiogram data to re-sample electrocardiogram data for the child subject and electrocardiogram data for the adult subject such that re-sampled electrocardiogram data for the child subject and re-sampled electrocardiogram data for the adult subject are equidistantly sampled in time.

23. The system according any one of claims 13 to 22, wherein the dyadic, parameter data comprises video data and the data storage device further stores computer program instructions operable to cause the processor to detect pose data of the child subject and pose data of the adult subject and analyze linear or non-linear contingencies between the pose data of the child subject and the pose data of the adult.

24. The system according to claim 23, wherein the data storage device further stores computer program instructions operable to cause the processor to analyze nonlinear contingencies between the pose data of the child subject and the pose data of the adult subject by applying multi-dimensional recurrence quantification analysis.

Images