Infant neural development state analysis method and system

Abstract

The invention discloses an infant neural development state analysis method and system, and belongs to the technical field of computer vision, and the method comprises the following steps: S1, data collection preparation; s2, collecting active motion data of the whole body; s3, collecting local induced motion data; and S4, multi-motion feature extraction and fusion decision making. On the basis of whole body spontaneous motion data acquisition and analysis, local motion data acquisition and analysis induced by visual and auditory sense organs are added, multi-modal behavioral features are obtained, and clinical classification accuracy is improved through later-stage fusion decision making of the modal features.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for analyzing the neural development status of an infant. Background Art

[0002] If infants (especially those in the early stages of infants, within 6 months) are harmed by high-risk factors such as premature birth, low body weight, asphyxia, hypoxic-ischemic encephalopathy, and intracranial hemorrhage, they may suffer from abnormal or disordered neurodevelopment, which in turn leads to cerebral palsy, intellectual disability, autism and other outcomes. With the improvement of medical standards, the survival rate of high-risk infants has continued to increase. How to reduce the incidence of neurodevelopmental status and reduce the degree of disability in high-risk infants has received increasing attention. At present, clinicians generally evaluate infant brain damage and clinical rehabilitation efficacy based on cranial ultrasound, MRI, CT, brainstem auditory evoked potentials, biochemical indicators and various scoring scales. However, imaging and electrophysiological assessments, as more complex technical assessments, require specific equipment. The sensitivity, specificity and accuracy of traditional evaluation scales in predicting neurological development outcomes vary greatly, and the diagnosis time is mostly after the child is one or even two years old. General movements (GMs) are spontaneous movements that exist from the early fetal period to about 20 weeks after full term. They last from a few seconds to a few minutes and are divided into normal movements and abnormal movements. GMs assessment can sensitively indicate specific neural damage, make reliable early predictions of neurological developmental disorders such as cerebral palsy and autism, and provide rehabilitation intervention as early as possible to improve their prognosis and reverse adverse neurodevelopmental outcomes, which has important clinical and social significance.

[0003] At present, GMs can only be visually assessed by trained clinicians with assessment licenses. These clinicians need a lot of training and years of practical assessment experience to achieve appropriate accuracy, so assessors are very scarce. This assessment method requires clinicians to observe GMs for a long time, which is easily affected by observer fatigue, making it difficult to conduct objective and quantitative assessments, affecting diagnostic accuracy and easily causing misdiagnosis and missed diagnosis. The manual and time-consuming nature of GMs assessment and the scarcity of assessors mean that GMs testing is currently only used in cases where there are medical problems, such as premature birth, hypoxia or congenital heart disease, and is not used as a physical examination screening tool for ordinary infants. To conduct a large number of infant physical examination screenings, automated quantitative assessment of GMs is required. Existing automated quantitative assessment methods based on GMs use position sensors and accelerometers attached to the limbs to assess the spontaneous movement of the limbs during GMs, analyze the periodicity of speed and acceleration in spontaneous movement, and use the extracted features to diagnose movement disorders. However, the above studies attach sensors or markers to infants, which interferes with the spontaneous occurrence of movement, and it is not easy to attach sensors to infants. Therefore, a method for analyzing the neurodevelopmental status of infants was proposed. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to analyze the neurodevelopmental status of an infant more accurately and conveniently, and provides a method for analyzing the neurodevelopmental status of an infant.

[0005] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0006] S1: Data collection preparation

[0007] Place the baby on the test bench, adjust the relative position between the side bracket and the test bench, so that the normal direction of the camera on the display module is perpendicular to the test bench and the field of view covers the whole body of the baby, and fix the audio module on the left and right sides of the baby's head on the test bench;

[0008] S2: Whole body active motion data collection

[0009] Use cameras to collect data on the baby's full body active movements;

[0010] S3: Locally evoked motion data acquisition

[0011] Using a display module and / or an audio module in conjunction with a camera to collect local induced movement data of the infant;

[0012] S4: Multi-motion feature extraction and fusion decision

[0013] Features are extracted according to the movement paradigm, and the first classification sub-decision and the second classification sub-decision are formed through feature layer fusion. The weighted average of the sub-decisions is used to obtain the final decision result, and then the neurodevelopmental status of the infant is analyzed and evaluated.

[0014] Furthermore, in step S2, the specific process is as follows:

[0015] S21: The computer controls the camera to detect the active whole-body movement of the infant in a natural state, and adjusts the relative position of the camera and the test bench so that the video contains the complete limbs and head information of the infant;

[0016] S22: Using a human posture estimation model to identify the video, identify key points of the infant's limbs, and extract motion feature data of the key points of the limbs.

[0017] Furthermore, in step S3, the specific process is as follows:

[0018] Step S31: The computer controls the display module to present a light spot and make a high-speed and large-amplitude horizontal movement within the display range. At the same time, the camera detects the head rotation posture of the infant during the visual following process, uses the human body posture estimation model to identify the key points of the infant's head, and extracts the motion feature data of the key points of the head;

[0019] Step S32: The computer controls the display module to present the light spot and make a low-speed and small-amplitude horizontal movement within the display range, while the camera detects the eye movement of the infant during the visual following process to obtain the movement trajectory of the infant's gaze point under the visual induction of the light spot;

[0020] Step S33: the computer controls the sound module to emit continuous sounds on both sides of the baby's head, and at the same time, the camera detects the head rotation posture of the baby during the auditory following process, uses the human body posture estimation model to identify the key points of the baby's head, and extracts the motion feature data of the key points of the head;

[0021] Step S34: The computer controls the display module and the audio module to stimulate synchronously, the display module presents a light spot and moves horizontally at a high speed and large amplitude within the display range, and the audio module emits a sound in the direction of the end point of the light spot movement. The camera detects the head rotation posture of the infant in the process of visual and auditory joint following, and uses the human body posture estimation model to identify the key points of the infant's head, and extract the above-mentioned head key point movement feature data;

[0022] Step S35: The computer controls the display module and the audio module to perform joint synchronous stimulation, the display module presents a light spot and moves horizontally at a low speed and a small amplitude within the display range, and at the same time, the audio module emits a sound in the direction of the end point of the light spot movement, and the camera detects the eye movement and pupil changes of the infant during the visual following process, and obtains the gaze point movement trajectory and pupil size under the joint induction of light spot vision and hearing;

[0023] Step S36: The computer controls the display module and the audio module to perform asynchronous stimulation, the display module presents a light spot and performs a high and large horizontal movement within the display range, and the audio module emits a sound in the direction of the starting point of the light spot movement. The camera detects the head rotation posture of the infant in the process of visual and auditory joint following, and uses the human body posture estimation model to identify the key points of the infant's head, and extract the above-mentioned head key point movement feature data;

[0024] Step S37: The computer controls the display module and the audio module to perform asynchronous stimulation together. The display module presents a light spot and moves horizontally at a low speed and with a small amplitude within the display range. At the same time, the audio module emits a sound in the direction of the starting point of the light spot movement. The camera detects the eye movement and pupil changes in the process of the infant's visual and auditory joint following, and obtains the gaze point movement trajectory and pupil size under the joint induction of light spot vision and hearing.

[0025] Furthermore, in step S4, the movement paradigm includes whole-body active movement and local induced movement; local induced movement includes local unimodal induced movement and local multimodal induced movement, wherein unimodal is visual or auditory, and multimodal is a combination of vision and hearing; local multimodal induced movement is divided into two modes: synchronous stimulation and asynchronous stimulation according to whether the directions of visual and auditory stimulation are consistent.

[0026] Furthermore, in step S4, the feature extracted from the whole body active motion data is F G , the feature extracted from the visual modality induced data is F L1 , the feature extracted from the auditory modality induced data is F L2 , the feature extracted from the audiovisual modality synchronization induced data is F L3 , the feature extracted from the asynchronous evoked data of the visual and auditory modalities is F L4 , wherein the whole-body active motion data is the limb key point motion feature data obtained in step S22, the visual modality induced data is the head key point motion feature data and the gaze point motion trajectory obtained in steps S31 and S32, the auditory modality induced data is the head key point motion feature data obtained in step S33, the visual and auditory modality synchronous induced data is the head key point motion feature data, the gaze point motion trajectory and the pupil size obtained in steps S34 and S35, and the visual and auditory modality asynchronous induced data is the head key point motion feature data, the gaze point motion trajectory and the pupil size obtained in steps S36 and S37.

[0027] Furthermore, feature F G The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's limbs are obtained, and the joint rotation angle, joint rotation angular velocity and joint rotation angular acceleration are further calculated to extract the feature F G as follows:

[0028] F G =[θ ij ω ij a ij ],i∈[1,12],j∈[1,6]

[0029] Wherein, θ, ω, a represent joint rotation angle, joint rotation angular velocity, and joint rotation angular acceleration, respectively; i represents 12 human joints, namely, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of each joint motion parameter, and the joint motion parameters include joint rotation angle, joint rotation angular velocity, or joint rotation angular acceleration;

[0030] Using logistic regression machine learning method based on feature FG The process of obtaining the first classification decision is as follows: for feature F G Standardization processing; Use the logistic regression model to train the fusion decision; Use the probability value output by the logistic regression model to obtain the first classification sub-decision. If the probability is greater than 0.5, the sub-decision is considered to be category 1, that is, abnormal neurodevelopment, otherwise it is considered to be category 0, that is, normal neurodevelopment.

[0031] Furthermore, feature F L1 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated. At the same time, the facial behavior analysis model is used to obtain the movement trajectory of the infant's gaze point. By calculating the distance from the movement trajectory of the stimulus light point, the movement trajectory error of the gaze point is obtained, and then the feature F is extracted. L1 as follows:

[0032] F L1 =[θ j ω j a j s j ],j∈[1,6]

[0033] Among them, θ, ω, a, and s represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, and fixation point motion trajectory error, respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of the head motion parameters and eye motion parameters. The head motion parameters include the head rotation angle, head rotation angular velocity, and head rotation angular acceleration. The eye motion parameters are the fixation point motion trajectory error.

[0034] Feature F L2 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the baby's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated to extract the feature F L2 as follows:

[0035] F L2 =[θ j ω j a j ],j∈[1,6]

[0036] Wherein, θ, ω, a represent the head rotation angle, head rotation angular velocity and head rotation angular acceleration respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value and standard deviation of the head motion parameters, and the head motion parameters include the head rotation angle, head rotation angular velocity and head rotation angular acceleration;

[0037] Feature FL3 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated. At the same time, the facial behavior analysis model is used to obtain the movement trajectory of the infant's gaze point. By calculating the distance from the movement trajectory of the stimulus light point, the movement trajectory error of the gaze point is obtained, and then the feature F is extracted. L3 as follows:

[0038] F L3 =[θ j ω j a j s j p j ],j∈[1,6]

[0039] Among them, θ, ω, a, s, and p represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, fixation point motion trajectory error, and pupil size, respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of the head movement parameters, eye movement parameters, and pupil parameters; the head movement parameters include the head rotation angle, head rotation angular velocity, and head rotation angular acceleration; the eye movement parameters are the fixation point motion trajectory error; and the pupil parameter is the pupil size;

[0040] Feature F L4 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated. At the same time, the facial behavior analysis model is used to obtain the movement trajectory of the infant's gaze point. By calculating the distance from the movement trajectory of the stimulus light point, the movement trajectory error of the gaze point is obtained, and then the feature F is extracted. L4 as follows:

[0041] F L3 =[θ j ω j a j s j p j ],j∈[1,6]

[0042] Among them, θ, ω, a, s, and p represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, fixation point motion trajectory error, and pupil size, respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of the head movement parameters, eye movement parameters, and pupil parameters; the head movement parameters include the head rotation angle, head rotation angular velocity, and head rotation angular acceleration; the eye movement parameters are the fixation point motion trajectory error; and the pupil parameter is the pupil size;

[0043] The support vector machine machine learning method is used based on the feature F L1 、F L2 、F L3 、F L4 The process of obtaining the first classification decision is as follows: for feature F L1 、F L2 、F L3 、F L4 Perform standardization processing; use the SVM model to train the fusion decision; classify the input features through the hyperplane trained by the SVM to obtain the second classification sub-decision.

[0044] Furthermore, the calculation formula of the gaze point motion trajectory error is as follows:

[0045]

[0046] Among them, A(x,y) and A'(x',y') are the corresponding points and their coordinates on the motion trajectory of the stimulus light spot and the motion trajectory of the infant's gaze point, respectively.

[0047] Furthermore, in step S4, each sub-decision is combined into a fused decision vector D, which is further analyzed to obtain a final decision on the task. The final result of the decision layer fusion calculation is based on the weighted average of multiple sub-decisions, and the process formula is as follows:

[0048]

[0049] Among them, d final is the final decision result, y is the weighted average, m is the number of sub-decision making, D is the fusion decision vector of sub-decision making combination, W=(w1,w2,…,w m ) is the weight vector, and d final A value of 0 indicates that the infant's neurological development is normal, and a value of 1 indicates that the infant's neurological development is abnormal.

[0050] The present invention also provides a system for analyzing the neurodevelopmental status of a young infant, which is used for the above-mentioned method for analyzing the neurodevelopmental status of a young infant, comprising: a test bench, a display module, an audio module, a camera, a computer and a table side fixing frame, wherein the camera is integrated on the display module; the audio module is arranged on the left and right sides of the infant's head; the display module is fixed on the table side fixing frame, and the camera can capture the whole body movement video of the infant by adjusting the relative position of the table side fixing frame and the test bench; the computer is used to control the display module and the audio module and to analyze the video captured by the camera, and the display module is a naked-eye VR device.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] Traditional methods mainly analyze abnormalities of spontaneous whole-body movements, which have a single modality and require comparative analysis of large-sample control group data. They are unable to solve clinical problems such as heterogeneity among individuals of neurodevelopmental diseases. The present invention adds the collection and analysis of local movement data induced by visual and auditory senses to the collection and analysis of whole-body spontaneous movement data, obtains multimodal behavioral characteristics, and improves clinical classification accuracy through post-fusion decision-making of modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic diagram of the structure of a system for analyzing the neurodevelopmental status of a young infant according to an embodiment of the present invention;

[0054] Figure 2 is a test schematic diagram of using a naked-eye VR device as a display module in an embodiment of the present invention;

[0055] Figure 3 1 is a schematic diagram of a flow chart of a method for analyzing the neurodevelopmental status of an infant in an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the process of induced motion detection and analysis in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0058] like Figure 1 As shown, this embodiment provides a technical solution: a system for analyzing the neurodevelopmental status of a young infant, for use in the following method for analyzing the neurodevelopmental status of a young infant, comprising: a test bench 1, a display module 2, an audio module 3, a camera 4, a computer 5 and a table side fixing frame 6, wherein the camera 4 is integrated on the display module 2; the audio module 3 is arranged on the left and right sides of the head of the young infant 7; the display module 2 is fixed on the table side fixing frame 6, and by adjusting the relative position of the table side fixing frame 6 and the test bench 1, the camera 4 can capture the whole body movement video of the young infant 7; the computer 5 is used to control the display module 2 and the audio module 3 and to analyze the video captured by the camera 3.

[0059] It should be noted that in order to increase the fidelity of the visual stimulation light spots and reduce the potential harm of electromagnetic radiation from traditional display screens to the eyesight of infants 7, the display module 2 in this embodiment adopts a naked-eye VR device. Figure 2 The naked-eye VR device is the "floating display module" in the Chinese invention patent application with application number 202410664169.0 and titled "A child autism rehabilitation training system and method based on floating interaction"

[0060] like Figure 3 As shown, this embodiment provides a method for analyzing the neurodevelopmental status of a young infant, comprising the following steps:

[0061] Step 1: Data collection preparation

[0062] Place the baby on the test bench, adjust the relative position between the side bracket and the test bench, so that the normal direction of the camera on the display module is perpendicular to the test bench and the field of view covers the whole body of the baby, and fix the audio module on the left and right sides of the baby's head on the test bench;

[0063] Step 2: Full body active motion data collection

[0064] Use cameras to collect data on the baby's full body active movements;

[0065] In this embodiment, the specific process of step 2 is as follows:

[0066] Step S21: The computer controls the camera to detect the active whole-body movement of the infant in a natural state, and adjusts the relative position of the camera and the test bench so that the video contains the complete limbs and head information of the infant;

[0067] Step S22: Use the human body posture estimation model to identify the video, identify the key points of the infant's limbs, and extract the above-mentioned limb key point motion feature data (limb key point coordinates, joint rotation angle, joint rotation angular velocity, joint rotation angular acceleration) to reflect the infant's whole body autonomous movement amplitude and rhythm.

[0068] It should be noted that for active whole-body movement, parameters such as the head rotation angle, angular velocity, angular acceleration, etc. are usually not directly analyzed. Instead, evaluation and diagnosis are made by observing the overall movement patterns of the infant's limbs (such as twisting movements, restless movements, etc.).

[0069] In this embodiment, the human body posture estimation model is a model such as Openpose, Media Pipe Blaze Pose, etc.

[0070] Step 3: Local evoked motion data acquisition

[0071] Using a display module and / or an audio module in conjunction with a camera to collect local induced movement data of the infant;

[0072] like Figure 4 As shown, in this embodiment, the specific process of step 3 is as follows:

[0073] Step S31: The computer controls the display module to present a light spot and make a high-speed and large-amplitude horizontal movement within the display range. At the same time, the camera detects the head rotation posture of the infant during the visual following process, uses the human body posture estimation model to identify the key points of the infant's head, and extracts the above-mentioned head key point motion feature data (head key point coordinates, head rotation angle, head rotation angular velocity, head rotation angular acceleration) to reflect the local induced motion amplitude and rhythm of the infant;

[0074] Step S32: The computer controls the display module to present the light spot and make a low-speed and small-amplitude horizontal movement within the display range, while the camera detects the eye movement of the infant during the visual following process to obtain the movement trajectory of the infant's gaze point under the visual induction of the light spot;

[0075] Step S33: The computer controls the sound module to emit continuous sounds on both sides of the infant's head. At the same time, the camera detects the head rotation posture of the infant during the auditory following process, and uses the human body posture estimation model to identify the key points of the infant's head, and extract the above-mentioned head key point motion feature data (head key point coordinates, head rotation angle, head rotation angular velocity, head rotation angular acceleration) to reflect the local induced movement amplitude and rhythm of the infant and the rotation reaction time;

[0076] Step S34: The computer controls the display module and the audio module to jointly and synchronously stimulate, the display module presents a light spot and moves horizontally at a high speed and large amplitude within the display range, and at the same time, the audio emits a sound in the direction of the end point of the light spot movement, the camera detects the head rotation posture of the infant in the process of visual and auditory joint following, uses the human body posture estimation model to identify the key points of the infant's head, extracts the above-mentioned head key point motion feature data (head key point coordinates, head rotation angle, head rotation angular velocity, head rotation angular acceleration), and reflects the local induced movement amplitude and rhythm of the infant and the rotation reaction time;

[0077] Step S35: The computer controls the display module and the audio module to perform joint synchronous stimulation, the display module presents a light spot and moves horizontally at a low speed and small amplitude within the display range, and at the same time, the audio module emits a sound in the direction of the end point of the light spot movement. The camera detects the eye movement and pupil changes of the infant during the visual following process, and obtains the gaze point movement trajectory and pupil size under the joint induction of light spot vision and hearing, reflecting the local induced movement amplitude and rhythm of the infant and the induced reaction time;

[0078] Step S36: The computer controls the display module and the audio module to perform asynchronous stimulation together, the display module presents a light spot and performs a highly large horizontal movement within the display range, and at the same time, the audio module emits a sound in the direction of the starting point of the light spot movement, the camera detects the head rotation posture of the infant in the process of visual and auditory joint following, and uses the human body posture estimation model to identify the key points of the infant's head, and extracts the above-mentioned head key point motion feature data (head key point coordinates, head rotation angle, head rotation angular velocity, head rotation angular acceleration), reflecting the local induced motion amplitude and rhythm of the infant and the rotation reaction time;

[0079] Step S37: The computer controls the display module and the audio module to perform asynchronous stimulation together. The display module presents a light spot and moves horizontally at a low speed and small amplitude within the display range. At the same time, the audio makes a sound in the direction of the starting point of the light spot movement. The camera detects the eye movement and pupil changes in the process of the infant's visual and auditory joint following, and obtains the gaze point movement trajectory and pupil size under the joint induction of light spot vision and hearing, reflecting the local induced movement amplitude and rhythm of the infant and the induced reaction time.

[0080] In this embodiment, a facial behavior analysis model such as OpenFace is used to calculate the gaze point motion trajectory by analyzing eye features and head posture.

[0081] In this embodiment, the amplitude is based on the displayable length of the display module in the horizontal direction, a large amplitude is defined as any value within the range of 70% to 100%, and a small amplitude is defined as any value within the range of 30% to 50%; the speed is based on the eye movement speed converted to the horizontal speed on the display module screen, a speed higher than 40 mm / s is defined as a high speed, otherwise, it is defined as a low speed.

[0082] In this embodiment, the pupil size is obtained by using a camera through a machine vision method, and the specific process is as follows:

[0083] 1) Image acquisition and preprocessing

[0084] Use the camera to collect face images and pre-process them;

[0085] 2) Eye position detection

[0086] Cascade classifier using Haar features to identify eyes and obtain location information from face images

[0087] 3) Pupil localization and segmentation

[0088] The pupil boundary is determined by thresholding and edge detection of the darkest point in the human eye area;

[0089] 4) Pupil feature extraction

[0090] Extract pupil diameter and area characteristic parameters, that is, obtain pupil size information.

[0091] Step 4: Multi-motion feature extraction and fusion decision

[0092] Based on the movement paradigm (whole-body active movement, local induced movement), features (aggregated limb movement features, head movement features, and eye movement features) are extracted, and classification sub-decisions are formed through feature layer fusion. The final decision result is obtained by weighted average of the sub-decisions, combined with the doctor's clinical judgment to assist in the analysis and evaluation of the infant's neurodevelopmental status.

[0093] In this embodiment, the movement paradigm includes whole-body active movement and local induced movement; local induced movement includes local unimodal induced movement and local multimodal induced movement, among which unimodal is visual or auditory, and multimodal is the combination of vision and hearing; local multimodal induced movement is divided into two modes: synchronous stimulation and asynchronous stimulation according to whether the directions of visual and auditory stimulation are consistent.

[0094] In this embodiment, the feature extracted from the whole body active motion data is F G , the feature extracted from the visual modality induced data is F L1 , the feature extracted from the auditory modality induced data is F L2 , the feature extracted from the audiovisual modality synchronization induced data is F L3 , the feature extracted from the asynchronous evoked data of the visual and auditory modalities is F L4 , wherein the whole-body active motion data is the limb key point motion feature data obtained in step S22, the visual modality induced data is the head key point motion feature data and the gaze point motion trajectory obtained in steps S31 and S32, the auditory modality induced data is the head key point motion feature data obtained in step S33, the visual and auditory modality synchronous induced data is the head key point motion feature data, the gaze point motion trajectory and the pupil size obtained in steps S34 and S35, and the visual and auditory modality asynchronous induced data is the head key point motion feature data, the gaze point motion trajectory and the pupil size obtained in steps S36 and S37.

[0095] In this embodiment, feature F G The extraction process is as follows:

[0096] Using human posture estimation models such as Openpose and Media Pipe Blaze Pose, we obtained the coordinate position data of the key points of the infant's limbs, and further calculated the spontaneous motion data such as joint rotation angle, joint rotation angular velocity, and joint rotation angular acceleration, as follows:

[0097] 1. Calculate the joint rotation angle:

[0098] The joint angle is obtained by calculating the angle between two adjacent bone vectors of the joint. The bone vector is defined by the corresponding joint position information:

[0099]

[0100] Among them, P A , P B , P C is the joint point coordinate;

[0101] The joint rotation angle is:

[0102]

[0103] 2. Calculate the angular velocity of the joint:

[0104] For two consecutive time points t1 and t2, and the corresponding joint rotation angles θ1 and θ2, the rotation angular velocity ω is:

[0105]

[0106] 3. Calculate the angular acceleration of the joint:

[0107] Similar to the calculation of joint rotation angular velocity, angular acceleration can be calculated by the rate of change of angular velocity:

[0108]

[0109] Based on the spontaneous motion data generated above, the whole body active motion feature F is further extracted. G as follows:

[0110] F G =[θ ij ω ij a ij ],i∈[1,12],j∈[1,6]

[0111] Among them, θ, ω, and a represent joint angles, joint angular velocity, and joint angular acceleration, respectively; i represents 12 human joints, including left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; j represents the maximum, minimum, mean, peak-to-peak value (the difference between the maximum and minimum values), root mean square value, standard deviation, and other characteristic values ​​of each joint motion parameter (angle, angular velocity, or angular acceleration).

[0112] In this embodiment, a logistic regression (LR) machine learning method is used to obtain a feature layer fusion decision 1 (sub-decision) based on whole-body autonomous motion features, and a support vector machine (SVM) machine learning method is used to obtain a feature layer fusion decision 2 (sub-decision) based on local induced motion features.

[0113] In this embodiment, the feature F is obtained GThe specific process of logistic regression (LR) fusion decision 1 is as follows:

[0114] 1.1 Feature vector preprocessing: feature F G Preprocessing is performed to ensure that the data has the same scale during training. The preprocessing standardization formula is as follows:

[0115]

[0116] Among them, X′ is the original data, μ is the mean, σ is the standard deviation, and X is the standardized data.

[0117] 1.2 Model training: Use the logistic regression model to train the fusion decision. Logistic regression obtains model parameters by solving the maximum likelihood estimation (MLE). The prediction formula of the logistic regression model is:

[0118]

[0119] Among them, w is the feature weight, X is the input feature, b is the bias term, and P(y=1|X) is the probability of being category 1 under the input feature X.

[0120] 1.3 Decision output: Through the probability value output by the model, the fusion decision 1 (sub-decision) can be obtained. Generally speaking, if the probability is greater than 0.5, the decision is considered to be category 1 (neural developmental abnormality), otherwise it is considered to be category 0 (neural developmental normal).

[0121] In this embodiment, feature F L1 The extraction process is as follows:

[0122] Using human posture estimation models such as Openpose and Media Pipe Blaze Pose, the coordinate position data of the key points of the infant's head are obtained, and the local induced (visual) motion data such as the head rotation angle, angular velocity and angular acceleration are further calculated. The specific process is similar to the aforementioned joint motion parameter calculation.

[0123] The OpenFace facial behavior analysis model is used to obtain the gaze trajectory of the infant. The gaze trajectory error is obtained by calculating the distance from the stimulus light trajectory:

[0124]

[0125] Among them, A(x,y) and A'(x',y') are the corresponding points and their coordinates on the motion trajectory of the stimulus light spot and the motion trajectory of the infant's gaze point, respectively.

[0126] Based on the local induced motion data, the local induced motion features F are further extracted. L1 as follows:

[0127] F L1 =[θ j ω j a j s j ],j∈[1,6]

[0128] Among them, θ, ω, a, and s represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, and gaze point motion trajectory error, respectively; j represents the maximum, minimum, mean, peak-to-peak value (the difference between the maximum and minimum values), root mean square value, standard deviation, and other characteristic values ​​of the head motion parameters (angle, angular velocity, or angular acceleration) and eye motion parameters (gaze point motion trajectory error).

[0129] In this embodiment, feature F L2 The extraction process is as follows:

[0130] Using human posture estimation models such as Openpose and Media Pipe Blaze Pose, the coordinate position data of the key points of the infant's head are obtained, and the local induced (auditory) motion data such as the head rotation angle, angular velocity and angular acceleration are further calculated. The specific process is similar to the aforementioned joint motion parameter calculation.

[0131] Based on the local induced motion data, the local induced motion features F are further extracted. L2 as follows:

[0132] F L2 =[θ j ω j a j ],j∈[1,6]

[0133] Among them, θ, ω, and a represent the head rotation angle, head rotation angular velocity, and head rotation angular acceleration, respectively; j represents the maximum, minimum, mean, peak-to-peak value (the difference between the maximum and minimum values), root mean square value, standard deviation, and other characteristic values ​​of the head motion parameters (angle, angular velocity, or angular acceleration).

[0134] In this embodiment, feature F L3 The extraction process is as follows:

[0135] Using human posture estimation models such as Openpose and Media Pipe Blaze Pose, the coordinate position data of the key points of the infant's head are obtained, and the local induced (auditory and visual synchronization) motion data such as the head rotation angle, angular velocity and angular acceleration are further calculated. The specific process is similar to the aforementioned joint motion parameter calculation.

[0136] The OpenFace facial behavior analysis model was used to obtain the gaze trajectory of the infant. The gaze trajectory error was obtained by calculating the distance from the stimulus light spot trajectory. The specific process was the same as that of the F L1 The extraction process is similar.

[0137] The method for calculating pupil size characteristics has been described above.

[0138] Based on the local induced motion data, the local induced motion features F are further extracted. L3 as follows:

[0139] F L3 =[θ j ω j a j s j p j ],j∈[1,6]

[0140] Among them, θ, ω, a, s, and p represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, gaze point motion trajectory error, and pupil size, respectively; j represents the maximum, minimum, mean, peak-to-peak value (the difference between the maximum and minimum values), root mean square value, standard deviation, and other characteristic values ​​of the head movement parameters (angle, angular velocity, or angular acceleration), eye movement parameters (gaze point motion trajectory error), and pupil parameters.

[0141] In this embodiment, feature F L4 The extraction process is as follows:

[0142] Using human posture estimation models such as Openpose and Media Pipe Blaze Pose, the coordinate position data of the key points of the infant's head are obtained, and the local induced (auditory and visual asynchronous) motion data such as the head rotation angle, angular velocity and angular acceleration are further calculated. The specific process is similar to the aforementioned joint motion parameter calculation.

[0143] The OpenFace facial behavior analysis model was used to obtain the gaze trajectory of the infant. The gaze trajectory error was obtained by calculating the distance from the stimulus light spot trajectory. The specific process was the same as that of the F L1 The extraction process is similar.

[0144] The method for calculating pupil size characteristics has been described above.

[0145] Based on the local induced motion data, the local induced motion features F are further extracted. L4 as follows:

[0146] F L4 =[θ j ω j a js j p j ],j∈[1,6]

[0147] Among them, θ, ω, a, s, and p represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, gaze point motion trajectory error, and pupil size, respectively; j represents the maximum, minimum, mean, peak-to-peak value (the difference between the maximum and minimum values), root mean square value, standard deviation, and other characteristic values ​​of the head movement parameters (angle, angular velocity, or angular acceleration), eye movement parameters (gaze point motion trajectory error), and pupil parameters.

[0148] Step 2: Based on feature F L1 、F L2 、F L3 、F L4 Support vector machine (SVM) fusion decision 2

[0149] 2.1 Data preprocessing: Similar to step 1.1, feature F L1 、F L2 、F L3 、F L4 Perform standardization. Connect the standardized feature vectors in series to obtain the composite feature vector F L ={F L1 ,F L2 ,F L3 ,F L4}, as model input.

[0150] 2.2 Support vector machine training: Use the SVM model to train fusion decisions based on local induced motion features. The goal of SVM is to find the best segmentation hyperplane by maximizing the classification interval. For the binary classification problem, the optimization goal of SVM is:

[0151]

[0152] Among them, w is the normal vector of the hyperplane, b is the bias term, and x i is the input feature, y i is the corresponding category label.

[0153] 2.3 Decision output: The input features are classified by the hyperplane trained by SVM to obtain fusion decision 2 (sub-decision). The prediction formula of SVM is:

[0154] f(x)=w T x+b

[0155] If the value of f(x) is greater than 0, then the classification is class 1; otherwise, it is class 0.

[0156] In this embodiment, the weighted average of the sub-decisions is used to obtain the final decision result, and whether the infant's neurological development is normal is determined based on the final decision result. If normal, the analysis of the infant's neurological development status is terminated; otherwise, the analysis of the infant's neurological development status is completed in combination with the doctor's clinical judgment.

[0157] Each sub-decision is combined into a fused decision vector D, which is further analyzed to obtain the final decision on the task. The final result of the decision layer fusion calculation is based on the weighted average of multiple sub-decisions, and the process formula is as follows:

[0158]

[0159] where d final is the final decision result, y is the weighted average, m is the number of sub-decision making, D is the fusion decision vector of sub-decision making combination, W=(w1,w2,…,w m ) is the weight vector, and d final A value of 0 indicates that the infant's neurological development is normal, while a value of 1 indicates that it is abnormal. The analysis of the infant's neurological development status needs to be assisted by the doctor's clinical judgment.

[0160] Decision layer fusion: In order to improve the fusion effect and automatically learn the nonlinear relationship between sub-decisions, the present invention introduces a multi-layer perceptron (MLP) model. The specific process is as follows:

[0161] The MLP network consists of multiple layers:

[0162] Input layer: Take the sub-decision D = (d1, d2, ..., dm) as the input vector of MLP. Each sub-decision can be a binary output (0 or 1) or a probability value.

[0163] Hidden layer: contains multiple neuron layers, and learns the complex relationship between sub-decisions through the ReLU activation function, where the activation function formula is:

[0164]

[0165] Output layer: Generates the final decision y, and maps the output value to the interval [0,1] through the Sigmoid activation function to represent the final decision result:

[0166] y=σ(W out ·h+b out )

[0167] Among them, h is the output of the hidden layer, W out is the weight of the output layer, b out is the bias term of the output layer, and σ is the Sigmoid activation function:

[0168]

[0169] After training the MLP, the final decision d final for:

[0170]

[0171] d final A value of 0 indicates that the infant's neurological development is normal, while a value of 1 indicates abnormality. Further analysis of the infant's neurological development status needs to be completed in conjunction with the doctor's clinical judgment.

[0172] In summary, the method for analyzing the neurodevelopmental status of infants in the above embodiment adds local motion data collection and analysis induced by visual and auditory senses on the basis of whole-body spontaneous motion data collection and analysis to obtain multimodal behavioral characteristics, and improves clinical classification accuracy through post-modal feature fusion decision-making.

[0173] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for analyzing the neurodevelopmental status of a young infant, characterized in that: The following steps are involved: S1: Data collection preparation Place the baby on the test bench, adjust the relative position between the side bracket and the test bench, so that the normal direction of the camera on the display module is perpendicular to the test bench and the field of view covers the whole body of the baby, and fix the audio module on the left and right sides of the baby's head on the test bench; S2: Whole body active motion data collection Use cameras to collect data on the baby's full body active movements; S3: Locally evoked motion data acquisition Using a display module and / or an audio module in conjunction with a camera to collect local induced movement data of the infant; S4: Multi-motion feature extraction and fusion decision Features are extracted according to the movement paradigm, and the first classification sub-decision and the second classification sub-decision are formed through feature layer fusion. The weighted average of the sub-decisions is used to obtain the final decision result, and then the neurodevelopmental status of the infant is analyzed and evaluated.

2. A method for analyzing the neurodevelopmental status of infants according to claim 1, characterized in that: In step S2, the specific process is as follows: S21: The computer controls the camera to detect the active whole-body movement of the infant in a natural state, and adjusts the relative position of the camera and the test bench so that the video contains the complete limbs and head information of the infant; S22: Using a human posture estimation model to identify the video, identify key points of the infant's limbs, and extract motion feature data of the key points of the limbs.

3. A method for analyzing the neurodevelopmental status of infants according to claim 2, characterized in that: In step S3, the specific process is as follows: Step S31: The computer controls the display module to present a light spot and make a high-speed and large-amplitude horizontal movement within the display range. At the same time, the camera detects the head rotation posture of the infant during the visual following process, uses the human body posture estimation model to identify the key points of the infant's head, and extracts the motion feature data of the key points of the head; Step S32: The computer controls the display module to present the light spot and make a low-speed and small-amplitude horizontal movement within the display range, while the camera detects the eye movement of the infant during the visual following process to obtain the movement trajectory of the infant's gaze point under the visual induction of the light spot; Step S33: the computer controls the sound module to emit continuous sounds on both sides of the baby's head, and at the same time, the camera detects the head rotation posture of the baby during the auditory following process, uses the human body posture estimation model to identify the key points of the baby's head, and extracts the motion feature data of the key points of the head; Step S34: The computer controls the display module and the audio module to stimulate synchronously, the display module presents a light spot and moves horizontally at a high speed and large amplitude within the display range, and the audio module emits a sound in the direction of the end point of the light spot movement. The camera detects the head rotation posture of the infant in the process of visual and auditory joint following, and uses the human body posture estimation model to identify the key points of the infant's head, and extract the above-mentioned head key point movement feature data; Step S35: The computer controls the display module and the audio module to perform joint synchronous stimulation, the display module presents a light spot and moves horizontally at a low speed and a small amplitude within the display range, and at the same time, the audio module emits a sound in the direction of the end point of the light spot movement, and the camera detects the eye movement and pupil changes of the infant during the visual following process, and obtains the gaze point movement trajectory and pupil size under the joint induction of light spot vision and hearing; Step S36: The computer controls the display module and the audio module to perform asynchronous stimulation, the display module presents a light spot and performs a high and large horizontal movement within the display range, and the audio module emits a sound in the direction of the starting point of the light spot movement. The camera detects the head rotation posture of the infant in the process of visual and auditory joint following, and uses the human body posture estimation model to identify the key points of the infant's head, and extract the above-mentioned head key point movement feature data; Step S37: The computer controls the display module and the audio module to perform asynchronous stimulation together. The display module presents a light spot and moves horizontally at a low speed and with a small amplitude within the display range. At the same time, the audio module emits a sound in the direction of the starting point of the light spot movement. The camera detects the eye movement and pupil changes of the infant's visual and auditory joint following process, and obtains the gaze point movement trajectory and pupil size under the joint induction of light spot vision and hearing.

4. A method for analyzing the neurodevelopmental status of infants according to claim 3, characterized in that: In step S4, the movement paradigm includes whole-body active movement and local induced movement; local induced movement includes local unimodal induced movement and local multimodal induced movement, wherein unimodal is visual or auditory, and multimodal is a combination of vision and hearing; local multimodal induced movement is divided into two modes: synchronous stimulation and asynchronous stimulation according to whether the directions of visual and auditory stimulation are consistent.

5. A method for analyzing the neurodevelopmental status of infants according to claim 4, characterized in that: In step S4, the feature extracted from the whole body active motion data is F G , the feature extracted from the visual modality induced data is F L1 , the feature extracted from the auditory modality induced data is F L2 , the feature extracted from the audiovisual modality synchronization induced data is F L3 , the feature extracted from the asynchronous evoked data of the visual and auditory modalities is F L4 , wherein the whole-body active motion data is the limb key point motion feature data obtained in step S22, the visual modality induced data is the head key point motion feature data and the gaze point motion trajectory obtained in steps S31 and S32, the auditory modality induced data is the head key point motion feature data obtained in step S33, the visual and auditory modality synchronous induced data is the head key point motion feature data, the gaze point motion trajectory and the pupil size obtained in steps S34 and S35, and the visual and auditory modality asynchronous induced data is the head key point motion feature data, the gaze point motion trajectory and the pupil size obtained in steps S36 and S37.

6. A method for analyzing the neurodevelopmental status of a young infant according to claim 5, characterized in that: Feature F G The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's limbs are obtained, and the joint rotation angle, joint rotation angular velocity and joint rotation angular acceleration are further calculated to extract the feature F G as follows: F G =[θ ij oh ij a ij ],i∈[1,12],j∈[1,6] Wherein, θ, ω, a represent joint rotation angle, joint rotation angular velocity, and joint rotation angular acceleration, respectively; i represents 12 human joints, namely, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of each joint motion parameter, and the joint motion parameters include joint rotation angle, joint rotation angular velocity, or joint rotation angular acceleration; Using logistic regression machine learning method based on feature F G The process of obtaining the first classification decision is as follows: for feature F G Standardization processing; Use the logistic regression model to train the fusion decision; Get the first classification decision through the probability value output by the logistic regression model.

7. A method for analyzing the neurodevelopmental status of infants according to claim 5, characterized in that: Feature F L1 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated. At the same time, the facial behavior analysis model is used to obtain the movement trajectory of the infant's gaze point. By calculating the distance from the movement trajectory of the stimulus light point, the movement trajectory error of the gaze point is obtained, and then the feature F is extracted. L1 as follows: F L1 =[θ j oh j a j s j ],j∈[1,6] Among them, θ, ω, a, and s represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, and fixation point motion trajectory error, respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of the head motion parameters and eye motion parameters. The head motion parameters include the head rotation angle, head rotation angular velocity, and head rotation angular acceleration. The eye motion parameters are the fixation point motion trajectory error. Feature F L2 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the baby's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated to extract the feature F L2 as follows: F L2 =[θ j oh j a j ],j∈[1,6] Wherein, θ, ω, a represent the head rotation angle, head rotation angular velocity and head rotation angular acceleration respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value and standard deviation of the head motion parameters, and the head motion parameters include the head rotation angle, head rotation angular velocity and head rotation angular acceleration; Feature F L3 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated. At the same time, the facial behavior analysis model is used to obtain the movement trajectory of the infant's gaze point. By calculating the distance from the movement trajectory of the stimulus light point, the movement trajectory error of the gaze point is obtained, and then the feature F is extracted. L3 as follows: F L3 =[θ j oh j a j s j p j ],j∈[1,6] Among them, θ, ω, a, s, and p represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, fixation point motion trajectory error, and pupil size, respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of the head movement parameters, eye movement parameters, and pupil parameters; the head movement parameters include the head rotation angle, head rotation angular velocity, and head rotation angular acceleration; the eye movement parameters are the fixation point motion trajectory error; and the pupil parameter is the pupil size; Feature F L4 The extraction process of is as follows: using the human posture estimation model, the coordinate position data of the key points of the infant's head are obtained, and the head rotation angle, head rotation angular velocity and head rotation angular acceleration are further calculated. At the same time, the facial behavior analysis model is used to obtain the movement trajectory of the infant's gaze point. By calculating the distance from the movement trajectory of the stimulus light point, the movement trajectory error of the gaze point is obtained, and then the feature F is extracted. L4 as follows: F L3 =[θ j oh j a j s j p j ],j∈[1,6] Among them, θ, ω, a, s, and p represent the head rotation angle, head rotation angular velocity, head rotation angular acceleration, fixation point motion trajectory error, and pupil size, respectively; j represents the maximum value, minimum value, mean value, peak-to-peak value, root mean square value, and standard deviation of the head movement parameters, eye movement parameters, and pupil parameters; the head movement parameters include the head rotation angle, head rotation angular velocity, and head rotation angular acceleration; the eye movement parameters are the fixation point motion trajectory error; and the pupil parameter is the pupil size; The support vector machine machine learning method is used based on the feature F L1 、F L2 、F L3 、F L4 The process of obtaining the first classification decision is as follows: for feature F L1 、F L2 、F L3 、F L4 Perform standardization processing; use the SVM model to train the fusion decision; classify the input features through the hyperplane trained by the SVM to obtain the second classification sub-decision.

8. A method for analyzing the neurodevelopmental status of infants according to claim 7, characterized in that: The calculation formula of the gaze point motion trajectory error is as follows: Among them, A(x,y) and A'(x',y') are the corresponding points and their coordinates on the motion trajectory of the stimulus light spot and the motion trajectory of the infant's gaze point, respectively.

9. A method for analyzing the neurodevelopmental status of infants according to claim 1, characterized in that: In step S4, each sub-decision is combined into a fused decision vector D, which is further analyzed to obtain the final decision on the task. The final result of the decision layer fusion calculation is based on the weighted average of multiple sub-decisions, and the process formula is as follows: Among them, d final is the final decision result, y is the weighted average, m is the number of sub-decision making, D is the fusion decision vector of sub-decision making combination, W=(w1,w2,…,w m ) is the weight vector, and d final A value of 0 indicates that the infant's neurological development is normal, and a value of 1 indicates that the infant's neurological development is abnormal.

10. A system for analyzing the neurodevelopmental status of infants, characterized in that: It is used for the above-mentioned method of analyzing the neurodevelopmental status of infants, and comprises: a test bench, a display module, an audio module, a camera, a computer and a table side fixing frame, wherein the camera is integrated on the display module; the audio module is arranged on the left and right sides of the infant's head; the display module is fixed on the table side fixing frame, and the camera can capture the whole body movement video of the infant by adjusting the relative position of the table side fixing frame and the test bench; the computer is used to control the display module and the audio module and to analyze the video captured by the camera, and the display module is a naked-eye VR device.

Images