A quantitative assessment system for infant brain development based on graph convolutional networks

Abstract

A quantitative assessment system for infant brain development based on a graph convolutional network includes a posture estimation module, a sequence preprocessing module, a feature extraction module, a graph convolution classification module, a multidimensional indicator analysis module, and a result visualization module. The posture estimation module receives an infant movement video as input, the posture estimation module output is connected to the sequence preprocessing module input, the sequence preprocessing module output is connected to the feature extraction module input, the feature extraction module output is connected to the graph convolution classification module input, the graph convolution classification module output is connected to the multidimensional indicator analysis module input, and the multidimensional indicator analysis module output is connected to the result visualization module input. The system output can output multiple abnormality categories and includes interpretable visualization results, thereby assisting physicians in targeted treatment intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of infant brain development assessment, and specifically relates to a quantitative assessment system for infant brain development based on a graph convolutional network. Background Art

[0002] Experts can observe the baby's bed and analyze the quality of the baby's spontaneous movements based on their experience to determine whether the baby's brain development is normal. However, this type of diagnosis relies heavily on the physician's professional experience, and training doctors takes a lot of time, which is not conducive to large-scale promotion. Inexperienced physicians have a high rate of misdiagnosis and missed diagnosis.

[0003] Although there are some intelligent means of screening infant brain development, there are the following shortcomings: First, some methods ([1]KDMcCay et al.,"A pose-based feature fusion and classification framework for the early prediction of cerebral palsy in infants,"IEEE Transactions on Neural Systems and Rehabilitation Engineering,vol.30,pp.8-19,2021; [2]D.Sakkos,KDMccay,C.Marcroft,NDEmbleton,S.Chattopadhyay,andE.S.Ho,"Identification of abnormal movements in infants:A deep neural networkfor body part-based prediction of cerebral palsy,"IEEE Access,vol.9,pp.94281-94292,2021; [3]KDMcCay,ESHo,HPShum,G.Fehringer,C.Marcroft,andN.D.Embleton,"Abnormal infant movements classification with deep learning onpose-based features,"IEEE Access, vol.8, pp.51582-51592, 2020.) Its technical solution is to use deep learning methods or traditional machine learning methods to classify the entire input video into two categories. For classification judgment, only the normal and abnormal classification is achieved, and the developmental level of the infant is unclear; secondly, there are also some methods ([1] Q.Wu et al., "Automatically measure the quality of infants'spontaneous movement via videosto predict the risk of cerebral palsy," IEEE Transactions on Instrumentation and Measurement, vol.70, pp.1-11, 2021; [2] Patent No.: ZL202011616536.8. (Unrelated text follows: "An Intelligent Assessment System for Infant Three-Dimensional Spontaneous Movement Based on Complex Networks"). Its technical solution uses complex networks to quantify infants' three-dimensional movement postures using a single indicator. This can quantify the quality of infants' spontaneous movements, but the single indicator is insufficient for accurate quantification to facilitate targeted intervention by doctors. Third, the existing methods mentioned above only provide results, and the interpretability of the results is limited.) Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a quantitative assessment system for infant brain development based on graph convolutional networks, which can realize the quantitative assessment of infant spontaneous movements with multi-dimensional indicators. At the same time, the assessment results have a certain degree of interpretability, which can facilitate physicians to carry out personalized intervention.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A quantitative assessment system for infant brain development based on a graph convolutional network includes a posture estimation module, a sequence preprocessing module, a feature extraction module, a graph convolution classification module, a multidimensional indicator analysis module, and a result visualization module; the input of the posture estimation module is an infant movement video, the output of the posture estimation module is connected to the input of the sequence preprocessing module, the output of the sequence preprocessing module is connected to the input of the feature extraction module, the output of the feature extraction module is connected to the input of the graph convolution classification module, the output of the graph convolution classification module is connected to the input of the multidimensional indicator analysis module, and the output of the multidimensional indicator analysis module is connected to the input of the result visualization module;

[0007] The baby movement video is a video of the baby's spontaneous movement in the supine position, including the baby's entire body and limbs; the baby movement video uses depth movement video and color movement video;

[0008] The posture estimation module realizes posture recognition and extraction of baby movement videos;

[0009] The sequence preprocessing module implements preprocessing such as interpolation, smoothing filtering and normalization of the baby posture sequence coordinates; the sequence preprocessing module includes a sequence interpolation module, a median filter and a mean smoothing filter module and a normalization module;

[0010] The feature extraction module realizes the digital feature representation of the baby posture sequence coordinates, including a sequence segmentation module and a feature calculation module;

[0011] The graph convolution classification module realizes classification of the segmented subsequences;

[0012] The multi-dimensional index analysis module realizes the statistics and analysis of the indicators of multiple limbs of the infant, including a multi-limb index statistics module and a multi-limb analysis module;

[0013] The result visualization module realizes the visualization of the infant's movement quality assessment results and assists doctors in making judgments.

[0014] The baby movement video is a two-dimensional posture or a three-dimensional posture of the baby lying down.

[0015] The posture estimation module divides the joint points of posture recognition into 4 body parts, namely part1 = {left shoulder, left elbow, left wrist}, part2 = {right shoulder, right elbow, right wrist}, part3 = {left hip, left knee, left ankle}, and part4 = {right hip, right knee, right ankle}.

[0016] The sequence interpolation module is used to calculate the joint point sequence of all body parts {J i,t (x, y)}, i is the definition of the joint point of the corresponding body part, t is the time frame in the sequence, x and y are the coordinates of the corresponding key points respectively; select the time window of duration and step length, and use linear interpolation to interpolate {J i,t (x,y)} interpolation;

[0017] The median filter and mean smoothing filter modules are used to filter the interpolated {J i,t (x, y)} is median filtered to remove the noise and jitter effects in wild points and posture recognition; a time window is selected and the {J i,t (x, y)} select the median as the coordinate of the current time t; then filter {J i,t (x, y)} is smoothed and filtered, a time window with a duration and a step size is selected, and the average value of the coordinates in the time window is calculated as the coordinate value at the current time t;

[0018] The normalization module first calculates the angles between the associated bones, normalizes all bones to a unit length of 1, and re-associates the corresponding bones with a length of 1 according to the calculated angles to form a normalized bone model.

[0019] The sequence segmentation module selects a time window of duration and step length, segments the four body parts respectively, and obtains a clip subset of the four body parts, which is defined as {C b |b=part1,part2,part3,part4}, experts pre-calibrated part {C b} are labeled as Class A: no motion, Class B: normal motion, Class C: abnormal motion, and a training dataset is constructed;

[0020] The feature calculation module uses the normalized coordinates, velocity, and acceleration motion parameters of each clip joint as features, and the coordinates, velocity, and acceleration can be freely combined;

[0021] The speed calculation formula is as follows:

[0022] V i，t =J i,t+1 -J i,t , where V i，t is the velocity of the i-th joint at time t, J i,t+1 is the coordinate of the i-th joint point at time t+1;

[0023] The acceleration calculation formula is as follows:

[0024] A i，t =J i,t+1 +J i,t-1 -2*J i,t , where A i，t is the acceleration of the i-th joint at time t, then the characteristic X of the i-th joint at time t is i，t Expressed as:

[0025] X i，t =[J i,t ,V i，t ,A i，t ]

[0026] Define the clip window length as L, then the feature sequence of each clip is expressed as {X i，t}=[X i，t ,X i，t+1 ,…,X i，t+L ].

[0027] The multi-limb part index statistics module uses the graph convolution model to b} classification results are counted to obtain the number of samples of the three categories of ABC for each body part, which are num(A b ), num(B b ) and num(C b );

[0028] The multi-limb part analysis module analyzes the movement quality of each body part and proposes the activity index M of each body part of the baby. b and normal exercise ratio N b , the two indicators are calculated by the following formula:

[0029] M b =(num(B b )+num(C b )) / (num(A b )+num(B b )+num(C b ))

[0030] N b =num(B b ) / (num(B b )+num(C b ))

[0031] When M b The bigger the value, the more active the baby's corresponding body part movement is; b The smaller the size, the smaller the activity of the corresponding body part of the baby, which means the baby's movement quality is poor; N b The larger the value, the more normal movement of the corresponding body part of the baby is, and the lower the risk of cerebral palsy is; on the contrary, the baby has a higher risk of cerebral palsy. b and N b Combined, the risk of infants suffering from different types of cerebral palsy is judged, and thresholds th1 and th2 are selected to achieve the final conclusion output.

[0032] The result visualization module performs result visualization output on the four body parts and can locate the position where the abnormality exists in each body part.

[0033] The beneficial effects of the present invention are as follows:

[0034] (1) Since the system of the present invention adopts multi-index evaluation, the output results can realize the output of multiple abnormal categories;

[0035] (2) Since the output of the present invention includes interpretable visual results, it is convenient to assist doctors in targeted treatment intervention;

[0036] (3) Since the system of the present invention uses automated processing of video input, the system efficiency is high and the system modules can realize automated calculation and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural block diagram of the present invention.

[0038] Figure 2 This is a diagram showing the position markings of human joints according to an embodiment of the present invention.

[0039] Figure 3 1 is a preprocessing change curve of the x-coordinate of a certain joint of an infant according to an embodiment of the present invention. FIG. 1 is the original curve for posture recognition; FIG. 1 is the curve after linear interpolation; and FIG. 1 is the curve after median filtering.

[0040] Figure 4 Schematic diagram of the graph convolution classification module.

[0041] Figure 5This is a schematic diagram of the visualization results of the video analysis results of the abnormal baby in the embodiment. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0043] Reference Figure 1 A quantitative evaluation system for infant brain development based on graph convolutional network includes a posture estimation module, a sequence preprocessing module, a feature extraction module, a graph convolution classification module, a multidimensional index analysis module and a result visualization module; the input of the posture estimation module is the infant movement video, the output of the posture estimation module is connected to the input of the sequence preprocessing module, the output of the sequence preprocessing module is connected to the input of the feature extraction module, the output of the feature extraction module is connected to the input of the graph convolution classification module, the output of the graph convolution classification module is connected to the input of the multidimensional index analysis module, and the output of the multidimensional index analysis module is connected to the input of the result visualization module; the infant movement video is a video of the infant's spontaneous movement in the supine position, including the infant's whole body limbs; the infant movement video uses depth motion video and color motion video.

[0044] The posture estimation module realizes posture recognition and extraction of infant movement videos, and divides the joint points of posture recognition into four body parts, such as Figure 2 As shown, part1 = {left shoulder, left elbow, left wrist}, part2 = {right shoulder, right elbow, right wrist}, part3 = {left hip, left knee, left ankle}, part4 = {right hip, right knee, right ankle}; after being processed by the posture estimation module, we can get Figure 2 A sequence of joint coordinates for all defined joint points.

[0045] Furthermore, the obtained sequence is input into a sequence preprocessing module, which implements preprocessing such as interpolation, smoothing filtering and normalization of the infant posture sequence coordinates. The sequence preprocessing module includes a sequence interpolation module, a median filter and a mean smoothing filter module and a normalization module.

[0046] like Figure 3 As shown in (a), the sequence interpolation module performs all joint point sequences of the body part {J i,t (x, y)}, i is the definition of the joint point of the corresponding body part, t is the time frame in the sequence, x and y are the coordinates of the corresponding key points; select the time window with appropriate duration and step size, and use linear interpolation to interpolate {J i,t (x,y)} interpolation.

[0047] The median filter and mean smoothing filter modules are used to filter the interpolated {J i,t(x, y)} is median filtered to remove the noise and jitter effects in wild points and posture recognition; an appropriate time window is selected to filter {J i,t (x,y)} select the median as the coordinate of the current time t; then Figure 3 As shown in (b), after median filtering, {J i,t (x,y)} is smoothed and filtered, and a time window with appropriate duration and step size is selected, such as Figure 3 As shown in (c), the average value of the coordinates in the time window is calculated as the coordinate value at the current time t.

[0048] The normalization module first calculates the angles between the associated bones, normalizes all bones to a unit length of 1, and re-associates the corresponding bones with a length of 1 according to the calculated angles to form a normalized bone model.

[0049] Next, all normalized joint sequences are subjected to feature extraction in the feature extraction module; the feature extraction module mainly realizes the digital feature representation of the coordinates of the infant posture sequence, including the sequence segmentation module and the feature calculation module. The sequence segmentation module selects a time window of a certain duration and step size, and segments the four body parts respectively, obtaining the four body part clip subsets, which are defined as {C b |b=part1,part2,part3,part4}, the clip set after segmentation is as follows Figure 4 As shown in the input; experts pre-calibrated some {C b}, which are Class A: no motion, Class B: normal motion, Class C: abnormal motion, and a training dataset of a certain scale is constructed.

[0050] The feature calculation module uses the normalized coordinates, velocity, and acceleration motion parameters of each clip joint as features;

[0051] The speed calculation formula is as follows:

[0052] V i，t =J i,t+1 -J i,t , where V i，t is the velocity of the i-th joint at time t, J i,t+1 is the coordinate of the i-th joint point at time t+1;

[0053] The acceleration calculation formula is as follows:

[0054] A i，t =J i,t+1 +J i,t-1 -2*J i,t , where A i，tis the acceleration of the i-th joint at time t. Then the characteristic X of the i-th joint at time t is i，t It can be expressed as:

[0055] X i，t =[J i,t ,V i，t ,A i，t ]

[0056] Define the clip window length as L, then the feature sequence of each clip can be expressed as {X i，t}=[X i，t ,X i，t+1 ,…,X i，t+L ].

[0057] The graph convolution classification module mainly realizes the classification of the segmented subsequences. The calculation process is as follows: Figure 4 As shown in the figure, the input is the features of all clip sets of each body part, the superposition of feature layers and activation functions (ReLU), and finally the classification result of each clip is obtained (0: no movement, 1: normal movement, 2: abnormal movement); by training each body part classification, four corresponding graph convolution classification models are obtained respectively. The feature sequence to be classified is input into the corresponding model, and the classification results of all clips of the corresponding body part to be analyzed can be obtained.

[0058] Next, a multi-dimensional index analysis module is used to perform statistical analysis on all clip classification results. The multi-dimensional index analysis module includes a multi-limb part index statistical module and a multi-limb part analysis module; the multi-limb part index statistical module converts the graph convolution model into {C b}, we can get the number of samples of the three categories of ABC for each body part, which are num(A b ), num(B b ) and num(C b ).

[0059] The multi-limb part analysis module can analyze the movement quality of each body part and propose the activity index M of each body part of the baby. b and normal exercise ratio N b , the two indicators are calculated by the following formula:

[0060] M b =(num(B b )+num(C b )) / (num(A b )+num(B b )+num(C b ))

[0061] N b = num(B b ) / (num(B b ) + num(C b ))

[0062] When M b is larger, it indicates that the baby's movement of the corresponding body part is more active; when M b is smaller, the baby's activity of the corresponding body part is less, indicating that the baby's movement quality is poor, the risk of suffering from cerebral palsy is higher, and it tends to be the non - movement type of cerebral palsy; when N b is larger, it means that the proportion of normal movement of the baby's corresponding body part is relatively large, and the risk of cerebral palsy is small; on the contrary, the baby has a greater risk of suffering from cerebral palsy; by combining M b and N b , the risk of the baby suffering from different types of cerebral palsy can be judged. By selecting appropriate thresholds th1 and th2, when M b > th1 and N b > th2, the baby has a lower risk of suffering from cerebral palsy; when M b < th1 and N b > th2, the baby has a certain risk of cerebral palsy, and it is recommended to strengthen observation and training; when M b > th1 and N b < th2, the baby has a higher risk of suffering from cerebral palsy and is highly suspected of having spastic cerebral palsy; when M b < th1 and N b < th2, the baby has a higher risk of suffering from cerebral palsy and is highly suspected of having non - movement - lacking cerebral palsy.

[0063] Finally, the classification results can be visualized using the above - mentioned result visualization module to assist the doctor in making judgments. Figure 5 It is a visualization schematic of the video analysis result of the abnormal baby in the embodiment. As can be seen from [[ID=​

Claims

1. A quantitative assessment system for infant brain development based on graph convolutional networks, characterized by: It includes a posture estimation module, a sequence preprocessing module, a feature extraction module, a graph convolution classification module, a multidimensional index analysis module and a result visualization module; the input of the posture estimation module is the infant movement video, the output of the posture estimation module is connected to the input of the sequence preprocessing module, the output of the sequence preprocessing module is connected to the input of the feature extraction module, the output of the feature extraction module is connected to the input of the graph convolution classification module, the output of the graph convolution classification module is connected to the input of the multidimensional index analysis module, and the output of the multidimensional index analysis module is connected to the input of the result visualization module; The baby movement video is a video of the baby's spontaneous movement in the supine position, including the baby's entire body and limbs; the baby movement video uses depth movement video and color movement video; The posture estimation module realizes posture recognition and extraction of baby movement videos; The sequence preprocessing module implements preprocessing such as interpolation, smoothing filtering and normalization of the baby posture sequence coordinates; the sequence preprocessing module includes a sequence interpolation module, a median filter and a mean smoothing filter module and a normalization module; The feature extraction module realizes the digital feature representation of the baby posture sequence coordinates, including a sequence segmentation module and a feature calculation module; The graph convolution classification module realizes classification of the segmented subsequences; The multi-dimensional index analysis module realizes the statistics and analysis of the indicators of multiple limbs of the infant, including a multi-limb index statistics module and a multi-limb analysis module; The result visualization module realizes the visualization of the infant's movement quality assessment results and assists the physician in making judgments; The sequence segmentation module selects a time window of duration and step length, segments the four body parts respectively, and obtains a clip subset of the four body parts, which is defined as {C b |b=part1,part2,part3,part4}, experts pre-calibrated part {C b } are labeled as Class A: no motion, Class B: normal motion, Class C: abnormal motion, and a training dataset is constructed; The feature calculation module uses the normalized coordinates, velocity, and acceleration motion parameters of each clip joint as features, and the coordinates, velocity, and acceleration can be freely combined; The speed calculation formula is as follows: V i，t =J i,t+1 -J i,t , where V i，t is the velocity of the i-th joint at time t, J i,t+1 is the coordinate of the i-th joint point at time t+1; The acceleration calculation formula is as follows: A i，t =J i,t+1 +J i,t-1 -2*J i,t , where A i，t is the acceleration of the i-th joint at time t, then the characteristic X of the i-th joint at time t is i，t Expressed as: X i，t =[J i,t ,V i，t ,A i，t ] Define the clip window length as L, then the feature sequence of each clip is expressed as {X i，t }=[X i，t ,X i，t+1 ,…,X i，t+L ].

2. The system according to claim 1, wherein: The baby movement video is a two-dimensional posture or a three-dimensional posture of the baby lying down.

3. The system according to claim 1, wherein: The posture estimation module divides the joint points of posture recognition into 4 body parts, namely part1 = {left shoulder, left elbow, left wrist}, part2 = {right shoulder, right elbow, right wrist}, part3 = {left hip, left knee, left ankle}, and part4 = {right hip, right knee, right ankle}.

4. The system according to claim 3, wherein: The sequence interpolation module is used to calculate the joint point sequence of all body parts {J i,t (x, y)}, i is the definition of the joint point of the corresponding body part, t is the time frame in the sequence, x and y are the coordinates of the corresponding key points respectively; select the time window of duration and step length, and use linear interpolation to interpolate {J i,t (x,y)} interpolation; The median filter and mean smoothing filter modules are used to filter the interpolated {J i,t (x,y)} to perform median filtering to remove the effects of noise and jitter in wild points and posture recognition; Select a time window and calculate {J i,t (x, y)} select the median as the coordinate of the current time t; then filter {J i,t (x, y)} is smoothed and filtered, a time window with a duration and a step size is selected, and the average value of the coordinates in the time window is calculated as the coordinate value at the current time t; The normalization module first calculates the angles between the associated bones, normalizes all bones to a unit length of 1, and re-associates the corresponding bones with a length of 1 according to the calculated angles to form a normalized bone model.

5. The system according to claim 1, wherein: The multi-limb part index statistics module uses the graph convolution model to b } classification results are counted to obtain the number of samples of the three categories of ABC for each body part, which are num(A b ), num(B b ) and num(C b ); The multi-limb part analysis module analyzes the movement quality of each body part and proposes the activity index M of each body part of the baby. b and normal exercise ratio N b , the two indicators are calculated by the following formula: M b =(num(B b )+num(C b )) / (num(A b )+num(B b )+num(C b )) N b =num(B b ) / (num(B b )+num(C b )) When M b The bigger the value, the more active the baby's corresponding body part movement is; b The smaller the size, the smaller the activity of the corresponding body part of the baby, which means the baby's movement quality is poor; N b The larger the value, the more normal movement of the corresponding body part of the baby, and the lower the risk of cerebral palsy. On the contrary, the baby is at greater risk of cerebral palsy; b and N b Combined, the risk of infants suffering from different types of cerebral palsy is judged, and thresholds th1 and th2 are selected to achieve the final conclusion output.

6. The system according to claim 5, characterized in that: The result visualization module performs result visualization output on the four body parts, and can locate the position where the abnormality exists in each body part.

Images