A quantitative assessment system for infant brain development based on a fusion model

Abstract

A fusion model-based quantitative assessment system for infant brain development includes a posture estimation module, a coordinate preprocessing module, a feature extraction module, a decision tree classification module, a graph convolution classification module, a multidimensional index analysis module, and a result visualization module. The posture estimation module receives an infant movement video as input, the posture estimation module output is connected to the coordinate preprocessing module input, the coordinate preprocessing module output is connected to the feature extraction module input, the feature extraction module output is connected to the decision tree classification module input, the decision tree classification module first output is connected to the graph convolution classification module input, the decision tree classification module second output and the graph convolution classification module output are connected to the multidimensional index analysis module input, and the multidimensional index analysis module output is connected to the result visualization module input. The present invention performs multidimensional and precise quantitative assessment of infant brain development, while providing interpretable results to assist physicians in conducting targeted treatment.

Description

Technical Field

[0001] The present invention belongs to the technical field of infant behavior analysis and evaluation, and particularly relates to a quantitative evaluation system for infant brain development based on a fusion model. Background Art

[0002] Pediatricians can observe and analyze the spontaneous movement patterns of infants at the bedside based on their experience and diagnose the brain development status of infants. However, this type of diagnosis relies on the subjective experience of physicians, resulting in a high rate of missed diagnosis and misdiagnosis. At present, there are also some video-based intelligent classification systems ([1] KDMcCay et al., "A pose-based feature fusion and classification framework for the early prediction of cerebral palsy ininfants," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 30, pp. 8-19, 2021; [2] D. Sakkos, KDMccay, C. Marcroft, NDEmbleton, S. Chattopadhyay, and ESHo, "Identification of abnormal movements in infants: A deep neural network for body part-based prediction of cerebral palsy ininfants," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 30, pp. 8-19, 2021; palsy,"IEEE Access, vol. 9, pp. 94281-94292, 2021; [3] KDMcCay, ESHo, HPShum, G. Fehringer, C. Marcroft, and NDEmbleton,"Abnormal infant movements classification with deep learning on pose-based features," IEEE Access, vol. 8, pp. 51582-51592, 2020.), which can input infant movement videos and output normal or abnormal conclusions. The technical solution is to use deep learning methods or traditional machine learning methods to classify the entire input video into two categories. However, this type of method has poor interpretability and a relatively rough evaluation, making it difficult to meet the multi-dimensional and fine quantitative evaluation of the quality of infant spontaneous behavior, further making personalized treatment difficult to achieve. Summary of the Invention

[0003] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a quantitative assessment system for infant brain development based on a fusion model, which can make multi-dimensional and precise quantitative judgments on the infant's brain development, while providing interpretable results to assist physicians in carrying out targeted treatment.

[0004] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0005] A quantitative assessment system for infant brain development based on a fusion model includes a posture estimation module 1, a coordinate preprocessing module 2, a feature extraction module 3, a decision tree classification module 4, a graph convolution classification module 5, a multidimensional index analysis module 6, and a result visualization module 7; the input of the posture estimation module 1 is an infant movement video, the output of the posture estimation module 1 is connected to the input of the coordinate preprocessing module 2, the output of the coordinate preprocessing module 2 is connected to the input of the feature extraction module 3, the output of the feature extraction module 3 is connected to the input of the decision tree classification module 4, the first output of the decision tree classification module 4 is connected to the input of the graph convolution classification module 5, the second output of the decision tree classification module 4 and the output of the graph convolution classification module 5 are connected to the input of the multidimensional index analysis module 6, and the output of the multidimensional index analysis module 6 is connected to the input of the result visualization module 7;

[0006] The posture estimation module 1 realizes posture extraction of the input baby movement video;

[0007] The coordinate preprocessing module 2 realizes interpolation, median filtering and smoothing filtering of the posture coordinate sequence;

[0008] The feature extraction module 3 realizes the segmentation and calculation of the motion parameters of each joint coordinate sequence to form a subsequence feature set of each joint;

[0009] The decision number classification module 4 realizes the binary classification of whether the joint subsequence set is moving or not, and inputs the subsequence set classified as moving into the graph convolution classification module 5, and inputs the subsequence set classified as non-moving into the multidimensional index analysis module 6;

[0010] The graph convolution classification module 5 realizes the binary classification of whether the motion pattern of the motion subsequence is normal, and inputs the result into the multi-dimensional indicator analysis module 6;

[0011] The multidimensional index analysis module 6 integrates the classification results of the decision tree classification module 4 and the graph convolution classification module 5 to form a series of comprehensive evaluation indicators to assist physicians in intelligently evaluating the quality of infant movement;

[0012] The result visualization module 7 realizes the comprehensive classification results of the decision tree classification module 4 and the graph convolution classification module 5, and marks the location where the abnormal motion pattern appears.

[0013] The advantages of the present invention are as follows:

[0014] (1) Since the system of the present invention can output multi-dimensional indicators, it can realize the precise and objective evaluation of infant movement patterns based on multi-dimensional indicators, making it easier for doctors to carry out targeted treatment.

[0015] (2) Since the system of the present invention can output interpretable visualization results, it can assist doctors in locating the location and joints of abnormal movement patterns, making it easier for doctors to carry out targeted treatment.

[0016] (3) The system of the present invention has the advantage of high efficiency due to the use of intelligent automated processing methods of fusion models, which is convenient for large-scale promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural block diagram of the present invention.

[0018] Figure 2 This is a diagram illustrating the position markings of the human body joints and different part definitions of embodiment 14 of the present invention.

[0019] Figure 3 Figure 2 is a curve showing the change of the x-coordinate of a joint of an infant according to an embodiment of the present invention. Figure (a) is the original curve for posture recognition; Figure (b) is the curve after linear interpolation; and Figure (c) is the curve after median filtering and mean filtering.

[0020] Figure 4 Schematic diagram of the fusion model of the present invention.

[0021] Figure 5 This is a schematic diagram of the visualization results of a certain infant according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0023] Reference Figure 1 , a quantitative assessment system for infant brain development based on a fusion model, comprising a posture estimation module 1, a coordinate preprocessing module 2, a feature extraction module 3, a decision tree classification module 4, a graph convolution classification module 5, a multidimensional index analysis module 6 and a result visualization module 7; the input of the posture estimation module 1 is an infant movement video, the output of the posture estimation module 1 is connected to the input of the coordinate preprocessing module 2, the output of the coordinate preprocessing module 2 is connected to the input of the feature extraction module 3, the output of the feature extraction module 3 is connected to the input of the decision tree classification module 4, the first output of the decision tree classification module 4 is connected to the input of the graph convolution classification module 5, the second output of the decision tree classification module 4 and the output of the graph convolution classification module 5 are connected to the input of the multidimensional index analysis module 6, and the output of the multidimensional index analysis module 6 is connected to the input of the result visualization module 7;

[0024] The posture estimation module 1 realizes posture extraction of the input baby movement video. The baby movement video is a two-dimensional or three-dimensional posture of the baby lying down, which is a color motion video or a depth motion video. The posture estimation module 1 divides the joint points of posture recognition into four body parts, such as Figure 2 As shown in the figure, they are defined as part1 = {left shoulder, left elbow, left wrist}, part2 = {right shoulder, right elbow, right wrist}, part3 = {left hip, left knee, left ankle}, and part4 = {right hip, right knee, right ankle}. After being processed by the posture estimation module, we can get Figure 2 Joint coordinate sequence of all defined joint points;

[0025] The coordinate preprocessing module 2 realizes interpolation, median filtering, smoothing filtering and normalization of the posture coordinate sequence; specifically: Figure 3 As shown in (a), the sequence interpolation is performed on the joint point sequence of all body parts {J i,t (x, y)}, i is the definition of the joint point of the corresponding body part, t is the time frame in the sequence, x and y are the coordinates of the corresponding key points; select the time window with appropriate duration and step size, and use linear interpolation to interpolate {J i,t (x,y)} interpolation;

[0026] Then, the interpolated {J i,t (x, y)} is median filtered to remove the noise and jitter effects in wild points and posture recognition; an appropriate time window is selected and the {J i,t (x,y)} select the median as the coordinate of the current time t; then Figure 3 As shown in (b), after median filtering, {J i,t (x,y)} is smoothed and filtered, and a time window with appropriate duration and step size is selected, such as Figure 3 As shown in (c), the average value of the coordinates in the time window is calculated as the coordinate value at the current time t;

[0027] Then the joint coordinate sequence is normalized. First, the angles between the associated bones are calculated and all bones are normalized to a unit length of 1. Then, the corresponding bones with a length of 1 are re-associated according to the calculated angles to form a normalized skeletal model.

[0028] The feature extraction module 3 realizes the segmentation and calculation of the motion parameters of each joint coordinate sequence, forms a subsequence feature set of each joint, and realizes the digital feature representation of the infant posture sequence coordinates; the feature extraction module 3 includes a sequence segmentation module and a feature calculation module;

[0029] The sequence segmentation module selects a time window with a certain duration and step size, and segments the four body parts respectively to obtain the clip subsets of the four body parts, which are defined as {C b |b=part1, part2, part3, part4};

[0030] The feature calculation module uses the normalized coordinates, velocity, and acceleration motion parameters of each clip joint as features, and the coordinates, velocity, and acceleration can be freely combined;

[0031] The speed calculation formula is as follows:

[0032] V i，t =J i,t+1 -J i,t , where V i，t is the velocity of the i-th joint at time t, J i,t+1 is the coordinate of the i-th joint point at time t+1;

[0033] The acceleration calculation formula is as follows:

[0034] A i，t =J i,t+1 +J i,t-1 -2*J i,t , where A i，t is the acceleration of the i-th joint at time t; then the characteristic X of the i-th joint at time t i，t Expressed as:

[0035] X i，t =[J i,t ,V i，t ,A i，t ]

[0036] Define the window length of clip as L, then the feature sequence of each clip can be expressed as:

[0037] {X i，t}=[X i，t ,X i，t+1 ,…,X i，t+L ];

[0038] The decision number classification module 4 realizes the binary classification of whether the joint subsequence set is moving or not, and inputs the subsequence set classified as moving into the graph convolution classification module 5, and inputs the subsequence set classified as non-moving into the multidimensional index analysis module 6; specifically: the decision tree classification module 4 is used to classify the input clip feature sequence, such as Figure 4As shown in the upper part, based on the speed characteristics of the input clip sequence, appropriate limit threshold th1 and ratio threshold th2 are set. When the proportion of frames in the clip with a speed less than the limit threshold th1 exceeds the ratio threshold th2, the clip is classified as non-moving and does not participate in subsequent classification; otherwise, the clip is classified as moving and participates in the subsequent convolution classification module 5 classification;

[0039] The graph convolution integration module 5 realizes the binary classification of whether the motion pattern of the motion subsequence is normal, and inputs the result into the multi-dimensional index analysis module 6; specifically, the graph convolution integration module 5 realizes the secondary classification of the clip subsequence classified as the motion state by the decision tree, such as Figure 4 In the lower part, the input is the features of all clips of each body part, the superposition of feature layers and activation functions (ReLU), and finally the classification result of each clip is obtained (0: normal movement, 1: abnormal movement); experts pre-calibrated some {C b}, with Class A: normal movement and Class B: abnormal movement. A certain scale of training data set was constructed. By training each body part classification, four corresponding graph convolution classification models were obtained. The feature sequence to be classified was input into the corresponding graph convolution classification model, and the classification results of all clips corresponding to the body part to be analyzed were obtained.

[0040] The multi-dimensional index analysis module 6 integrates the classification results of the decision tree classification module 4 and the graph convolution classification module 5 to form a series of comprehensive evaluation indicators to assist doctors in intelligently evaluating the quality of infant movement. Specifically, the multi-dimensional index analysis module 6 performs statistical analysis on all clip classification results, and classifies the clips classified as non-movement in the decision tree classification module 4 and the clips classified as {C b}, we can get the number of samples of the three categories (A: normal movement, B: abnormal movement, C: no movement) for each body part, which are num(A b ), num(B b ) and num(C b ); Next, analyze the movement quality of each body part and propose the activity index M of each body part of the baby b and normal exercise ratio N b , the two indicators are calculated by the following formula:

[0041] M b =(num(A b )+num(B b )) / (num(A b )+num(B b )+num(Cb ))

[0042] N b = num(A b ) / (num(A b ) + num(B b ))

[0043] When M b is larger, it indicates that the baby's movement of the corresponding body part is more active; when M b is smaller, the baby's activity of the corresponding body part is less, indicating that the baby's movement quality is poor, and the risk of suffering from cerebral palsy is higher, tending to the absence of不安运动 (it should be a specific term, and the accurate translation is missing here) type of cerebral palsy; when N b is larger, it means that the proportion of normal movement of the baby's corresponding body part is relatively large, and the risk of cerebral palsy is smaller; on the contrary, the baby has a greater risk of suffering from cerebral palsy; combining M b and N b , the risk of the baby suffering from different types of cerebral palsy can be judged. Select appropriate thresholds th3 and th4. When M b > th3, N b > th4, the baby has a lower risk of suffering from cerebral palsy; when M b < th3, N b > th4, the baby has a certain risk of cerebral palsy, and it is recommended to strengthen observation and training; when M b > th3, N b < th4, the baby has a higher risk of suffering from cerebral palsy and is highly suspected of having spastic cerebral palsy; when M b < th3, N b < th4, the baby has a higher risk of suffering from cerebral palsy and is highly suspected of having the absence of不安运动 (it should be a specific term, and the accurate translation is missing here) type of cerebral palsy;

[0044] The result visualization module 7 realizes the comprehensive classification results of the decision tree classification module 4 and the graph convolution classification module 5, and marks the positions where abnormal movement patterns appear; specifically: using the result visualization module 7 to visualize the classification results to assist the doctor in judgment, Figure 5 It is a visualization schematic diagram of the video analysis result of a normal baby in the embodiment. It can be seen from the figure that only the lower limbs of the baby's four limbs bodypart have a little abnormal movement, and the proportion of normal movement of the four limbs is relatively high, so it can be determined that the baby is in a normal brain development state.

Claims

1. A quantitative assessment system for infant brain development based on a fusion model, characterized by: The method comprises a posture estimation module (1), a coordinate preprocessing module (2), a feature extraction module (3), a decision tree classification module (4), a graph convolution classification module (5), a multidimensional index analysis module (6) and a result visualization module (7); the input of the posture estimation module (1) is a baby movement video, the output of the posture estimation module (1) is connected to the input of the coordinate preprocessing module (2), the output of the coordinate preprocessing module (2) is connected to the input of the feature extraction module (3), the output of the feature extraction module (3) is connected to the input of the decision tree classification module (4), the first output of the decision tree classification module (4) is connected to the input of the graph convolution classification module (5), the second output of the decision tree classification module (4) and the output of the graph convolution classification module (5) are connected to the input of the multidimensional index analysis module (6), and the output of the multidimensional index analysis module (6) is connected to the input of the result visualization module (7); The posture estimation module (1) realizes posture extraction of the input baby movement video; The coordinate preprocessing module (2) implements interpolation, median filtering and smoothing filtering of the posture coordinate sequence; The feature extraction module (3) realizes the segmentation and calculation of the motion parameters of each joint coordinate sequence to form a subsequence feature set of each joint; The decision number classification module (4) realizes the binary classification of whether the joint subsequence set is moving or not, and inputs the subsequence set classified as moving into the graph convolution classification module (5), and inputs the subsequence set classified as non-moving into the multidimensional index analysis module (6); The graph convolution classification module (5) realizes the binary classification of whether the motion pattern of the motion subsequence is normal, and inputs the result into the multi-dimensional index analysis module (6); The multi-dimensional index analysis module (6) integrates the classification results of the decision tree classification module (4) and the graph convolution classification module (5) to form a series of comprehensive evaluation indicators to assist physicians in intelligently evaluating the quality of infant movement; The result visualization module (7) realizes the comprehensive classification results of the decision tree classification module (4) and the graph convolution classification module (5), and marks the location where the abnormal motion pattern appears; The feature extraction module (3) selects a time window of duration and step length, and segments the four body parts respectively to obtain a clip subset of the four body parts, which is defined as {C b |b=part1, part2, part3, part4}; calculate the normalized coordinates, velocity, and acceleration motion parameters of each clip joint as features, and the coordinates, velocity, and acceleration can be freely combined; The speed calculation formula is as follows: V i，t =J i,t+1 -J i,t , where V i，t is the velocity of the i-th joint at time t, J i,t+1 is the coordinate of the i-th joint point at time t+1; The acceleration calculation formula is as follows: A i，t =J i,t+1 +J i,t-1 -2*J i,t , where A i，t is the acceleration of the i-th joint at time t, then the characteristic X of the i-th joint at time t is i，t Expressed as: X i，t =[J i,t ,V i，t ,A i，t ] Define the clip window length as L, then the feature sequence of each clip is expressed as {X i，t }=[X i，t ,X i，t+1 ,…,X i，t+L ].

2. The system according to claim 1, wherein: The baby movement video is a two-dimensional posture or a three-dimensional posture of the baby lying down.

3. The system according to claim 1, wherein: The posture estimation module (1) divides the joint points of posture recognition into four body parts, namely part1 = {left shoulder, left elbow, left wrist}, part2 = {right shoulder, right elbow, right wrist}, part3 = {left hip, left knee, left ankle}, and part4 = {right hip, right knee, right ankle}.

4. The system according to claim 3, wherein: The coordinate preprocessing module (2) first processes the joint point sequences {J i,t (x, y)}, i is the definition of the joint point of the corresponding body part, t is the time frame in the sequence, x and y are the coordinates of the corresponding key points respectively; select the time window of duration and step length, and use linear interpolation to interpolate {J i,t (x,y)} interpolation; Then, the interpolated {J i,t (x,y)} to perform median filtering to remove the effects of noise and jitter in wild points and posture recognition; Select a time window and calculate {J i,t (x, y)} select the median as the coordinate of the current time t; then filter the median {J i,t (x, y)} is smoothed and filtered, a time window with a duration and a step size is selected, and the average value of the coordinates in the time window is calculated as the coordinate value at the current time t; The joint coordinate sequence is then normalized. First, the angles between the associated bones are calculated, and all bones are normalized to a unit length of 1. The corresponding bones with a length of 1 are re-associated according to the calculated angles to form a normalized skeletal model.

5. The system according to claim 4, characterized in that: The decision tree classification module (4) classifies the clips according to the speed features of the input clips, and the classification results are two states: no motion and motion.

6. The system according to claim 5, characterized in that: The multi-dimensional index analysis module (6) integrates the decision tree classification module (4) and the graph convolution classification module (5) to b }, and the number of samples of the three categories of ABC for each body part is obtained, which are num(A b ), num(B b ) and num(C b ); Analyze the movement quality of each body part and propose the activity index M of each body part of the baby b and normal exercise ratio N b , the two indicators are calculated by the following formula: M b =(num(A b )+num(B b )) / (num(A b )+num(B b )+num(C b )) N b =num(A b ) / (num(A b )+num(B b )) Where A represents normal movement, B represents abnormal movement, and C represents no movement; When M b The bigger the value, the more active the baby's corresponding body part movement is; b The smaller the size, the smaller the activity of the corresponding body part of the baby, which means the baby's movement quality is poor; N b The larger the value, the more normal movement of the corresponding body part of the baby, and the lower the risk of cerebral palsy. On the contrary, the baby is at greater risk of cerebral palsy; b and N b Combined, the risk of infants suffering from different types of cerebral palsy is judged, and thresholds th1 and th2 are selected to achieve the final conclusion output.

7. The system according to claim 6, characterized in that: The result visualization module (7) performs result visualization output on the four body parts and can locate the position where the abnormality exists in each body part.

Images