A new-born baby limb movement monitoring method based on a multi-task classification network

Abstract

The application discloses a newborn limb movement monitoring method based on a multi-task classification network. The application comprises a main task and an auxiliary task; wherein the main task is mainly to detect the movement of the limbs and head of a newborn, and to perform a multi-label classification task on the five categories of limbs and head through a classification network; the auxiliary task is mainly a multi-classification task for three movement states of the newborn, i.e. stillness, single-limb movement and multi-limb movement. Compared with a traditional monitoring method based on sensors and artificial clinical inspection, the application can assist a clinician in real-time monitoring and analysis of the limbs of a newborn on the basis of zero contact with the newborn. The application adopts a combined classification task and multi-label classification task to share features, thereby limiting the ambiguity in the identification of unconscious subtle movements of the newborn and improving the accuracy of limb movement identification. The application adopts a multi-label hard coding mode to mark the label of a video sample of limb movement of an infant, so that the unconscious limb movement of the infant can be better modeled.

Description

Technical Field

[0001] This invention belongs to the field of action recognition and intelligent medical auxiliary analysis, and proposes an intelligent video monitoring method for the limb movements of newborns based on multi-task and multi-label.

[0002] Background Story

[0003] According to the latest report from the World Health Organization (WHO), approximately one in ten babies worldwide are born prematurely (before 37 weeks of gestation). Compared to full-term infants, premature infants have immature organs, require external equipment support, and have potential pathological conditions. These issues need to be detected through early clinical observation, making the monitoring of physiological indicators and limb behavior in premature infants particularly important. However, the limb movements of premature infants are sporadic and irregular. Monitoring their condition often relies on regular inspections of incubators by caregivers. This monitoring data is often inaccurate and unrealistic due to staff fatigue and inter-clinical changes, making it difficult to respond promptly to emergencies. Therefore, there is an urgent need for an intelligent, automated monitoring system to achieve real-time limb monitoring of newborns.

[0004] Traditional neonatal intelligent monitoring mainly relies on wearable sensors, such as triaxial accelerometers, gyroscopes, and magnetometers, to record the limb movements of newborns. However, these contact-based sensors have drawbacks, including restricting infant movement, causing skin infections, and requiring repeated manual settings and calibrations, failing to meet monitoring needs in terms of comfort and convenience. This invention proposes a multi-task, multi-label learning-based neonatal intelligent video monitoring system using a contactless video monitoring method combined with a deep convolutional neural network for neonatal limb movement recognition. Compared to contactless sensors, this invention offers advantages such as contactlessness, real-time performance, and accuracy. Furthermore, it employs a joint multi-label classification network to achieve accurate classification of the infant's multi-limb coordinated movements; and uses multi-label soft coding for continuous frame-by-frame analysis of infant limb movements, overcoming the shortcomings of traditional monitoring methods that rely on manual intervention and state initialization. This invention plays a crucial role in real-time clinical monitoring of infants, enabling medical staff to understand the infant's physiological state, reduce medical stress, and improve medical efficiency. Summary of the Invention

[0005] This invention proposes a method for monitoring neonatal limb movements based on a multi-task classification network. The primary task involves detecting the movement of the newborn's limbs and head through multi-label classification of five categories using a classification network. The auxiliary task involves multi-classification of three movement states: stillness, single-limb movement, and multiple-limb movement. This invention enables video stream-based neonatal limb movement classification and real-time monitoring of neonatal limb movements, effectively suppressing multi-label classification errors and thus meeting the needs for intelligent video monitoring of neonatal limb movements and physiological states.

[0006] The technical solution of this invention mainly includes the following steps:

[0007] Step 1: Processing the newborn video dataset

[0008] First, the newborn monitoring videos are cropped according to limb movement categories, resulting in several video clips containing newborn limb movements. Based on the limb movements in the video clips, it is determined whether there are movements corresponding to the limbs and head, and these are arranged in an orderly fashion [head, left hand, right hand, left leg, right leg] as the primary multi-label classification task. For example, the label [0, 1, 0, 1, 0] means that the video clip contains a situation where the newborn's left hand and left foot move simultaneously. Based on the number of limb movements, three categories are divided: no limb movement, single limb movement, and multiple limb movements, serving as labels for the auxiliary classification task. Each video clip corresponds to only one limb movement quantity classification label.

[0009] Step 1 is performed as follows:

[0010] 1-1. Video clips are trimmed according to different newborns and different limb movement patterns to enhance the diversity of the dataset distribution. Based on the length requirements of subsequent network input data, the length of video clips is controlled within a few seconds to ensure the validity of the dataset.

[0011] 1-2. Define the limb movements of each video clip as labels such as [head, left hand, right hand, left leg, right leg], where 1 represents movement at that position and 0 represents no movement. Divide the number of limb movements into three categories: no limb movement, single limb movement, and multiple limb movement.

[0012] Step 2: For H×W×T video frame data, a 3D convolutional residual neural network is selected as the backbone network to construct a feature extraction network to extract spatial and temporal features. Hidden layers are shared between the two tasks using a hard parameter sharing approach. At the same time, the output layers of the limb movement quantity classification network specific to the auxiliary task and the output layers of the limb movement multi-label classification network specific to the main task are retained. The cross-entropy loss functions of the main task and the auxiliary task are used to train the network model together.

[0013] Step 2 is specifically operated as follows:

[0014] 2-1. A 3D convolutional residual neural network is selected as the backbone network, which consists of several 3D residual convolutional blocks. Each 3D residual convolutional block fuses low-level and high-level features and uses the fused features as the input to the next adjacent 3D residual convolutional block. This 3D convolutional residual neural network can also extract spatial and temporal features from video data for action recognition. These 3D residual convolutional blocks operate in the spatial and temporal dimensions to capture motion information in the video stream.

[0015] 2-2. The auxiliary task branch for classifying the number of limb movements consists of global average pooling, a fully connected layer, and a softmax layer. The softmax formula is as follows:

[0016]

[0017] Here, zj represents the value of the j-th category in the auxiliary classification task category z. The auxiliary classification task is divided into three categories based on the number of limb movements: no limb movement, single limb movement, and multiple limb movements. These categories serve as labels for the auxiliary classification task, and each video segment corresponds to only one limb movement quantity category label. The last dimension of the fully connected layer is set to the number of predicted limb movement quantity categories, and then the softmax layer outputs the probability value of each category.

[0018] 2-3. For the main task branch of multi-label classification of limb movements, a 3D convolutional residual block is used again to extract higher-level shared knowledge and progressively separate task-related parameters, making joint representation learning and sharing more efficient and flexible. The main task branch consists of global average pooling, fully connected layers, and a sigmoid function, as shown in the following formula:

[0019]

[0020] Here, x is the input, and this function allows us to generate confidence scores for multiple label categories of the head and limbs, and then use a fixed threshold to determine whether there is movement.

[0021] 2-4. A multi-task learning mechanism is used for training. The model employs a hard parameter sharing mechanism, meaning that the network layers before the output layers of the two tasks share parameters, while only the output layers correspond to their respective network parameters. Since the auxiliary classification task corresponds to the Coss-Etropy Loss (CEL) function, and the main multi-label classification task corresponds to the Binary Cross-Entropy Loss (BCEL) function, the loss function used is L... totalThis is a weighted superposition of the cross-entropy loss function and the binary cross-entropy loss function. Each loss function is defined as follows:

[0022] L total =λ1L lmnc +λ2L amc (3)

[0023] Where L lmnc Cross-entropy is defined as the loss function for the limb motion number classification task, as follows:

[0024]

[0025] Where x' is the logistic regression value input to the cross-entropy loss function, class is the label for the limb movement quantity classification, and L amc The binary cross-entropy loss function for action multi-label classification is described as follows:

[0026]

[0027] w=weight[c]·1{c≠ignore_index} (6)

[0028] Where, z' n Let y represent the prediction score for the nth sample as a positive example. n w represents the label of the nth sample. n Let λn represent the weight value of the nth sample, δ be the sigmoid activation function, and λ1 and λ2 be the weights. The aim is to balance the contributions of classification and multi-label classification tasks to the joint loss.

[0029] Step 3: Subsequently, several short video clips are cropped into video frame data at equal intervals, and the size of each frame image is adjusted to H×W. After adjusting the video frame data to the corresponding size to meet the network input requirements, it is fed into the network and the model is trained. Finally, it is determined whether the limbs and head are moving to achieve the function of assisted monitoring.

[0030] Step 3 is performed as follows:

[0031] 3-1. Randomly select T overlapping and continuous video segments from the video data generated in step 1 as network input, with input dimensions of C×T×H×W. During model training, the input video segments are obtained by using the difference frame extraction in random cropping, and then fed into a 3D convolutional residual neural network for feature extraction.

[0032] 3-2. The network architecture obtains feature maps from multiple information channels of adjacent video frame data, and performs convolution and subsampling in each channel. It convolves a cube formed by stacking multiple consecutive frames with a 3D kernel. The calculation formula is as follows:

[0033]

[0034] in, This represents the calculated value of the j-th feature map in the i-th layer, where tanh is the hyperbolic tangent function, and b... ij , P i Q i R i These represent the bias, weights, kernel height, kernel width, and kernel depth, respectively. The formula for calculating the dimension of each feature map is as follows:

[0035]

[0036]

[0037]

[0038] Among them, D out H out and W out These represent the time dimension, height, and width of the output feature map, respectively. p represents the padding parameter, d represents the dilation coefficient, k represents the kernel size, s represents the stride, and D... in H in and W in These represent the depth, height, and width of the input feature map, respectively.

[0039] In convolutional layers, pixel values ​​(feature values) of feature maps are obtained by weighting the local receptive fields of all channels from the previous layer using the convolutional kernel. A 3D convolutional kernel can only extract one type of feature at a time. Throughout the convolution process, the kernel weights are adjusted and parameters are shared through loss and feedback gradients during model training. By constructing spatiotemporal convolutional layers, the network model extracts initial video sequence features from consecutive frames of the initial video, achieving preliminary extraction of action information from the video sequence.

[0040] 3-3. After passing through a feature extraction network composed of several 3D convolutional residual blocks, the output features are separated by a task-specific output layer to train relevant task parameters and output predicted values. The specific steps are as follows:

[0041] 3-3-1. The convolution operation is represented as follows:

[0042]

[0043] Among them, F tr(.,θ) represents the standard convolution operation, θ represents the parameters of the convolution kernel, Y represents the input feature map of the convolutional layer with dimensions D'×H'×W'×C', and U represents the output feature map of the convolutional layer with dimensions D×H×W×C.

[0044] 3-3-2. The compression operation is a global average pooling operation performed on the output feature map U. The compression operation is represented as follows:

[0045]

[0046] During the compression operation, the output feature map of size D×H×W×C undergoes spatial compression, converting each three-dimensional feature channel into a one-dimensional vector, resulting in a one-dimensional vector of size C×1×1×1; c Represents feature maps U and F. sq This represents the compression operation function.

[0047] 3-3-3. The data is then fed into a fully connected layer. In the limb movement quantity classification task, the softmax layer outputs the probability value of each category, and the category with the highest probability is selected. In the limb movement multi-label classification task, the sigmoid function outputs the probability value of each limb movement, which is then judged and output based on a fixed threshold.

[0048] 3-4. Network training performance is assessed using average per-class Precision (CP), average per-class Recall (CR), average per-class F1 (CF1), and average overall Precision (OP), average overall Recall (OR), and average overall F1 (OF1). The formulas for each parameter are as follows:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] in For the i-th class, predict the number of videos that are correctly predicted. It is the number of videos predicted for the i-th class. This represents the number of real videos in the i-th category. Each indicator ranges from 0 to 1; the closer to 1, the better the model's prediction performance, and the closer to 0, the worse the model's prediction performance.

[0056] The beneficial effects of this invention are as follows:

[0057] This invention analyzes video motion in infant monitoring video streams based on a multi-task classification network. The social benefits of this invention are as follows: (1) Compared with traditional monitoring methods based on sensors and manual clinical inspections, this invention can assist clinicians in real-time monitoring and analysis of newborn limbs without contact with the newborn. (2) By using joint classification tasks and multi-label classification tasks to share features, the ambiguity in the recognition of unconscious subtle movements of newborns is limited, thus improving the accuracy of limb movement recognition. (3) By using multi-label hard coding to label the video samples of infant limb movements, it is possible to better model the unconscious limb movements of infants.

[0058] This invention enables the establishment of a precise and efficient video-based limb movement recognition architecture for newborns, overcoming the limitations of previous monitoring methods that relied solely on manual labor or devices attached to newborns. It significantly assists doctors in clinical monitoring, allowing medical staff to communicate promptly based on the different states of newborns, thereby quickly issuing medical orders, timely transferring infants, and addressing problems, potentially reducing medical workload and improving medical efficiency. Attached Figure Description

[0059] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0060] The implementation steps of the newborn intelligent monitoring and auxiliary analysis method based on video multi-task learning have been described in detail in the invention content. The specific network structure is shown in the figure. Figure 1 As shown in the accompanying drawings and specific embodiments, the present invention will now be described in detail. The technical solution of the present invention mainly includes the following steps:

[0061] Step 1: Preprocessing of the newborn video dataset

[0062] First, the collected videos of neonatal limb movements were cropped to obtain multiple 1-3 second video clips of neonatal limb movements. The video clips were then categorized by observing the neonatal movements, and multi-label classification tags were generated for a multi-limb classification task. Based on the number of limb movements, three categories were defined: no limb movement, single limb movement, and multiple limb movements. The number of limb movements was selected as the label for the auxiliary classification task.

[0063] Step 2: For the video segment data, a convolutional residual neural network in R2Plus1D is selected as the backbone network to extract spatiotemporal features from the input tensor. This invention uses a hard parameter sharing approach to share the weights for extracting video features between the two tasks, while retaining the output layer of the limb quantity classification network specific to the auxiliary task and the output layer of the multi-label limb motion classification network for the main task, such as... Figure 1 As shown, the main task branch and auxiliary task branch are combined with the cross-entropy loss function to realize the network model to achieve single-label and multi-label classification tasks.

[0064] Step 3: The video clips are then cropped into equal-length video sequences at equal intervals, adjusting the frame space size to 171×128. The video sequence data is cropped to the corresponding size to fit the network input, and the SGD neural network optimizer is selected for network training and optimization. The network model built in Step 2 is then fed into the model for iterative training. After training, the model can perform real-time inference and output the infant's limb movements, thus assisting in monitoring the newborn's limb movement status.

[0065] Step 1 is performed as follows:

[0066] 1-1. Video segments are trimmed according to the number of limb movements in different newborns. The video dataset contains all clinical newborn limb activities, reflecting the diversity of newborn limb movement distribution. Based on the training environment and network structure's requirements for the number of input video frames, the video segment length is controlled between 1-3 seconds to ensure efficient use of the video dataset.

[0067] 1-2. Observe the movement of newborns in video clips and assign two labels to each video: a limb category movement label and a multi-label classification label. The limb movement in each video clip is defined as the movement of [head, left hand, right hand, left leg, right leg], and encoded using a One-Hot method as [0,1,0,0,0] (when the left hand is moving). When multiple limbs are moving, the number of limb movements is used as the encoding. The encoding uses 0 and 1 to represent whether a specific limb is moving; 0 represents no movement, and 1 represents movement. The multi-label classification label uses the number of limb movements as the label to distinguish between no limb, single limb, and multiple limb movement categories. The dataset encoding process uses both multi-label classification and multi-label classification labels for dual labeling based on the movement within the video clips.

[0068] Step 2 is specifically operated as follows:

[0069] 2-1. Network Feature Extraction Structure: The R2Plus1D network is selected as the main network to extract spatial and temporal features from video data for action recognition. The main network performs separable convolutional operations in the spatial and temporal dimensions to capture motion information in the video stream. The main network mainly consists of three residual convolutional blocks, the structure of which is as follows: Figure 1 Conv3 x As shown, the model consists of spatial and spatiotemporal convolution steps, where the convolutional kernel is 1×3×3 with a stride of {1, 2, 2} and the convolutional kernel is 3×1×1 with a stride of {2, 1, 1}. Then, it is normalized using the 3D batch normalization (BN) function BatchNorm3d, and the activation function is ReLU. The 1×3×3 and 3×1×1 convolutional layers are combined in parallel to replace the standard 3×3×3 convolutional layer, aiming to reduce the computational efficiency of the model. Simultaneously, during the initial training phase, the model uses a pre-trained model on ImageNet for weight initialization.

[0070] 2-2. In the network branch for assisting limb movement, the model uses global average pooling to enhance the generalization effect of the fully connected layers. This replaces traditional fully connected layers to reduce model parameters, prevent overfitting, and thus accelerate network training. The last dimension of the fully connected layer is set to 3 nodes, representing the three categories of no limb, single limb, and multi-limb movement, respectively. The softmax layer outputs the probability values ​​of all categories and selects the category with the highest probability value for output.

[0071] 2-3. In the multi-class classification task of limb movement, the class information of the shared feature maps of the previous model is first extracted through Conv5_x convolutional blocks, and feature information irrelevant to the classification is gradually separated. The model makes joint representation learning and parameter sharing efficient and flexible. A sigmoid function is inserted in the last layer to calculate the classification probability. This function generates confidence scores for multiple class nodes, including head (H), left hand (LH), right hand (RH), left leg (LL), and right leg (RL), representing the probability of movement of each part. Then, a classification threshold of 0.5 is used to determine whether the limb is moving.

[0072] 2-4. The model is trained using a multi-task learning mechanism. A hard parameter sharing mechanism is employed to provide shared network layer weights for the output layers of the two subsequent tasks, and to provide the respective feature extraction results for the output layers of the branch networks. Since the auxiliary classification task corresponds to the Cross-Entropy Loss (CEL) loss function, and the multi-label classification task corresponds to the Binary Cross-Entropy Loss (BCEL) loss function, the loss function used is L...total This is a weighted superposition of the cross-entropy loss function and the binary cross-entropy loss function, specifically defined as follows:

[0073] L total =λ1L lmnc +λ2L amc (19)

[0074] Where L lmnc Cross-entropy is defined as the loss function for the limb motion number classification task, as follows:

[0075] L lmnc = -x[class] + log(∑ j exp([j])) (20)

[0076] Where x is the input to the cross-entropy loss function, class is the label for the limb movement quantity classification, and L amc The binary cross-entropy loss function for action multi-label classification is described as follows:

[0077]

[0078] w=weight[c]·1{c≠ignore_index} (22)

[0079] Where z n Let yn represent the predicted score of the nth sample as a positive example, δ be the sigmoid activation function, and λ1 and λ2 be the weight coefficients, which aim to balance the weights of classification and multi-label classification in the joint loss. In this model, λ1 = 1 and λ2 = 1 are used.

[0080] Step 3 is performed as follows:

[0081] 3-1. Randomly select 16 non-overlapping and consecutive video frame segments from the data generated in step 1 as network input. The input dimension is 3×16×128×171. During training, data augmentation is used to adjust the dimension to 3×16×112×112. The model selects the Stochastic Gradient Descent (SGD) optimizer, selecting one batch of video sequences each time instead of all samples. The model uses gradient descent to update the model parameters to speed up training. Then, the model loss is calculated according to the updated weights, and training continues iteratively.

[0082] 3-2. The model performs spatiotemporal convolution calculations sequentially on a single batch of video sequences. The information flow diagram is attached. Figure 1As shown, feature maps for multiple information channels are generated. Convolutional kernel weighting and subsampling operations are performed in each channel. By simultaneously stacking the weighted sum of feature maps from multiple channels, a B×C×D×H×W feature map is formed and then convolved with the next 3D kernel. The calculation formula is as follows:

[0083]

[0084] in, Let denot be the feature value of the j-th feature map in the i-th convolutional layer, where tanh is the hyperbolic tangent function, and b ij , P i Q i R i These represent the bias, weight, kernel height, kernel width, and kernel depth, respectively.

[0085] The formula for calculating the size of each feature map is as follows:

[0086]

[0087]

[0088]

[0089] Among them, D out H out and W out These represent the depth, height, and width of the output feature map, respectively. p represents the padding parameter, d represents the dilation coefficient (default is 1), k represents the pooling kernel size, s represents the stride, and D... in H in and W in These represent the depth, height, and width of the input feature map, respectively.

[0090] The value at a certain location in a feature map is obtained by convolving the local receptive fields of the same location in three consecutive channels of the previous layer. A 3D convolutional kernel can only extract one type of feature at a time, and the weights of the kernel remain the same throughout the entire convolution process (shared weights). Through this construction, the feature maps on the convolutional layer are connected to multiple consecutive frames from the previous layer, thereby capturing motion information.

[0091] 3-3. After passing through a feature extraction network consisting of three 3D convolutional residual blocks, the output features are separated by a task-specific output layer to train relevant task parameters and output predicted values. The specific steps are as follows:

[0092] 3-3-1. The convolution operation is represented as follows:

[0093]

[0094] Among them, F tr (.,θ) represents the standard convolution operation, θ represents the parameters of the convolution kernel, Y represents the input feature map of the convolutional layer with dimensions D'×H'×W'×C', and U represents the output feature map of the convolutional layer with dimensions D×H×W×C. The feature map output by the feature extraction network has dimensions of 256×4×14×14.

[0095] 3-3-2. The compression operation is a global average pooling operation performed on the output feature map U. The compression operation is represented as follows:

[0096]

[0097] During the compression operation, the output feature map of size D×H×W×C undergoes spatial compression, converting each three-dimensional feature channel into a one-dimensional vector, resulting in a one-dimensional vector of size C×1×1×1; c Represents feature maps U and F. sq This represents the compression operation function.

[0098] 3-3-3. The data is then passed to a fully connected layer. In the limb movement quantity classification task, the softmax layer outputs the probability value of each category, and the category with the highest probability is taken as the output category. In the limb movement multi-label classification task, the sigmoid function outputs the probability value of each limb movement, which is then judged and output using a 0.5 threshold, as shown below. Figure 1 As shown.

[0099] 3-4. The network training performance is measured using the average per-class Precision (CP), average per-class Recall (CR), average per-class F1 (CF1), and average overall Precision (OP), average overall Recall (OR), and average overall F1 (OF1). The formulas for each parameter are as follows:

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] in, For the i-th class, predict the number of videos that are correctly predicted. It is the number of videos predicted for the i-th class. This represents the number of real videos in the i-th category. Each metric ranges from 0 to 1; the closer to 1, the better the model's prediction performance, and the closer to 0, the worse the model's prediction performance.

[0107] The present invention also provides a newborn intelligent video monitoring system based on multi-task multi-label learning, specifically including a newborn video dataset processing module, a model building module, and a model training module;

[0108] The neonatal video dataset processing module first crops the collected clinical neonatal limb movement videos to obtain multiple 1-3s neonatal limb movement video clips. By observing the neonatal movements in the video clips, the video clips are labeled with multiple categories, and multi-label classification labels are generated for multi-limb classification tasks. Based on the number of limb movements, they are divided into three categories: no limb movement, single limb movement, and multiple limb movements. The number of limb movements is selected as the label for the auxiliary classification task.

[0109] The model building module, for video segment data, selects the convolutional residual neural network in R2Plus1D as the backbone network for extracting spatiotemporal features from the input tensor. This invention employs a hard parameter sharing approach to share the weights for extracting video features between the two tasks, while retaining the output layer of the limb quantity classification network specific to the auxiliary task and the output layer of the multi-label limb motion classification network for the main task, such as... Figure 1 As shown, the main task branch and auxiliary task branch are combined with the cross-entropy loss function to realize the network model to achieve single-label and multi-label classification tasks.

[0110] Model training module: Video clips are cropped into equal-length video sequences at equal intervals, adjusting the frame space size to 171×128. The video sequence data is cropped to the corresponding size to fit the network input, and the SGD neural network optimizer is selected for network training and optimization. The data is then fed into the network model constructed in step 2 for iterative training. After training, the model can perform real-time inference and output the infant's limb movements, thus assisting in monitoring the newborn's limb movement status.

[0111] To achieve better predictive effects in intelligent neonatal monitoring, the following section will introduce the selection and design of parameters for practical applications, serving as a reference for the application of this invention in other fields:

[0112] This invention uses a fixed 16 frames of video data only to facilitate model training. After training, the model can handle video data of any length in practical applications. The auxiliary task is only used for model training and can be removed during inference without increasing the complexity of the model, thus improving the accuracy of the main task. The video data obtained from practical applications is processed in step 3, cropped into video frames at equal intervals, and the frame size is adjusted to 171×128. Then, 16 non-overlapping and continuous video frame segments are randomly selected and adjusted to 3×16×112×112, which are then fed into a 3D convolutional residual neural network. Finally, the sigmoid function outputs the probability value of each limb movement, which is then judged and output using a 0.5 threshold.

[0113] In this invention, when only a single multi-label classification task model is used, the evaluation metrics are CF1 = 85.65% and OF1 = 84.18%. Comparing the model's predicted values ​​with the actual values ​​reveals that some samples have significant prediction errors. In contrast, the evaluation metrics for multi-task learning are CF1 = 90.00% and OF1 = 88.04%, and the number of samples with large prediction errors is significantly reduced. In summary, the neonatal intelligent monitoring auxiliary analysis method based on video multi-task learning in this invention can provide scientific and objective evaluation results for neonatal clinical monitoring, filling the gap in the reliance on manual labor in neonatal monitoring.

Claims

1. A method for monitoring neonatal limb movements based on a multi-task classification network, characterized in that... This method includes a main task and an auxiliary task. The main task is to detect the movement of the newborn's limbs and head, and to perform multi-label classification of the limbs and head into five categories using a classification network. The auxiliary task is a multi-classification task for three movement states of the newborn: static, single-limb movement, and multi-limb movement. The specific implementation includes the following steps: Step 1: Processing the newborn video dataset First, the newborn monitoring video is cropped according to the category of limb movement to obtain several video clips containing newborn limb movements. Based on the limb movements of the newborn in the video clips, it is determined whether there are movements corresponding to the limbs and head, and they are arranged in order [head, left hand, right hand, left leg, right leg] as the multi-label classification label for the main task. Based on the number of limb movements, they are divided into three categories: no limb movement, single limb movement, and multiple limb movement, which are used as labels for auxiliary classification tasks. Each video clip corresponds to only one limb movement quantity classification label. Step 2: For H×W×T video frame data, a 3D convolutional residual neural network is selected as the backbone network to construct a feature extraction network to extract spatial and temporal features. Hidden layers are shared between the two tasks using a hard parameter sharing approach. At the same time, the output layers of the limb motion quantity classification network specific to the auxiliary task and the output layers of the limb motion multi-label classification network specific to the main task are retained. The cross-entropy loss functions of the main task and the auxiliary task are used to train the network model together. Step 2 is performed as follows: 2-1. A 3D convolutional residual neural network is selected as the backbone network. It consists of several 3D residual convolutional blocks, which can extract spatial and temporal features from video data for action recognition and fuse low-level and high-level features together through the residual network to prevent network degradation. 2-2. The auxiliary task branch for classifying the number of limb movements consists of global average pooling, a fully connected layer, and a softmax layer; 2-3. For the main task branch of multi-label classification of limb movement, higher-level shared knowledge is extracted by 3D convolutional residual blocks and task-related parameters are separated step by step to make joint representation learning and sharing more efficient and flexible. At the same time, a sigmoid function is inserted in the last layer. 2-4. A multi-task learning mechanism is used for training, and the model applies a hard parameter sharing mechanism, meaning that the network layers before the output layers of the two tasks share parameters, while only the output layers correspond to their respective network parameters. Since the auxiliary classification task corresponds to the cross-entropy loss function, and the main multi-label classification task corresponds to the binary cross-entropy loss function, the loss function used is L... total It is a weighted superposition of the cross-entropy loss function and the binary cross-entropy loss function; Step 3: Cut several short video clips into video frame data at equal intervals, and adjust the size of each frame image to H×W; after adjusting the video frame data to the corresponding size to meet the network input requirements, send it into the network and train the model, and finally obtain whether the limbs and head are moving to realize the function of assisted monitoring.

2. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 1, characterized in that... Step 1 is performed as follows: 1-1. Video clips are cropped according to different newborns and different limb movement patterns to enhance the diversity of the dataset distribution; 1-2. Define the limb movements of each video clip as labels such as [head, left hand, right hand, left leg, right leg], where 1 represents movement at that position and 0 represents no movement. Divide the number of limb movements into three categories: no limb movement, single limb movement, and multiple limb movement.

3. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 2, characterized in that... The softmax formula mentioned in step 2-2 is as follows: Where z j Let be the value of the j-th category in z; set the last dimension of the fully connected layer to the number of predicted limb movement categories, and then output the probability value of the category by the softmax layer.

4. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 3, characterized in that... The sigmoid function described in steps 2-3 has the following formula: Where x is the input, this function allows generating confidence scores for multiple label categories of the head and limbs, and then using a fixed threshold to determine whether movement has occurred.

5. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 4, characterized in that, The loss function for steps 2-4 is defined as follows: L total =λ1L lmnc +λ2L amc (3) Where L lmnc Cross-entropy is defined as the loss function for classifying the number of limb movements, as follows: L lmnc =loss(x,class)=-x[class]+log(∑ j exp(x[j])) (4) Where x is the logistic regression value input to the cross-entropy loss function, class is the label for the limb movement quantity classification, and L... amc Binary cross-entropy is used as the loss function for multi-label classification of limb movements, and is described as follows: w=weight[c]·1{c≠ignore_index} (6) Where z n Let y represent the prediction score for the nth sample as a positive example. n Let δ represent the label of the nth sample, δ be the sigmoid activation function, and λ1 and λ2 be the weights. The goal is to balance the contributions of classification and multi-label classification tasks to the joint loss.

6. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 5, characterized in that... Step 3 is performed as follows: 3-1. Randomly select T frames and overlapping, continuous video segments from the video data generated in step 1 as network input, with the input dimension being C×T×H×W. During model training, the input video segments are obtained by extracting the difference frames in the random cropping and then fed into a 3D convolutional residual neural network for feature extraction. 3-2. The network architecture obtains feature maps of multiple information channels from adjacent video sequences, and performs convolution and subsampling in each information channel. It convolves a cube formed by stacking multiple consecutive frames with a 3D kernel. 3-3. After passing through a feature extraction network consisting of several 3D convolutional residual blocks, the output features are separated by a task-specific output layer to train relevant task parameters and output predicted values. 3-4. Indicators of network training effectiveness.

7. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 6, characterized in that... The calculation formula in step 3-2 is as follows: in, This represents the calculated value of the j-th feature map in the i-th layer, where tanh is the hyperbolic tangent function, and b... ij , P i Q i R i These represent the bias, weights, kernel height, kernel width, and kernel depth, respectively; the formula for calculating the dimension of each feature map is as follows: Among them, D out H out and W out These represent the time dimension, height, and width of the output feature map, respectively; p represents the padding parameter, d represents the dilation coefficient, k represents the kernel size, s represents the stride, and D represents the depth of motion. in H in and W in These represent the depth, height, and width of the input feature map, respectively.

8. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 7, characterized in that... Step 3-3: The specific steps are as follows: 3-3-1. The convolution operation is represented as follows: Among them, F tr (.,θ) represents the standard convolution operation, θ represents the parameters of the convolution kernel, Y represents the input feature map of the convolutional layer with dimensions D'×H'×W'×C', and U represents the output feature map of the convolutional layer with dimensions D×H×W×C. 3-3-2. The compression operation is a global average pooling operation performed on the output feature map U. The compression operation is represented as follows: During the compression operation, the output feature map of size D×H×W×C undergoes spatial compression, converting each three-dimensional feature channel into a one-dimensional vector, resulting in a one-dimensional vector of size C×1×1×1; c Represents feature maps U and F. sq This represents the compression operation function; 3-3-3. Then, the data is passed to a fully connected layer. In the task of classifying the number of limb movements, the softmax layer outputs the probability value of the category, and the category with the highest probability is selected. In the task of classifying the multiple labels of limb movements, the sigmoid function outputs the probability value of each limb movement, and then a fixed threshold is used to judge and output the result.

9. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 8, characterized in that... Steps 3-4 are detailed below: The network training performance is measured by the average CP, CR, CF1 per class, and the average overall OP, OR, OF1. The formulas for each parameter are as follows: in For the i-th class, predict the number of videos that are correctly predicted. It is the number of videos predicted for the i-th class. is the number of real videos in the i-th category; each indicator is between 0 and 1, the closer to 1, the better the model's prediction effect, and the closer to 0, the worse the model's prediction effect.

Images