A method for preterm infant motion recognition based on local region discrimination and AUC collaborative optimization
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-04-14
- Publication Date
- 2026-06-26
Smart Images
Figure CN122024334B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of non-contact video monitoring and premature infant motion recognition, and relates to a multi-label learning method designed for premature infant motion recognition tasks based on the co-optimization of local region discrimination and AUC (Area Under the ROC Curve) loss. Background Technology
[0002] Premature infants are those born before 37 weeks of gestation. Due to insufficient gestation time, the various systems of premature infants are not fully developed. Compared to full-term newborns, premature infants require more medical care to detect potential physical or psychological problems as early as possible and provide timely medical intervention.
[0003] The observation and assessment of the physiological condition of premature infants mainly relies on their physiological data and spontaneous limb movements. Traditional monitoring of premature infant movement primarily depends on sensors or sporadic observations by medical staff. Because premature infants have very delicate and fragile skin, coupled with an immature immune system, using sensors that come into direct contact with the skin may irritate, damage, or cause allergic reactions, potentially leading to pressure necrosis in severe cases. Furthermore, the presence of sensors may prevent limbs from fully extending, interfering with spontaneous movement. While observation by medical staff is highly professional and allows for immediate assessment and analysis of the premature infant's condition, hospital staff already have heavy workloads and cannot provide 24-hour continuous monitoring of the infant's limb movements. Therefore, using non-contact cameras for real-time monitoring of premature infants is a more ideal and reliable method.
[0004] Existing non-contact motion recognition methods are primarily designed for adults. However, the limb movements of premature infants differ from the conscious control of adult limbs; their movements are unconscious and spontaneous, lacking clear semantic relationships. Furthermore, different limb movements in premature infants exhibit a high degree of similarity in spatiotemporal patterns. In addition, the body proportions of premature infants differ from adults, with longer trunks and shorter limbs, while adults typically have longer limbs. These factors lead to poor performance when mainstream motion recognition methods designed for adults are directly applied to non-contact monitoring of premature infants. Designing a non-contact intelligent monitoring system for motion recognition in premature infants is a key problem to be solved. Summary of the Invention
[0005] To address the shortcomings of existing preterm infant movement recognition technologies, this invention provides a multi-label limb movement classification method for preterm infants based on the co-optimization of local region discrimination and logarithmic AUC loss function.
[0006] The technical solution of this invention mainly includes the following steps:
[0007] In a first aspect, embodiments of this application provide a method for recognizing the actions of preterm infants by co-optimizing local region discrimination and AUC, comprising the following steps:
[0008] Step 1: Preprocess the acquired preterm infant monitoring video data to convert it into a multi-label limb movement dataset for preterm infants, containing five body parts: head, left hand, right hand, left foot, and right foot. The preprocessing includes video category sorting, cropping, and multi-label annotation.
[0009] Step 2: Construct a multi-label limb movement classification model for premature infants, including a video feature extraction module, a feature mapping module, a category-aware feature enhancement module, and a classifier module. The category-aware feature enhancement module includes dimensionality transformation operations, L2 normalization, and a multi-head class-specific residual attention mechanism.
[0010] Step 3: Based on the constructed multi-label limb movement classification model for preterm infants, introduce AUC loss and construct a fusion loss function that combines binary cross-entropy and AUC loss.
[0011] Step 4: Input the preprocessed multi-label limb movement data of premature infants into the constructed multi-label limb movement classification model of premature infants for recognition training and save the trained model.
[0012] In one possible implementation, step 1 is specifically implemented as follows:
[0013] 1-1: The acquired monitoring video data of premature infants was precisely cut according to the limb movement categories to obtain video segments for each limb movement category, and these video segments were then classified and organized according to category. Based on the classification and organization results, the training set, validation set, and test set were completed.
[0014] 1-2: Extract consecutive image frames from the divided training, validation, and test set video data frame by frame, and adjust the images to a fixed size. Divide the extracted image frames into multiple consecutive image frames of fixed length.
[0015] 1-3: Define a fixed-length continuous image frame after the sliding window as a single data sample. Perform One-Hot category encoding on each sample to convert the limb action category label into a binary multi-label form. The encoded action categories include head, left hand, right hand, left leg, and right leg. Specifically, a one-dimensional array is used to encode the five limb parts of the body in the order of head, left hand, right hand, left leg, and right leg into a 1×5 array consisting of 0s and 1s. If a limb part has moved, the corresponding position is encoded as 1; otherwise, if no limb part has moved, it is encoded as 0.
[0016] In one possible implementation, step 2 is specifically implemented as follows:
[0017] 2-1: The video feature extraction module uses a video feature extraction backbone network as a feature extractor, takes multi-label sample data of premature infants as input, and extracts the spatiotemporal features of the samples.
[0018] 2-2: The spatiotemporal features extracted by the feature extractor are fed into the feature mapping module. Fully connected transformations and Dropout operations are performed on the sample spatiotemporal features to generate projection vectors.
[0019] 2-3: Input the spatiotemporal features extracted by the feature extractor into the category-aware feature enhancement module to obtain class-specific residual features.
[0020] 2-4: Based on the input projection vector and class-specific residual features, the classification result is calculated by the classifier module.
[0021] In one possible implementation, the category-aware feature enhancement module operates as follows:
[0022] First, the video's spatiotemporal features are mapped and transformed using 3D convolution operations to adjust the number of feature channels to match the number of classes, resulting in a class response tensor. The L2 norm is calculated for each class weight vector of the 3D convolution, and the class response tensor is then element-wise divided by the L2 norm according to its corresponding channel norm to obtain the normalized class response. Furthermore, the spatiotemporal dimension of the class response is flattened into a single dimension, yielding the flattened spatiotemporal feature vector.
[0023] The spatiotemporal feature vectors, after being flattened, are input into a multi-head class-specific residual attention mechanism for weighted aggregation of key spatiotemporal regions. This mechanism consists of multiple independent class-specific residual attention units operating in parallel, each employing a two-branch parallel structure. For each unit, in one branch, the flattened spatiotemporal features are average-pooled to obtain class-independent average-pooled features; in the other branch, these features are input into a softmax function controlled by a corresponding temperature parameter to obtain a class-specific attention score. The attention score is then weighted and summed with the flattened feature vector to obtain the class-specific features. Finally, the class-independent average-pooled features and the class-specific features are weighted and summed to obtain the class-specific discriminant vector for a single unit.
[0024] In one possible implementation, the classifier module operates as follows:
[0025] The projection vector generated by the feature mapping module is input into the fully connected layer to obtain fully connected layer mapped features consistent with the number of categories. Simultaneously, the class-specific discriminant vectors output by each unit in the multi-head class-specific residual attention mechanism are element-wise summed according to category to obtain the discriminant scores for each category. The discriminant scores of all categories together constitute the discriminant score vector. The discriminant score vector The features mapped by the fully connected layer are weighted and fused according to category to obtain the fused multi-label prediction score distribution.
[0026] The Sigmoid function is used to perform element-wise mapping of the multi-label prediction score distribution, independently mapping each category's prediction score to... Within the interval, the final predicted probability distribution containing all categories is obtained. Based on a preset judgment threshold of 0.5, the probability distribution is binary judged. If the output predicted probability is greater than 0.5, it is judged as a positive sample, that is, the corresponding limb movement is generated; otherwise, it is regarded as a negative sample, and no corresponding limb movement is generated.
[0027] In one possible implementation, step 4 is specifically implemented as follows:
[0028] 4-1: Adjust the image size of the preprocessed multi-label limb movement data of premature infants to the input size of the video feature extraction module.
[0029] 4-2: The adjusted data is fed into the constructed preterm infant multi-label limb movement classification model, and end-to-end collaborative training is performed based on the constructed fusion loss function.
[0030] The training process uses stochastic gradient descent for optimization, with an initial learning rate set at 0.5%. Batch size is The learning rate is dynamically adjusted using a StepLR scheduler with a specified step size. During training iterations, the model adapts to inputs of length... Forward propagation is performed on image frame sequence samples to generate a probability distribution. This distribution is then combined with the ground truth labels to calculate the fusion loss, which is used to update the model parameters based on the backpropagation algorithm. Simultaneously, a validation set is used for monitoring during training to prevent overfitting.
[0031] Secondly, embodiments of this application provide a preterm infant motion recognition device that combines local region discrimination and AUC co-optimization, comprising the following modules:
[0032] Preprocessing module: This module preprocesses the acquired preterm infant monitoring video data, converting it into a multi-label limb movement dataset for preterm infants, containing five body parts: head, left hand, right hand, left foot, and right foot. The preprocessing includes video category sorting, cropping, and multi-label annotation.
[0033] Model building module: Constructs a multi-label limb movement classification model for preterm infants, including a video feature extraction module, a feature mapping module, a category-aware feature enhancement module, and a classifier module. The category-aware feature enhancement module includes dimensionality transformation operations, L2 normalization, and a multi-head class-specific residual attention mechanism.
[0034] Loss function construction module: Based on the constructed multi-label limb movement classification model for preterm infants, AUC loss is introduced to construct a fusion loss function that combines binary cross-entropy and AUC loss.
[0035] Training module: The preprocessed multi-label limb movement data of premature infants is fed into the constructed multi-label limb movement classification model of premature infants for recognition training, and the trained model is saved.
[0036] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory;
[0037] The memory is used to store computer programs.
[0038] When the processor executes the program stored in the memory, it implements any of the premature infant action recognition methods described in this application.
[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the premature infant action recognition methods described in this application.
[0040] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the premature infant action recognition methods described in this application.
[0041] The beneficial effects of this invention are as follows:
[0042] This invention proposes a method based on the collaborative optimization of local region discrimination and logarithmic AUC loss for limb movement classification tasks in premature infants in the intensive care unit. Multi-label class-specific residual attention is employed to address the high similarity between different categories in spatiotemporal patterns. Simultaneously, the logarithmic AUC loss function is introduced to alleviate the class imbalance caused by the unconscious spontaneous limb movements of premature infants, thereby synergistically improving the model's recognition performance when dealing with limb movements of premature infants.
[0043] Compared with existing technologies for recognizing limb movements in premature infants, the innovations of this invention are as follows: (1) The limb movements of premature infants are different from those of adults. They do not have significant semantic relationships, and there is a high degree of similarity between different categories in terms of spatiotemporal patterns. A multi-head class-specific residual attention mechanism for videos is designed to adaptively aggregate the discriminative region information for each category, thereby enhancing the model's region discrimination ability at the model level and enabling the model to achieve better limb movement recognition accuracy. (2) Since the limb movements of premature infants are unconscious spontaneous movements without human or external interference, the generation of their movements is completely uncontrolled, resulting in a natural long-tail distribution problem in the collected data. A fusion loss based on the combination of AUC and binary cross-entropy is proposed to improve the model's ability to recognize a small number of samples. (3) This invention proposes a feasible technical method for non-contact movement monitoring of premature infants, and this method improves the model's performance without increasing the model's network parameters. Attached Figure Description
[0044] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.
[0045] Figure 2 This is a network structure diagram of a C3D-based multi-label limb movement classification model for premature infants, as described in an embodiment of the present invention. Detailed Implementation
[0046] The flowchart of the preterm infant motion recognition method based on local region discrimination and AUC co-optimization is as follows: Figure 1 As shown in the accompanying drawings and specific embodiments, the technical solution of the present invention will be further described below.
[0047] The core innovation of this invention lies in overcoming the difficulty of directly applying existing algorithms due to the differences in limb movement characteristics between premature infants and adults. Specifically, there are significant differences between the limb movements of premature infants and adults, mainly manifested in the absence of clear semantic relationships in premature infant limb movements, such as playing basketball, holding a water cup, or writing, where limb movements semantically constitute higher-level behaviors. However, due to the incomplete development of their bodily systems, premature infants' limb movements are spontaneous and not subject to deliberate human intervention, and these movements cannot form higher-level semantic behaviors. Furthermore, different limb movements exhibit high similarity in spatiotemporal patterns. To address these issues, this invention designs a multi-head class-specific residual attention mechanism for multi-label learning at the model level to enhance the model's attention to local discrimination regions. Further, this invention introduces logarithmic AUC loss to alleviate the long-tail distribution problem faced by premature infant limb movement recognition tasks, achieving a collaborative optimization model of local region discrimination and logarithmic AUC loss.
[0048] To facilitate the explanation of this invention, subsequent embodiments use a C3D (Convolutional 3D) backbone network as an example. However, it should be noted that this invention is not limited to this type of backbone network. The local region discrimination mechanism and the fusion loss function combining logarithmic AUC loss proposed in this invention can be applied to any video feature extraction backbone network.
[0049] This invention employs C3D as the backbone network for video feature extraction, then utilizes a multi-head class-specific residual attention mechanism to extract features from different object categories. These features are then fused with the backbone network features, and a logarithmic AUC loss combined with binary cross-entropy is used as the fusion loss function for model optimization. The above design of this invention enables better completion of multi-label limb movement classification tasks for premature infants in the intensive care unit.
[0050] The clinical data used for experimental verification of the results of this invention comes from videos of spontaneous limb movements of premature infants in real neonatal intensive care units. Based on this data, this invention achieves video-based multi-label limb movement recognition for premature infants, which can be used for non-contact intelligent monitoring of premature infants.
[0051] This application provides a method for recognizing the actions of preterm infants by co-optimizing local region discrimination and AUC, including the following steps:
[0052] Step 1: Preprocess the monitoring video data of premature infants obtained from real neonatal intensive care units, converting it into multi-label limb movement data of premature infants, including five body parts: head, left hand, right hand, left foot, and right foot. The preprocessing includes video categorization, cropping, and multi-label annotation. The specific implementation is as follows:
[0053] 1-1: The acquired monitoring video data of premature infants is cut according to the category of limb movement to obtain video segments of each limb movement category, and these video segments are then classified and organized according to category. Based on the classification and organization results, the training set, validation set, and test set are completed.
[0054] 1-2: Extract consecutive image frames frame by frame from the divided training, validation, and test set video data, and adjust the image size to 171×128. With a step size of 8 and a sampling window of 16, the extracted consecutive image frames are divided into multiple consecutive image frames of fixed length 16, thereby increasing the number of samples while maintaining the temporal continuity of limb movements.
[0055] 1-3: Define a fixed-length continuous image frame after the sliding window as a single data sample. Perform One-Hot category encoding on each sample to convert the limb action category labels into a binary multi-label form. The encoded action categories include head, left hand, right hand, left leg, and right leg. Specifically, a one-dimensional array is used to encode the five limb parts of the body in the order of head, left hand, right hand, left leg, and right leg into a 1×5 array consisting of 0s and 1s. If a limb part has moved, the corresponding position is encoded as 1; otherwise, if no limb part has moved, it is encoded as 0.
[0056] Step 2: Construct a multi-label limb movement classification model for premature infants, including a video feature extraction module, a feature mapping module, a category-aware feature enhancement module, and a classifier module. The designed category-aware feature enhancement module includes dimensionality transformation operations, L2 normalization, and a multi-head class-specific residual attention mechanism. The specific implementation is as follows:
[0057] 2-1: For ease of explanation, this invention uses C3D as the video feature extraction backbone network in a specific embodiment. The backbone network includes the following five module stages.
[0058] Stage-0: The preprocessed consecutive image frame samples are resized to 112×112 and normalized before being input into the video feature extraction module. The input tensor dimensions are (16, 3, 16, 112, 112), representing the batch size, the number of RGB image channels, the number of frames per sample, and the image height and width, respectively. This stage includes a 3×3×3 3D convolutional block, a ReLU activation function layer, and a 3D max-pooling layer. The 3D convolutional block has 3 input channels and 64 output channels, used to extract the basic spatiotemporal features of the input frame sequence. The final output tensor size of this stage is (16, 64, 16, 56, 56).
[0059] Stage-1: Stage-1 consists of a 3×3×3 3D convolutional block, a ReLU activation function layer, and a 3D max pooling layer with a (2×2×2) kernel. Taking the output feature tensor of stage-0 as input, after processing by stage-1, the size of the output feature tensor is (16, 128, 8, 28, 28).
[0060] Stage-2: Stage-2 consists of two 3×3×3 3D convolutional layers, two ReLU activation function layers, and a 3D max pooling layer with a (2×2×2) kernel. Taking the output feature tensor of stage-1 as input, after processing by stage-2, the size of the output feature tensor is (16, 256, 4, 14, 14).
[0061] Stage-3: Stage-3 consists of two 3×3×3 3D convolutional layers, two ReLU activation function layers, and a 3D max pooling layer with a (2×2×2) kernel. Taking the output feature tensor of stage-2 as input, after processing by stage-3, the output feature tensor size is (16, 512, 2, 7, 7).
[0062] Stage-4: Stage-4 consists of two 3×3×3 3D convolutional layers, two ReLU activation layers, and a 3D max-pooling layer with a (2×2×2) kernel. Taking the output feature tensor of Stage-3 as input, after processing by Stage-4, the output is a sample spatiotemporal feature of size (16, 512, 2, 4, 4). .
[0063] 2-2: The output feature tensor from stage-4 is fed into the feature mapping module. This module contains two linear layers, two ReLU activation function layers, and two regularization layers with a probability of 0.5 to prevent overfitting. The output is a projection vector of size (16, 4096). .
[0064] 2-3: The output feature tensor of stage-4 is fed into the category-aware feature enhancement module, which sequentially includes a 3D convolutional layer, an L2 normalization layer, and a multi-head class-specific residual attention mechanism.
[0065] 2-3-1: First, a 3D convolutional layer is used to map and transform the spatiotemporal features of the video, adjusting the number of feature channels to the number of classes, to obtain the class response tensor. Let 3D convolution be the first... The class weight vector is For each class of weights, the L2 norm By dividing the class response tensor element by element according to its corresponding channel norm by the L2 norm, the normalized class response can be obtained. Furthermore, the spatiotemporal dimensions of the normalized class response are flattened into a single dimension to obtain the flattened spatiotemporal features. .in, Represents the number of categories. , , , , These represent the category index, time index, height index, width index, and spatiotemporal location index, respectively.
[0066] 2-3-2: Spatiotemporal feature vectors after dimensionality flattening A multi-head class-specific residual attention mechanism is used to perform weighted aggregation on key spatiotemporal regions. This mechanism comprises four parallel class-specific residual attention units, each employing a dual-branch parallel structure. One branch is an average pooling layer branch, used to obtain class-independent average pooling features. Secondly, it is determined by temperature parameters. The controlled softmax function branch is used to compute class-specific attention scores. The class-specific attention score The calculation process can be described as follows:
[0067]
[0068] In the formula, Represents cell index, The length is the time dimension. , They represent height and width respectively. The temperature parameter is used to control the sharpness of the function. When this happens, the above formula degenerates into max pooling.
[0069] Further increase attention score With the flattened feature vector Perform a weighted summation to obtain class-specific features. Finally, class-independent average pooling features are used. With class-specific features The weighted sums are used to obtain the class-specific discriminant vector for each individual unit. :
[0070]
[0071] in, The fusion weights are specific to class characteristics.
[0072] 2-4: Feed the outputs of 2-2 and 2-3 into the classifier module, which outputs the predicted probability distribution. Then, input the projection vector generated by the feature mapping module in 2-2. The input is fed into a fully connected layer to obtain fully connected layer mapping features consistent with the dimension of the number of categories. Simultaneously, the class-specific discriminant vectors of each unit in the 2-3 multi-head class-specific residual attention mechanism are... The discrimination scores for each category are obtained by summing the elements one by one according to the category. The discriminant scores of all categories together constitute the discriminant score vector. The discriminant score vector Mapping features with fully connected layers Weighted fusion by category yields the fused multi-label prediction score distribution. .
[0073] 2-5: To facilitate loss calculation together with the true labels, the Sigmoid function is used to distribute the output multi-label prediction scores. Perform element-wise mapping to independently map the predicted scores of each category to Within the interval, the final predicted probability distribution containing all categories is obtained. The probability distribution is binary-valued based on a preset judgment threshold of 0.5. If the output predicted probability is greater than 0.5, it is judged as a positive sample, that is, the corresponding limb movement is generated; otherwise, it is regarded as a negative sample, and no corresponding limb movement is generated.
[0074] Step 3: Based on the multi-label limb movement classification model for premature infants built in Step 2, construct a fusion loss function that can both alleviate the data imbalance problem and ensure accurate matching of model information. The specific implementation is as follows:
[0075] 3-1: Use the logarithmic AUC loss function to alleviate the problem of long-tailed data distribution.
[0076] To address the issue of limb movement imbalance in premature infants, logarithmic AUC loss is used to optimize the model. A set of positive samples is defined. and negative sample set ,in Indicates the first The sample corresponds to the first Real labels for limb movement categories in premature infants. When the sample... Belongs to the When there are multiple action categories Conversely, if the value is greater than or equal to 0, then the value is 0. Therefore, the calculation process for the logarithmic AUC can be expressed as:
[0077]
[0078]
[0079] in, and These represent the number of positive and negative samples, respectively. Let be a penalty threshold, and satisfy . , This is a prediction function for a multi-label limb movement classification model for premature infants, used to output the probability distribution of samples.
[0080] 3-2: In order to take into account the accurate matching information of the model, the logarithmic AUC loss is used. Based on the binary cross-entropy loss Therefore, the fusion loss function for optimizing the model is:
[0081]
[0082] in The weighting coefficients represent the binary cross-entropy.
[0083] Step 4: Input the preprocessed multi-label limb movement data of premature infants into the constructed multi-label limb movement classification model for recognition training, and save the trained model. The specific implementation is as follows:
[0084] 4-1: After preprocessing in step 1, the images in the multi-label limb movement data of preterm infants were cropped to a fixed size. To match the input of the model, the image size was adjusted to... .
[0085] 4-2: Measure the size as A multi-label limb movement dataset of preterm infants was fed into a pre-constructed multi-label limb movement classification model for preterm infants, and end-to-end co-training was performed based on a constructed fusion loss function. The training process employed stochastic gradient descent for optimization, with an initial learning rate set to [value missing]. Batch size is The learning rate is dynamically adjusted using a StepLR scheduler with a specified step size of 20. During training iterations, the model performs forward propagation on input image frame sequences of length 16 to generate a probability distribution, calculates the fusion loss using the ground truth labels, and then updates the model parameters based on the backpropagation algorithm. Simultaneously, a validation set is used for monitoring during training to prevent overfitting.
[0086] Step 5: After the model training is completed, the model performance is evaluated using the test set. The experimental results are shown in Table 1 and Table 2.
[0087] Table 1: Ablation experiments using the local region discrimination and AUC co-optimization method
[0088]
[0089] Table 2: Comparative Experiment Results
[0090]
[0091] This invention was based on experiments conducted using a constructed dataset of real preterm infant limb movements. In multi-label classification tasks, since each sample may belong to multiple categories simultaneously, ordinary single-class or multi-class evaluation metrics are no longer applicable. Therefore, multi-label evaluation metrics such as mAP (Mean Average Precision), OP (overall precision), OR (overall recall), and OF1 (overall F1) were used. Using multi-label class-specific residual attention alone improved mAP by 1.46%, and using a fusion loss combined with logarithmic AUC loss alone improved mAP by 1.01%. When class-specific residual attention and fusion loss were co-optimized, mAP reached 87.46%, an improvement of 3.27%, while OP, OR, and OF1 reached 79.81%, 80.98%, and 80.39%, respectively, achieving the best results among the comparative models. Therefore, the local region discrimination and AUC co-optimization method for preterm infant multi-label limb movement recognition designed in this invention can effectively aggregate the discriminative region information for each limb movement category, enhance the model's ability to distinguish between different categories, and strengthen the model's recognition of a small number of samples in conjunction with the designed loss function, further improving the model's recognition ability in the face of imbalanced fine-grained multi-label data of preterm infants.
[0092] This application embodiment also provides a preterm infant motion recognition device that combines local region discrimination and AUC co-optimization, including the following modules:
[0093] Preprocessing module: This module preprocesses the acquired preterm infant monitoring video data, converting it into a multi-label limb movement dataset for preterm infants, containing five body parts: head, left hand, right hand, left foot, and right foot. The preprocessing includes video category sorting, cropping, and multi-label annotation.
[0094] Model building module: Constructs a multi-label limb movement classification model for preterm infants, including a video feature extraction module, a feature mapping module, a category-aware feature enhancement module, and a classifier module. The category-aware feature enhancement module includes dimensionality transformation operations, L2 normalization, and a multi-head class-specific residual attention mechanism.
[0095] Loss function construction module: Based on the constructed multi-label limb movement classification model for preterm infants, AUC loss is introduced to construct a fusion loss function that combines binary cross-entropy and AUC loss.
[0096] Training module: The preprocessed multi-label limb movement data of premature infants is fed into the constructed multi-label limb movement classification model of premature infants for recognition training, and the trained model is saved.
[0097] In one possible implementation, the preprocessing module is specifically implemented as follows:
[0098] 1-1: The acquired monitoring video data of premature infants was precisely cut according to the limb movement categories to obtain video segments for each limb movement category, and these video segments were then classified and organized according to category. Based on the classification and organization results, the training set, validation set, and test set were completed.
[0099] 1-2: Extract consecutive image frames from the divided training, validation, and test set video data frame by frame, and resize the images to a fixed size. Set the stride to [value missing]. The sampling window is The extracted image frames are divided into multiple fixed-length segments. A series of image frames.
[0100] 1-3: Define a fixed-length continuous image frame after the sliding window as a single data sample. Perform One-Hot category encoding on each sample to convert the limb action category label into a binary multi-label form. The encoded action categories include head, left hand, right hand, left leg, and right leg. Specifically, a one-dimensional array is used to encode the five limb parts of the body in the order of head, left hand, right hand, left leg, and right leg into a 1×5 array consisting of 0s and 1s. If a limb part has moved, the corresponding position is encoded as 1; otherwise, if no limb part has moved, it is encoded as 0.
[0101] In one possible implementation, the specific operation of the preterm infant multi-label limb movement classification model is as follows:
[0102] 2-1: Using a video feature extraction backbone network as the feature extractor, the multi-label sample data of premature infants was processed. As input, extract the spatiotemporal features of the sample. .in Representing batch, Represents the number of channels in an RGB image. Represents the length of the time dimension. , These represent the height and width of a single frame image, respectively. , , , These represent the number of channels, the length of the time dimension, the height of a single frame, and the width, respectively, after passing through the feature extractor.
[0103] 2-2: The spatiotemporal features extracted by the feature extractor are fed into the feature mapping module. This involves processing the spatiotemporal features of the samples. Perform a fully connected transformation and Dropout operation to generate a projection vector. .
[0104] 2-3: Input the spatiotemporal features extracted by the feature extractor into the category-aware feature enhancement module to obtain class-specific residual features.
[0105] 2-3-1: First, the video's spatiotemporal features are mapped and transformed using 3D convolution operations to adjust the number of feature channels to the number of classes, thus obtaining the class response tensor. Let 3D convolution be the first... The class weight vector is Calculate the L2 norm for each type of weight vector. By dividing the class response tensor element by element according to its corresponding channel norm by the L2 norm, the normalized class response can be obtained. Furthermore, the spatiotemporal dimension of the class response is flattened into a single dimension, resulting in the flattened spatiotemporal feature vector. .in, Represents the number of categories. , , , , These represent the category index, time index, height index, width index, and spatiotemporal location index, respectively.
[0106] 2-3-2: Spatiotemporal feature vectors after dimensionality flattening A multi-head class-specific residual attention mechanism is used to perform weighted aggregation of key spatiotemporal regions. This mechanism consists of multiple independent class-specific residual attention units operating in parallel, each employing a two-branch parallel structure. For each unit, the spatiotemporal features after dimension flattening are processed within one branch. Perform average pooling to obtain class-independent average pooling features. In another branch, the spatiotemporal features after dimension flattening are... Input consists of temperature parameters By controlling the softmax function, the corresponding class-specific attention score is obtained. Further increase attention score With the flattened feature vector Perform a weighted summation to obtain class-specific features. Finally, class-independent average pooling features are applied. With class-specific features The weighted sums are used to obtain the class-specific discriminant vector for each individual unit. Among them, parameters Used to control the sharpness of the softmax function. Representing the Units.
[0107] 2-4: Calculate the classification result based on the classifier module. The projection vector generated by the feature mapping module... The input is fed into a fully connected layer to obtain fully connected layer mapping features consistent with the dimension of the number of categories. Simultaneously, the class-specific discriminant vectors output by each unit in the multi-head class-specific residual attention mechanism are... The discrimination scores for each category are obtained by summing the elements one by one according to the category. The discriminant scores of all categories together constitute the discriminant score vector. The discriminant score vector Mapping features with fully connected layers Weighted fusion by category yields the fused multi-label prediction score distribution. .
[0108] 2-5: Using the Sigmoid function to distribute multi-label prediction scores Perform element-wise mapping to independently map the predicted scores of each category to Within the interval, the final predicted probability distribution containing all categories is obtained. The probability distribution is binary-valued based on a preset judgment threshold of 0.5. If the output predicted probability is greater than 0.5, it is judged as a positive sample, that is, the corresponding limb movement is generated; otherwise, it is regarded as a negative sample, and no corresponding limb movement is generated.
[0109] In one possible implementation, the loss function construction module is specifically implemented as follows:
[0110] To address the issue of limb movement imbalance in premature infants, a fusion loss function combining logarithmic AUC loss and binary cross-entropy loss was used to optimize the model.
[0111] 3-1: Use the logarithmic AUC loss function to alleviate the problem of long-tailed data distribution.
[0112] Define the set of positive samples and negative sample set ,in Indicates the first The sample corresponds to the first Real labels for limb movement categories in premature infants. When the sample... Belongs to the When there are multiple action categories Conversely, if the value is greater than or equal to 0, then the value is 0. Therefore, the calculation process for the logarithmic AUC can be expressed as:
[0113]
[0114]
[0115] in, and These represent the number of positive and negative samples, respectively. Let be a penalty threshold, and satisfy . , This is a prediction function for a multi-label limb movement classification model for premature infants, used to output the probability distribution of samples.
[0116] 3-2: To ensure accurate model matching, a binary cross-entropy loss is combined with the logarithmic AUC loss. Therefore, the optimized fusion loss function is:
[0117]
[0118] in The weighting coefficients represent the binary cross-entropy.
[0119] In one possible implementation, the training module is specifically implemented as follows:
[0120] 4-1: After preprocessing, the images in the multi-label limb movement data of preterm infants were cropped to a fixed size. To match the model input, the image size was adjusted to... .
[0121] 4-2: Measure the size as The preterm infant multi-label limb movement dataset is fed into the constructed preterm infant multi-label limb movement classification model, and end-to-end collaborative training is performed based on the constructed fusion loss function.
[0122] The training process uses stochastic gradient descent for optimization, with an initial learning rate set at 0.5%. Batch size is The learning rate is dynamically adjusted using a StepLR scheduler with a specified step size. During training iterations, the model adapts to inputs of length... Forward propagation is performed on image frame sequence samples to generate a probability distribution. This distribution is then combined with the ground truth labels to calculate the fusion loss, which is used to update the model parameters based on the backpropagation algorithm. Simultaneously, a validation set is used for monitoring during training to prevent overfitting.
[0123] This application also provides an electronic device, including a processor and a memory.
[0124] The memory is used to store computer programs.
[0125] When the processor executes a program stored in the memory, it implements any of the methods described in this application.
[0126] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.
[0127] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.
[0128] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0129] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0130] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0131] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.
[0132] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.
[0133] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).
[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0135] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.
[0136] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A method for recognizing preterm infant movements using local region discrimination and AUC co-optimization, characterized in that, Includes the following steps: Step 1: Preprocess the acquired preterm infant monitoring video data to convert it into a preterm infant multi-label limb movement dataset containing five body parts: head, left hand, right hand, left foot, and right foot; the preprocessing includes video category sorting, cropping, and multi-label annotation; Step 2: Construct a multi-label limb movement classification model for premature infants, which includes a video feature extraction module, a feature mapping module, a category-aware feature enhancement module, and a classifier module; wherein the category-aware feature enhancement module includes dimensionality transformation operation, L2 normalization, and a multi-head class-specific residual attention mechanism; The specific implementation of step 2 is as follows: 2-1: The video feature extraction module uses a video feature extraction backbone network as a feature extractor, taking multi-label sample data of premature infants as input to extract the spatiotemporal features of the samples; 2-2: The spatiotemporal features extracted by the feature extractor are fed into the feature mapping module; the fully connected transformation and Dropout operation are performed on the spatiotemporal features of the samples to generate projection vectors; 2-3: Input the spatiotemporal features extracted by the feature extractor into the category-aware feature enhancement module to obtain class-specific residual features; 2-4: Based on the input projection vector and class-specific residual features, the classification result is calculated by the classifier module; The specific operation of the category-aware feature enhancement module is as follows: First, the video's spatiotemporal features are mapped and transformed using 3D convolution operations to adjust the number of feature channels to the number of classes, resulting in a class response tensor. The L2 norm is calculated for each class weight vector of the 3D convolution, and the class response tensor is divided element-wise by the L2 norm according to its corresponding channel norm to obtain the normalized class response. The spatiotemporal dimension of the class response is further flattened into one dimension to obtain the flattened spatiotemporal feature vector. The spatiotemporal feature vectors, after being flattened, are input into a multi-head class-specific residual attention mechanism for weighted aggregation of key spatiotemporal regions. This mechanism consists of multiple independent class-specific residual attention units operating in parallel, each employing a dual-branch parallel structure. For each unit, in one branch, the flattened spatiotemporal features are average-pooled to obtain class-independent average-pooled features. In the other branch, the flattened spatiotemporal features are input into a softmax function controlled by a corresponding temperature parameter to obtain a class-specific attention score. This attention score is then weighted and summed with the flattened feature vector to obtain class-specific features. Finally, the class-independent average-pooled features and class-specific features are weighted and summed to obtain the class-specific discriminant vector for a single unit. Step 3: Based on the constructed multi-label limb movement classification model for preterm infants, introduce AUC loss and construct a fusion loss function that combines binary cross-entropy and AUC loss; Step 4: Input the preprocessed multi-label limb movement data of premature infants into the constructed multi-label limb movement classification model of premature infants for recognition training and save the trained model.
2. The method for recognizing preterm infant movements by local region discrimination and AUC co-optimization according to claim 1, characterized in that, The specific implementation of step 1 is as follows: 1-1: The acquired monitoring video data of premature infants is precisely cut according to the category of limb movement to obtain video segments of each limb movement category, and these video segments are classified and organized according to the category; based on the classification and organization results, the training set, validation set and test set are divided. 1-2: Extract consecutive image frames from the divided training, validation, and test set video data frame by frame, and adjust the images to a fixed size; divide the extracted image frames into multiple consecutive image frames of fixed length; 1-3: Define a fixed-length continuous image frame after the sliding window as a single data sample. Perform One-Hot category encoding on each sample to convert the limb action category label into a binary multi-label form. The encoded action categories include head, left hand, right hand, left leg, and right leg. Specifically, a one-dimensional array is used to encode the five limb parts of the body in the order of head, left hand, right hand, left leg, and right leg into a 1×5 array consisting of 0s and 1s. If a limb part moves, the corresponding position is encoded as 1; otherwise, if no limb part moves, it is encoded as 0.
3. The method for recognizing preterm infant movements by local region discrimination and AUC co-optimization according to claim 1, characterized in that, The specific operation of the classifier module is as follows: The projection vector generated by the feature mapping module is input into the fully connected layer to obtain fully connected layer mapped features consistent with the number of categories. Simultaneously, the class-specific discriminant vectors output by each unit in the multi-head class-specific residual attention mechanism are element-wise summed according to category to obtain the discriminant scores for each category. The discriminant scores of all categories together constitute the discriminant score vector. ; Discriminant score vector The features mapped from the fully connected layer are weighted and fused according to category to obtain the fused multi-label prediction score distribution; The Sigmoid function is used to perform element-wise mapping of the multi-label prediction score distribution, independently mapping each category's prediction score to... Within the interval, the final predicted probability distribution containing all categories is obtained; based on the preset judgment threshold of 0.5, the probability distribution is binary judged. If the output predicted probability is greater than 0.5, it is judged as a positive sample, that is, the corresponding limb movement is generated; otherwise, it is regarded as a negative sample and no corresponding limb movement is generated.
4. The method for recognizing preterm infant movements by local region discrimination and AUC co-optimization according to claim 1, characterized in that, The specific implementation of step 4 is as follows: 4-1: Adjust the image size of the preprocessed multi-label limb movement data of premature infants to the input size of the video feature extraction module; 4-2: The adjusted data is fed into the constructed multi-label limb movement classification model for premature infants, and end-to-end collaborative training is performed based on the constructed fusion loss function; The training process uses stochastic gradient descent for optimization, with an initial learning rate set at 0.5%. Batch size is The learning rate is dynamically adjusted using a StepLR scheduler with a specified step size. During training iterations, the model adapts to input lengths of... Forward propagation is performed on image frame sequence samples to generate a probability distribution. The fusion loss is calculated by combining the real labels, and then the model parameters are updated based on the backpropagation algorithm. At the same time, a validation set is used for monitoring during training to avoid overfitting.
5. A preterm infant motion recognition device with local region discrimination and AUC co-optimization, used to implement the method of claim 1, characterized in that, Includes the following modules: Preprocessing module: preprocesses the acquired monitoring video data of premature infants, converting it into a multi-label limb movement dataset of premature infants containing five body parts: head, left hand, right hand, left foot, and right foot; the preprocessing includes video category sorting, cropping, and multi-label annotation; Model building module: Constructs a multi-label limb movement classification model for preterm infants, including a video feature extraction module, a feature mapping module, a category-aware feature enhancement module, and a classifier module; wherein the category-aware feature enhancement module includes dimensionality transformation operation, L2 normalization, and a multi-head class-specific residual attention mechanism; Loss function construction module: Based on the constructed multi-label limb movement classification model of preterm infants, AUC loss is introduced to construct a fusion loss function that combines binary cross-entropy and AUC loss; Training module: The preprocessed multi-label limb movement data of premature infants is fed into the constructed multi-label limb movement classification model of premature infants for recognition training, and the trained model is saved.
6. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the premature infant movement recognition method according to any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the premature infant movement recognition method according to any one of claims 1-4.
8. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the premature infant movement recognition method according to any one of claims 1-4.