Newborn limb action recognition method based on double-branch multi-scale feature fusion

Abstract

The application provides a newborn limb action recognition method based on double-branch multi-scale feature fusion, video of newborns in different limb action states is collected to construct a newborn limb action video sample set; a newborn limb action recognition model based on double-branch multi-scale feature fusion is constructed and comprises a first branch feature extraction network, a second branch feature extraction network, a feature fusion module and a classifier; a newborn limb action recognition model is obtained; action recognition is performed on newly input test video by using the obtained newborn limb action recognition model; the method uses double branches to extract action features from video with different frame rates, improves the capture ability of subtle action changes, and adopts a multi-scale convolution attention mechanism to focus on key features of limb actions, thereby effectively improving the accuracy of newborn limb action recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to a new-born limb action recognition method based on double-branch multi-scale feature fusion, and belongs to the technical field of action recognition. BACKGROUND

[0002] Action is a special way of communication analysis, which can reveal their inner state and needs by observing their subtle body movements and posture changes when they cannot express in language. This reaction usually occurs naturally and unconsciously, and is difficult to hide or control. It is often directly related to a person's true emotions and can reveal their true feelings and psychological conditions. New-born limb action, as a branch of action recognition research, focuses on analyzing and interpreting the body language of infants when they cannot express in language. It can provide more accurate and timely care for new-borns by providing a deeper understanding of their non-verbal communication methods. Therefore, new-born related action recognition has very important applications in human-computer interaction, early health detection, medical diagnosis assistance and family care.

[0003] New-born limb action recognition is a new research field that has attracted the attention of many researchers. Although researchers have tried to introduce attention mechanisms and time pyramid networks for action recognition, existing methods often have subtle and short-lived limb actions for new-borns, and existing technologies may not be able to accurately capture the start and end of these small actions.

[0004] In actual application scenarios, the actions of new-borns are disturbed by complex backgrounds and large model computation, which may cause existing methods to be unable to effectively distinguish between actions and backgrounds, making it difficult to obtain information. Therefore, it is difficult to train an action recognition model with high accuracy and strong robustness, resulting in low action recognition accuracy. SUMMARY

[0005] The purpose of the present application is to provide a new-born limb action recognition method based on double-branch multi-scale feature fusion to solve the problem of low capture ability of subtle action changes and low action recognition accuracy in the prior art.

[0006] The technical solution of the present application is:

[0007] A new-born limb action recognition method based on double-branch multi-scale feature fusion, comprising the following steps,

[0008] S1. Collect videos of newborns in different limb motion states, segment the videos into video segments according to time intervals T, and use the first frame of each video segment as a key frame. Detect the key limb parts of the newborn in the key frame, including the head, upper limbs, and lower limbs, generate corresponding bounding boxes, annotate the newborn's limb motion labels, integrate the video segments and labels, and construct a newborn limb motion video sample set;

[0009] S2. Construct a newborn limb movement recognition model based on dual-branch multi-scale feature fusion. The newborn limb movement recognition model based on dual-branch multi-scale feature fusion includes a first-branch feature extraction network, a second-branch feature extraction network, a feature fusion module, and a classifier, wherein:

[0010] The first branch feature extraction network: uses the deep convolutional neural network Net1 to extract action features F at N levels from the input video segment. 1，p , where p = 1, 2, ..., N, and the action features F extracted at the Nth level are 1，N Output to the feature fusion module; where N is an integer ranging from 3 to 5;

[0011] The second branch feature extraction network: the input video segment is frame-sampled to obtain a low frame rate video. The low frame rate video is extracted by the deep convolutional neural network Net2 at N levels to extract action features F 2，p , and the action features F extracted from the first three levels 2，p With action feature F 1，p The features obtained by introducing the multi-scale convolution attention mechanism are then combined with the action features F 2，p After the residual connection, the residual connection feature F is obtained r，p , as the input feature of the next 3D residual block in the deep convolutional neural network Net2, the action feature F extracted at the Nth level 2，N Output to the feature fusion module;

[0012] Feature fusion module: used to fuse the newborn limb movement features output by the first branch feature extraction network and the second branch feature extraction network at the Nth level to obtain the fused feature F;

[0013] Classifier: classifies and identifies the fused feature F and outputs the body movement category;

[0014] S3, using the newborn limb movement video sample set obtained in step S1 to train the newborn limb movement recognition model based on dual-branch multi-scale feature fusion constructed in step S2, to obtain a newborn limb movement recognition model;

[0015] S4. Use the newborn limb movement recognition model obtained in step S3 to perform movement recognition on the newly input test video.

[0016] Further, in step S1, the neonatal limb action labels include head shaking, finger spreading, fist clenching, arm waving, leg kicking, and stillness.

[0017] Further, in step S1, the time interval T is in the range of 8 frames to 64 frames.

[0018] Further, in step S2, the second branch feature extraction network includes a frame extraction module, a deep convolutional neural network Net2, a deep convolutional neural network Net2, three multi-scale convolutional attention modules, and three residual connection modules,

[0019] Frame extraction module: interval frame extraction is performed on the input video segment to obtain a low frame rate video output to the deep convolutional neural network Net2;

[0020] Deep convolutional neural network Net2: action features F 2，p are extracted from the low frame rate video at N levels respectively.

[0021] Multi-scale convolutional attention module: multi-scale convolutional attention mechanism is introduced into the input action features F 1，p and the action features F 2，p respectively, and a multi-scale feature tensor Fpˋ is output.

[0022] Residual connection module: the input multi-scale feature tensor Fpˋ is connected in residual with the action features F 2，p output by the previous level of the deep convolutional neural network Net2, and is output to the 3D residual block of the next level of the deep convolutional neural network Net2.

[0023] Further, in the multi-scale convolutional attention module, multi-scale convolutional attention mechanism is introduced into the input action features F 1，p and the action features F 2，p respectively, and a multi-scale feature tensor Fpˋ is output, specifically,

[0024] 1) The action features F 1，p and the action features F 2，p are spliced in the channel dimension to obtain a feature tensor Fp;

[0025] 2) The attention weight ω p of the feature tensor F p is obtained by using multi-scale convolutional attention mechanism as follows:

[0026]

[0027] where σ represents the Sigmoid activation function, Conv 1×1×1point-wise convolution with a convolution kernel of 1×1×1, L is the number of depthwise separable convolutions, L∈{2, 3, 4, 5}; DW j depthwise separable convolutions with the jth strip kernel of k×1×1, 1×k×1 and 1×1×k, k∈{3, 5, 7, 9, 11};

[0028] 3) the attention weight ω p is multiplied by the feature tensor F p , and is convolved by point-wise convolution with a convolution kernel of 1×1×1, outputting a multi-scale feature tensor F p ′:

[0029] F p ′=Conv 1×1×1 (ω p ·F p )

[0030] where Conv 1×1×1 represents point-wise convolution with a convolution kernel of 1×1×1.

[0031] Further, the feature fusion module includes an SE attention module and an efficient feature fusion block,

[0032] the SE attention module: the output of the first feature extraction network and the output of the second branch feature extraction at the Nth level are respectively subjected to squeeze-and-excitation operation to obtain an enhanced first action feature tensor and an enhanced second action feature tensor;

[0033] the efficient feature fusion block: after self-adaptive pooling operation and convolution processing are respectively performed on the enhanced first action feature tensor and the enhanced second action feature tensor, a convolved first action feature and a convolved second action feature are obtained, a weight is obtained by using an activation function Sigmoid, and the weight is multiplied by the enhanced action feature tensor and the enhanced second action tensor respectively to obtain a first intermediate action feature and a second intermediate action feature, and the first intermediate action feature and the second intermediate action feature are spliced to obtain a fused feature F.

[0034] Further, the classifier includes a pooling block and a fully connected layer,

[0035] the pooling block: the fused feature F is subjected to maximum pooling and average pooling operations to obtain a pooled feature, and the pooled feature is subjected to stretching and flattening operations to output a feature vector V;

[0036] the fully connected layer: the feature vector V is mapped to a probability distribution corresponding to a predicted category and outputs a limb action category.

[0037] Furthermore, in step S3, when training the newborn limb movement recognition model based on dual-branch multi-scale feature fusion constructed in step S2, the focal loss function is used to optimize the training:

[0038]

[0039] Among them, loss i,c represents the binary cross entropy loss, FL represents the focal loss, K represents the total number of categories, c∈[1,K], y i,c Indicates the true value that the i-th sample belongs to the c-th category, i∈[1,n], n represents the total number of samples, It represents the model prediction value that the i-th sample belongs to the c-th category, α represents the balance factor, which is used to adjust the weights of positive and negative samples, and β represents the adjustment factor, which is used to make the labels that are difficult to classify have a larger weight.

[0040] The beneficial effects of the present invention are:

[0041] 1. Compared with existing methods, this method for recognizing newborn limb movements based on dual-branch multi-scale feature fusion uses dual branches to extract movement features from videos with different frame rates, improving the ability to capture subtle movement changes. It also adopts a multi-scale convolutional attention mechanism to focus on the key features of limb movements, effectively improving the accuracy of newborn limb movement recognition and its robustness to complex scenes, while requiring less computation.

[0042] 2. In order to address the problem that the sampling frequency of a single branch cannot flexibly adapt to the diversity of actions, which leads to information redundancy or insufficiency and affects the training speed, the present invention uses dual branches to extract action features from videos with different frame rates, capture rapidly changing short-term dynamic details, alleviate the huge imbalance between effective action features and invalid background features, and enhance the synergy of features at different time scales.

[0043] 3. This method of neonatal limb movement recognition based on dual-branch multi-scale feature fusion introduces a multi-scale convolutional attention mechanism, integrates the feature extraction capabilities of different receptive fields, and combines the weighted strategy of global and local information to achieve accurate capture of multi-scale features, effectively improving the model's ability to recognize detailed features in complex scenes, and using depthwise separable convolution instead of traditional convolution to effectively reduce the model's computational complexity. In the task of neonatal limb movement recognition, the multi-scale convolutional attention mechanism can further improve the accuracy and effectiveness of the model.

[0044] IV. The new-born limb action recognition method based on the dual-branch multi-scale feature fusion can effectively highlight the expression ability of the key feature region by dynamically adjusting the weight of the feature channel through the fusion of the action features of the dual branch, can make the model focus on the key features of the limb action, can dynamically adapt to the action of different time scales, can accurately capture the subtle action difference, and can effectively improve the accuracy and robustness of the action recognition. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 FIG. 1 is a structural schematic diagram of the new-born limb action recognition method based on the dual-branch multi-scale feature fusion according to an embodiment of the present application.

[0046] Figure 2 FIG. 2 is an explanatory schematic diagram of the new-born limb action recognition model based on the dual-branch multi-scale feature fusion according to an embodiment.

[0047] Figure 3 FIG. 3 is an explanatory schematic diagram of the multi-scale convolution attention module according to an embodiment.

[0048] Figure 4 FIG. 4 is an explanatory schematic diagram of the feature fusion module according to an embodiment. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0050] An embodiment of the present application is a new-born limb action recognition method based on the dual-branch multi-scale feature fusion, which comprises the following steps: Figure 1

[0051] S1, collect new-born videos in different limb action states, segment the videos according to a time interval T to obtain video segments, take the first frame of each video segment as a key frame, detect the key limb parts of the new-born in the key frame, including the head, upper limbs and lower limbs, generate the corresponding bounding box, label the new-born limb action label, integrate the video segments and the label, and construct a new-born limb action video sample set.

[0052] In step S1, the YOLO model is used to detect the key limb parts of the new-born in the key frame, including the head, upper limbs and lower limbs, to generate the corresponding bounding box. The position and size of the bounding box are fine-tuned by using the labeling tool VGG Image Annotator, and the new-born limb action label is labeled on the bounding box. The new-born limb action label includes head shaking, finger opening, fist clenching, arm waving, leg kicking and being still. The value range of the time interval T is 8 frames to 64 frames. In this embodiment, T is equal to 32 frames. The new-born limb action video sample set in this embodiment contains new-born limb action video samples and labels with a time length of 10 seconds and a frame rate of 30 fps.

[0053] ​S2, construct a neonatal limb action recognition model based on a dual-branch multi-scale feature fusion, the neonatal limb action recognition model based on a dual-branch multi-scale feature fusion comprises a first branch feature extraction network, a second branch feature extraction network, a feature fusion module and a classifier, wherein:

[0054] The first branch feature extraction network: a deep convolutional neural network Net1 is used to extract action features F 1，p at N levels respectively for the input video segment, p = 1, 2,..., N, and the action features F 1，N extracted at the Nth level are output to the feature fusion module; wherein the value range of N is an integer of 3-5.

[0055] As Figure 2 , the first branch feature extraction network takes N = 3 and the deep convolutional neural network Net1 as an example using 3DResNet-50 as follows: input a neonatal video segment, the first branch passes through the deep convolutional neural network Net1, and the first branch captures more temporal information by capturing fast motion information in the video. ResNet-50 contains 1 3D convolution layer, 1 max pooling layer, 16 3D residual blocks, and the 16 3D residual blocks are respectively represented as 3D residual block B1-3D residual block B 16 . The output features of the N 3D residual blocks are taken as the action features F 1，p extracted at N levels respectively. Wherein, p represents the serial number of the feature level, p ∈ {1, 2, 3}, H p represents the height of the feature map, W p represents the width of the feature map, C 1,p represents the number of channels of the deep convolutional neural network Net1, Q represents the channel scaling ratio, which is used to realize the lightweight of the model, and the value range of Q is an integer of 2-16, C 2,prepresents the number of channels of the deep convolutional neural network Net2. The time interval T is specified as 32 frames, Q = 8, the output channel number of the 3D convolutional layer is set to 8, and the light weight of the model is realized by using a smaller number of channels and a weaker spatial information processing capability to provide the first branch with less redundant spatial information. The 3D convolutional layer uses a 5x7x7 convolution kernel to convolve the input feature tensor of the branch, and the stride is 1x2x2; the pooling layer uses a 1x3x3 max pooling layer to pool the input feature tensor; the 3D residual block uses three different 3D convolutional layers, and the 3D residual block also uses m2 3x1x1, 1x3x3 and 1x1x1 convolution kernels to convolve the input feature tensor, where m2 can be selected from 16, 32 and 64. Among the 16 3D residual blocks, the 3D residual block B3, the 3D residual block B7 and the 3D residual block B 13 The output feature tensor F 1,p is input into the multi-scale convolution attention module, and the feature tensor output by the three residual blocks has a size of

[0056] Second branch feature extraction network: interval frame extraction is performed on the input video segment to obtain a low frame rate video, and the low frame rate video is extracted by the deep convolutional neural network Net2 at N levels to obtain action features wherein p represents the sequence number of the feature level, H p represents the height of the feature map, W p represents the width of the feature map, and C 2,p represents the number of channels of the deep convolutional neural network Net2, and the action features F 2，p extracted by the first three levels are input into the multi-scale convolution attention mechanism to obtain the feature F 1，p corresponding to the action feature F 2，p is spliced to obtain the residual connection feature F r，p , which is used as the input feature of the next 3D residual block in the next deep convolutional neural network Net2. The action feature F 2，N extracted at the Nth level is output to the feature fusion module.

[0057] In step S2, the second branch feature extraction network includes a frame extraction module, a deep convolutional neural network Net2, three multi-scale convolution attention modules and three residual connection modules:

[0058] Frame extraction module: interval frame extraction is performed on the input video segment at a frame extraction frequency M to obtain a low frame rate video, which is output to the deep convolutional neural network Net2; the frame extraction frequency M is an integer in the range of 2-16.

[0059] Deep convolutional neural network Net2: low frame rate video is extracted at N levels to obtain action features F 2，p ;

[0060] Multi-scale convolutional attention module: respectively input action features F 1，p and action feature F 2，p Correspondingly, a multi-scale convolutional attention mechanism is introduced and a multi-scale feature tensor Fpˋ is output; Figure 3 , specifically:

[0061] 1) The action feature F 1，p and action feature F 2，p Splice and get the feature tensor F p ;

[0062] 2) The feature tensor F p The input is fed into three parallel depth-wise separable convolution blocks, where the three depth-wise separable convolution blocks use m 3 3 groups of 3×3×3 convolution kernels and m3 groups of 3 band convolution kernels are used to extract multi-scale information. The optional values ​​of the band convolution kernel are 7, 9, and 11, and the optional values ​​of m3 are 128, 256, and 512. The three feature tensors output are the same as the feature tensor F. p Add and use the activation function to get the attention weight ω p for:

[0063]

[0064] Among them, σ represents the Sigmoid activation function, Conv 1×1×1 Denotes a point-by-point convolution with a convolution kernel of 1×1×1, L is the number of depth-wise separable convolution blocks, L∈{2,3,4,5}; DW j represents a depthwise separable convolution with the jth banded convolution kernels being k×1×1, 1×k×1, and 1×1×k, where k∈{3,5,7,9,11}. In this embodiment, L=3, the sizes of the banded convolution kernels in the first depthwise separable convolution are 7×1×1, 1×7×1, and 1×1×7, the sizes of the banded convolution kernels in the second depthwise separable convolution are 9×1×1, 1×9×1, and 1×1×9, and the sizes of the banded convolution kernels in the third depthwise separable convolution are 11×1×1, 1×11×1, and 1×1×11.

[0065] Depthwise Separable Convolution (DW) j By transforming the feature tensor F in the channel dimension p Divide into features and features Among them, feature G p,0 Stay still, Feature G p,1 Split equally into features in the channel dimension Among them, s represents the sequence number of the segmentation feature, s∈{1,2,3,4}, and a 3×3×3 convolution kernel is used to check the feature. Convolution is performed, and the remaining features are processed using depth convolution of k×1×1, 1×k×1 and 1×1×k banded convolution kernels. Perform convolution operation, concatenate the outputs of each branch in the channel dimension to obtain the feature tensor, the depth-separable convolution table DW j The expression is:

[0066]

[0067] Among them, Concat represents feature concatenation, Dw 3×3×3 Denotes a depthwise convolution with a convolution kernel of 3×3×3, Dw k×1×1 Denotes a depthwise convolution with a banded convolution kernel of k×1×1, Dw 1×k×1 Denotes a depthwise convolution with a striped convolution kernel of 1×k×1, Dw 1×1×k Indicates that the strip convolution kernel is a depth convolution of 1×1×k;

[0068] 3) Weight ω p Channel by channel and feature tensor F p Multiply and perform convolution processing with a convolution kernel of 1×1×1 point by point to output a multi-scale feature tensor for:

[0069] F p ′=Conv 1×1×1 (ω p ·F p )

[0070] Among them, Conv 1×1×1 Indicates point-by-point convolution with a convolution kernel of 1×1×1.

[0071] Taking N=3 as an example, the following is explained: Among the three multi-scale convolutional attention modules, the first multi-scale convolutional attention module is used to extract the shallow feature tensor F output by the first branch and the second branch feature extraction network. 1,1 and F 2,1 The multi-scale convolution attention mechanism is introduced to focus on the key features of subtle body movements; the second multi-scale convolution attention module is used to extract the intermediate layer feature tensor F output by the first branch and the second branch feature extraction network. 1,2 and F 2,2 The multi-scale convolution attention mechanism is introduced to enhance the information interaction between low frame rate features and high frame rate features; the third multi-scale convolution attention module is used to extract the deep feature tensor F output by the first branch and the second branch feature extraction network. 1,3 and F 2,3 A multi-scale convolutional attention mechanism is introduced to focus on the key features of large-scale limb movements.

[0072] Residual connection module: respectively connect the input multi-scale feature tensor Fp' and the action feature Fp output by the previous level of the deep convolutional neural network Net2 2，p Residual connection output residual connection feature to the 3D residual block of the next level of the deep convolutional neural network Net2.

[0073] As Figure 2 , the second branch feature extraction network takes N = 3 and the deep convolutional neural network Net2 as an example of 3DResNet-50 as follows: input the neonatal video segment, the second branch captures the spatial semantic features in static or slowly changing content, uses sparse sampling to reduce the calculation cost while retaining the information of the key frames, and the spatiotemporal size of the unprocessed video segment is 32x224x224, wherein 32 represents the time interval T, and 224x224 represents the height and width after cropping. The second branch passes through the frame extraction module and the deep convolutional neural network Net2, and the features output by the 3D ResNet-50 at the first 3 different levels are represented as F 2，p , wherein p represents the serial number of the feature level, and p∈{1,2,3}. The 3D ResNet-50 contains 1 3D convolutional layer, 1 maximum pooling layer, and 16 3D residual blocks, and the output features of N 3D residual blocks in the deep convolutional neural network Net2 are taken as the action features F 2，p , Figure 2 Only 3 3D residual blocks are shown in the deep convolutional neural network Net2, and the 3 3D residual blocks are respectively represented as residual block A1 to residual block A 16 In this embodiment, the frame extraction frequency M = 8, the frame extraction module extracts a frame every 8 frames, reduces the frame rate of the video, and the time interval T is reduced from 32 frames to 4 frames. The output channel number of the 3D convolutional layer is set to 64, and the stride is 1x2x2; the 3D convolutional layer uses a 1x7x7 convolution kernel to convolve the input feature tensor; the pooling layer uses a 1x3x3 maximum pooling layer to pool the input feature tensor, and the stride is 1x2x2; the residual block uses 3 different 3D convolutional layers, and each residual block uses m1 1x1x1, 1x3x3 and 1x1x1 convolution kernels to convolve the input feature tensor, wherein m1 can be selected from 128, 256 and 512, and residual blocks A3, A7 and A 13 The output feature tensor F 2,p is respectively input into the pth multi-scale convolution attention module, and the output F p of the multi-scale convolution attention module is respectively connected with F 2,p to output the residual connection feature F The feature tensor output by the 3 residual blocks has a scale of

[0074] Residual connection module: respectively connect the input feature Fp' and the action feature F 2，p output by the previous deep convolutional neural network Net2.

[0075] In this embodiment, since N=3 is consistent with the number of multi-scale convolutional attention, the second branch feature extraction network connects the residual feature F r,3 output by the first branch feature extraction network as the action feature F 2，N input to the feature fusion module; when N>3, the feature input to the feature fusion module by the second branch feature extraction network is F 2,N .

[0076] Feature fusion module: used for feature fusion of the newborn limb action features output by the first branch feature extraction network and the second branch feature extraction network at the Nth level, to obtain the fused feature F.

[0077] As Figure 4 , the feature fusion module includes an SE attention module and an efficient feature fusion block:

[0078] SE attention module: respectively adopts the squeeze-and-excitation operation on the output of the first feature extraction network and the second branch feature extraction at the Nth level to obtain the enhanced first action feature tensor and the enhanced second action feature tensor. The convolution layer in the SE attention module adopts a 3x3x3 convolution kernel for convolution, wherein the channel number of the first convolution layer becomes the original channel number which is changed back to the original channel number after the second convolution layer.

[0079] Efficient feature fusion block: respectively performs adaptive pooling operation and convolution processing on the enhanced first action feature tensor and the enhanced second action feature tensor to obtain the convolved first action feature and the convolved second action feature, adopts the activation function Sigmoid to obtain the weight, and performs product operation on the weight and the enhanced action feature tensor and the enhanced second action tensor respectively to obtain the first intermediate action feature and the second intermediate action feature, and splices to obtain the fused feature F.

[0080] As Figure 4 , the efficient feature fusion block includes an adaptive average pooling layer, a convolution layer and an activation function Sigmoid, which are respectively used for multi-scale feature extraction, smoothing and strengthening tasks. In the efficient feature fusion block, the adaptive pooling operation is performed to compress the spatial dimension of the output feature, and a convolution layer with a convolution kernel size of C k is adopted to perform convolution processing on the obtained feature, wherein the size of the variable convolution kernel C kThe size of the convolution kernel can be dynamically calculated according to the number of channels t of the feature tensor, and the expression is:

[0081]

[0082] The weight is obtained by using the Sigmoid activation function, and the weight is multiplied with the original feature tensor channel by channel, and the obtained features are spliced to output the fused features wherein, H N represents the height of the feature map, W N represents the width of the feature map, C 1,N and C 2,N represent the number of channels. In the embodiment, the number of channels t includes the number of channels t1 of the first branch and the number of channels t2 of the second branch, wherein t1 = 2048 and t2 = 256.

[0083] Classifier: classifying and recognizing the action feature vector V and outputting the limb action category. The classifier includes a pooling block and a fully connected layer, wherein the pooling block: obtaining the pooled features by performing maximum pooling and average pooling operations on the fused features F, and outputting the feature vector V after stretching and flattening operation; the fully connected layer: mapping the feature vector V to the probability distribution of the corresponding predicted category and outputting the limb action category.

[0084] S3, using the neonatal limb action video sample set obtained in step S1 to train the neonatal limb action recognition model based on the double-branch multi-scale feature fusion constructed in step S2, to obtain the neonatal limb action recognition model.

[0085] In step S3, when training the neonatal limb action recognition model based on the double-branch multi-scale feature fusion constructed in step S2, a focal loss function is used to optimize the training:

[0086]

[0087] wherein, loss i,c represents the binary cross-entropy loss, FL represents the focal loss, K represents the total number of categories, c [1, K], y i,c represents the true value of the i-th sample belonging to the c-th category, i [1, n], n represents the total number of samples, represents the model prediction value of the i-th sample belonging to the c-th category, a represents a balance factor for adjusting the weight of positive and negative samples, and β represents an adjustment factor for making the weight of difficult-to-classify labels larger. The balance factor a can be selected from 0 to 2, and the adjustment factor β can be selected from 0 to 5. In the embodiment, a = 0.75 and β = 2.

[0088] In step S3, the classifier optimizes the training of the model using a focal loss function, the output of the model is converted into a prediction probability of each class through a Sigmoid activation function, and the prediction probability is converted into a binary label according to a preset threshold value 0.5, a class with a probability greater than the threshold value is marked as 1, otherwise marked as 0, so that it can be directly compatible with the calculation formula of the focal loss function, effectively handle the class imbalance problem, and improve the attention to difficult classification samples.

[0089] S4, using the newborn limb action recognition model obtained in step S3 to recognize the action of the newly input test video.

[0090] The newborn limb action recognition method based on double-branch multi-scale feature fusion uses a double-branch feature extraction module to extract action features from videos of different frame rates, improves the ability to capture subtle action changes, and uses a multi-scale convolution attention module to focus on key features of limb actions, effectively improving the accuracy of newborn limb action recognition and robustness to complex scenes, and the computational complexity is relatively small.

[0091] The newborn limb action recognition method based on double-branch multi-scale feature fusion introduces a multi-scale convolution attention mechanism, fuses the feature extraction capabilities of different receptive fields, combines the weighting strategy of global and local information, realizes accurate capture of multi-scale features, effectively improves the model's ability to identify detailed features in complex scenes, and uses depth separable convolution instead of traditional convolution, effectively reducing the computational complexity of the model. In the task of newborn limb action recognition, the multi-scale convolution attention mechanism can further improve the accuracy and effectiveness of the model.

[0092] The newborn limb action recognition method based on double-branch multi-scale feature fusion fuses the action features of the double branches, dynamically adjusts the weights of the feature channels, effectively highlights the expression ability of the key feature region, makes the model focus on the key features of the limb action, and can dynamically adapt to actions of different time scales, accurately capture subtle action differences, and effectively improve the accuracy and robustness of action recognition.

[0093] The above is only a specific implementation in the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can understand and think of the transformation or replacement within the technical scope disclosed by the present application, which should be covered within the scope of the present application, therefore, the protection scope of the present application should be the protection scope of the claims.

Claims

1. A method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion, characterized by: The following steps are included: S1. Collect videos of newborns in different limb motion states, segment the videos into video segments according to time intervals T, and use the first frame of each video segment as a key frame. Detect the key limb parts of the newborn in the key frame, including the head, upper limbs, and lower limbs, generate corresponding bounding boxes, annotate the newborn's limb motion labels, integrate the video segments and labels, and construct a newborn limb motion video sample set; S2. Construct a newborn limb movement recognition model based on dual-branch multi-scale feature fusion. The newborn limb movement recognition model based on dual-branch multi-scale feature fusion includes a first-branch feature extraction network, a second-branch feature extraction network, a feature fusion module, and a classifier, wherein: The first branch feature extraction network: uses the deep convolutional neural network Net1 to extract action features F at N levels from the input video segment. 1，p , where p = 1, 2, ..., N, and the action features F extracted at the Nth level are 1，N Output to the feature fusion module; where N is an integer ranging from 3 to 5; The second branch feature extraction network: the input video segment is frame-sampled to obtain a low frame rate video. The low frame rate video is extracted by the deep convolutional neural network Net2 at N levels to extract action features F 2，p , and the action features F extracted from the first three levels 2，p With action feature F 1，p The features obtained by introducing the multi-scale convolution attention mechanism are then combined with the action features F 2，p After the residual connection, the residual connection feature F is obtained r，p , as the input feature of the next 3D residual block in the deep convolutional neural network Net2, the action feature F extracted at the Nth level 2，N Output to the feature fusion module; Feature fusion module: used to fuse the newborn limb movement features output by the first branch feature extraction network and the second branch feature extraction network at the Nth level to obtain the fused feature F; Classifier: classifies and identifies the fused feature F and outputs the body movement category; S3, using the newborn limb movement video sample set obtained in step S1 to train the newborn limb movement recognition model based on dual-branch multi-scale feature fusion constructed in step S2, to obtain a newborn limb movement recognition model; S4. Use the newborn limb movement recognition model obtained in step S3 to perform movement recognition on the newly input test video.

2. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to claim 1, characterized in that: In step S1, the newborn's limb movement labels include head shaking, fingers opening, fist clenching, arm waving, leg kicking and standing still.

3. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to claim 1, characterized in that: In step S1 , the time interval T ranges from 8 frames to 64 frames.

4. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to any one of claims 1 to 3, characterized in that: In step S2, the second branch feature extraction network includes a frame extraction module, a deep convolutional neural network Net2, three multi-scale convolutional attention modules and three residual connection modules. Frame extraction module: extracts frames from the input video segment at intervals to obtain low frame rate video output to the deep convolutional neural network Net2; Deep convolutional neural network Net2: extracts action features F at N levels for low frame rate videos 2，p ; Multi-scale convolutional attention module: respectively input action features F 1，p and action feature F 2，p Correspondingly, a multi-scale convolutional attention mechanism is introduced and a multi-scale feature tensor Fp` is output; Residual connection module: The input multi-scale feature tensor Fp` is combined with the action feature F output by the previous layer of the deep convolutional neural network Net2. 2，p After the residual connection, the output is sent to the 3D residual block of the next layer of the deep convolutional neural network Net2.

5. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to claim 4, characterized in that: In the multi-scale convolutional attention module, the input action features F 1，p and action feature F 2，p Correspondingly, a multi-scale convolutional attention mechanism is introduced and a multi-scale feature tensor Fp` is output, specifically, 1) The action feature F 1，p and action feature F 2，p Splicing is performed on the channel dimension to obtain the feature tensor Fp; 2) Use the multi-scale convolution attention mechanism to obtain the attention weight ω of the feature tensor Fp p for: Among them, σ represents the Sigmoid activation function, Conv 1×1×1 Denotes a point-by-point convolution with a convolution kernel of 1×1×1, L is the number of depth-wise separable convolutions, L∈{2,3,4,5}; DW j represents the depthwise separable convolution of the jth banded convolution kernels k×1×1, 1×k×1, and 1×1×k, k∈{3,5,7,9,11}; 3) The attention weight ω p Channel by channel and feature tensor F p Multiply and perform point-by-point convolution with a convolution kernel of 1×1×1 to output a multi-scale feature tensor F p 'for: F p ′=Conv 1×1×1 (ω p ·F p ) Among them, Conv 1×1×1 Indicates point-by-point convolution with a convolution kernel of 1×1×1.

6. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to any one of claims 1 to 3, characterized in that: The feature fusion module includes the SE attention module and the efficient feature fusion block. SE attention module: The squeeze-excitation operation is applied to the output of the first feature extraction network and the second branch feature extraction at the Nth level to obtain the enhanced first action feature tensor and the enhanced second action feature tensor; Efficient feature fusion block: After adaptive pooling and convolution processing on the enhanced first action feature tensor and the enhanced second action feature tensor, the convolution first action feature and the convolution second action feature are obtained. The activation function Sigmoid is used to obtain the weights, and the weights are multiplied channel by channel with the enhanced action feature tensor and the enhanced second action tensor to obtain the first intermediate action feature and the second intermediate action feature. After splicing, the fused feature F is obtained.

7. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to any one of claims 1 to 3, characterized in that: The classifier consists of pooling blocks and fully connected layers. Pooling block: The fused feature F is subjected to maximum pooling and average pooling operations to obtain the pooled feature. The pooled feature is stretched and flattened to output the feature vector V. Fully connected layer: maps the feature vector V to the probability distribution of the corresponding predicted category and outputs the body movement category.

8. The method for recognizing neonatal limb movements based on dual-branch multi-scale feature fusion according to any one of claims 1 to 3, characterized in that: In step S3, when training the newborn limb movement recognition model based on dual-branch multi-scale feature fusion constructed in step S2, the focal loss function is used to optimize the training: Among them, loss i,c represents the binary cross entropy loss, FL represents the focal loss, K represents the total number of categories, c∈[1,K], y i,c Indicates the true value that the i-th sample belongs to the c-th category, i∈[1,n], n represents the total number of samples, It represents the model prediction value that the i-th sample belongs to the c-th category, α represents the balance factor, which is used to adjust the weights of positive and negative samples, and β represents the adjustment factor, which is used to make the labels that are difficult to classify have a larger weight.

Images