Baby crying classification method based on baby facial feature extraction

Abstract

The invention discloses a baby crying classification method based on baby facial feature extraction, which realizes intelligent classification of baby crying reasons by introducing a multi-scale information fusion network (MIFNet) to extract and classify baby facial features. The method comprises the following steps: firstly, extracting local features and global features of a baby face in parallel through a CNN branch and a self-attention branch; secondly, fusing the features by using a spatial global attention module and a channel attention module; and finally, performing feature classification by combining adaptive depth center loss with SoftMax loss, thereby obtaining different crying types of the baby, including but not limited to hunger, restlessness, anthenia, pain, discomfort and the like. Through the method, different types of crying of the baby can be accurately classified and identified, and parents or nursing personnel can timely know and meet the requirements of the baby. Compared with the prior art, the method has high classification accuracy and real-time performance through facial feature extraction and a deep learning algorithm.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and in particular to a method for classifying infant crying based on extraction of infant facial features. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of intelligent care systems in infant care is becoming more and more extensive. Since the language central nervous system of infants is not yet mature, they mainly express their needs and discomfort through facial expressions and crying. However, due to the high similarity between infant facial expressions and the complex and diverse reasons for crying, traditional expression recognition methods are difficult to effectively distinguish different emotions and needs of infants. Therefore, it is of great significance to propose a crying classification method based on infant facial feature extraction to intelligently recognize and classify infant facial expressions. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for classifying infant crying based on extraction of infant facial features in view of the defects involved in the background technology.

[0004] The present invention adopts the following technical solutions to solve the above technical problems:

[0005] A method for classifying infant crying based on extraction of infant facial features comprises the following steps:

[0006] Step 1), data preprocessing, normalization and data enhancement processing of infant facial image data;

[0007] Step 2), feature extraction, through a dual-branch feature extraction structure, using a convolutional neural network (CNN) branch and a self-attention branch to extract local features and global features of the baby's face;

[0008] Step 3), feature fusion, using the spatial global attention module and the channel attention module to fuse local features with global features in the spatial domain and channel domain;

[0009] Step 4), feature classification, using adaptive deep center loss combined with SoftMax loss for feature classification to determine the type of baby crying.

[0010] As a further optimization scheme of the baby crying classification method based on baby facial feature extraction of the present invention, the CNN branch of the dual-branch feature extraction structure in step 2) adopts the backbone network of ResNet18, and the self-attention branch adopts the backbone network of Swin Transformer.

[0011] As a further optimization scheme of the baby crying classification method based on baby facial feature extraction of the present invention, the spatial global attention module in step 3), namely the SGA-Net module, is applied to the baby crying classification task, which obtains the spatial weight information of the input features by fusing the feature map generated by dimensionality reduction to guide the allocation of feature weights;

[0012] The SGA-Net module adds the local features of infant expressions extracted by the CNN branch and the global features extracted by the self-attention branch to obtain the features. The global maximum pooling operation of branch a is used to obtain the maximum element value in each channel vector of the feature to represent the feature information; in The global average pooling operation of branch b is used to obtain the average element value in each channel vector of the feature to represent the feature information, and the two are concatenated to obtain the feature X in In a representation in the spatial domain, 7×7 convolution is used to enhance the splicing features, and finally the fusion feature i is activated by the Sigmoid function. The detailed calculation process is shown in formula (1);

[0013] I=S(f 7×7 [AvgPool (X in );MaxPool (X in )]) (1)

[0014] Among them, S represents the activation function Sigmoid, f 7×7 represents a 7×7 convolution, [·] represents matrix concatenation, AvgPool represents a global average pooling operation, and MaxPool represents a global maximum pooling operation;

[0015] At the same time X in Through the 1×1 convolution of branch c and branch d, channel transformation is performed to realize the dimension reduction processing of the baby's facial expression features and convert them into a one-dimensional vector. The dimension reduction features obtained by branch c are transposed and then multiplied by the dimension reduction features obtained by branch d to fuse the low-dimensional information of the expression features. Finally, the normalized feature J is obtained through the SoftMax function. The detailed calculation process is shown in formula (2);

[0016]

[0017] Among them, δ represents the SoftMax function, f 1×1 represents 1×1 convolution, and * represents matrix multiplication;

[0018] Feature J is transposed and then multiplied by feature I to promote the interaction of information between the two, and the obtained features are restored to the dimensions of the original input features of the baby's facial expressions, so that the information of the features in the spatial domain is enhanced;

[0019] In addition, step 3) uses a channel attention module, namely the SK-Net module, to process feature channel information;

[0020] The SK-Net module adds the local features and global features of the baby's expression to obtain the features Input 3×3 dilated convolution and 3×3 grouped convolution at the same time to get the feature With features In order to achieve the effect of adaptively adjusting the receptive field range, the information flow of multiple branches is controlled by the gating unit, and the information of multiple branches is integrated to obtain

[0021] The integrated feature U is obtained by global average pooling. Then through Function generates vector And calculated by formula (3) and formula (4) and

[0022]

[0023] Among them, a c and b c The matrix in and Initialization is required before training;

[0024] a c and b c The sum of the function values ​​is 1. When setting the feature map weights of the branches, the proportion of the two branches is controlled.

[0025] A c and b c and Multiply Integration to obtain feature maps

[0026]

[0027] V=[V1,V2,…,V c ] (6)

[0028] Among them, a c and b c yes and The corresponding attention vectors, A c With B c Represents the cth row of A and B respectively, a c With b c represents the cth element of a and b respectively, and [·] represents the concatenation of matrices;

[0029] In the SGA-Net module, each channel of the feature is considered equally, and the information exchange between channels is ignored; in the SK-Net module, each channel of the feature is directly analyzed as a whole, and the exchange of spatial information is not considered; the SGA-Net module and the SK-Net module are connected in parallel to obtain a hybrid domain attention module, namely the HDAM module, which processes features in the channel domain and the spatial domain respectively, retains the key information of the feature channel and space, and enhances the expressiveness of the features.

[0030] As a further optimization scheme of the method for classifying infant crying based on extraction of infant facial features of the present invention, in the step 4), an adaptive deep center loss is adopted to reduce the weight of common features between different categories of crying by using the attention network, and increase the weight of different categories of crying difference features. The network is optimized by combining the SoftMax loss to realize accurate identification of infant crying types. The specific calculation is shown in formula (7):

[0031] L=L S +γL SC (7)

[0032] Among them, the balance coefficient γ is set to 0.6, L S is the SoftMax loss, L SC is the adaptive depth center loss;

[0033] The features extracted by the dual-branch feature extraction structure are passed through the HDAM module to obtain fused features. The final d-dimensional deep feature vector is extracted through a feature pooling layer, which is used for SoftMax loss and adaptive deep center loss to judge the crying category. At the same time, it is input into the attention network as context to estimate the attention weight, and the obtained weight is used to guide the center loss to achieve intra-class compactness and inter-class separation of infant crying features in the feature element subset, and is optimized in conjunction with SoftMax loss.

[0034] Not all elements in a feature vector are relevant to classification. The goal is to reduce irrelevant elements in the deep feature vector of the baby's crying expression, so as to obtain a subset of elements in the vector to assist classification. In order to filter out irrelevant features and common features of different crying categories of babies during the classification process, the adaptive deep center loss, namely SDCL, is used. Based on the center loss calculation method, the Euclidean distance of each dimension is weighted. The specific calculation is shown in formula (8):

[0035]

[0036] Among them, ⊙ represents element-wise multiplication, a ijrepresents the weight of the i-th deep feature along dimension j∈{1,…,d} computed by the deep attention network in the embedding space;

[0037] A deep attention network is connected to the HDAM module to dynamically estimate the attention weights of the adaptive deep center loss according to the input baby crying features. ij Guided center loss; The deep attention network mainly consists of two components: the context encoding module and the multi-head binary classification module. The context encoding module uses the fused feature map obtained by the HDAM module as input to generate a potential representation vector; the multi-head binary classification module uses the potential representation vector to estimate the attention weight; The deep attention network processes the features at the convolutional feature level to retain the spatial information of the baby's crying features;

[0038] The upper branch structure of the context encoding module of the deep attention network is composed of stacking the global maximum pooling layer and three trainable fully connected linear layers, and the calculation is shown in formula (9); the lower branch structure is composed of stacking three trainable fully connected linear layers, and the calculation is shown in formula (10):

[0039]

[0040] Among them, X fusion represents the fused feature map obtained after the input baby expression image passes through the HDAM module, that is, the context feature of the i-th sample; operator α: converts the feature map X fusion Perform global maximum pooling and flattening; Operator β: transform the feature map X fusion Flattening is performed; W1 and b1 represent the weight and bias of the lth linear layer of the upper branch of the context coding module, where l = 1, 2, 3; W′1 and b′1 represent the weight and bias of the lth linear layer of the lower branch of the context coding module, where l = 1, 2, 3;

[0041] The batch normalization layer BN(·) and the rectified linear unit ReLU(·) are inserted between each network layer of the context encoding module of the deep attention network. The purpose is to capture the nonlinear relationship between layers. Tanh(·) is used as the activation function to retain positive and negative activation values, making the gradient flow in the network smoother. The potential feature vector of the i-th sample of the upper branch and the lower branch of the context encoding module is calculated by equations (8) and (9) respectively. and And the low-latitude potential feature vector is calculated by formula (11): To eliminate irrelevant information; the context encoding module can be adjusted according to the hierarchical parameters to match the specific task;

[0042] q i =tanh (q i1+q i2 ) (11)

[0043] The multi-head binary classification module of the deep attention network is located after the context encoding module, and the potential d′-dimensional feature vector q i Shared among d linear units, that is, each unit has two outputs to calculate the deep feature X fusion The calculation process of the two raw scores along dimension j is shown in formula (12):

[0044]

[0045] in, and Represents the learnable weights and biases of each classification head in the multi-head binary classification module. The subscript in represents inclusion, and the subscript ex represents exclusion. and Represents X fusion The inclusion score and exclusion score of the jth dimension in ;

[0046] The two raw scores calculated by formula (10) are normalized using the SoftMax function to normalize the output of each head, and the corresponding attention weight of each dimension is obtained, that is, the attention weight a of the adaptive deep center loss ij , the SoftMax function is used to fine-tune the original score and limit the estimated attention weight value to the range of (0,1). The calculation process is shown in formula (13);

[0047]

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] The present invention solves the limitation of the prior art that infant crying classification only relies on sound signals. By introducing facial feature analysis, the accuracy and reliability of classification are improved, and different types of infant crying can be accurately classified and identified so as to better take care of the infant's needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram for feature map visualization;

[0051] Figure 2 This is the schematic diagram of the SGA-Net module;

[0052] Figure 3 This is a schematic diagram of the SK-Net module;

[0053] Figure 4 It is a schematic diagram of the HDAM framework;

[0054] Figure 5Schematic diagram of the loss framework of MIFNet;

[0055] Figure 6 Schematic diagram of deep attention network. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0057] The present invention can be implemented in many different forms and should not be considered to be limited to the embodiments described herein. On the contrary, these embodiments are provided to make this disclosure thorough and complete, and will fully express the scope of the present invention to those skilled in the art. In the accompanying drawings, components are enlarged for clarity.

[0058] As shown in the accompanying drawings, a method for classifying infant crying based on infant facial feature extraction comprises the following steps:

[0059] Data preprocessing: The collected infant facial image data is normalized and enhanced through rotation, translation, scaling, etc. to increase data diversity.

[0060] Feature extraction: The present invention adopts a dual-branch feature extraction structure, in which the CNN branch adopts the backbone network of ResNet18, and the self-attention branch adopts the backbone network of Swin Transformer. Figure 1 The invention shows the baby crying features extracted at each stage of the dual-branch feature extraction structure, as well as the fusion features obtained after the feature fusion module. The features extracted by the CNN branch highlight the local detail features more and lack the information of the global features, while the features extracted by the self-attention branch establish the global dependency between the pixels, but lack the attention to the local information and cannot accurately capture the local detail information. Therefore, the model used in the present invention combines the characteristics of the CNN structure and the self-attention structure, and uses the CNN branch and the self-attention branch to extract the local features and the global features of the baby's expression respectively, which are used for the subsequent processing of the network.

[0061] Feature fusion: This paper uses a spatial global attention network (SGA-Net) to apply to the task of infant cry classification. It obtains the spatial weight information of the input features by fusing the feature map generated by dimensionality reduction to guide the allocation of feature weights. Its structure is as follows: Figure 2 shown.

[0062] The SGA-Net module adds the local features of infant expressions extracted by the CNN branch and the global features extracted by the self-attention branch to obtain the features. The global maximum pooling operation of branch a is used to obtain the maximum element value in each channel vector of the feature to represent the feature information. Similarly, X in The global average pooling operation of branch b is used to obtain the average element value in each channel vector of the feature to represent the feature information, and the two are concatenated to obtain the feature X in In the spatial domain, a 7×7 convolution is used to enhance the splicing features, and finally the fusion feature I is activated by the Sigmoid function. The detailed calculation process is shown in formula (1). in Through the 1×1 convolution of branches c and d, channel transformation is performed to realize the dimension reduction processing of the baby's facial expression features and convert them into a one-dimensional vector. The dimension reduction features obtained by branch c are transposed and then multiplied by the dimension reduction features obtained by branch d to fuse the low-dimensional information of the expression features. Finally, the normalized feature J is obtained through the SoftMax function. The detailed calculation process is shown in formula (2). Feature J is transposed and multiplied by feature I to promote the interaction of information between the two, and the obtained features are restored to the dimension of the original input features of the baby's facial expression, so that the information of the features in the spatial domain is enhanced.

[0063] I=S(f 7×7 [AvgPool (X in );MaxPool (X in )]) (1)

[0064] Among them, S represents the activation function Sigmoid, f 7×7 represents a 7×7 convolution, [·] represents matrix concatenation, AvgPool represents a global average pooling operation, and MaxPool represents a global maximum pooling operation.

[0065]

[0066] Among them, δ represents the SoftMax function, f 1×1 represents 1×1 convolution, and * represents matrix multiplication.

[0067] In combination with the characteristics of the channel attention mechanism, the present invention uses the channel attention module SK-Net (selective kernel network) to process feature channel information. SK-Net starts from the perspective of multi-scale features, uses two different convolution kernel branches to learn features at different scales, obtains the weights of important feature scale information, and enhances the network's attention to important channel features. Its structure is as follows: Figure 3 shown.

[0068] The SK-Net module adds the local features and global features of the baby's expression to obtain the features At the same time, input 3×3 empty convolution (receptive field is 5×5) and 3×3 grouped convolution to get the feature With features In order to achieve the effect of adaptively adjusting the receptive field range, the information flow of multiple branches is controlled by the gating unit, and the information of multiple branches is integrated to obtain The integrated feature U is obtained by global average pooling. Then through Function generates vector And calculated by formula (3) and formula (4) and

[0069]

[0070] Among them, a c and b c The matrix in and Initialization is required before training.

[0071] a c and b c The sum of the function values ​​is 1. When setting the feature map weights of the branches, the weights of the two branches are controlled. c and b c and Multiply Integration to obtain feature maps

[0072]

[0073] V=[V1,V2,…,V c ] (6)

[0074] Among them, a c and b c yes and The corresponding attention vectors, A c With B c Represents the cth row of A and B respectively, a c With b c They represent the cth element of a and b respectively, and [·] represents the concatenation of matrices.

[0075] In the spatial attention module, each channel of the feature is considered equally, and the information exchange between channels is ignored; in the channel attention module, each channel of the feature is directly analyzed as a whole, and the exchange of spatial information is not considered. Therefore, the present invention connects the spatial domain attention module SGA-Net and the channel domain attention module SK-Net in parallel to obtain a hybrid domain attention module (HDAM), whose structure is as follows Figure 4 The features are processed separately in the channel domain and the spatial domain, retaining the key information of the feature channel and space, and enhancing the expressiveness of the features.

[0076] Feature classification: In view of the high similarity between infant facial expressions, this paper adopts an adaptive deep center loss, uses the attention network to reduce the weight of common features between different types of crying, increases the weight of different types of crying features, and optimizes the network with SoftMax loss to achieve accurate recognition of infant crying types. The specific calculation is shown in formula (7).

[0077] L=γL S +γL SC (7)

[0078] Among them, the balance coefficient γ is set to 0.6, γL S is the SoftMax loss, L SC is the adaptive depth center loss.

[0079] The loss function used by the MIFNet network is as follows Figure 5 As shown in the figure, the features extracted by the dual-branch structure are fused after the HDAM module, and the final d-dimensional deep feature vector is extracted through a feature pooling layer, which is used for SoftMax loss and adaptive deep center loss to judge the crying category. At the same time, it is input into the attention network as context to estimate the attention weight, and the obtained weight is used to guide the center loss to achieve intra-class compactness and inter-class separation of infant crying features in the feature element subset, and optimized with SoftMax loss.

[0080] Not all elements in a feature vector are relevant to classification. The goal is to reduce irrelevant elements in the deep feature vector of the baby's crying expression, so as to obtain a subset of elements in the vector to assist classification. In order to filter out irrelevant features and common features of different crying categories of babies during the classification process, the present invention adopts self-adaptive deep center loss (SDCL), which performs weighted calculation of the Euclidean distance of each dimension based on the center loss calculation method. The specific calculation is shown in formula (8).

[0081]

[0082] Among them, ⊙ represents element-wise multiplication, a ij represents the weight of the i-th deep feature along dimension j∈{1,…,d} computed by the deep attention network in the embedding space.

[0083] The present invention adopts an adaptive and flexible method to estimate the weight of input features to adapt to the task of baby cry recognition. To this end, the present invention uses a deep attention network (DAN) connected to the HDAM module to dynamically estimate the attention weight a of the adaptive deep center loss according to the input baby cry features. ij Guided by the center loss. The structure of the deep attention network is as follows Figure 6 As shown, it mainly includes two components: context encoder module (CEU), which uses the fusion feature map obtained by the HDAM module as input to generate a potential representation vector; multi-head binary classification module (MBCM), which uses the potential representation vector to estimate the attention weight. The deep attention network used in the present invention processes the features at the convolutional feature level, so the spatial information of the baby's crying features is retained.

[0084] The upper branch structure of the dual-branch context encoding module of the deep attention network is composed of stacking a global maximum pooling layer and three trainable fully connected linear layers, and the calculation is shown in formula (9); the lower branch structure is composed of stacking three trainable fully connected linear layers, and the calculation is shown in formula (10).

[0085]

[0086] Among them, X fusion represents the fused feature map obtained after the input baby expression image passes through the HDAM module, that is, the context feature of the i-th sample. Operator α: Convert the feature map X fusion Perform global maximum pooling and flattening; Operator β: transform the feature map X fusion Flattening is performed. W1 and b1 represent the weight and bias of the lth linear layer of the upper branch of the context coding module, where l = 1, 2, 3. W′1 and b′1 represent the weight and bias of the lth linear layer of the lower branch of the context coding module, where l = 1, 2, 3.

[0087] The batch normalization layer BN(·) and the rectified linear unit ReLU(·) are inserted between each network layer of the context encoding module of the deep attention network. The purpose is to capture the nonlinear relationship between layers. Tanh(·) is used as the activation function to retain positive and negative activation values, making the gradient flow in the network smoother. The potential feature vector of the i-th sample of the upper branch and the lower branch of the context encoding module is calculated by equations (8) and (9) respectively. and And the low-latitude potential feature vector is calculated by formula (11): It is used to eliminate irrelevant information and retain important feature information. The context encoding module can be adjusted according to the hierarchical parameters to achieve matching to specific tasks.

[0088] q i =tanh (q i1 +q i2 ) (11)

[0089] The multi-head binary classification (inclusion / exclusion) module of the deep attention network is located after the context encoding module, and the potential d′-dimensional feature vector q i Shared among d linear units, that is, each unit has two outputs to calculate the deep feature X fusion The two raw scores along dimension j are calculated as shown in formula (12).

[0090]

[0091] in, and Represents the learnable weights and biases of each classification head in the multi-head binary classification module. The subscript in represents inclusion, and the subscript ex represents exclusion. and Represents X fusion The inclusion score and exclusion score of the j-th dimension in .

[0092] The two raw scores calculated by formula (10) are normalized using the SoftMax function to normalize the output of each head, and the corresponding attention weight of each dimension is obtained, that is, the attention weight a of the adaptive deep center loss ij , the SoftMax function is used to fine-tune the original score and limit the estimated attention weight value to the range of (0,1). The calculation process is shown in formula (13).

[0093]

[0094] Through the above embodiments, the present invention can effectively identify different reasons for the crying of infants, and provides an efficient and accurate method for classifying infant crying. It can monitor the infant's condition in real time, and provide timely feedback on the classification results, so as to help parents or caregivers take better care of the infant, meet their needs, and improve the infant's quality of life and health level.

[0095] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0096] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for classifying infant crying based on extraction of infant facial features, characterized in that: The following steps are involved: Step 1), data preprocessing, normalization and data enhancement processing of infant facial image data; Step 2), feature extraction, through a dual-branch feature extraction structure, using a convolutional neural network (CNN) branch and a self-attention branch to extract local features and global features of the baby's face; Step 3), feature fusion, using the spatial global attention module and the channel attention module to fuse local features with global features in the spatial domain and channel domain; Step 4), feature classification, using adaptive deep center loss combined with SoftMax loss to perform feature classification and determine the type of baby crying.

2. The method for classifying infant crying based on infant facial feature extraction according to claim 1, characterized in that: In step 2), the CNN branch of the dual-branch feature extraction structure uses the backbone network of ResNet18, and the self-attention branch uses the backbone network of SwinTransformer.

3. The method for classifying crying babies based on extraction of baby facial features according to claim 2, characterized in that: The spatial global attention module in step 3), namely the SGA-Net module, is applied to the task of classifying infant crying, and obtains the spatial weight information of the input features by fusing the feature graph generated by dimensionality reduction to guide the allocation of feature weights; The SGA-Net module adds the local features of infant expressions extracted by the CNN branch and the global features extracted by the self-attention branch to obtain the features. The global maximum pooling operation of branch a is used to obtain the maximum element value in each channel vector of the feature to represent the feature information; X in The global average pooling operation of branch b is used to obtain the average element value in each channel vector of the feature to represent the feature information, and the two are concatenated to obtain the feature X in In a representation in the spatial domain, 7×7 convolution is used to enhance the splicing features, and finally the fusion feature i is activated by the Sigmoid function. The detailed calculation process is shown in formula (1); I=S(f 7×7 [AvgPool (X in );MaxPool (X in )]) (1) Among them, S represents the activation function Sigmoid, f 7×7 represents a 7×7 convolution, [·] represents matrix concatenation, AvgPool represents a global average pooling operation, and MaxPool represents a global maximum pooling operation; At the same time X in Through the 1×1 convolution of branch c and branch d, channel transformation is performed to realize the dimension reduction processing of the baby's facial expression features and convert them into a one-dimensional vector. The dimension reduction features obtained by branch c are transposed and then multiplied by the dimension reduction features obtained by branch d to fuse the low-dimensional information of the expression features. Finally, the normalized feature J is obtained through the SoftMax function. The detailed calculation process is shown in formula (2); Among them, δ represents the SoftMax function, f 1×1 represents 1×1 convolution, and * represents matrix multiplication; Feature J is transposed and then multiplied by feature I to promote the interaction of information between the two, and the obtained features are restored to the dimensions of the original input features of the baby's facial expressions, so that the information of the features in the spatial domain is enhanced; In addition, step 3) uses a channel attention module, namely the SK-Net module, to process feature channel information; The SK-Net module adds the local features and global features of the baby's expression to obtain the features Input 3×3 dilated convolution and 3×3 grouped convolution at the same time to get the feature With features In order to achieve the effect of adaptively adjusting the receptive field range, the information flow of multiple branches is controlled by the gating unit, and the information of multiple branches is integrated to obtain The integrated feature U is obtained by global average pooling. Then through Function generates vector And calculated by formula (3) and formula (4) and Among them, a c and b c The matrix in and Initialization is required before training; a c and b c The sum of the function values ​​is 1. When setting the feature map weights of the branches, the proportion of the two branches is controlled. A c and b c and Multiply Integration to obtain feature maps V=[V1,V2,…,V c ] (6) Among them, a c and b c yes and The corresponding attention vectors, A c With B c Represents the cth row of A and B respectively, a c With b c represents the cth element of a and b respectively, and [·] represents the concatenation of matrices; In the SGA-Net module, each channel of the feature is considered equally, and the information exchange between channels is ignored; in the SK-Net module, each channel of the feature is directly analyzed as a whole, and the exchange of spatial information is not considered; the SGA-Net module and the SK-Net module are connected in parallel to obtain a hybrid domain attention module, namely the HDAM module, which processes features in the channel domain and the spatial domain respectively, retains the key information of the feature channel and space, and enhances the expressiveness of the features.

4. The method for classifying infant crying based on infant facial feature extraction according to claim 3, characterized in that: In step 4), in view of the high similarity between infant facial expression classes, an adaptive deep center loss is used, and the attention network is used to reduce the weight of the common features between different types of crying, and increase the weight of the difference features of different types of crying. The network is optimized in conjunction with the SoftMax loss to achieve accurate recognition of infant crying types. The specific calculation is shown in formula (7): L=L S +γL SC (7) Among them, the balance coefficient γ is set to 0.6, γL S is the SoftMax loss, L SC is the adaptive depth center loss; The features extracted by the dual-branch feature extraction structure are passed through the HDAM module to obtain fused features. The final d-dimensional deep feature vector is extracted through a feature pooling layer, which is used for SoftMax loss and adaptive deep center loss to judge the crying category. At the same time, it is input into the attention network as context to estimate the attention weight, and the obtained weight is used to guide the center loss to achieve intra-class compactness and inter-class separation of infant crying features in the feature element subset, and is optimized in conjunction with SoftMax loss. Not all elements in a feature vector are relevant to classification. The goal is to reduce irrelevant elements in the deep feature vector of the baby's crying expression, so as to obtain a subset of elements in the vector to assist classification. In order to filter out irrelevant features and common features of different crying categories of babies during the classification process, the adaptive deep center loss, namely SDCL, is used. Based on the center loss calculation method, the Euclidean distance of each dimension is weighted. The specific calculation is shown in formula (8): Among them, ⊙ represents element-wise multiplication, a ij represents the weight of the i-th deep feature along dimension j∈{1,…,d} computed by the deep attention network in the embedding space; A deep attention network is connected to the HDAM module to dynamically estimate the attention weights of the adaptive deep center loss according to the input baby crying features. ij Guided center loss; The deep attention network mainly consists of two components: the context encoding module and the multi-head binary classification module. The context encoding module uses the fused feature map obtained by the HDAM module as input to generate a potential representation vector; the multi-head binary classification module uses the potential representation vector to estimate the attention weight; The deep attention network processes the features at the convolutional feature level to retain the spatial information of the baby's crying features; The upper branch structure of the context encoding module of the deep attention network is composed of stacking the global maximum pooling layer and three trainable fully connected linear layers, and the calculation is shown in formula (9); the lower branch structure is composed of stacking three trainable fully connected linear layers, and the calculation is shown in formula (10): Among them, X fusion represents the fused feature map obtained after the input baby expression image passes through the HDAM module, that is, the context feature of the i-th sample; operator α: converts the feature map X fusion Perform global maximum pooling and flattening; Operator β: transform the feature map X fusion Flattening is performed; W1 and b1 represent the weight and bias of the lth linear layer of the upper branch of the context encoding module, where l = 1, 2, 3; W1 ′ and b ′ 1 represents the weight and bias of the lth linear layer in the lower branch of the context encoding module, where l = 1, 2, 3; The batch normalization layer BN(·) and the rectified linear unit ReLU(·) are inserted between each network layer of the context encoding module of the deep attention network. The purpose is to capture the nonlinear relationship between layers. Tanh(·) is used as the activation function to retain positive and negative activation values, making the gradient flow in the network smoother. The potential feature vector of the i-th sample of the upper branch and the lower branch of the context encoding module is calculated by equations (8) and (9) respectively. and And the low-latitude potential feature vector is calculated by formula (11): To eliminate irrelevant information; the context encoding module can be adjusted according to the hierarchical parameters to match the specific task; q i =tanh (q i1 +q i2 ) (11) The multi-head binary classification module of the deep attention network is placed after the context encoding module, and the potential d ′ dimensional feature vector q i Shared among d linear units, that is, each unit has two outputs to calculate the deep feature X fusion The calculation process of the two raw scores along dimension j is shown in formula (12): in, and Represents the learnable weights and biases of each classification head in the multi-head binary classification module. The subscript in represents inclusion, and the subscript ex represents exclusion. and Represents X fusion The inclusion score and exclusion score of the jth dimension in ; The two raw scores calculated by formula (10) are normalized using the SoftMax function to normalize the output of each head, and the corresponding attention weight of each dimension is obtained, that is, the attention weight a of the adaptive deep center loss ij , the SoftMax function is used to fine-tune the original score and limit the estimated attention weight value to the range of (0,1). The calculation process is shown in formula (13);

Images