Clustering-based autism diagnosis method for cross-domain facial expression recognition

Abstract

The invention discloses an autism diagnosis method for performing cross-domain facial expression recognition based on clustering. The method comprises the following steps: S1 to S4 are source domain model training parts; s1, acquiring a source domain facial expression image; s2, using a large model CLIP to initialize a pre-training model, and preprocessing an input face image; s3, initializing deep learning training conditions, freezing large model parameters, and only training Adapter and a classification module; s4, after the model is trained, facial expression images can be classified, and a trained source domain model weight is also obtained; s5-S10 are target domain model training and autism diagnosis parts. S5, inputting a face video (image sequence) of each child; s6, using the trained source domain model weight to initialize the model, and preprocessing a non-label face image; s7, training a deep learning model, freezing large model parameters, and only training Adapter and a classification module; and S8, according to expressions and facial features classified by the model, domain self-adaption is carried out through a clustering-based dynamic pseudo tag generation module. S9, classifying a facial expression sequence by using the model subjected to domain self-adaption, and performing autism classification by using a long-short-term memory network; and S10, obtaining a diagnosis result.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and pattern recognition, and in particular relates to an autism diagnosis method for cross-domain facial expression recognition based on clustering. Background Art

[0002] Autism spectrum disorder (ASD) is a common neurodevelopmental disorder that appears in early childhood and is characterized by social impairments and repetitive behaviors. While facial expression recognition has shown high application value in ASD diagnosis, previous studies have overlooked the key domain shift between ASD and traditional facial expression datasets, limiting the accuracy of diagnosis. In addition, ASD data are often unlabeled, highly sensitive expressions, making it difficult to directly access ASD data for model training. To address this challenge, we propose a clustering-based approach for cross-domain facial expression recognition for autism diagnosis, which uses seven expressions from a low-entropy group as cluster centers and generates high-quality pseudo-labels from unlabeled data via semi-supervised learning. The approach includes a dynamic memory module that continuously optimizes feature representations based on new inputs, resulting in iterative improvements in cross-domain recognition performance. Summary of the invention

[0003] The present invention proposes an autism diagnosis method based on clustering for cross-domain facial expression recognition to solve the problems mentioned in the above background technology. The technical solution steps of the present invention are as follows:

[0004] Step 1: Use the labeled public dataset RAFDB 2.0 to train the source domain model of the large model CLIP to obtain the trained source domain model weights.

[0005] Step 1.1: Use the trained CLIP weights to initialize the deep learning model, perform image preprocessing on the face image, and standardize the face image.

[0006] Step 1.2: Freeze the weights of the large model CLIP and do not perform parameter training. Add the adapter module and the classifier module. These two modules are learnable and trainable and participate in the training.

[0007] Step 1.3: Select the highest classification accuracy in the test result after training as the weight of the trained source domain model, which will be used as the initial weight for subsequent target domain learning.

[0008] Step 2: Use facial video data of autistic children without facial expression labels as target domain data.

[0009] Step 2.1: The autism data is video data about each child. Use open-cv to frame the video of each child to obtain a facial image sequence of each child.

[0010] Step 2.2: Use MTCNN to further detect and crop the face image of the same size.

[0011] Step 3: Use the trained source domain model weights to initialize the CLIP model, including the trainable adapter module and classifier module, and standardize the facial image data.

[0012] Step 4: Input the processed facial image into the initialized model to obtain the corresponding expression classification and facial features that have not been classified. Then input it into the dynamic pseudo-label generation module to generate the corresponding pseudo-label. The pseudo-labeled image will participate in the next model training, so as to continuously iterate and optimize the model and increase the proportion of low entropy groups.

[0013] Step 4.1: The model structure is the same as the source domain model. The target domain image will not be trained when it is first input into the model. After being processed by the model, facial expression classification and unclassified facial features can be directly obtained.

[0014] Step 4.2: Input the facial expression classification probability and the unclassified facial features into the dynamic pseudo-label generation module, which will obtain the average entropy value based on the facial expression classification probability, and divide it into high entropy group and low entropy group based on this standard. (High entropy: the expression classification determination value of the image is not high. Low entropy: the expression classification determination value of the image is high). The image feature with the lowest entropy value for each expression is used as the cluster center point of K-Means. The low entropy group is reclassified into seven clusters, and the label of the cluster center is used as the pseudo label of other images in the cluster.

[0015] Step 4.3: Re-input the pseudo-labeled images into the model and perform adaptive training on the model. The trainable parameters are still the adapter and the classifier module. Then test the trained model on the target domain data. Repeat step 4.2 and continuously iterate and update to increase the proportion of the low entropy group. We use 30 rounds of training and perform an iterative update on the low entropy group every 5 rounds.

[0016] Step 4.3.1: Calculate the centroid of each class in the target domain. Here, xt represents the target domain sample, gt represents the network feature extractor, ft represents the classifier, and δk represents the model's predicted probability that the sample is the kth class. These centroids can characterize the distribution of different classes in the target domain. The formula is as follows:

[0017]

[0018] Step 4.3.2: Obtain pseudo labels by calculating the distance between the feature and the centroid, where, represents the generated pseudo-label, Df(gt(xt),ck (0) ) represents the cosine distance between the target domain sample feature and the centroid. The formula is as follows:

[0019]

[0020] Step 4.3.3: Update the target centroid based on the generated pseudo label, and update the pseudo label to obtain the final pseudo label. represents the conversion function, only when When the condition holds, the function value is 1, and when the condition does not hold, the function value is 0. It is called self-supervised pseudo label. The formula is as follows:

[0021]

[0022] Step 4.3.4: The samples in the low entropy group and their corresponding pseudo labels will participate in the training, and the label smoothed cross entropy loss is used as the classification loss function. Where ∈ represents the smoothing factor, represents the probability of the pseudo label being in the kth category, pi represents the model's predicted probability that the sample is in the kth category, and N represents the number of categories. The formula is as follows:

[0023]

[0024] Step 4.3.5: Minimize the entropy loss to make the target output approximate a one-hot vector. Where pk represents the probability that the target sample is the kth class, Indicates the size of the entropy of the prediction result:

[0025]

[0026] Step 4.3.6: One problem with the loss is that it may cause most samples to be classified into the same category. To avoid this problem, this chapter introduces the diversity loss Regularization is performed. Among them, is the average probability of all target samples in the kth category. Diversity loss can force the model to predict a more uniform distribution of categories, achieving the goal of predicting diverse categories. The formula is as follows:

[0027]

[0028] Step 4.3.7: The overall optimization goal of this network is as shown in the following formula:

[0029]

[0030] Step 5: Use the deep neural network model with updated weight parameters as the expression recognition model, input the autistic face image sequence to be recognized into the expression recognition model, and obtain the corresponding expression sequence. Use the long short-term memory network to classify this sequence and obtain the diagnosis result.

[0031] An autism diagnosis method for cross-domain facial expression recognition based on clustering, characterized by comprising a preprocessing module, a deep neural network initialization module, a CLIP fine-tuning module, a dynamic pseudo-label generation module, and an autism diagnosis module;

[0032] Preprocessing module: Use open-cv to process each child's video into frames, and then use MTCNN to crop the facial area and preprocess the face image to be recognized, and output the standardized face image after face alignment to the input module;

[0033] Deep neural network initialization module: In the source domain model training phase, the CLIP pre-trained model (ViT-B / 16) is used to initialize the weights of the deep network. In the target domain adaptation phase, the trained source domain model is used to initialize the weights of the deep network;

[0034] Dynamic pseudo-label generation module: The facial expression classification probability and unclassified facial features are input into the dynamic pseudo-label generation module. The module will obtain the average entropy value based on the facial expression classification probability, and divide it into high entropy group and low entropy group based on this standard. (High entropy: the expression classification determination value of the image is not high. Low entropy: the expression classification determination value of the image is high). The image feature with the lowest entropy value of each expression is used as the cluster center point of K-Means. The low entropy group is reclassified into seven clusters, and the label of the cluster center is used as the pseudo label of other images in the cluster.

[0035] Autism diagnosis module: The facial expression sequence output by the model is input into the long short-term memory network to obtain the diagnosis result.

[0036] The autism diagnosis method for cross-domain facial expression recognition based on clustering of the present invention has the following beneficial effects:

[0037] The autism diagnosis method proposed in this invention for cross-domain facial expression recognition based on clustering solves the problem of facial expression recognition of autistic children on unlabeled target domain data, and achieves efficient and accurate expression classification through cross-domain migration and dynamic pseudo-label generation. Compared with traditional methods, this method greatly improves recognition accuracy without labels, ensures the reliability of labels, reduces category bias, makes the model applicable to diverse expression data, and enhances the support effect of autism diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flow chart of the autism diagnosis method for cross-domain facial expression recognition based on clustering of the present invention;

[0039] Figure 2 A framework diagram of the autism diagnosis method for cross-domain facial expression recognition based on clustering according to the present invention;

[0040] Figure 3 The adapter framework diagram of the present invention for cross-domain facial expression recognition based on clustering; DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with preferred embodiments. More details are elaborated in the following description to facilitate a full understanding of the present invention. However, the present invention can obviously be implemented in a variety of other ways different from the description. Those skilled in the art can make similar generalizations and deductions based on actual application situations without violating the connotation of the present invention. Therefore, the protection scope of the present invention should not be limited by the content of this specific embodiment.

[0042] A clustering-based method for cross-domain facial expression recognition in autism diagnosis, such as Figure 1 As shown, the following steps are included:

[0043] S1, obtain source domain facial expression images;

[0044] S2: Use the large model CLIP to initialize the pre-trained model and pre-process the input face image;

[0045] S2.1: Initialize the deep neural network using the public open source CLIP pre-trained model (ViT-B / 16).

[0046] S2.2: Use the face detector MTCNN to detect the face image of the original image to be recognized. If the face area is detected, the rectangular face area is cropped as the face image to be recognized;

[0047] S3: Load the pre-trained weight parameters of the CLIP deep neural network model and freeze its parameters. No parameter update is performed during training. However, the adapter and classification module continuously update weight parameters during training.

[0048] S3.1: Use the RAFDB 2.0 face recognition dataset to pre-train the deep neural network model and obtain the source domain model weight parameters of the deep neural network model;

[0049] S4: After training, the model can classify facial expression images, and a trained source domain model weight is also obtained;

[0050] S5: Autism data is video data about each child. Open-cv is used to divide the video of each child into frames to obtain a facial image sequence of each child as model input.

[0051] S6: Initialize the model using the trained source domain model weights and preprocess unlabeled facial images:

[0052] S6.1: Initialize the deep neural network using the source domain model trained on RAFDB 2.0.

[0053] S6.2: Use MTCNN to crop and preprocess the facial area, preprocess the face image to be recognized, and output the standardized face image after face alignment to the input module;

[0054] S7: Train the deep learning model, freeze the large model parameters and only train the adapter and classification modules;

[0055] S8: Based on the expressions and facial features classified by the model, domain adaptation is performed through a clustering-based dynamic pseudo-label generation module.

[0056] S8.1: The facial expression classification probability and the unclassified facial features are input into the dynamic pseudo-label generation module. The module will obtain the average entropy value based on the facial expression classification probability, and divide it into high entropy group and low entropy group based on this standard. (High entropy: the expression classification determination value of the image is not high. Low entropy: the expression classification determination value of the image is high). The image feature with the lowest entropy value for each expression is used as the cluster center point of K-Means. The low entropy group is reclassified into seven clusters, and the label of the cluster center is used as the pseudo label of other images in the cluster.

[0057] S8.2: Calculate the centroid of each class in the target domain. Where xt represents the target domain sample, gt represents the network feature extractor, ft represents the classifier, and δk represents the model's predicted probability that the sample is of the kth class. These centroids can characterize the distribution of different classes in the target domain. The formula is as follows:

[0058]

[0059] S8.3: Obtain pseudo labels by calculating the distance between the feature and the centroid, where represents the generated pseudo-label, Df(gt(xt),ck (0) ) represents the cosine distance between the target domain sample feature and the centroid. The formula is as follows:

[0060]

[0061] S8.4: Update the target centroid according to the generated pseudo label, and update the pseudo label to obtain the final pseudo label. represents the conversion function, only when When the condition holds, the function value is 1, and when the condition does not hold, the function value is 0. It is called self-supervised pseudo label. The formula is as follows:

[0062]

[0063] S8.5: The samples in the low entropy group and their corresponding pseudo labels will participate in the training, and the label smoothed cross entropy loss is used as the classification loss function. Where ∈ represents the smoothing factor, represents the probability of the pseudo label being in the kth category, pi represents the model's predicted probability that the sample is in the kth category, and N represents the number of categories. The formula is as follows:

[0064]

[0065] S8.6: Entropy minimization loss makes the target output approximate a one-hot vector. Where pk represents the probability that the target sample is of the kth class, Indicates the size of the entropy of the prediction result:

[0066]

[0067] S8.7: One problem with the loss is that it may cause most samples to be classified into the same category. To avoid this problem, this chapter introduces the diversity loss Regularization is performed. Among them, is the average probability of all target samples in the kth category. Diversity loss can force the model to predict a more uniform distribution of categories, achieving the goal of predicting diverse categories. The formula is as follows:

[0068]

[0069] S8.8: The overall optimization goal of this network is as shown in the following formula:

[0070]

[0071] S9: Use the domain-adapted model to classify facial expression sequences and use long short-term memory networks for autism classification;

[0072] S10, obtaining a diagnosis result;

[0073] A framework diagram of an autism diagnosis method based on clustering for cross-domain facial expression recognition, as shown in the figure. Figure 2 As shown, the following steps are included:

[0074] Phase 1: Pseudo-label generation. The deep neural network initialized by the trained source domain model has a preliminary facial expression recognition capability. Therefore, the facial expression recognition of autistic facial sequences can be performed, and the corresponding expression classification and facial features can be obtained. The average entropy value is obtained according to the facial expression probability, and the class is divided into high and low entropy groups. The low entropy group is further clustered by K-Means to obtain the corresponding pseudo-label.

[0075] Phase 2: The pseudo-labeled images are re-entered into the model for model training. Domain adaptation is performed on autism data. By continuously updating the model parameters, different expression classifications can be obtained and different facial features can be extracted. This can also be used as the material for the next pseudo-label generation. Continuously iterate to increase the proportion of the low entropy group.

[0076] Phase 3: Use the model trained in the target domain to classify the facial expressions of the data sequence, input the classified expression sequence into the long short-term memory network, and diagnose autism on the facial expression sequence.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A method for diagnosing autism based on clustering for cross-domain facial expression recognition, characterized in that: The specific steps are as follows: Step 1: Use the labeled public dataset RAFDB 2.0 to train the source domain model of the large model CLIP to obtain the trained source domain model weights. Step 2: The facial video data of autistic children without facial expression labels is used as the target domain data. The CLIP model is initialized using the trained source domain model weights, including the trainable adapter module and classifier module, and the facial image data is standardized. Step 3: Input the processed facial image into the initialized model to obtain the corresponding expression classification and facial features that have not been classified. Then input it into the dynamic pseudo-label generation module to generate the corresponding pseudo-label. The pseudo-labeled image will participate in the next model training, so as to continuously iterate and optimize the model and increase the proportion of low entropy groups. Step 4: Use the deep neural network model with updated weight parameters as the expression recognition model, input the autistic face image sequence to be recognized into the expression recognition model, and obtain the corresponding expression sequence. Use the long short-term memory network to classify this sequence and obtain the diagnosis result.

2. The autism diagnosis method for cross-domain facial expression recognition based on clustering according to claim 1, characterized in that: In step 1, the sub-steps of training the source domain model by the large model CLIP are: Step 1.1: Use the trained CLIP weights to initialize the deep learning model, perform image preprocessing on the face image, and standardize the face image. Step 1.2: Freeze the weights of the large model CLIP and do not perform parameter training. Add the adapter module and the classifier module. These two modules are learnable and trainable and participate in the training.

3. The autism diagnosis method for cross-domain facial expression recognition based on clustering according to claim 1, characterized in that: In step 2, the sub-steps of initializing the CLIP model with the source domain model weights are: Step 2.1 Pre-train the deep neural network model using the labeled public dataset RAFDB2.0, obtain the model weight parameters of the deep neural network model, and initialize the deep neural network model using the pre-trained weight parameters as initial values; The deep neural network model is a ViT-B / 16 neural network model accompanied by an adapter; The image preprocessing method includes image enhancement methods such as grayscale transformation, random horizontal flipping, and adding Gaussian noise.

4. The autism diagnosis method for cross-domain facial expression recognition based on clustering according to claim 1, characterized in that: In step 3, the sub-steps of the dynamic pseudo-label generation module are: Step 3.1: Input the facial expression classification probability and the unclassified facial features into the dynamic pseudo-label generation module. The module will obtain the average entropy value based on the facial expression classification probability, and divide it into high entropy group and low entropy group based on this standard. (High entropy: the expression classification determination value of the image is not high. Low entropy: the expression classification determination value of the image is high). The image feature with the lowest entropy value of each expression is used as the cluster center point of K-Means. The low entropy group is reclassified into seven clusters, and the label of the cluster center is used as the pseudo label of other images in the cluster; Step 3.2: Re-input the pseudo-labeled images into the model and perform adaptive training on the model. The trainable parameters are still the adapter and the classifier module. Then test the trained model on the target domain data. Repeat step 4.2 and continuously iterate and update to increase the proportion of the low entropy group. We use 30 rounds of training and perform an iterative update on the low entropy group every 5 rounds.

5. The autism diagnosis method for cross-domain facial expression recognition based on clustering according to claim 4, characterized in that: The sub-steps of clustering according to the low entropy group features in step 3.1 are: Step 3.1.1: Calculate the centroid of each class in the target domain. Here, xt represents the target domain sample, gt represents the network feature extractor, ft represents the classifier, and δk represents the model's predicted probability that the sample is the kth class. These centroids can characterize the distribution of different classes in the target domain. The formula is as follows: Step 3.1.2: Obtain pseudo labels by calculating the distance between the feature and the centroid, where represents the generated pseudo-label, Df(gt(xt),ck (0) ) represents the cosine distance between the target domain sample feature and the centroid. The formula is as follows: Step 3.1.3: Update the target centroid according to the generated pseudo label, and update the pseudo label to obtain the final pseudo label. represents the conversion function, only when When the condition holds, the function value is 1, and when the condition does not hold, the function value is 0. It is called self-supervised pseudo label. The formula is as follows:

6. The autism diagnosis method for cross-domain facial expression recognition based on clustering according to claim 4, characterized in that: The sub-steps of adjusting the model parameters according to the loss value in step 3.2 are: Step 3.2.1: The samples in the low entropy group and their corresponding pseudo labels will participate in the training, and the label smoothed cross entropy loss is used as the classification loss function. Among them, ε represents the smoothing factor, represents the probability of the pseudo label being in the kth category, pi represents the model's predicted probability that the sample is in the kth category, and N represents the number of categories. The formula is as follows: Step 3.2.2: Minimize the entropy loss to make the target output approximate a one-hot vector. Where pk represents the probability that the target sample is the kth class, Indicates the size of the entropy of the prediction result: Step 3.2.3: One problem with the loss is that it may cause most samples to be classified into the same category. To avoid this problem, this chapter introduces the diversity loss Regularization is performed. Among them, is the average probability of all target samples in the kth category. Diversity loss can force the model to predict a more uniform distribution of categories, achieving the goal of predicting diverse categories. The formula is as follows: Step 3.2.4: The overall optimization goal of this network is as shown in the following formula:

7. A method for diagnosing autism based on clustering for cross-domain facial expression recognition, characterized in that: It includes preprocessing module, deep neural network initialization module, CLIP fine-tuning module, dynamic pseudo-label generation module, and autism diagnosis module; Preprocessing module: Use open-cv to process each child's video into frames, and then use MTCNN to crop the facial area and preprocess the face image to be recognized, and output the standardized face image after face alignment to the input module; Deep neural network initialization module: In the source domain model training phase, the CLIP pre-trained model (ViT-B / 16) is used to initialize the weights of the deep network. In the target domain adaptation phase, the trained source domain model is used to initialize the weights of the deep network; Dynamic pseudo-label generation module: The facial expression classification probability and unclassified facial features are input into the dynamic pseudo-label generation module. The module will obtain the average entropy value based on the facial expression classification probability, and divide it into high entropy group and low entropy group based on this standard. (High entropy: the expression classification determination value of the image is not high. Low entropy: the expression classification determination value of the image is high). The image feature with the lowest entropy value of each expression is used as the cluster center point of K-Means. The low entropy group is reclassified into seven clusters, and the label of the cluster center is used as the pseudo label of other images in the cluster. Autism diagnosis module: The facial expression sequence output by the model is input into the long short-term memory network to obtain the diagnosis result.

Images