A method to assist communication disorders in children with autism based on facial expression recognition
Through facial expression recognition technology, combined with custom shooting and image enhancement, the problem of emotional interpretation difficulties in social interaction among children with autism is solved, emotional recognition and transmission is achieved, and social communication skills and treatment effects are improved.
Patent Information
- Application Number
- CN202111351567.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-11-16
AI Technical Summary
It is difficult for children with autism to interpret emotional information during social interaction, resulting in communication disorders. The existing facial expression recognition technology has developed slowly and has limited application.
Adopting an auxiliary method based on facial expression recognition, facial images are collected in real time through a custom shooting module, using JAFFE and CK+ facial expression databases to train the SVM classifier, combined with local binary mode algorithm and image enhancement technology, to realize emotion recognition, and through sound Play modules deliver emotional states to children with autism.
It improves the ability of autistic children to identify the emotions of the communication subjects, enhances social communication skills, helps them better integrate into society, and reduces the economic and spiritual burden of families and governments.
Smart Images

Figure CN114067398B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of face recognition and application technology, and in particular relates to a method for assisting communication disorders in children with autism based on facial expression recognition. Background Art
[0002] Autism is a public health issue that requires urgent attention today. It brings huge economic and mental burdens to society and families. Data show that there are 35 million people with autism worldwide, 40% of whom are children.
[0003] The typical clinical manifestation of autism is a severe lack of emotional contact with others, that is, the inability to resonate with others. Autism is a pervasive developmental disorder, one of the core obstacles of which is the persistent impairment of social communication and interaction ability, which is mainly manifested in difficulty in interpreting emotional information and thus inability to communicate with social people, hence the name autism.
[0004] Facial expression is the result of one or more movements or states of facial muscles. The movement of these facial muscles reflects the individual's emotional state to the observer. Facial expression is a form of non-verbal communication. It is the main means of expressing social information between humans, and it also occurs in most other mammals and other species. There are at least 21 human facial expressions. In addition to the 6 common ones of happiness, surprise, sadness, anger, disgust and fear, there are 15 distinguishable complex expressions such as surprise (happiness + surprise) and grief (sadness + anger).
[0005] Facial expression recognition technology has only been gradually developed in recent decades. Due to the diversity and complexity of facial expressions, and the involvement of physiology and psychology, expression recognition is very difficult. Therefore, compared with other biometric technologies such as fingerprint recognition, iris recognition, and face recognition, it has developed relatively slowly and is not widely used. However, expression recognition has important value for human-computer interaction, so many research institutions and scholars at home and abroad are committed to research in this area and have achieved certain results.
[0006] Emotion is a state that integrates a person's feelings, thoughts and behaviors. It plays an important role in interpersonal communication, including a person's psychological response to external or self-stimuli and the corresponding physiological response. The most common emotion recognition method is facial expression recognition based on non-physiological signals. Facial expression recognition is to identify different emotions based on the correspondence between a person's facial expression and emotions. Assisting autistic children to recognize the facial expressions of the communication partner is an urgent need to solve the problem of autistic children's social interaction disorders. Summary of the invention
[0007] The technical problem to be solved by the present invention is to provide a method for assisting communication disorders in children with autism based on facial expression recognition in view of the deficiencies of the above-mentioned prior art.
[0008] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0009] A method for assisting communication disorders in children with autism based on facial expression recognition, comprising:
[0010] Step 1: The custom camera module monitors the daily behavior of autistic children in real time and collects facial images of the communication object for later analysis;
[0011] Step 2: The real-time processing module controls the communication partner to switch on or off emotion recognition according to the needs of the autistic child;
[0012] Step 3: The facial expression recognition module performs face recognition and emotion recognition on the images captured by the custom shooting module;
[0013] Step 4: The sound playback module transmits the emotional state identified by the facial expression recognition module to the autistic child.
[0014] To optimize the above technical solutions, the specific measures taken also include:
[0015] The above-mentioned custom shooting module monitors the daily behavior and activities of autistic children in real time to determine whether there is any danger when there is no communication object and no shooting is required;
[0016] When a communication partner appears, the autistic child collects the facial image of the communication partner according to his or her needs based on the real-time processing module for later analysis.
[0017] The above-mentioned custom shooting module utilizes a portable micro camera (glass-style pinhole camera) based on a real-time processing module to enable autistic children to autonomously shoot the person they are talking to and collect images.
[0018] The facial expression recognition module first uses feature extraction to train the SVM classifier based on the JAFFE facial expression database and the CK+ facial expression database to obtain a facial expression recognition model;
[0019] Secondly, the images captured by the custom shooting module are used for face recognition, and the recognized face images are saved and pre-processed;
[0020] Finally, the facial expression recognition model is used to perform emotion recognition on the preprocessed face images.
[0021] The above-mentioned feature extraction is used to train the SVM classifier based on the JAFFE facial expression database and the CK+ facial expression database to obtain a facial expression recognition model, including:
[0022] Step (1): uniformly preprocess the images in the JAFFE facial expression database and the CK+ facial expression database;
[0023] Step (2): Extract features from the preprocessed image using a local binary pattern algorithm, including:
[0024] Step (3): Classify the expressions in the JAFFE facial expression database and the CK+ facial expression database into seven types, namely anger, fear, disgust, surprise, happiness, sadness, and neutral;
[0025] Step (4): The seven expression pictures are randomly grouped, and after feature extraction, the SVM classifier is trained and adjusted using the ten-fold cross-validation method to obtain a facial expression recognition model that meets the accuracy requirements.
[0026] The above step (2) uses a local binary pattern (LBP) algorithm to extract features from the preprocessed image, including:
[0027] Step (2.1): Divide the preprocessed face image into blocks;
[0028] Step (2.2): extract LBP features from the divided small image areas;
[0029] Step (2.3): Use LBP histogram vector as a description of facial features;
[0030] Step (2.4): Convert the extracted feature values into feature values monitored by the facial expression recognition model.
[0031] The above preprocessing includes grayscale processing, face alignment and image enhancement.
[0032] The above-mentioned sound playback module sends the emotional state recognized by the facial expression recognition module to the Bluetooth device, and the Bluetooth device uses the set sound mode for voice broadcasting to transmit the emotional state of the communication object to the autistic child.
[0033] The present invention has the following beneficial effects:
[0034] The present invention can help autistic children recognize the emotions of the communication partner, use the camera and the camera button to selectively capture, and transmit the emotional state of the communication partner analyzed in the facial expression recognition module to the autistic children through the sound playback module, so as to assist in analyzing the emotions of the communication partner and help them to better carry out social activities.
[0035] In actual scenarios, considering the user experience, faces are generally photographed under unrestricted conditions. Faces photographed in such environments have various problems such as large angles, dark light, strong light, backlight, occlusion, blur, noise, etc. These low-quality images will greatly affect the recognition accuracy of the face algorithm. In order to improve the accurate recognition rate of the algorithm, in addition to improving the robustness of the algorithm, the present invention starts from the source camera to improve the image acquisition quality, and at the same time pre-processes the collected face images, and uses image enhancement to improve the image quality.
[0036] The present invention makes up for the deficiencies of autistic children in social communication and interaction ability to a certain extent. During the conversation, autistic children can communicate and interact normally, which helps autistic children increase information, thereby improving the treatment effect of autism, and can help autistic children better integrate into society and reduce the burden on the government and families. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 The schematic diagram of the method of the present invention. DETAILED DESCRIPTION
[0038] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.
[0039] See also Figure 1 The present invention provides a method for assisting communication disorders in children with autism based on facial expression recognition, comprising:
[0040] Step 1: The custom camera module monitors the daily behavior of autistic children in real time and collects facial images of the communication object for later analysis;
[0041] In the embodiment, the custom shooting module can monitor the daily behavior and activities of autistic children in real time to determine whether there is any danger when there is no communication object and shooting is not required;
[0042] When a communication partner appears, the autistic child collects the facial image of the communication partner according to his or her needs based on the real-time processing module for later analysis.
[0043] In the custom shooting module, first: choose a portable camera, so that autistic children can communicate in any place without restrictions, which is conducive to normal communication;
[0044] Second, ordinary cameras are too large and will attract the attention of the communicator and the autistic child, so a pinhole camera is used. Since the pinhole camera is carried by the autistic child, the video will be motion blurred. To address this shortcoming, an image restoration algorithm is used to reduce this effect.
[0045] The custom shooting module uses a portable micro camera (glasses-style pinhole camera) based on a real-time processing module to enable autistic children to autonomously shoot the person they are talking to and collect images (this operation is performed with the consent of the person they are talking to). The camera controls its shooting time through software.
[0046] The use of a miniature camera can reduce the interference of the camera to autistic children and the communication partners, thus ensuring that the communication proceeds more naturally.
[0047] At the same time, using selective shooting instead of video capture can reduce the time of software implementation to a certain extent and achieve a near real-time effect.
[0048] Step 2: The real-time processing module controls the communication partner to switch on and off emotion recognition according to the needs of autistic children, and realizes the combination of software and hardware through hardware devices;
[0049] In the embodiment, the real-time processing module provides a hardware support for the custom shooting module, and the two need to be used in combination.
[0050] Step 3: The facial expression recognition module performs face recognition and emotion recognition on the images captured by the custom shooting module;
[0051] In the embodiment, feature extraction is first used to train the SVM classifier based on the JAFFE facial expression database and the CK+ facial expression database to obtain a facial expression recognition model;
[0052] Before performing facial expression recognition, you first need to train the model. Using the ten-fold cross-validation method, divide the images in the two data sets into ten groups with roughly the same number of images. These ten groups are used as test sets, and the remaining nine groups are used as training sets.
[0053] Perform LBP feature extraction on ten sets of pictures respectively:
[0054] Step 1: Divide the preprocessed face image into blocks;
[0055] Step 2: Perform LBP feature extraction transformation on each small image area after segmentation. The feature points include: the corners of the eyes, nose, mouth, and chin.
[0056] Step 3: Use LBP histogram vector as a description of facial features.
[0057] Step 4: Convert the extracted feature values into a format that can be monitored by the SVM trained model.
[0058] Generally, the more blocks there are, the better the facial expression effect will be. However, as the number of blocks increases, the dimension of the feature vector will increase directly, which will greatly increase the computational complexity.
[0059] The feature values obtained by feature extraction are converted into a data set that can be trained by the SVM classifier. Ten-fold cross validation is used to train ten sets of trainable data to obtain the optimal solution.
[0060] SVM (Support Vector Machine) is a supervised learning model and related learning algorithm for analyzing data in classification and regression analysis. Given a set of training examples, each training example is labeled as belonging to one or the other of two categories, the SVM training algorithm creates a model that assigns new examples to one of the two categories, making it a non-probabilistic binary linear classifier. When using SVM for training, the larger the number of training samples, the better the recognition effect.
[0061] Using a hybrid face database, namely the JAFFE face database and the CK+ face database, human emotions are divided into seven states, namely anger, fear, disgust, surprise, happiness, sadness, and neutral.
[0062] When an autistic child touches the camera switch, a face picture will be obtained. After the original picture is recognized, a face picture is obtained. After pre-processing, feature extraction is performed on the face picture and it is converted into a data set that can be used for the SVM classifier. The data is put into the model for recognition, and the facial expression of the communication object is obtained, so as to judge his or her emotional state.
[0063] That is, the feature extraction is used to train the SVM classifier based on the JAFFE facial expression database and the CK+ facial expression database to obtain a facial expression recognition model, including:
[0064] Step (1): uniformly preprocess the images in the JAFFE facial expression database and the CK+ facial expression database;
[0065] The pre-processing comprises:
[0066] Grayscale processing: Face recognition is difficult to achieve high accuracy under strong or weak light conditions. Therefore, grayscale processing is performed first to reduce the interference of light on recognition during the recognition process.
[0067] When processing color images, it is often necessary to process the three channels (R, G, B) in sequence, which takes a lot of time. Therefore, in order to increase the processing speed of the entire system, it is necessary to grayscale the color image to reduce the amount of data that needs to be processed.
[0068] In the RGB model, if R=G=B, the color represents a grayscale color, where the value of R=G=B is called the grayscale value. Therefore, each pixel of the grayscale image only needs one byte to store the grayscale value, and the grayscale value can range from 0 to 255. There are generally four methods to grayscale color images: component method, maximum value method, average method, and weighted average method. Among them, the weighted average method is to weighted average the three components with different weights according to their importance and other indicators. Since the human eye is most sensitive to green and least sensitive to blue, a more reasonable grayscale image can be obtained by weighted averaging the three RGB components according to the following formula. The formula is as follows:
[0069] Conversion algorithm L = R*299 / 1000+G*587 / 1000+B*114 / 1000;
[0070] Face alignment: Face alignment needs to be achieved through face key points. Based on these key points, the face is calibrated. The so-called key points are some points with characteristic meanings, such as: the corners of the eyes, the nose, the outline of the face, and the chin. With these feature points, the face can be aligned using affine transformation, which can largely eliminate the errors caused by different postures. Therefore, due to the shooting reasons, the angle and position of the face are disordered. In order to better perform face recognition, face alignment operations are required. This part is mainly implemented using code.
[0071] Image enhancement: For low-quality images, dark light enhancement, super-resolution, denoising, motion blur removal and other methods can be used for image enhancement. The purpose of image enhancement is to improve the visual effect of the image. For different applications of different images, the overall or local characteristics of the image are purposefully emphasized, the original unclear image becomes clear or certain interesting features are emphasized, the difference between the features of different objects in the image is enlarged, and the features of no interest are suppressed, so as to improve the image quality, enrich the amount of information, strengthen the image interpretation and recognition effect, and meet the needs of certain special analysis.
[0072] Image enhancement can be divided into two categories: frequency domain methods and spatial domain methods.
[0073] Frequency domain method: The frequency of an image refers to the speed of change. The various parameters of different pixels in an image are different, so the image actually changes in space. Images are divided into high-frequency images and low-frequency images. Low-frequency images are images whose colors change slowly, that is, images whose grayscale changes slowly; high-frequency images correspond to images that change very quickly, and the grayscale between adjacent areas varies greatly. The frequency domain method is an indirect image enhancement algorithm that regards the image as a two-dimensional signal and performs signal enhancement based on two-dimensional Fourier transform. The use of a low-pass filter can only allow low-frequency signals to pass through, thereby achieving the effect of removing noise in the image; the use of a high-pass filter can enhance high-frequency signals such as edges, making blurred images clear.
[0074] Spatial domain method: It is a direct image enhancement algorithm, which is divided into point operation algorithm and neighborhood denoising algorithm.
[0075] Point operation algorithms, namely grayscale transformation (gamma transformation, logarithmic enhancement) and histogram correction, aim to make the image uniform, or to expand the dynamic range of the image and extend the contrast.
[0076] Neighborhood enhancement algorithms are divided into two types: image smoothing and sharpening. Smoothing is generally used to eliminate image noise, but it is also easy to cause edge blur. Common algorithms include mean filtering and median filtering. Sharpening is generally used to highlight the edge of an object to facilitate target recognition. Common algorithms include gradient method, operator method, mask matching method, statistical difference method, high-pass filtering method, etc.
[0077] Step (2): Extract features from the preprocessed image using a local binary pattern (LBP) algorithm, including:
[0078] Step (2.1): Divide the preprocessed face image into blocks;
[0079] Step (2.2): Perform LBP feature extraction on the divided small image areas; there are many feature extraction methods, and LBP feature extraction is used here.
[0080] Step (2.3): Use LBP histogram vector as a description of facial features;
[0081] Step (2.4): Convert the extracted feature values into feature values monitored by the facial expression recognition model.
[0082] Step (3): Classify the expressions in the JAFFE facial expression database and the CK+ facial expression database into seven types, namely anger, fear, disgust, surprise, happiness, sadness, and neutral;
[0083] Step (4): The seven expression pictures are randomly grouped, and after feature extraction, the SVM classifier is trained and adjusted using the ten-fold cross-validation method to obtain a facial expression recognition model that meets the accuracy requirements.
[0084] Ten-fold cross validation:
[0085] Divide all the data into ten parts, and then use each part as a validation set and the others as training sets for training and validation. In this process, keep the hyperparameters consistent, and then take the average training loss and average validation loss of the 10 models to measure the quality of the hyperparameters. Finally, after obtaining a satisfactory hyperparameter, use all the data as a training set and use the hyperparameters to train a model.
[0086] The purpose of using ten-fold cross validation is to reduce the contingency caused by a single partition of the training set and the validation set, and to make full use of the existing data set to perform multiple partitions to avoid the selection of accidental hyperparameters and models that do not have generalization capabilities due to special partitions. Cross validation can reduce contingency and improve generalization capabilities.
[0087] After performing the above steps, a facial expression recognition model can be obtained.
[0088] After obtaining an accurate training model, the collected images can be used for facial expression recognition and pre-processing (including grayscale processing, face alignment and image enhancement) according to the needs of autistic children, and then placed in the model for recognition to obtain the emotional state of the current communication partner.
[0089] Secondly, the images captured by the custom shooting module are used for face recognition, and the recognized face images are saved and pre-processed;
[0090] Finally, the facial expression recognition model is used to perform emotion recognition on the preprocessed face images.
[0091] People have various facial expressions. By analyzing and comparing them in the included JAFFE database, we can obtain the facial expressions of the current communication partner and achieve accurate recognition.
[0092] Step 4: The sound playback module transmits the emotional state identified by the facial expression recognition module to the autistic child.
[0093] In an embodiment, the sound playback module sends the emotional state recognized by the facial expression recognition module to the Bluetooth device, and the Bluetooth device uses a set sound mode for voice broadcasting to transmit the emotional state of the communication partner to the autistic child.
[0094] Bluetooth devices (mainly Bluetooth headsets) are used to output the identified emotional state to autistic children without hindering the normal conversation between the autistic children and the communication partners.
[0095] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.
Claims
1. A method for assisting communication disorders in children with autism based on facial expression recognition, characterized in that: include: Step 1: The custom camera module monitors the daily behavior of autistic children in real time and collects facial images of the communication object for later analysis; Step 2: The real-time processing module controls the communication partner to switch on or off emotion recognition according to the needs of the autistic child; Step 3: The facial expression recognition module performs face recognition and emotion recognition on the images captured by the custom shooting module; Step 4: The sound playback module transmits the emotional state identified by the facial expression recognition module to the autistic child; The custom shooting module monitors the daily behavior of autistic children in real time to determine whether there is any danger when there is no communication object and no shooting is required; When a communication partner appears, the autistic child collects the communication partner's facial image for later analysis based on his or her needs based on the real-time processing module; The custom shooting module uses a portable micro camera and a real-time processing module to enable the autistic child to autonomously shoot the person being talked to and collect images.
2. The method for assisting communication disorders in children with autism based on facial expression recognition according to claim 1, characterized in that: The facial expression recognition module firstly uses feature extraction to train the SVM classifier based on the JAFFE facial expression database and the CK+ facial expression database to obtain a facial expression recognition model; Secondly, the images captured by the custom shooting module are used for face recognition, and the recognized face images are saved and pre-processed; Finally, the facial expression recognition model is used to perform emotion recognition on the preprocessed face images.
3. The method for assisting communication disorders in children with autism based on facial expression recognition according to claim 2, characterized in that: The feature extraction is used to train the SVM classifier based on the JAFFE facial expression database and the CK+ facial expression database to obtain a facial expression recognition model, including: Step (1): Perform unified preprocessing on the images in the JAFFE facial expression database and the CK+ facial expression database; Step (2): Extract features from the preprocessed image using a local binary pattern algorithm, including: Step (3): Classify the expressions in the JAFFE facial expression database and the CK+ facial expression database into seven types, namely anger, fear, disgust, surprise, happiness, sadness, and neutral; Step (4): Randomly group the seven expression pictures, and after feature extraction, use the ten-fold cross-validation method to train and adjust the SVM classifier to obtain a facial expression recognition model that meets the accuracy requirements.
4. The method for assisting communication disorders in children with autism based on facial expression recognition according to claim 3, characterized in that: The step (2) of extracting features from the preprocessed image using a local binary pattern (LBP) algorithm comprises: Step (2.1): Divide the preprocessed face image into blocks; Step (2.2): Perform LBP feature extraction on the divided small image areas; Step (2.3): Use LBP histogram vector as a description of facial features; Step (2.4): Convert the extracted feature values into feature values monitored by the facial expression recognition model.
5. The method for assisting communication disorders in children with autism based on facial expression recognition according to claim 3, characterized in that: The preprocessing includes grayscale processing, face alignment and image enhancement.
6. The method for assisting communication disorders in children with autism based on facial expression recognition according to claim 1, characterized in that: The sound playing module sends the emotional state recognized by the facial expression recognition module to the Bluetooth device, and the Bluetooth device uses a set sound mode to perform voice broadcasting to transmit the emotional state of the communication object to the autistic child.
Citation Information
Patent Citations
Robot for adjuvant therapy of infantile autism
CN107283389A
Auxiliary communication device for children with autism
CN107307865A
Emotion and capability intervention system for children having autism spectrum disorder
CN107785061A
Facial emotion identification method, apparatus and storage medium
WO2019033573A1
