Personalized emotion perception method for autistic children based on tactile interaction

Abstract

The invention provides an autism child personalized emotion perception method based on tactile interaction, and the method comprises an identification stage: employing a trained autism child personalized emotion identification model to identify tactile behavior data in a man-machine interaction process of an autism child and a human-shaped accompanying robot, and obtaining an emotion classification result; in the model training stage, the personalized emotion recognition model of the autism children comprises a public feature extractor, a domain independent feature extraction and classifier, a domain general feature extraction and classifier and an aggregation prediction classifier. The training process of the personalized emotion recognition model for children with autism comprises the following steps: acquiring tactile behavior data; dividing the tactile behavior data into source domain data and target domain data; obtaining a source domain feature and a target domain feature through a common feature extractor; obtaining a domain independent classification result through a domain independent feature extraction and classifier, and obtaining a domain general classification result through a domain general feature extraction and classifier; and obtaining a classification result of the emotion category through an aggregation prediction classifier.

Description

Technical Field

[0001] This application relates to the fields of human-computer interaction, emotion recognition, and transfer learning, etc., and particularly relates to a personalized emotion perception method for autistic children based on tactile interaction. Background Art

[0002] Autism is a neurodevelopmental disorder that seriously affects the physical and mental development of children. Autistic children not only have obvious deficiencies in language development and behavioral performance, but also face huge challenges in emotional cognition and expression. They often have difficulty accurately understanding and expressing their own emotions, and cannot respond appropriately to the emotions of others like normal children, which seriously hinders their effective communication and social interaction with the outside world, and greatly affects their quality of life and social integration ability. For example, in daily life, autistic children may make inappropriate behaviors due to their inability to understand the angry emotions of others, or due to their own emotional expression disorders, their needs cannot be understood by others in time, and then they may generate negative emotions such as anxiety and irritability, forming a vicious cycle.

[0003] Currently, most emotion computing methods and emotion recognition models are developed based on the behavioral and physiological data of normal children or adults. Normal children have relatively consistent patterns in emotion expression, such as conveying emotions through facial expressions, language expressions, and typical body movements. However, the emotion expression methods of autistic children are significantly different from those of normal children. They may have characteristics such as dull facial expressions, limited language expressions, and stereotyped body movements, making it difficult for traditional emotion recognition technologies developed based on the normal population to be directly applied to autistic children. For example, normal children will naturally smile, laugh and have lively body movements when they are happy, while autistic children may only show slight facial muscle changes or specific repetitive actions, and these subtle and unique manifestations cannot be accurately captured by traditional emotion recognition models. Moreover, there is a high degree of symptom heterogeneity among autistic children, and there are significant differences among different children in terms of symptom types, degrees, and emotion expression patterns. Even in the same emotional state, the behavioral manifestations of different autistic children may vary greatly. Some children may relieve their anxiety by touching specific items, while others may show self-harm behaviors. This huge difference between individuals makes the existing emotion recognition models have serious deficiencies in generalization ability when facing autistic children. A single model is difficult to adapt to the diverse emotion expressions of different autistic children, resulting in low accuracy in cross-subject emotion recognition tasks, and unable to meet the requirements of clinical diagnosis and intervention treatment for emotion recognition accuracy and personalization.

[0004] Human-computer interaction technology provides a new approach for the emotion recognition and intervention of children with autism. Robots have characteristics such as repeatability, consistency, and patience, and can provide a stable and predictable interaction environment for children with autism. For example, humanoid robots can interact with children with autism through simplified facial expressions and body languages, attract their attention, increase a sense of intimacy, and thus improve their willingness to participate in the interaction. At the same time, robots can accurately record various behavioral data of children during the interaction process, such as touch actions, gaze tracking, behavioral response time, etc., which makes it possible to objectively and quantitatively evaluate the emotions and behaviors of children. Although human-computer interaction technology shows certain potential in the research of children with autism, it is still in the exploratory stage in terms of emotion recognition. Most of the existing research focuses on the analysis of the facial expression data of children with autism, lacking in-depth exploration of the complex relationship between the tactile behaviors and emotions of children with autism. Tactile behaviors not only include touch positions and actions, but also involve multi-dimensional information such as touch force, frequency, and duration, and the relationship between this information and emotions has not been fully studied. In addition, there are significant differences among different individuals in the process of emotion expression, and this difference may be more obvious among children with autism. There is also little research on the systems and methods for personalized emotion recognition of children with autism.

[0005] In summary, considering the particularity of the emotion expression of children with autism and the limitations of the existing technology in solving this problem, there is an urgent need to develop an innovative technology specifically for children with autism that can accurately perceive their personalized emotions. Summary of the Invention

[0006] In view of this, it is necessary to provide a method for personalized emotion perception of children with autism based on tactile interaction, which can at least overcome one of the above defects.

[0007] In the first aspect, the embodiment of the present application provides a method for personalized emotion perception of children with autism based on tactile interaction, including: Recognition stage: Obtain the tactile behavior data during the human-computer interaction between a child with autism and a humanoid companion robot, preprocess the tactile behavior data, and input the preprocessed tactile behavior data into a trained personalized emotion recognition model for children with autism to obtain an emotion classification result; Model training stage: The personalized emotion recognition model for children with autism includes a common feature extractor, a domain-independent feature extraction and classifier, a domain-general feature extraction and classifier, and an aggregated prediction classifier; The training process of the personalized emotion recognition model for children with autism includes: S1. Obtain the tactile behavior data during the human - robot interaction between autistic children and humanoid companion robots, and pre - process the tactile behavior data; the tactile behavior data is the interaction feedback data of autistic children touching the humanoid companion robot during the interaction, and each key body part feeds back one modality of tactile behavior data; S2. Divide the pre - processed tactile behavior data into multiple source - domain data and one target - domain data; S3. Input both the source - domain data and the target - domain data into a common feature extractor for feature extraction to obtain source - domain features and target - domain features; S4. Input both the source - domain features and the target - domain features into a domain - independent feature extractor and classifier to obtain a domain - independent classification result; meanwhile, also input both the source - domain features and the target - domain features into a domain - general feature extractor and classifier to obtain a domain - general classification result; S5. Input the domain - independent classification result and the domain - general classification result into an aggregated prediction classifier, and adopt a confidence - weighted aggregation prediction method to obtain the classification result of the emotion category and the confidence of the corresponding emotion category.

[0008] Based on the above, the common feature extractor includes a data window - slicing sub - module, a feature extraction sub - module, an attention fusion sub - module, a global feature fusion sub - module, and a feature representation output sub - module; The data window - slicing sub - module is used to slice the pre - processed tactile sensor data into data segments of multiple time steps using a sliding window method; The feature extraction sub - module uses a two - layer 3D CNN network structure to extract features from the data segments of each time step to obtain spatio - temporal fusion features; The attention fusion sub - module inputs the spatio - temporal fusion features extracted by the feature extraction sub - module to learn the complementary information of the spatio - temporal fusion features of different modalities; Take the spatio - temporal fusion feature of each modality at each time step as Q, the spatio - temporal fusion features of other modalities as K and V, and fuse them through a multi - head attention mechanism to obtain multiple groups of fusion features at each time step; Integrate the multiple groups of fusion features at each time step through addition or concatenation operations into a feature vector to obtain the local fusion feature at each time step; Fuse the spatio - temporal fusion features of all time steps through a multi - head attention mechanism respectively to obtain the spatio - temporal fusion features of each time step; The global feature fusion sub - module uses a Transformer encoder to fuse the local fusion features of each time step to obtain the global fusion features of each time step; The described feature representation output sub-module aggregates the global fusion features of all time steps using average pooling to obtain the global joint representation of the tactile behavior data.

[0009] Based on the above, the domain-independent feature extraction and classifier includes a domain-independent feature extractor and a domain-independent classifier established for the source domain and the target domain; The domain-independent feature extractor is used to map the source domain features and the target domain features to a specific feature space, and uses the maximum mean discrepancy (MMD) to reduce the difference in feature distributions between different domains, obtaining domain-personalized features; The domain-independent classifier is used to classify the domain-personalized features to obtain the domain-independent classification results of the source domain and the target domain.

[0010] Based on the above, the domain-general feature extraction and classifier includes a domain-general feature extractor and a domain-general classifier; The domain-general feature extractor is used to perform spatial mapping and consistency encoding on the source domain features and the target domain features to obtain domain-general features; The domain-general classifier is used to classify the domain-general features to obtain the domain-general classification results of the source domain and the target domain.

[0011] Based on the above, the method for obtaining different-modal tactile behavior data in the human-robot interaction process between autistic children and humanoid companion robots includes: Build a human-robot interaction environment and use a humanoid companion robot to interact with autistic children; install tactile sensors at key body parts of the humanoid companion robot to collect tactile behavior data of autistic children during the interaction; Design interaction tasks and emotion induction schemes to stimulate different emotional states of autistic children; continuously collect tactile sensor data during the human-robot interaction process; According to the collected multi-modal tactile behavior data, annotate the emotional state of autistic children in each interaction segment to determine the emotional category to which the tactile behavior data belongs.

[0012] Based on the above, the method for preprocessing tactile behavior data includes: Clean the collected tactile sensor data to remove outliers caused by sensor failures or interference; Use a filtering algorithm for denoising to reduce the noise components in the data; Normalize the data and map it to a specific interval.

[0013] In a second aspect, an embodiment of the present application provides a personalized emotion perception system for autistic children based on tactile interaction, including: A humanoid companion robot is used for human - machine interaction with autistic children during the recognition stage and the model training stage, and feeds back the tactile behavior data during the human - machine interaction process, and also pre - processes the tactile behavior data; the tactile behavior data is the interaction feedback data of autistic children touching the humanoid companion robot during the interaction process, and in the model training stage, each key body part feeds back one modality of tactile behavior data. A construction module is used for constructing an individualized emotion recognition network for autistic children; the individualized emotion recognition network for autistic children includes: a training data manager, a common feature extractor, a domain - independent feature extraction and classifier, a domain - general feature extraction and classifier, and an aggregation prediction classifier. The training data manager is used for receiving and storing the tactile behavior data, and dividing the pre - processed tactile behavior data into source domain data and target domain data. The common feature extractor is connected to the training data manager and is used for extracting features from the input source domain data and target domain data to obtain source domain features and target domain features. The domain - independent feature extraction and classifier is connected to the common feature extractor and is used for independent feature extraction and classification of the source domain features and target domain features to obtain domain - independent classification results. The domain - general feature extraction and classifier is connected to the common feature extractor and is used for general feature extraction and classification of the source domain features and target domain features to obtain domain - general classification results. The aggregation prediction classifier is respectively connected to the domain - independent feature extraction and classifier and the domain - general feature extraction and classifier, and is used for adopting a confidence - weighted aggregation prediction method for the domain - independent classification results and the domain - general classification results to obtain the classification results of emotion categories and the confidence levels corresponding to the emotion categories. An acquisition module is connected to the humanoid companion robot and is used for acquiring the pre - processed tactile behavior data during the model training stage, and dividing the tactile behavior data into multiple source domain data and one target domain data; it is also used for acquiring the pre - processed tactile behavior data during the recognition stage. A training module is used for training the individualized emotion recognition network for autistic children by using all the source domain data and the target domain data to obtain a trained individualized emotion recognition network for autistic children. A recognition module is used for, during the recognition stage, performing emotion recognition on the acquired pre - processed tactile behavior data through the trained individualized emotion recognition network for autistic children to obtain emotion classification results.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including: A memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned personalized emotion perception method for autistic children based on tactile interaction.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned personalized emotion perception method for autistic children based on tactile interaction.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, they implement the above-mentioned personalized emotion perception method for autistic children based on tactile interaction.

[0017] The present invention has prominent substantial features and significant progress compared with the prior art. Specifically: (1) By processing the tactile interaction data between autistic children and humanoid companion robots, the present invention perceives and recognizes the emotional states of autistic children, provides a basis for the regulation of real-time intervention strategies for autistic children, and helps to improve the intervention effect.

[0018] (2) The common feature extractor proposed by the present invention adopts a tactile emotion feature encoding and decoding method based on multi-sensor fusion, realizes the efficient fusion of local features through an attention model, highlights key information, and at the same time uses a Transformer for global feature modeling to capture long-distance dependencies and context information, significantly improving the expression ability of emotion features.

[0019] (3) The cross-subject emotion recognition model based on multi-source domain adaptation of the present invention adopts a multi-source domain adaptation method based on shared and specific feature projections. Not only does the domain-independent feature extractor solve the emotional differences between different individuals by aligning feature distributions and provide domain-personalized features, but also the domain-general feature extractor extracts domain-general features through spatial mapping and consistency encoding of all source domain features, constructs classifiers for these two types of features respectively, and uses a confidence-weighted aggregation method to significantly improve the generalization ability and robustness of the model. This method does not require a large amount of target domain annotation data, has high efficiency and practicality, and can be widely applied to scenarios such as children's emotion intervention and mental health assessment, providing technical support for cross-subject emotion recognition. Description of the Drawings

[0020] Figure 1 It is a tactile emotion feature encoding model based on multi-sensor fusion.

[0021] Figure 2 It is the method framework of the cross-attention fusion module.

[0022] Figure 3 It is a method framework for a cross-subject emotion recognition model based on multi-source domain adaptation. Specific implementation manner

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0024] The terms "including" and "having" in the specification and claims of the present application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but may include steps or modules not listed.

[0025] Embodiment 1 This embodiment provides a personalized emotion perception method for autistic children based on tactile interaction, including: Recognition stage: Obtain the tactile behavior data during the human-machine interaction between autistic children and a humanoid companion robot, preprocess the tactile behavior data, and input the preprocessed tactile behavior data into a trained personalized emotion recognition model for autistic children to obtain an emotion classification result.

[0026] Model training stage: The personalized emotion recognition model for autistic children includes a common feature extractor, a domain-independent feature extractor and classifier, a domain-general feature extractor and classifier, and an aggregated prediction classifier; The training process of the personalized emotion recognition model for autistic children includes: S1. Obtain the tactile behavior data during the human-machine interaction between autistic children and a humanoid companion robot, and preprocess the tactile behavior data; the tactile behavior data is the interaction feedback data of autistic children touching the humanoid companion robot during the interaction, and each key body part feeds back a modality of tactile behavior data; S2. Divide the preprocessed tactile behavior data into multiple source domain data and one target domain data; S3. Input both the source domain data and the target domain data into the common feature extractor for feature extraction to obtain source domain features and target domain features; S4. Input both the source domain features and the target domain features into the domain-independent feature extractor and classifier to obtain a domain-independent classification result; at the same time, Both the source domain features and the target domain features are also input into the domain-general feature extractor and classifier to obtain the domain-general classification result; S5. Input the domain-independent classification result and the domain-general classification result into the aggregated prediction classifier, and adopt the confidence-weighted aggregation prediction method to obtain the classification result of the emotion category and the confidence of the corresponding emotion category.

[0027] Specifically, the training process of the personalized emotion recognition model for children with autism includes: a) Data collection and emotion annotation Interactive scenario setting: Build a human-computer interaction environment, and use a humanoid companion robot to interact with children with autism. Tactile sensors are installed at the key parts of the robot's body. In this embodiment, tactile sensors are installed on the head, arms, and back respectively to collect the tactile behavior data of children during the interaction.

[0028] Emotion induction and data collection: Design a series of diverse interaction tasks and emotion induction schemes to stimulate different emotional states of children with autism. For example, by playing different types of music (lively, soothing, exciting, etc.), showing interesting animations or stories, and playing simple games, etc., to induce children to produce emotions such as joy, calmness, boredom, anger, etc. During the interaction, continuously collect the data of the tactile sensors.

[0029] Emotion category annotation: Invite experienced autism rehabilitation experts, affective computing experts, and the parents of children to participate in the emotion annotation work together. They annotate the emotional state of children in each interaction segment according to the collected multimodal data, and determine the emotion category to which it belongs (such as joy, boredom, anger, calmness, etc.). The annotation process follows strict standards and procedures to ensure the accuracy and consistency of the annotation.

[0030] b) Data preprocessing After preprocessing the collected tactile sensor data, a tactile interaction emotion dataset for children with autism is constructed. The preprocessing includes operations such as data cleaning, denoising, and normalization. Data cleaning mainly removes the outliers generated due to sensor failures or interference during the collection process; denoising uses a filtering algorithm to reduce the noise components in the data; normalization maps the data to a specific interval (such as) for subsequent processing.

[0031] c) Tactile emotion feature encoding model based on multi-sensor fusion As Figure 1 shown, the tactile emotion feature encoding model based on multi-sensor fusion is mainly implemented by a common feature extractor, and the common feature extractor includes a data windowing and slicing sub-module, a feature extraction sub-module, an attention fusion sub-module, a global feature fusion sub-module, and a feature representation output sub-module.

[0032] Multi-sensor data windowing: Implemented by the data windowing slicing sub-module.

[0033] To extract the local features of each tactile sensor data, the sliding window method is used to process the head tactile sensor data X H , arm tactile sensor data X A and back tactile sensor data X B ( , T representing the time series length, w × h indicating the spatial array of the sensors) are sliced. Set the time window width and the scanning step size to be both s , and the data of each sensor after windowing can be represented as X 1, X 2, …, X j ,…], where is the data segment at the th time step.

[0034] Local fusion feature extraction: Implemented by the feature extraction sub-module.

[0035] Adopt a two-layer 3D CNN network structure to extract features from the tactile sensor data at each time step X Hj , X Aj and X Bj to obtain spatio-temporal fusion features H j , A j and B j .

[0036] Attention fusion: Implemented by the attention fusion sub-module.

[0037] As Figure 2 shown, input the spatio-temporal fusion features of three tactile sensors H j , A j and B j , so as to learn the complementary information of different modalities; take the spatio-temporal fusion feature of one tactile sensor at each time step as Q, the spatio-temporal fusion feature of another tactile sensor as K, V, and fuse them through the multi-head attention mechanism to obtain six groups of fusion features HA j ,AH j , HB j , BH j , AB j and BA j ; The six groups of fused features HA j , AH j , HB j , BH j , AB j and BA j are added or concatenated and integrated into a feature vector to finally obtain the local fused feature Z j ; Repeat the above steps for the spatio-temporal fused features of the tactile sensors at all time steps, and finally obtain the spatio-temporal fused features at each time step Z 1, Z 2, …, Z j ,…].

[0038] Global feature fusion: It is implemented by the global feature fusion sub-module.

[0039] Use the Transformer encoder to perform global feature fusion on the spatio-temporal fused features Z 1, Z 2, …, Z j ,…].[[]]

[0040] First, perform positional encoding on the feature Z j to incorporate the positional information into the feature, obtaining ; Use the multi-head attention mechanism to focus on the features at different positions and calculate the attention weight matrix: where, , W Q , W K and W V are trainable weight matrices.

[0041] Then, the output of the multi-head self-attention is added to the original input (residual connection) and then layer normalization is performed: Furthermore, the features at each time step independently pass through a feed-forward neural network: where W 1 , W 2 , b 1 , b 2 are trainable parameters.

[0042] Furthermore, the output of the FFN is added to the input features and normalized: .

[0043] Global joint representation output: Is implemented by the feature representation output sub-module.

[0044] After the output of the Transformer encoder Has fused the information between each time step, mean pooling is used to aggregate the features of all time steps to obtain the global joint representation of the three tactile sensors: .

[0045] d) Construction and training of a cross-subject emotion recognition model based on multi-source domain adaptation The construction and training process is as Figure 3 shown.

[0046] Training data preparation: Select k the data of X S1 subjects X S2 , …, X Sk from the constructed tactile interaction emotion dataset of autistic children as the source domain, and the subjects to be tested are the target domain. Collect a certain amount of target domain data, and select part of the data (remove the class labels) X T to assist in training, and the remaining data is used as the test set.

[0047] Common feature extractor: Use the above-mentioned tactile emotion feature encoding model based on multi-sensor fusion as the common feature extractor, then the features obtained by the source domain and the target domain through common feature extraction are respectively represented as F S1 , F S2 , …,F Sk and F T 。

[0048] Domain-independent Feature Extraction and Classifier: Build a domain-independent feature extractor and a domain-independent classifier for each pair of source domain and target domain. In the domain-independent feature extractor, the data features of each pair of source domain and target domain (such as F S1 and F T ) are mapped to a specific feature space, and the Maximum Mean Discrepancy (MMD) is used to reduce the difference in feature distributions between different domains to obtain domain-personalized features. The aligned features are respectively represented as ( D S1 , D T1 ), ( D S2 , D T2 ), …, ( D Sk , D Tk ). Subsequently, build a domain-independent classifier for each source domain, and its classification results are represented as O S1 , O S2 , …, O Sk .

[0049] Domain-general Feature Extraction and Classifier: Include a domain-general feature extractor and a domain-general classifier; The domain-general feature extractor is used to perform spatial mapping and consistency encoding on all source domain features to extract domain-general features, represented as G S .

[0050] The domain-general classifier is used to classify the domain-general features to obtain the domain-general classification results of the source domain and the target domain O G .

[0051] Aggregate Prediction: Since the domain-independent classifier is trained on each source domain and the domain-general classifier is trained on all source domains, for a target sample, the classification results of each classifier may be different. Therefore, a confidence-weighted aggregation prediction method is adopted, and the final classification result is represented as . Among them, w GDenote the weights of the outputs of the domain-general classifier. w j Denote the weights of the outputs of each domain-independent classifier.

[0052] To obtain the output weights of each classifier, it is necessary to calculate the output confidence of each classification: The output weights of each classifier are expressed as: Among them, O G is the probability distribution of the output of the domain-general classifier, c G is the output confidence of the domain-general classifier; O Sj is the probability distribution of the output of the domain-independent classifier, c j is the output confidence of the domain-independent classifier.

[0053] Embodiment 2 This embodiment provides a personalized emotion perception system for autistic children based on tactile interaction, including: A humanoid companion robot, which is used to interact with autistic children during the recognition stage and the model training stage, and feedback the tactile behavior data during the human-computer interaction process, and also preprocess the tactile behavior data; the tactile behavior data is the interaction feedback data of autistic children touching the humanoid companion robot during the interaction process, and each body key part feedbacks one modality of tactile behavior data during the model training stage; A construction module, which is used to construct a personalized emotion recognition network for autistic children; the personalized emotion recognition network for autistic children includes: a training data manager, a common feature extractor, a domain-independent feature extraction and classifier, a domain-general feature extraction and classifier, and an aggregated prediction classifier; The training data manager is used to receive and store the tactile behavior data, and divide the preprocessed tactile behavior data into source domain data and target domain data; The common feature extractor is connected to the training data manager, and is used to extract features from the input source domain data and target domain data to obtain source domain features and target domain features; The domain-independent feature extraction and classifier is connected to the common feature extractor, and is used to perform independent feature extraction and classification on the source domain features and target domain features to obtain domain-independent classification results; The domain-general feature extraction and classifier is connected to the common feature extractor, and is used to perform general feature extraction and classification on the source domain features and target domain features to obtain domain-general classification results; The aggregation prediction classifier is respectively connected to the domain-independent feature extraction and classifier and the domain-general feature extraction and classifier, and is used to adopt a confidence-weighted aggregation prediction method for the domain-independent classification result and the domain-general classification result to obtain the classification result of the emotion category and the confidence of the corresponding emotion category; An acquisition module, connected to the humanoid companion robot, is used to acquire preprocessed tactile behavior data in the model training stage, and divide the tactile behavior data into multiple source domain data and one target domain data; it is also used to acquire preprocessed tactile behavior data in the recognition stage; A training module is used to train the personalized emotion recognition network for autistic children by using all the source domain data and the target domain data to obtain a trained personalized emotion recognition network for autistic children; A recognition module is used to perform emotion recognition on the acquired preprocessed tactile behavior data through the trained personalized emotion recognition network for autistic children in the recognition stage to obtain an emotion classification result.

[0054] It should be noted that the system embodiment of this embodiment is similar to the above method embodiment, so the description is relatively simple. For related parts, please refer to the above method embodiment.

[0055] Embodiment 3 This embodiment provides an electronic device, including: One or more processors; A memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the steps of the method for personalized emotion perception of autistic children based on tactile interaction as described in Embodiment 1.

[0056] This application embodiment also provides a computer-readable storage medium, on which computer programs / instructions are stored, and when the computer programs / instructions are executed by a processor, the steps of the method for personalized emotion perception of autistic children based on tactile interaction disclosed in Embodiment 1 of this application are implemented.

[0057] This application embodiment also provides a computer program product, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the method for personalized emotion perception of autistic children based on tactile interaction disclosed in Embodiment 1 of this application are implemented.

[0058] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to describe the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0059] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present application can take the form of all-hardware embodiments, all-software embodiments, or embodiments combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of methods, systems, devices, storage media, and program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate means for realizing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0061] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A personalized emotion perception method for autistic children based on tactile interaction, characterized in that, Including: Recognition stage: Obtain the tactile behavior data during the human-robot interaction between the autistic child and the humanoid companion robot, preprocess the tactile behavior data, and input the preprocessed tactile behavior data into the trained personalized emotion recognition model for autistic children to obtain the emotion classification result; Model training stage: The personalized emotion recognition model for autistic children includes a common feature extractor, a domain-independent feature extraction and classifier, a domain-general feature extraction and classifier, and an aggregation prediction classifier; The training process of the personalized emotion recognition model for autistic children includes: S1. Obtain the tactile behavior data during the human-robot interaction between the autistic child and the humanoid companion robot, and preprocess the tactile behavior data; the tactile behavior data is the interaction feedback data of the autistic child touching the humanoid companion robot during the interaction, and each body key part feeds back a modality of tactile behavior data; S2. Divide the preprocessed tactile behavior data into multiple source domain data and one target domain data; S3. Input both the source domain data and the target domain data into the common feature extractor for feature extraction to obtain source domain features and target domain features; S4. Input both the source domain features and the target domain features into the domain-independent feature extraction and classifier to obtain the domain-independent classification result; meanwhile, also input both the source domain features and the target domain features into the domain-general feature extraction and classifier to obtain the domain-general classification result; S5. Input the domain-independent classification result and the domain-general classification result into the aggregation prediction classifier, and use the confidence-weighted aggregation prediction method to obtain the classification result of the emotion category and the confidence of the corresponding emotion category.

2. The personalized emotion perception method for autistic children based on tactile interaction according to claim 1, characterized in that The common feature extractor includes a data window slicing sub-module, a feature extraction sub-module, an attention fusion sub-module, a global feature fusion sub-module, and a feature representation output sub-module; The data window slicing sub-module is used to slice the preprocessed tactile sensor data into multiple time-step data segments by using a sliding window method; The feature extraction sub-module uses a two-layer 3D CNN network structure to extract features from each time-step data segment to obtain spatio-temporal fusion features; The attention fusion sub-module inputs the spatio-temporal fusion features extracted by the feature extraction sub-module to learn the complementary information of the spatio-temporal fusion features of different modalities; Take the spatio-temporal fusion feature of each modality at each time step as Q, the spatio-temporal fusion features of other modalities as K and V, and perform fusion through the multi-head attention mechanism to obtain multiple groups of fusion features at each time step; Integrate the multiple groups of fusion features at each time step by addition or concatenation operations into a feature vector to obtain the local fusion feature at each time step; Perform multi-head attention mechanism fusion on the spatio-temporal fusion features of all time steps respectively to obtain the spatio-temporal fusion features of each time step; The global feature fusion sub-module uses a Transformer encoder to fuse the local fusion features of each time step to obtain the global fusion features of each time step; The feature representation output sub-module uses average pooling to aggregate the global fusion features of all time steps to obtain the global joint representation of the tactile behavior data.

3. The personalized emotion perception method for autistic children based on tactile interaction according to claim 1, wherein, The domain-independent feature extraction and classifier include a domain-independent feature extractor and a domain-independent classifier established for the source domain and the target domain; The domain-independent feature extractor is used to map the source domain features and the target domain features to a specific feature space, and adopt the maximum mean discrepancy (MMD) to reduce the difference in feature distributions between different domains, so as to obtain domain-personalized features; The domain-independent classifier is used to classify the domain-personalized features to obtain the domain-independent classification results of the source domain and the target domain.

4. The personalized emotion perception method for autistic children based on tactile interaction according to claim 1, characterized in that The domain-general feature extraction and classifier include a domain-general feature extractor and a domain-general classifier; The domain-general feature extractor is used to perform spatial mapping and consistency encoding on the source domain features and the target domain features to obtain domain-general features; The domain-general classifier is used to classify the domain-general features to obtain the domain-general classification results of the source domain and the target domain.

5. The personalized emotion perception method for autistic children based on tactile interaction according to any one of claims 1-4, characterized in that, The method for obtaining tactile behavior data of different modalities in the human-computer interaction process between autistic children and humanoid companion robots includes: Construct a human-computer interaction environment and use a humanoid companion robot to interact with autistic children; install tactile sensors at the key body parts of the humanoid companion robot to collect tactile behavior data of autistic children during the interaction process; Design interaction tasks and emotion induction schemes to stimulate different emotional states of autistic children; during the human-computer interaction process, continuously collect tactile sensor data; According to the collected multi-modal tactile behavior data, annotate the emotional state of autistic children in each interaction segment to determine the emotional category to which the tactile behavior data belongs.

6. The personalized emotion perception method for autistic children based on tactile interaction according to any one of claims 1-4, characterized in that, The method for preprocessing tactile behavior data includes: Clean the collected tactile sensor data to remove outliers caused by sensor failures or interference; Adopt a filtering algorithm for denoising to reduce the noise components in the data; Normalize the data and map it to a specific interval.

7. A personalized emotion perception system for autistic children based on tactile interaction, characterized in that, It includes: A humanoid companion robot, which is used to perform human-computer interaction with autistic children in the recognition stage and the model training stage, and feedback the tactile behavior data during the human-computer interaction process, and also preprocess the tactile behavior data; the tactile behavior data is the interaction feedback data of autistic children touching the humanoid companion robot during the interaction process, and each key body part feeds back one modality of tactile behavior data in the model training stage; A construction module, which is used to construct an autistic child personalized emotion recognition network; The autistic child personalized emotion recognition network includes: a training data manager, a common feature extractor, a domain-independent feature extraction and classifier, a domain-general feature extraction and classifier, and an aggregation prediction classifier; The training data manager is used to receive and store the tactile behavior data, and divide the preprocessed tactile behavior data into source domain data and target domain data; The common feature extractor is connected to the training data manager and is used to extract features from the input source domain data and target domain data to obtain source domain features and target domain features; The domain-independent feature extraction and classifier is connected to the common feature extractor and is used to perform independent feature extraction and classification on the source domain features and the target domain features to obtain domain-independent classification results; The domain-general feature extractor and classifier, which is connected to the common feature extractor, is used to perform general feature extraction and classification on source domain features and target domain features to obtain domain-general classification results; The aggregated prediction classifier, which is respectively connected to the domain-independent feature extractor and classifier and the domain-general feature extractor and classifier, is used to adopt a confidence-weighted aggregation prediction method for the domain-independent classification results and the domain-general classification results to obtain the classification results of emotion categories and the confidence levels of the corresponding emotion categories; The acquisition module, which is connected to the humanoid companion robot, is used to acquire preprocessed tactile behavior data during the model training phase and divide the tactile behavior data into multiple source domain data and one target domain data; it is also used to acquire preprocessed tactile behavior data during the recognition phase; The training module is used to train the personalized emotion recognition network for autistic children by using all the source domain data and the target domain data to obtain a trained personalized emotion recognition network for autistic children; The recognition module is used to perform emotion recognition on the acquired preprocessed tactile behavior data through the trained personalized emotion recognition network for autistic children during the recognition phase to obtain emotion classification results.

8. An electronic device, characterized in that, It includes: A memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for personalized emotion perception of autistic children based on tactile interaction according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by the processor, it implements the method for personalized emotion perception of autistic children based on tactile interaction according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, it implements the method for personalized emotion perception of autistic children based on tactile interaction according to any one of claims 1 to 7.

Images