A multi-modal fusion emotion recognition method and system for social robots

Abstract

The application relates to a multi-modal fusion emotion recognition method and system for a social robot, and the method comprises the following steps: obtaining real-time sensing information of target personnel, wherein the real-time sensing information comprises image information, tactile information and sound information; pre-processing the real-time sensing information to obtain pre-processed sensing information; and obtaining an emotion recognition result according to the pre-processed sensing information. Compared with the prior art, the application fuses visual, voice and tactile sensing data for emotion recognition, uses a microphone and an electromyography sensor to acquire voice sensing data, and uses multiple key points to capture human parameters for visual sensing data, so that the recognition accuracy is improved and the method is suitable for more environments.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a multimodal fusion emotion recognition method and system for social robots. Background Technology

[0002] Emotions play a crucial role in human-computer interaction. In healthcare, understanding a patient's emotional state, especially those with expression difficulties, allows for tailored care and improved treatment quality. Similarly, if social robots can recognize user emotions, human-machine interaction will become more friendly and natural.

[0003] Emotion recognition can be made using information such as facial features, posture, and voice, but most current emotion recognition methods are based on only one of these types. Current research related to robotics largely focuses on robot movement and human interaction, with relatively little research on emotion recognition methods that integrate multiple information sources.

[0004] Chinese patent application CN202210010258.4 discloses a multimodal facial emotion recognition method and device. The method includes: acquiring video data of workers in a target scene; extracting frames from the video data at preset time intervals to obtain a multimodal information sequence; extracting visual modal information from each frame to obtain a facial expression feature sequence about key facial points; processing the spectrogram corresponding to each frame of auditory modal information using a convolutional neural network model to obtain a speech feature sequence; inputting the normalized facial expression feature sequence and the normalized speech feature sequence into an attention-based temporal learning model for fusion encoding to obtain a temporal fusion feature vector; inputting the temporal fusion feature vector into a multimodal emotion recognition model to obtain an emotion score, and determining the worker's facial emotion based on the emotion score. This method uses a convolutional neural network model to fuse visual and auditory information to achieve emotion recognition. However, this method only fuses visual and speech perception data, resulting in limited sources of perception data and insufficient data for emotion judgment. Furthermore, the speech perception data only uses spectrograms as input information, leading to low accuracy in the judgment results.

[0005] In summary, existing emotion recognition methods have the following drawbacks:

[0006] (1) Most recognition methods only use one or more of visual perception data and speech perception data as the basis for emotion recognition. The types of perception data are limited, and the accuracy of emotion recognition results is low.

[0007] (2) Most of the speech perception data in existing methods are obtained by using microphones or other sound pickup devices, and the source of speech perception data is relatively singular;

[0008] (3) Most existing methods for processing visual perception data only consider information about a human body or face and lack the capture of human body feature point information. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art by providing a method and system for multimodal emotion recognition through the collection of sensory data, including visual perception data, speech perception data and tactile perception data.

[0010] The objective of this invention can be achieved through the following technical solutions:

[0011] One aspect of the present invention provides a multimodal fusion emotion recognition method for social robots, comprising the following steps: obtaining real-time perception information of a target person, the real-time perception information including image information, tactile information and sound information; preprocessing the real-time perception information to obtain preprocessed perception information; and inputting the preprocessed perception information into a pre-trained multimodal fusion emotion recognition model to obtain an emotion recognition result.

[0012] As a preferred technical solution, the preprocessing process includes the following steps: obtaining human body key point coordinate information and facial key point coordinate information based on the image information; obtaining contact force information based on the tactile information; obtaining speech information based on the sound information; and obtaining the preprocessed perceptual information based on the human body key point coordinate information, facial key point coordinate information, contact force information, and speech information.

[0013] As a preferred technical solution, the key points of the human body include one or more of the following: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The key points of the face include one or more of the following: eyebrows, eyes, nose, mouth, and key parts: forehead, cheeks, eye bags, upper and lower jaws, and lips.

[0014] As a preferred technical solution, the process of obtaining the human body key point coordinate information and the face key point coordinate information includes the following steps: based on the image information, after image enhancement, data encoding and transmission, smoothing, edge sharpening, segmentation, feature extraction and image recognition and understanding, optimized image information is obtained; the optimized image information is input into a pre-trained key point recognition model to obtain the human body key point coordinate information and the face key point coordinate information.

[0015] As a preferred technical solution, the sound information is obtained from a microphone and an electromyography sensor.

[0016] Another aspect of the present invention provides a multimodal fusion emotion recognition system for social robots, comprising: a visual processing module including a visual sensor and an image processing unit, wherein the image processing unit is used to acquire real-time image signals from the visual sensor and acquire human body key point coordinate information and facial key point coordinate information; a tactile processing module including a tactile sensor and a tactile signal processing unit, wherein the tactile signal processing unit is used to acquire real-time tactile signals from the tactile sensor and acquire contact force information; an auditory processing module including a speech sensor and a speech signal processing unit, wherein the speech signal processing unit is used to acquire real-time auditory signals from the speech sensor and acquire speech information; and an emotion recognition module connected to the image processing unit, the tactile signal processing unit, and the speech signal processing unit, respectively, for acquiring the human body key point coordinate information, facial key point coordinate information, contact force information, and speech information, inputting them into a pre-trained multimodal fusion emotion recognition model, and outputting emotion recognition results.

[0017] As a preferred technical solution, the voice sensor includes one or more of an electromyography sensor and a sound sensor.

[0018] As a preferred technical solution, the visual sensor includes a camera.

[0019] As a preferred technical solution, the tactile sensor includes a pressure sensor.

[0020] As a preferred technical solution, an interactive interface is also included, which is connected to the emotion recognition module.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] (1) Comprehensive collection of real-time sensory information, including image information, tactile information and sound information, as the information source for emotion recognition. Due to the richer types of information, the accuracy of recognition can reach a higher level.

[0023] (2) Sound information is obtained through electromyography sensor and sound sensor, rather than solely by sound sensor, to avoid the sound sensor being unable to obtain accurate speech information in noisy environments;

[0024] (3) The key point coordinate information of human body and face is obtained by using feature point recognition, which can be applied to the monitoring of various faces and human bodies, and further improve the recognition accuracy. Attached Figure Description

[0025] Figure 1 This is a schematic diagram showing the connections of each part in the embodiment;

[0026] Figure 2 This is a schematic diagram of the working environment in the embodiment.

[0027] Among them, 1. Interactive personnel, 2. Social robot, 3. Interactive environment, 4. Visual processing module, 5. Tactile processing module, 6. Auditory processing module, 7. Emotion recognition module, 201. Camera, 202. Tactile sensor, 203. Microphone. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] Example 1

[0030] This embodiment presents a multimodal fusion emotion recognition method for social robots, comprising the following steps: obtaining real-time perception information of the target person, including image information, tactile information, and sound information; based on the image information, performing image enhancement, data encoding and transmission, smoothing, edge sharpening, segmentation, feature extraction, and image recognition and understanding to obtain optimized image information; inputting the optimized image information into a pre-trained keypoint recognition model based on a deep neural convolutional network, using the optimized image information as input to the keypoint recognition model to extract parameters of the worker's human body key points and facial key parts, and obtaining... The system obtains the coordinates of key points on the human body and the face; based on tactile information, incomplete and non-standard tactile data are removed and standardized to obtain contact force information; based on sound information, a combination of sound and electromyographic signals is used to recognize speech signals, removing incomplete and non-standard sound information and standardizing it to obtain speech information; based on the coordinates of key points on the human body, the coordinates of key points on the face, the contact force information, and the speech information, preprocessed perceptual information is obtained; the preprocessed perceptual information is input into a pre-trained multimodal fusion emotion recognition model to obtain the emotion recognition result.

[0031] Among them, the key point recognition model and the multimodal fusion emotion recognition model both use existing residual neural network models for feature extraction.

[0032] Key points of the human body include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. Key points of the face include the eyebrows, eyes, nose, mouth, and key areas such as the forehead, cheeks, eye bags, upper and lower jaws, and lips.

[0033] Example 2

[0034] like Figure 2 The diagram illustrates an environment for a multimodal fusion emotion recognition system for social robots. The system includes: a visual processing module comprising a visual sensor and an image processing unit. The image processing unit stores an existing residual neural network model for extracting parameters of key human body points and key facial features. The image processing unit acquires real-time image signals from the visual sensor and obtains coordinate information of key human body points and key facial features. A tactile processing module comprising a tactile sensor and a tactile signal processing unit. The tactile signal processing unit acquires real-time tactile signals from the tactile sensor and obtains contact force information. An auditory processing module comprising a speech sensor and a speech signal processing unit. The speech signal processing unit acquires real-time auditory signals from the speech sensor and obtains speech information. An emotion recognition module internally stores an existing residual neural network model for multimodal fusion emotion recognition. It acquires coordinate information of key human body points, key facial features, contact force information, and speech information, inputs these into the model, and obtains the emotion recognition result. An interactive interface connected to the emotion recognition module allows for interaction with the user, including the emotion recognition result. Among them, the voice sensor includes an electromyography sensor and a sound sensor, the vision sensor includes a camera, and the tactile sensor includes a pressure sensor.

[0035] like Figure 1 The above is a schematic diagram of the composition of a multimodal fusion emotion recognition system for social robots in this embodiment. It describes a control method and system for establishing trust in a social robot based on tactile interaction. The system includes a visual processing module 4, a tactile processing module 5, an auditory processing module 6, an emotion recognition module 7, and a social robot. The visual processing module 4 includes a visual sensor and an image processing unit. The visual sensor captures real-time image information of the worker, and the image processing unit extracts the coordinates of key human body points, key facial points, and key facial features, and transmits them to the emotion recognition module 7. The tactile processing module 5 includes a tactile sensor and a sensor signal processing unit. The tactile sensor captures real-time contact force information between the worker and the robot, including pressure value, sensor contact area, and effective touch time. The tactile sensor signal processing unit processes the signals from multiple tactile sensors and transmits them to the emotion recognition module 7. The auditory processing module 6 includes a voice acquisition device and a voice signal processing unit. The voice acquisition device captures and filters sound signals, and the voice signal processing unit processes the signals transmitted by the voice acquisition device and transmits them to the emotion recognition module 7.

[0036] In practical use, the system operation process includes the following steps:

[0037] Step S1: The visual processing module 4 collects image information of the human-machine working scene in real time through the visual sensor. The image processing unit processes the collected image information, including image enhancement, data encoding and transmission, smoothing, edge sharpening, segmentation, feature extraction, image recognition and understanding, etc. Then, it uses the existing residual neural network model to extract the key points of the human body and the key parts of the face of the workers and transmits them to the emotion recognition module and the interactive environment 3.

[0038] Step S2: The tactile processing module 5 captures real-time contact force information between the worker and the robot through the tactile sensor, performs data preprocessing on the collected tactile information, including deleting incomplete and non-standard tactile data, performing standardization processing, and transmitting the obtained tactile feature information to the emotion recognition module and the interactive environment 3.

[0039] Step S3: The voice acquisition device acquires the voice signal. The voice acquisition device includes a microphone and a noise filter. The voice processing unit recognizes the voice signal by combining sound and electromyography (EMG) signals. The preprocessing of sound data and surface EMG data includes deleting incomplete and non-standard voice data and performing standardization processing to obtain sound and surface EMG feature information, which is then transmitted to the emotion recognition module 7.

[0040] Step S4: The emotion recognition module 7 makes an emotion recognition judgment based on the parameters of key points of the human body and key parts of the face, as well as the information data of voice and surface electromyography.

[0041] The visual sensor is a CCD camera, containing 17 key human body points: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. This data comprehensively describes the position of each part of the human body. Facial key points and key facial features are used to extract facial emotional characteristics. These key points include eyebrows, eyes, nose, and mouth, while key facial features include the forehead, cheeks, eye bags, upper and lower jaw, and lips. The tactile sensor uses an RX-F2632 piezoresistive flexible thin-film pressure sensor, serving as the sensor for the robot arm.

[0042] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal fusion emotion recognition method for social robots, characterized in that, This is achieved based on a multimodal fusion emotion recognition system, including the following steps: Obtain real-time perception information of the target person, including image information, tactile information, and sound information; The real-time sensing information is preprocessed to obtain preprocessed sensing information; The preprocessed perceptual information is input into a pre-trained multimodal fusion emotion recognition model to obtain the emotion recognition result. The preprocessing process includes the following steps: Based on the image information, obtain the coordinate information of human body key points and the coordinate information of facial key points; Based on the tactile information, contact force information is obtained; Based on the sound information, obtain the voice information; Based on the human body key point coordinate information, facial key point coordinate information, contact force information, and voice information, the preprocessed perception information is obtained. The aforementioned key points of the human body include one or more of the following: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The aforementioned key points of the face include one or more of the following: eyebrows, eyes, nose, mouth, and key areas: forehead, cheeks, eye bags, upper and lower jaw, and lips. The process of obtaining the coordinate information of human body key points and facial key points includes the following steps: Based on the image information, optimized image information is obtained through image enhancement, data encoding and transmission, smoothing, edge sharpening, segmentation, feature extraction, and image recognition and understanding. The optimized image information is input into a pre-trained keypoint recognition model to obtain the coordinate information of human body keypoints and facial keypoints. The sound information is acquired from a microphone and an electromyography sensor. The multimodal fusion emotion recognition system includes: The vision processing module includes a vision sensor and an image processing unit. The image processing unit is used to acquire real-time image signals from the vision sensor and acquire human body key point coordinate information and face key point coordinate information. The tactile processing module includes a tactile sensor and a tactile signal processing unit. The tactile signal processing unit is used to acquire real-time tactile signals from the tactile sensor and acquire contact force information. The auditory processing module includes a speech sensor and a speech signal processing unit, wherein the speech signal processing unit is used to acquire real-time auditory signals from the speech sensor and acquire speech information; The emotion recognition module is connected to the image processing unit, tactile signal processing unit, and speech signal processing unit, respectively. It obtains the coordinate information of key human body points, the coordinate information of key facial points, contact force information, and speech information, inputs these into a pre-trained multimodal fusion emotion recognition model, and outputs the emotion recognition result. The voice sensor includes one or more of an electromyography (EMG) sensor and a sound sensor. It also includes an interactive interface connected to the emotion recognition module.

2. The multimodal fusion emotion recognition method for social robots according to claim 1, characterized in that, The visual sensor mentioned includes a camera.

3. The multimodal fusion emotion recognition method for social robots according to claim 1, characterized in that, The tactile sensor includes a pressure sensor.

Images