Service robot emotional interaction system using deep learning

Abstract

The application discloses a service robot emotional interaction system using deep learning, and relates to the technical artificial intelligence field.The application comprises an emotional analysis unit, a robot interaction unit, an instruction implementation unit, a power supply and an interface.The emotional analysis unit comprises an environment perception module, a scene analysis module, a voice recognition module, a semantic recognition module, a face recognition module, a data fusion module, an emotional analysis module, a grade division module, an emotion generation module, an intelligent learning module, a storage module and an emotional feedback module.The application can analyze voice intonation, facial expressions, body language, surrounding environment and the scene where the user is, understand the emotional state of the user, and make corresponding responses, which enables the robot to communicate with people more naturally.Through the intelligent learning module, the individual characteristics and preferences of the user are learned, and customized services and responses are provided according to the information, which can improve the satisfaction and interactive experience of the user.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a service robot emotional interaction system that utilizes deep learning. Background Technology

[0002] Currently, the field of service robots has made great progress, but most robots can only complete preset tasks and lack the ability to perceive and respond to users' emotions, thus failing to provide users with more emotional value. Robots with emotional interaction capabilities can better understand user needs, provide more humanized services, and improve user experience, thereby realizing real human-to-human interaction based on robots. Therefore, we propose a service robot emotional interaction system using deep learning. Summary of the Invention

[0003] The purpose of this invention is to provide a service robot emotional interaction system that utilizes deep learning. By setting up an emotion analysis unit, it can analyze voice tone, facial expressions, body language, surrounding environment and situation to understand the user's emotional state and respond accordingly. This enables the robot to communicate with people more naturally and establish emotional connections more easily. It solves the problem that most existing robots can only complete preset tasks, lack the ability to perceive and respond to user emotions, and cannot provide users with more emotional value assistance.

[0004] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0005] This invention is a service robot emotion interaction system that utilizes deep learning, comprising an emotion analysis unit, a robot interaction unit, an instruction execution unit, a power supply, and an interface;

[0006] The emotion analysis unit includes an environment perception module, a scene analysis module, a speech recognition module, a semantic recognition module, a facial recognition module, a data fusion module, an emotion analysis module, a level classification module, an emotion generation module, an intelligent learning module, a storage module, and an emotion feedback module. The environment perception module, scene analysis module, speech recognition module, semantic recognition module, and facial recognition module are each unidirectionally connected to the data fusion module. The speech recognition module is unidirectionally connected to the semantic recognition module. The data fusion module is unidirectionally connected to the emotion analysis module. The emotion analysis module is bidirectionally connected to the storage module. The emotion analysis module is unidirectionally connected to both the level classification module and the intelligent learning module. The level classification module and the intelligent learning module are unidirectionally connected to the emotion generation module. The emotion generation module is unidirectionally connected to the emotion feedback module. The robot interaction unit includes a microphone, a camera, a speaker, and a human-computer interaction interface. The system includes a facial and animated expression generation module; an emotion feedback module is unidirectionally connected to a speaker, a human-computer interaction interface, and the animated expression generation module; a microphone is unidirectionally connected to a speech recognition module; a camera is unidirectionally connected to a facial recognition module, an environmental perception module, and a scene analysis module; the animated expression generation module is unidirectionally connected to the human-computer interaction interface; the command implementation unit includes a signal receiving module, a controller, sensors, a command driving module, a silicone simulated arm, and a mechanical simulated lower limb; the emotion feedback module is unidirectionally connected to the signal receiving module; the signal receiving module is unidirectionally connected to the controller; the controller is unidirectionally connected to the sensors; the sensors are unidirectionally connected to the command driving module; the command driving module is unidirectionally connected to the silicone simulated arm and the mechanical simulated lower limb; the interface is unidirectionally connected to a power supply; and the power supply is unidirectionally connected to the emotion analysis unit, the robot interaction unit, and the command implementation unit.

[0007] As a preferred technical solution, the environmental perception module is responsible for observing the user's current environment through a camera, collecting and analyzing various information in the environment, such as light and sound. This information helps the robot determine the emotional atmosphere of the environment. The scene analysis module is responsible for enabling the environmental perception module to identify different scenes, such as home, office, restaurant, or other public scenes, and adjusting emotional interaction strategies according to the scene. For example, in a home scene, the robot behaves more friendly and warm; in an office scene, the robot behaves more professional and efficient; in a hospital scene, the robot behaves more serious and solemn. The speech recognition module is responsible for receiving the user's voice and converting the user's speech input into text so that the robot can understand the user's instructions and requirements. The semantic recognition module is responsible for analyzing the collected user voice and judging the user's emotions based on the literal meaning of the words, the tone, rhythm, and intensity of the voice, and understanding the meaning and emotional color of the user's sentences. The facial recognition module is responsible for analyzing the user's facial expressions through camera observation, thereby recognizing expressions such as joy, anger, worry, thought, sadness, fear, and surprise, and judging the user's current emotion based on these expressions. The data fusion module is responsible for arranging the data collected by the above modules and integrating data from different modalities. First, cross-validation and complementary analysis are performed to improve the robustness and accuracy of the emotion recognition model. The emotion analysis module analyzes the fused and arranged data, using information from different modalities to complement and validate each other, reducing the false positive rate. The grading module assesses the intensity of the identified emotions, judging the strength of the user's emotions, such as mild happiness or intense sadness. This assessment helps the robot understand the user's emotions more accurately and make more nuanced responses, thus providing targeted responses to their emotions. The emotion generation module generates appropriate emotional feedback based on the user's emotions, making the robot more natural and engaging. The robot effectively communicates with humans. The intelligent learning module continuously learns user preferences and adjusts emotional expression strategies to enhance emotional resonance between the robot and the user. Based on the user's emotional characteristics, it constructs a personalized user emotional model, thereby providing more accurate and personalized emotional interaction services. The storage module stores the learned emotional changes and personalized emotional services, which can be retrieved later. The emotional feedback module expresses different emotions using different tones, vocabulary, and intonations based on the analyzed user emotions, providing anthropomorphic responses through facial expressions, body language, and vocal tone, and converting these into signals for transmission.

[0008] As a preferred technical solution, the microphone is responsible for recording the user's current voice; the camera is responsible for recording the user's current behavioral state or surrounding scene and environment; the speaker is responsible for conveying different emotions expressed by the robot using different tones, words, and intonations based on the user's current mood; the human-computer interaction interface is responsible for providing the user with a visual interface on the robot, allowing the user to observe their current emotional state and correct any errors in the emotion analysis; the animated expression generation module is responsible for generating animated expressions on the visual interface based on the user's current mood. Animated expressions can more intuitively express the robot's emotions, such as happiness, sadness, anger, and confusion, making it easier for the user to understand the robot's emotional state. Through changes in expression, the robot can establish a deeper emotional connection with the user, enhancing the user's sense of intimacy and trust.

[0009] As a preferred technical solution, the signal receiving module is responsible for receiving and parsing instruction signals from the emotional feedback module; the controller is responsible for receiving the parsed instruction signals and coordinating the robot's behavior, such as adjusting tone of voice, facial expressions, and body movements, to achieve more natural and fluent emotional interaction; the sensors are responsible for collecting motion data of the robot arm, such as angle, position, and torque. Commonly used sensors include angle sensors, torque sensors, and inertial measurement units. These sensors are responsible for monitoring the current state of the robot arm in real time, providing basic data for subsequent control; the instruction driving module is responsible for receiving control instruction signals and outputting instructions to the robot according to the driving instructions; the silicone simulated arm is responsible for simulating the natural movements of a human arm, providing more intuitive visual feedback by cooperating with the signals output by the instruction driving module, such as shaking hands, patting, stroking, and waving, allowing users to feel the robot's real presence and emotional expression. The silicone material can simulate the soft touch of a human hand, enhancing the intimacy and comfort of human-computer interaction; the mechanical simulated lower limbs are responsible for enabling the robot to move, facilitating the robot's access to the user.

[0010] As a preferred technical solution, the socket is responsible for inputting power to the power supply line, thereby charging the power supply line.

[0011] As a preferred technical solution, the power supply is responsible for providing power to the emotion analysis unit, the robot interaction unit, and the instruction execution unit.

[0012] The present invention has the following beneficial effects:

[0013] This invention, by setting up an emotion analysis unit, can analyze voice tone, facial expressions, body language, surrounding environment, and scene to understand the user's emotional state and respond accordingly. This enables the robot to communicate with people more naturally and establish emotional connections more easily. Through the intelligent learning module, it learns the user's personality traits and preferences and provides customized services and responses based on this information, which can improve user satisfaction and interactive experience.

[0014] This invention, by setting up a robot interaction unit, allows users to express their current emotions, and the system quickly and accurately analyzes the user's current state based on their facial expressions. This enables the system to respond to users more quickly and provide targeted responses. The system can also use this information to answer user questions and solve user problems, providing users with more convenient and effective services. Furthermore, it can correct any errors in the analysis of emotional data.

[0015] This invention, by setting up an instruction implementation unit, receives data from the emotional feedback module and, in conjunction with the signals output by the instruction implementation drive, adjusts the tone of voice, facial expressions, and body movements to achieve a more natural and fluid emotional interaction, such as handshakes, pats, stroking, and waving. It provides more intuitive visual feedback, allowing users to feel the robot's real presence and emotional expression. The silicone material can simulate the soft touch of a human hand, enhancing the intimacy and comfort of human-computer interaction. The mechanically simulated lower limbs can follow the user's movements, improving the robot's flexibility.

[0016] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This invention relates to a service robot emotional interaction system that utilizes deep learning. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, the present invention is a service robot emotion interaction system using deep learning, including an emotion analysis unit, a robot interaction unit, an instruction execution unit, a power supply, and an interface;

[0021] The sentiment analysis unit includes an environmental perception module, a scene analysis module, a speech recognition module, a semantic recognition module, a facial recognition module, a data fusion module, a sentiment analysis module, a grading module, an emotion generation module, an intelligent learning module, a storage module, and an sentiment feedback module. The environmental perception module, scene analysis module, speech recognition module, semantic recognition module, and facial recognition module are each unidirectionally connected to the data fusion module; the speech recognition module is unidirectionally connected to the semantic recognition module; the data fusion module is unidirectionally connected to the sentiment analysis module; the sentiment analysis module is bidirectionally connected to the storage module; the sentiment analysis module is unidirectionally connected to both the grading module and the intelligent learning module; the grading module and the intelligent learning module are unidirectionally connected to the emotion generation module; and the emotion generation module is unidirectionally connected to the sentiment feedback module. The robot interaction unit includes a microphone, camera, speaker, and human-machine interface. The system includes an interactive interface and an animated expression generation module; an emotion feedback module that is unidirectionally connected to the speaker, human-computer interaction interface, and animated expression generation module; a microphone that is unidirectionally connected to the speech recognition module; a camera that is unidirectionally connected to the facial recognition module, environmental perception module, and scene analysis module; an animated expression generation module that is unidirectionally connected to the human-computer interaction interface; and a command implementation unit comprising a signal receiving module, a controller, sensors, a command driving module, a silicone simulated arm, and a mechanical simulated lower limb. The emotion feedback module is unidirectionally connected to the signal receiving module; the signal receiving module is unidirectionally connected to the controller; the controller is unidirectionally connected to the sensors; the sensors are unidirectionally connected to the command driving module; the command driving module is unidirectionally connected to the silicone simulated arm and the mechanical simulated lower limb; an interface that is unidirectionally connected to the power supply; and the power supply that is unidirectionally connected to the emotion analysis unit, the robot interaction unit, and the command implementation unit.

[0022] The environmental perception module is responsible for observing the user's current environment through a camera, collecting and analyzing various information such as light and sound. This information helps the robot determine the emotional atmosphere of the environment. The scene analysis module enables the environmental perception module to identify different scenes, such as home, office, restaurant, or other public scenes, and adjust emotional interaction strategies accordingly. For example, in a home scene, the robot behaves more friendly and warm; in an office scene, the robot behaves more professionally and efficiently; and in a hospital scene, the robot behaves solemnly and seriously. The speech recognition module receives the user's voice and converts the user's speech input into text so that the robot can understand the user's instructions and requests. The semantic recognition module analyzes the collected user voice and judges the user's emotions based on the literal meaning of the words, the tone, rhythm, and intensity of the voice, understanding the meaning and emotional color of the user's sentences. The facial recognition module analyzes the user's facial expressions through camera observation, thereby identifying expressions such as joy, anger, worry, thought, sadness, fear, and surprise, and judging the user's current emotion based on these expressions. The data fusion module is responsible for arranging the data collected by the above modules, integrating data from different modalities, and cross-referencing them. Verification and complementary analysis improve the robustness and accuracy of the emotion recognition model; the emotion analysis module analyzes the fused and arranged data, using information from different modalities to complement and verify each other, reducing the false positive rate; the grading module assesses the intensity of identified emotions, judging the strength of user emotions, such as mild happiness or intense sadness. This assessment helps the robot understand user emotions more accurately and make more nuanced responses, thus providing targeted emotional responses; the emotion generation module generates appropriate emotional feedback based on user emotions, allowing the robot to communicate with humans more naturally and engagingly; the intelligent learning module continuously learns user preferences and adjusts emotional expression strategies to enhance emotional resonance between the robot and users, constructing personalized user emotion models based on user emotional characteristics, thereby providing more accurate and personalized emotional interaction services; the storage module stores the learned emotional changes and personalized emotional services for later retrieval; the emotional feedback module expresses different emotions using different tones, vocabulary, and intonations based on the analyzed user emotions, providing anthropomorphic responses through facial expressions, body language, and vocal tone, and converting these into signals for transmission.

[0023] The microphone is responsible for recording the user's current voice; the camera is responsible for recording the user's current behavior or surrounding scene and environment; the speaker is responsible for conveying different emotions expressed by the robot using different tones, words, and intonations based on the user's current mood; the human-computer interaction interface is responsible for providing a visual interface on the robot for the user to observe their current emotional state and to correct any errors in the emotion analysis; the animated expression generation module is responsible for generating animated expressions on the visual interface based on the user's current mood. Animated expressions can more intuitively express the robot's emotions, such as happiness, sadness, anger, and confusion, making it easier for users to understand the robot's emotional state. Through changes in expressions, the robot can establish a deeper emotional connection with the user, enhancing the user's sense of intimacy and trust.

[0024] The signal receiving module receives and parses command signals from the emotional feedback module. The controller receives the parsed command signals and coordinates the robot's behavior, such as adjusting tone of voice, facial expressions, and body movements, to achieve more natural and fluid emotional interaction. Sensors collect motion data of the robot arm, such as angle, position, and torque. Commonly used sensors include angle sensors, torque sensors, and inertial measurement units. These sensors monitor the current state of the robot arm in real time, providing basic data for subsequent control. The command driving module receives control command signals and outputs commands to the robot according to the driving commands. The silicone simulated arm simulates the natural movements of a human arm, providing more intuitive visual feedback by cooperating with the command-driven output signals, such as shaking hands, patting, stroking, and waving. This allows users to feel the robot's real presence and emotional expression. The silicone material simulates the soft touch of a human hand, enhancing the intimacy and comfort of human-computer interaction. The mechanical simulated lower limbs enable the robot to move, facilitating its access to the user.

[0025] The socket is responsible for inputting power to the power supply line, thereby charging the power supply line.

[0026] The power supply is responsible for providing power to the emotion analysis unit, robot interaction unit, and command execution unit.

[0027] One specific application of this embodiment is as follows: Data fusion and analysis are performed on the user's current language, facial expressions, surrounding environment, and scene using a camera and microphone. The analyzed emotional intensity is then categorized into levels to determine the strength of the user's emotions. Corresponding emotions are generated for the robot, and feedback signals are sent. The instruction execution unit receives data from the emotional feedback module and, in conjunction with the signals driven by the instruction execution, adjusts the tone of voice, facial expressions, and body movements to achieve a more natural and fluid emotional interaction. Examples include handshakes, pats, stroking, and waving, providing more intuitive visual feedback. This allows the user to feel the robot's real presence and emotional expression. The silicone material simulates the soft touch of a human hand, enhancing the intimacy and comfort of human-computer interaction. The mechanically simulated lower limbs can follow the user's movements, improving the robot's flexibility. Users can also observe their current emotional state through a visual interface to better understand their own physical condition.

[0028] It is worth noting that the various units included in the above system embodiments are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0029] Furthermore, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0030] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A service robot emotion interaction system utilizing deep learning, comprising an emotion analysis unit, a robot interaction unit, an instruction execution unit, a power supply, and an interface; characterized in that: The sentiment analysis unit includes an environment perception module, a scene analysis module, a speech recognition module, a semantic recognition module, a facial recognition module, a data fusion module, a sentiment analysis module, a level classification module, an emotion generation module, an intelligent learning module, a storage module, and an sentiment feedback module. The environment perception module, scene analysis module, speech recognition module, semantic recognition module, and facial recognition module are each unidirectionally connected to the data fusion module. The speech recognition module is unidirectionally connected to the semantic recognition module. The data fusion module is unidirectionally connected to the sentiment analysis module. The sentiment analysis module is bidirectionally connected to the storage module. The sentiment analysis module is unidirectionally connected to both the level classification module and the intelligent learning module. The level classification module and the intelligent learning module are unidirectionally connected to the emotion generation module. The emotion generation module is unidirectionally connected to the sentiment feedback module. The robot interaction unit includes a microphone, camera, speaker, human-computer interaction interface, and animated expression generation module; The emotion feedback module is unidirectionally connected to the speaker, the human-computer interaction interface, and the animated expression generation module; the microphone is unidirectionally connected to the speech recognition module; the camera is unidirectionally connected to the facial recognition module, the environmental perception module, and the scene analysis module; and the animated expression generation module is unidirectionally connected to the human-computer interaction interface. The instruction implementation unit includes a signal receiving module, a controller, a sensor, an instruction driving module, a silicone simulated arm, and a mechanical simulated lower limb; the emotional feedback module is unidirectionally connected to the signal receiving module; the signal receiving module is unidirectionally connected to the controller; the controller is unidirectionally connected to the sensor; the sensor is unidirectionally connected to the instruction driving module; and the instruction driving module is unidirectionally connected to both the silicone simulated arm and the mechanical simulated lower limb. The interface is unidirectionally connected to the power supply. The power supply is unidirectionally connected to the emotion analysis unit, the robot interaction unit, and the instruction execution unit, respectively.

2. The service robot emotional interaction system using deep learning according to claim 1, characterized in that, The environmental perception module is responsible for observing the user's current environment through the camera, collecting and analyzing various information in the environment, such as light and sound. This information can help the robot judge the emotional atmosphere of the environment. The scene analysis module is responsible for enabling the environment perception module to identify different scenes, such as home, office, restaurant, or other public scenes, and adjust the emotional interaction strategy according to the scene. For example, in a home scene, the robot behaves more friendly and warm; in an office scene, the robot behaves more professional and efficient; and in a hospital scene, the robot behaves more serious and solemn. The speech recognition module is responsible for receiving the user's voice and converting the user's speech input into text so that the robot can understand the user's instructions and requirements. The semantic recognition module is responsible for analyzing the collected user voices and judging the user's emotions based on the literal meaning of the words, the tone, rhythm and intensity of the voice, and understanding the meaning and emotional color of the user's sentences. The facial recognition module is responsible for analyzing the user's facial expressions through the camera, thereby identifying expressions such as joy, anger, worry, thought, sadness, fear, and surprise, and judging the user's current emotion based on these expressions; The data fusion module is responsible for arranging the data collected by the above modules, integrating data from different modalities, performing cross-validation and complementary analysis, and improving the robustness and accuracy of the emotion recognition model. The sentiment analysis module is responsible for analyzing the fused and arranged data, using information from different modalities to complement and verify each other, thereby reducing the false judgment rate. The rating module is responsible for assessing the intensity of the identified emotions and judging the strength of the user's emotions, such as mild happiness or intense sadness. This assessment helps the robot to understand the user's emotions more accurately and make more detailed responses, thereby providing targeted responses to the user's emotions. The emotion generation module is responsible for generating appropriate emotional feedback based on the user's emotions, allowing the robot to communicate with humans more naturally and engagingly. The intelligent learning module is responsible for continuously learning the user's preferences and interests, adjusting emotional expression strategies, enhancing the emotional resonance between the robot and the user, and constructing a personalized user emotional model based on the user's emotional characteristics, thereby providing more accurate and personalized emotional interaction services. The storage module is responsible for storing the learned emotional changes and personalized emotional services, and retrieving them when needed later. The emotional feedback module is responsible for expressing different emotions using different tones, words, and intonations based on the analyzed user emotions. It provides anthropomorphic responses through facial expressions, body language, and tone of voice, and converts these responses into signals for transmission.

3. The service robot emotional interaction system using deep learning according to claim 2, characterized in that, The microphone is responsible for recording the user's current voice; The camera is responsible for recording the user's current behavior or the surrounding scene and environment; The speaker is responsible for conveying different emotions by using different tones, words, and intonations based on the user's current mood. The human-computer interaction interface is responsible for providing a visual interface on the robot for users to observe their current emotional state and to correct any errors in the emotion analysis. The animated expression generation module is responsible for generating animated expressions on the visual interface based on the user's current emotions. Animated expressions can more intuitively express the robot's emotions, such as happiness, sadness, anger, and confusion, making it easier for users to understand the robot's emotional state. Through changes in expressions, the robot can establish a deeper emotional connection with the user, enhancing the user's sense of intimacy and trust.

4. The service robot emotional interaction system using deep learning according to claim 3, characterized in that, The signal receiving module is responsible for receiving and parsing the instruction signals from the emotion feedback module; The controller is responsible for receiving and parsing instruction signals and coordinating the robot's behavior, such as adjusting the tone of voice, facial expressions, and body movements, to achieve more natural and fluent emotional interaction. The sensors are responsible for collecting motion data of the robot arm, such as angle, position, and torque. Commonly used sensors include angle sensors, torque sensors, and inertial measurement units. These sensors are responsible for monitoring the current state of the robot arm in real time, providing basic data for subsequent control. The instruction drive module is responsible for receiving control instruction signals and outputting instructions to the robot according to the drive instructions; The silicone simulated arm is responsible for simulating the natural movements of a human arm. By cooperating with the signals output by the execution of commands, such as shaking hands, patting, stroking, and waving, it provides more intuitive visual feedback, allowing users to feel the real presence and emotional expression of the robot. The silicone material can simulate the soft touch of a human hand, enhancing the intimacy and comfort of human-computer interaction. The mechanically simulated lower limbs are responsible for enabling the robot to move and reach the user.

5. The service robot emotional interaction system using deep learning according to claim 4, characterized in that, The socket is responsible for inputting power to the power supply line, thereby charging the power supply line.

6. The service robot emotional interaction system using deep learning according to claim 5, characterized in that, The power supply is responsible for providing power to the emotion analysis unit, the robot interaction unit, and the instruction execution unit.

Images