Humanoid technical system for predicting and stimulating human emotional reactions during interview

A humanoid robotic system with electromechanical actuators and neural networks processes multimodal data to predict and stimulate emotional reactions, addressing limitations in existing systems by improving accuracy and reducing delay.

UA164612UActive Publication Date: 2026-10-21SHAMOV OLEKSII ANATOLIIOVYCH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
UA202502855U
Authority / Receiving Office
UA · UA
Patent Type
Utility models
Current Assignee / Owner
Filing Date
2025-06-16
Publication Date
2026-10-21
Estimated Expiration
2035-06-16

AI Technical Summary

Technical Problem

Existing robotic systems lack the ability to physically reproduce subtle emotional reactions, predict emotional states in real-time, and adapt to socio-cultural contexts, limiting their effectiveness in dynamic interactions and interviews.

Method used

A humanoid technical system with a multimodal design, incorporating electromechanical actuators, CMOS video cameras, microphones, and a multi-core processor with neural networks, processes multimodal sensory data to predict and stimulate emotional reactions in real-time, using psychological techniques for personalized interaction.

Benefits of technology

Enhances the accuracy of emotional analysis and physical feedback, reducing reaction delay to 150 ms, allowing for adaptive and culturally sensitive interactions.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

Humanoid technical system for predicting and stimulating human emotional reactions during interview includes a speech module and a built-in computing module. The system is designed as a housing shaped like a human body, comprising a head with a face, a neck, shoulders, a torso, and limbs. The facial expression system includes a set of electromechanical actuators with a resolution of at least 0.5°, mounted beneath a layer of artificial facial skin, as well as video cameras based on CMOS sensors with a resolution of at least 1080p and a frame rate of at least 30 frames per second. The microphones have a sensitivity of at least –40 dB and a frequency range of 20–20,000 Hz. The speakers have a frequency range of 80–8,000 Hz, are installed in the head region, and are connected to the speech module. The inertial sensors consist of three-axis accelerometers and gyroscopes located in the neck and shoulder regions. The computing module contains a multi-core processor with a neural network trained using methods from evaluation theory, affective control theory, and statement reliability analysis. The computing module is designed to process multimodal data obtained from video cameras, microphones, and inertial sensors, and to generate control signals for reproducing facial, vocal, and behavioral responses in real time.
Need to check novelty before this filing date? Find Prior Art

Description

The utility model belongs to the field of affective robotics, artificial intelligence and human- Human-Robot Interaction (HRI), in particular to technical systems designed for recognizing, predicting and stimulating human emotional reactions to improve communication, revealing hidden information, assessing truthfulness, and maintaining psycho-emotional contact. The utility model belongs to the field of affective robotics, artificial intelligence and human- machine interaction (HRI), in particular to technical means and systems designed for recognition, predicting and stimulating human emotional reactions in real time. The relevance of developments in this area is due to the growing need for intelligent systems capable of to deep social, psychological and emotional interaction with a person. Such systems are necessary for applications in psychology, security, training, HR selection, forensics and other areas where effective communication with a person is crucial. Creating models that can simulate human behavior, predict emotional state and adapt to human reactions, opens up new capabilities for automated analysis of emotions and behavior in complex social situations. Known patent US8996429B1 relates to a method, device and system for providing individuality to a robotic device based on personalized information about the user obtained from other devices. The system receives user communication and behavioral data (word choice, sentence structure sentences, symbols) from the second device, processes them using cloud computing, determines the desired personality traits of the robot and changes its behavior, language, reactions or other characteristics to adapt to The user's personality can take into account the context of time, place, historical emotions, visual indicators, etc. This allows you to create adaptive, emotionally tuned robots- companions or digital assistants capable of personalizing interaction with the user. The disadvantages of the analogue are as follows: first, it does not provide a physical representation of emotions through facial expressions, gestures or change in voice intonation, limited only to modifying the interface or personality device. Secondly, the system does not predict the emotional state before it is expressed, which limits its effectiveness in interviewing or dynamic interaction scenarios. Third, emotional analysis is based on linguistic and behavioral patterns, without integration of multimodal sensors or neuropsychological models that allow for the detection of microexpressions, timbre or body language. In addition, the personalization algorithm critically depends on connectivity to the cloud infrastructure, which makes it difficult to use in remote or standalone environments. Also, there are no scenarios multi-user interaction, and the consideration of context is superficial (for example, time of day or mood from historical data), without taking into account deeper psychological changes or cross-cultural factors (patent for invention US8996429B1, IPC G06F 16 / 214, published 31.03.2015). A well-known patent from Amazon Technologies US11260536B1 describes a robotic device that simulates emotional state by analyzing commands, sensory data, and previous experience: the device determines the input function based on the previous emotional state and the effect of completing the task, calculates emotion using a nonlinear (e.g., sigmoid) function, generates the corresponding behavior and displays it through audio, visual, or other devices; the system takes into account historical time series, emotion derivatives, battery status, duration of inactivity, device status changes, and others triggers, which allows the robot to adapt behavior to the context. The main disadvantages of the analogue are the following: the lack of a mimic system capable of physically reproducing emotional reactions, limitations in predicting and stimulating human emotions as an active form interactions, lack of complex processing of multimodal sensory signals (in particular microexpressions, voice tone, body movements), insufficient personalization of interaction taking into account sociocultural characteristics of the interlocutor, as well as the lack of use of modern psychological methods for assessing credibility and emotional authenticity, which significantly narrows the scope application of the analogue in interviewing, profiling, law enforcement or psychological activities compared to the utility model (patent for invention US11260536B1, IPC B25J 11 / 001, publ. 01.03.2022). The well-known patent of Samsung Electronics US9311680B2 is about the development of a device and a method of generating emotional information that allows a terminal (e.g., smartphone, computer) to or other electronic device) to detect user interaction during the performance of a certain function, analyze this interaction to determine the user's emotional state (using emotional intelligence) models, estimates, stochastic score), store this data in connection with the device functions, and use them to make recommendations for future actions or features. The device contains a processor, function and emotion detectors, a storage unit, a display unit, and an output device for recommendations. All these components interact to form a personalized emotional profile of the user, based on which the system can adapt the terminal functions to emotional state. This can be applied in UX research systems, adaptive interfaces, personalized services or digital mental support. At the same time, the specified analogue has a number of limitations compared to the utility model. In particular, it does not involves physical emotional feedback with the user (facial expressions, gestures, postures), not implemented in the form of a humanoid device, does not provide multimodal interaction in the mode real-time, and its functionality is focused on building an emotional profile for UX- adaptation, rather than on the active stimulation of emotional reactions during dialogue. Thus, the technical the result of a useful model is increasing the accuracy of emotion analysis and ensuring physical reverse emotional impact in real time - not achieved in a known analogue (patent for invention US9311680B2, IPC G06F 3 / 01, published 12.04.2016). The well-known patent US11074491B2 describes a robotic companion device equipped with processor with an emotional core that is capable of reading, analyzing, and modeling emotional and semantic user states to provide personalized interactions. The device operates in active learning mode (through direct communication) and passive mode (analyzing digital traces of the user, in particular on social networks). Based on input data from sensors (visual, audio, etc.), the device classifies the user's emotional state into categories (emotions, mood, attitudes, personality traits), correlates them with internal emotional state and external events, and generates behavioral and emotional responses accordingly. The system can take into account cultural characteristics, has cloud data processing, remembers previous interactions with users, supports distributed architecture and adaptively forms the robot's personality based on aggregated data from many users. The disadvantage of the analogue is that the device does not have a realistic mimic system with high resolution. ability to physically reproduce subtle or mixed emotions through facial expression, intonation, and gestures, as provided in the utility model. Also, the design lacks a bodily anthropomorphic an embodiment with full motor skills, which limits the physical impact on the user during the interaction process. Highly accurate sensory reading of microexpressions, voice tone, and body movements is not provided due to inertial sensors – the analysis of affective information is mostly based on general audio- visual stream. In addition, the analogue does not involve predicting the emotional state before it occurs. expression and formation of physical actions in response, instead focuses on context-dependent semantic analysis. The system is software-adaptive, but does not provide real-time reactions from with a delay of no more than 150 ms. The utility model also implements neural network processing with using psychological techniques, which is not provided for in the specified analogue, which reduces its effectiveness in psychologically stressful scenarios, in particular during interviews (patent for invention US11074491B2, IPC G06N 3 / 006, publ. 07 / 27 / 2021). The well-known patent Hanson Robitics Inc US7113848B2 describes a system and methods for creating an artificial facial expressions for humanoid robots, in particular based on the use of flexible "artificial "skin" and its activation mechanisms, which ensure the expressiveness of the robot's face. In the device mechanical actuators located under an elastic coating are used to simulate the main human emotions, such as smile, surprise, sadness, etc. Scope of application - entertainment and service robots, educational systems, social companion robots. Specifications include controlled skin tension nodes that provide multiaxial deformation to create facial expressions expressions. The main disadvantages are low resolution of expressions, dependence on mechanics without deep sensory or cognitive processing, lack of adaptive behavior or emotion prediction, limited combination with psychological or neural network analysis models (patent pending) US7113848B2, IPC G16Z 99 / 00, publ. 26.09.2006). The well-known Nokia Technologies patent EP3709131A1 relates to a device comprising means for receiving data from an inertial measurement module (IMM) placed on the user's head (for example, in the form of headphones or earable), and the definition information about facial expressions based on this data. The system uses algorithms machine learning, including convolutional neural networks (CNN) and long-short-term networks memory (LSTM) or hidden Markov models (HMM), for analyzing data from a gyroscope or other IVMs. The device can control the functions of electronic devices based on detected facial expressions, in including generating feedback, recommendations for changing the task or the way it is done execution, or interpret facial expressions as a user command. Implementation is possible as in the form of portable electronic device, as well as as part of larger systems. The main disadvantages of the above device are the limited accuracy of recognizing complex or mixed emotions due to the use of only inertial sensors (e.g. gyroscopes) without combined analysis of video or audio information, which reduces the reliability of facial expression detection; Binding to the earable form factor limits the variability of applications in humanoid robots; and also the lack of a physical actuator or emotional feedback system that would allow for non- not only read, but also express or modulate emotions in response - which fundamentally distinguishes it from utility model with multimodal interaction and active influence (patent for invention EP3709131A1, IPC G06F 3 / 011, publ. 16.09.2020). The famous patent CN106650621A is an emotion recognition system based on deep learning contains modules for collecting a frontal image of an employee's face, processing the image to the format RGB 227×227, emotion recognition using pre-trained deep convolutional VGGNet neural networks (with 16 layers, 3×3 kernels, 2×2 pooling and initialization based on the VGG- Face), as well as an early warning module that analyzes the employee's historical emotional data and detects abnormal states among the seven basic emotions (joy, peace, sadness, anger, surprise, disgust, fear), with the possibility of generating signals of psychological intervention in case of repeated negative emotions. The main disadvantages of deep learning-based emotion recognition systems, such as described in the aforementioned patent, is: a narrow focus only on visual facial data without taking into account vocal, physiological, or behavioral signals, which limits the accuracy of the interpretation of affective states; over-reliance on pre-trained VGGNet, which may not take into account individual or cultural differences in emotional expression; lack of adaptive interaction in real-time — the system does not affect the user's emotional state, but only records and warns; not feedback in the form of voice, facial expressions or gestures is provided; the system does not have a humanoid physical form, which makes it impossible to use it effectively in interviews or therapy, where important non-verbal social signals (patent for invention CN106650621A, IPC G06V 40 / 172, publ. 10.05.2017). The well-known patent US8209182B2 describes the analysis of verbal and non-verbal signs using emotion ontologies. The patent describes a system and method for analyzing human emotional behavior by processing verbal and nonverbal signals using a multimodal approach. The system analyzes cues such as tone of voice, pauses, nonverbal responses, and uses emotional ontology to representing emotions as intensity gradients or mixed states. In technical The characteristics indicate the use of a database of emotional signs, noise filtering mechanisms, and tools for classifying emotions based on semantic and acoustic parameters. Scope of application includes distance learning systems, virtual agents, smart assistants, support services customers, stress management and adaptive interfaces. The limitation is the lack of physical interaction, the dependence of analysis accuracy on the quality of input signals, contextual ambiguity of interpretations, and also weak real-time adaptability without full integration with external behavioral feedback communication (patent for invention US8209182B2, IPC G10L17 / 26, published 06 / 26 / 2012). The closest analogue of a utility model is SONY CORP patent US7379871B2, which describes a system and method for generating speech in response to user behavioral events. The system comprises information processing computing module, message text generation module, databases dictionaries and emotional models, as well as a speech synthesis module with an audio output device. The device can interact with the user through input and output means that ensure the perception of user behavioral events and the formation of appropriate speech reactions system. Based on information about the user's personal characteristics (type, gender, age, temperament, etc.) the system generates the message text and synthesizes speech taking into account emotional state and the frequency of its changes. For this, mechanisms of word substitution are used from corresponding dictionaries and algorithms for modifying language parameters. Thus, the closest analogue contains a computing module for information processing, text message generation module, speech synthesis module and means of interaction with by the user through the input and output interface. The closest analogue has functional similarity to the utility model by the criterion personalized emotional response of a technical device in the process of interaction with a person. However, it has a number of significant technical differences that result in a different technical result. In particular, the closest analogue does not contain a high-resolution mimicry system, capable of physically reproducing subtle or mixed emotions; it lacks sensors to read microexpressions, head or body movements; no neural network processing of multimodal signals using psychological techniques; there are no means of predicting emotional state human to its manifestation and generation of stimulating behavior in response. In addition, it only implements language channel of influence on the user without the use of non-verbal means (facial expressions, postures, intonations) and does not adapt the interview to the socio-cultural or psychophysiological characteristics of the person (patent for invention US7379871B2, IPC G10L 13 / 00, publ. 27.05.2008). Thus, the utility model is characterized by a complex multimodal design, the presence of an electromechanical mimic system, a sensor module that includes video cameras, made based on CMOS (Complementary Metal-Oxide-Semiconductor) matrices, microphones and inertial sensors, and a computing module with a multi-core processor with a neural network that provide analysis of multimodal sensory data and coordinated control of facial expressions, voice and gesture interaction in real time, which allows for increased accuracy reproduction of facial expressions, increase the sensitivity of the sensor module and reduce reaction delay systems during interaction with a person. The basis of the utility model is the task of creating a human-like technical system capable of predict and stimulate a person's emotional reactions during an interview in real time. The task is solved by the fact that a humanoid technical system for forecasting and stimulation of a person's emotional reactions during an interview, containing a speech module built-in the computing module, according to the utility model, is made in the form of a body-shaped housing of a person, containing a head with a face, neck, shoulders, torso and limbs, with a facial system contains a set of electromechanical actuators with a resolution of at least 0.5 °, installed under layer of artificial facial skin, video cameras based on CMOS matrices with a resolution of resolution of at least 1080 p and a frame rate of at least 30 fps; microphones have a sensitivity of at least worse than minus 40 dB and a frequency range of 20-20000 Hz; speakers have a frequency range of 80-8000 Hz, installed in the head area and connected to the speech module; inertial sensors are implemented in the form of three-axis accelerometers and gyroscopes located in the neck and shoulder areas; computing module contains a multi-core processor with a neural network trained using estimation theory, theory affective control and analysis of the reliability of statements, while the computing module is designed to process multimodal data obtained from video cameras, microphones and inertial sensors, and the formation of control signals for the reproduction of facial expressions, voice and behavioral reactions in real time. According to the utility model, it is additionally equipped with an interface for entering biographical and contextual data of the interlocutor with a visual menu. The difference of the utility model is the combination of high-precision electromechanical drives of the mimic system, made in the form of a set of actuators located under a layer of artificial facial skin, with with a resolution of at least 0.5 °, video cameras based on CMOS matrices, microphones, inertial accelerometers, and a computing module that performs neural network processing multimodal signals using psychological techniques: Appraisal theory Theory) [1; 2] – as an algorithmic model for assessing the significance of stimuli based on input data about facial expressions, voice and behavior; Affect Control Theory [3] – as a mechanism building emotional feedback by comparing expected and actual behavior interviewer; Statement Validity Analysis [4] and content analysis by Criteria-Based Content Analysis [5; 6] – as modules of linguistic analysis of verbal statements for assessing credibility; Reality Monitoring [7] – as a subsystem detection of fictional facts based on context, sensory detail, and coherence; methods Cognitive Load Approaches [8] – as an assessment system cognitive load by analyzing pauses, speech rate, and nonverbal cues; Strategic Use of Evidence [9] – as an interview-navigation model that manages information disclosure to identify reactions to key facts. This allows not only to capture the emotional state of the interlocutor, but also to adaptively influence him through intonation, facial expressions, gestures, posture and verbal interaction. Additionally, the system can take into account biographical and cultural human characteristics for personalized customization of interview scenarios, which increases accuracy of emotional analysis and effectiveness of communication within the framework of psychological or profiling dialogue. The utility model consists in implementing a humanoid technical system comprising a body in the form of human body, containing head with face, neck, shoulders, torso and limbs, with facial expressions, module speech and sensor module and is designed with the ability to use the built-in computing module with a neural network to predict emotional state in real time a person even before his expression based on the analysis of microexpressions, voice timbre, head and body movements. The mimic system of the system is a set of electromechanical actuators located under a layer of artificial facial skin, which ensures the reproduction of facial movements by controlled movement of individual areas of artificial leather in accordance with control signals computing module. The system provides feedback to the person by reproducing the corresponding facial expressions. reactions, voice intonations, poses and gestures. At the same time, the entire interaction is recorded in the form of sensor data, analytical conclusions, control signals and corresponding reactions on the machine carriers. The system contains electromechanical drives with a resolution of at least 0.5 °, video cameras, made on the basis of CMOS matrices, microphones with a frequency range of 20-20000 Hz, speakers with frequency range 80-8000 Hz, three-axis inertial sensors and a computing module with digital input / output ports that ensure reliability and accuracy of operation, adaptability to human reactions, reducing reaction delay to 150 ms and the ability to influence emotional state person through facial expressions, intonation, posture, and gestures. The technical result consists in increasing the accuracy of reproducing facial expressions to within an error positioning of no more than 0.5° due to the use of electromechanical drives with appropriate angular resolution, reducing the system response delay to 150 ms by reducing signal processing time and transmission of control commands between the sensor module and actuators elements, as well as in increasing the sensitivity of the sensor module to human microreactions by recording changes with an amplitude corresponding to the minimum detected signals of video cameras, performed based on CMOS matrices, microphones and three-axis accelerometers. The specified technical result is achieved through the coordinated operation of sensory, computational and executive components, implementing real-time signal processing and minimizing data and control transmission delays influences. Sources of information: 1. Lazarus, RS Emotion and adaptation. Oxford University Press, 1991. 2. Scherer, KR "Appraisal considered as a process of multilevel sequential checking." In Appraisal processes in emotion theory, Oxford University Press, 2001. 3. Heise, DR Expressive Order: Confirming Sentiments in Social Actions. Springer, 2007. 4. Vrij, A. Detecting Lies and Deceit: Pitfalls and Opportunities. John Wiley & Sons, 2008. 5. Steller, M., Köhnken, G. "Criteria-based content analysis." In Raskin, D. C. (Ed.), Psychological methods in criminal investigation and evidence. Springer, 1989. 6. Vrij, A. "Criteria-based content analysis: A review of the evidence." Legal and Criminological Psychology, 10(1), 1–15, 2005. 7. Johnson, M. K., Raye, C. L. "Reality monitoring." Psychological Review, 88(1), 67–85, 1981. 8. Sweller, J. "Cognitive load during problem solving: Effects on learning." Cognitive Science, 12(2), 257– 285, 1988. 9. Walczyk, J. J., et al. "Lying person-to-person about life events: A cognitive framework for lie detection." Personnel Psychology, 58(1), 141–170, 2005.

Claims

1. A human-like technical system for predicting and stimulating emotional reactions a person during an interview, containing a speech module, a built-in computing module, which is distinguished by the fact that the system is made in the form of a housing in the shape of a human body, which contains the head with face, neck, shoulders, torso and limbs, while the facial system contains a set of electromechanical actuators with a resolution of at least 0.5°, installed under a layer of artificial facial skin, video cameras made on the basis of CMOS- matrices with a resolution of at least 1080 p and a frame rate of at least 30 fps; microphones have a sensitivity of no worse than minus 40 dB and a frequency range of 20-20,000 Hz; The speakers have a frequency range of 80-8000 Hz, are installed in the head area and connected to a speech module; inertial sensors are implemented in the form of three-axis accelerometers and gyroscopes located in the neck and shoulder areas; the computing module contains multi-core processor with a neural network trained using estimation theory, theory affective control and analysis of the reliability of statements, while computational the module is designed to process multimodal data received from video cameras, microphones and inertial sensors, and generating control signals for playback facial, vocal and behavioral reactions in real time.

2. The system according to claim 1, characterized in that it is additionally equipped with an input interface biographical and contextual data of the interlocutor with a visual menu.