A User State Inference System Based On Multimodal User Response Recognition

Abstract

A multimodal user response recognition-based user state inference system includes an input unit that receives a user's response signal in a multiple modal form, and a control unit that stores multiple judgment information for determining each user state of a multiple modal including a language modal, a voice modal, a visual modal, and a gaze and posture modal, determines the stability of each of the multiple modal inputs input to the input unit, calculates the usage reliability of each of the multiple modal inputs according to the judgment result, and infers the state of the user for the multiple modals based on the calculated usage reliability.

Description

Technology Field

[0001] The present invention relates to a user state inference system based on multimodal user response recognition, and more specifically, to a user state inference system based on multimodal user response recognition that infers the emotional, cognitive, and behavioral states of a user by analyzing multiple modal inputs, including language, voice, visual, and gaze and posture information, using an artificial intelligence model. Background Technology

[0003] With the recent advancements in artificial intelligence and sensor technologies, interest in user response recognition technology is increasing to facilitate more natural and efficient interaction between users and systems. In particular, fields such as user interfaces, virtual assistants, distance education, remote conferencing, healthcare, counseling systems, smart devices, and robotics require technologies that can identify users' emotions, concentration, cognitive load, and behavioral states in real time and respond adaptively.

[0004] Conventional user state recognition technologies have generally relied on determining the user's state based primarily on single-modal input. For example, methods have been used to infer emotions by analyzing the content or tone of speech, or to analyze facial expressions or gaze information using cameras. However, these single-modal-based recognition methods had limitations in accurately reflecting the actual user state, as they were vulnerable to external environmental factors such as ambient noise, lighting changes, facial occlusion, and missing speech, as well as changes in user behavior.

[0005] To address this, multimodal user recognition technology that utilizes a combination of language, voice, and visual information has been proposed; however, conventional multimodal technology also had the problem of failing to sufficiently consider the input quality or reliability of each modal. For example, even when a specific modal input is distorted due to noise, occlusion, or errors, it is fused with the same weight, which actually leads to a decrease in the accuracy of user state determination.

[0006] Furthermore, conventional technologies often judged the user's state at the level of simple emotion classification, which limited the ability to precisely distinguish cognitive and behavioral states such as concentration, cognitive load, avoidance, and defense. Consequently, there was a problem in that adaptive interface control based on user response or the provision of personalized services could not be sufficiently achieved.

[0007] Furthermore, techniques for evaluating the reliability of feature information extracted from various modal inputs and dynamically adjusting modal contributions accordingly to infer user states have not been sufficiently presented. As a result, it is currently difficult to achieve stable and reliable user state inference under imperfect input conditions that occur in actual usage environments.

[0008] Therefore, there is a need to develop a new user state recognition technology that comprehensively recognizes user responses based on multiple modal inputs, fuses them while considering the stability and reliability of each modal input, and thereby precisely infers the user's emotional, cognitive, and behavioral states. Prior art literature

[0010] Republic of Korea Registered Patent No. 10-2139855 Emotionally intelligent personal assistant system capable of conversing with a user, understanding their inner state, and establishing a close relationship (Registration Date: July 24, 2020) The problem to be solved

[0011] Therefore, the objective of the present invention is to provide a multimodal user response recognition-based user state inference system capable of precisely inferring detailed user states such as neutrality, attention, tension, cognitive load, avoidance, frustration, and defense by performing multimodal fusion by applying contribution weights based on usage reliability calculated based on features extracted from a plurality of modal inputs including language modal, speech modal, visual modal, and gaze and posture modal, and the stability of each modal. means of solving the problem

[0013] A user state inference system based on multimodal user response recognition according to the present invention for achieving the above objective comprises: an input unit that receives a user's response signal in the form of multiple modals; and a control unit that stores multiple judgment information for determining each user state of a plurality of modals including a language modal, a voice modal, a visual modal, and a gaze and posture modal, determines the stability of each of the plurality of modal inputs input to the input unit, calculates the reliability of use of each of the plurality of modal inputs according to the judgment result, and infers the state of the user for the plurality of modals based on the calculated reliability of use. Since the user state can be inferred comprehensively from the perspective of multiple modals rather than relying on a single modal, the reliability and accuracy of user state inference can be improved by more precisely reflecting subtle changes or mutually contradictory information inherent in the user response, and stable user state inference can be achieved even in the event of problems such as noise, temporary occlusion, or sensor errors by reducing the influence of modals with low input quality and selectively utilizing information from modals with high reliability.

[0014] Here, it is desirable that the control unit, by identifying user states regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense, enables high-level user state inference that includes cognitive and behavioral states beyond simple emotion classification, thereby allowing for the precise provision of user-customized responses in various application fields such as user interface adaptation, learning systems, and counseling systems.

[0015] Furthermore, it is desirable that the control unit extracts features from each modal input, including facial expression parameters, gaze vectors, speech rate, tone, energy, emotion keywords, intention expressions, posture angles, and movement frequencies used for determining the user state, so that the expressiveness of the information used for determining the user state is improved and the quantitative and qualitative information of individual modals can be richly reflected during user state inference, thereby enhancing the sensitivity and discriminability of state classification.

[0016] And the above control unit is desirable because it can calculate reliability that reflects modal characteristics by determining the stability of the language modal including sentence omission and utterance overlap, the stability of the speech modal including noise ratio and silent interval, the stability of the visual modal including face tracking stability and occlusion, and the stability of the gaze and posture modal including gaze retention rate and posture stability, thereby more accurately reflecting the quality of actual user response compared to a simple statistical reliability calculation method, and thus improving the reliability of overall user state inference.

[0017] Here, the control unit generates modal contribution weights in proportion to the usage reliability, and by performing multimodal fusion by applying the generated contribution weights to the usage reliability, adaptive fusion is possible that gives greater weight to the most reliable modal information. This allows for flexible response to quality deviations by modal due to environmental changes or user movements, thereby improving the consistency and accuracy of user state inference results, which is desirable.

[0018] Furthermore, the control unit is desirable because, by inferring user states regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense based on the performed multimodal fusion, it is possible to determine the user state that comprehensively reflects the reliability and characteristics of each modal, thereby providing a state inference result that has a high correlation with actual user behavior.

[0019] A user state inference method based on multimodal user response recognition according to the present invention for achieving the above objective comprises: a step of storing a plurality of judgment information for determining each user state of a plurality of modals including a language modal, a voice modal, a visual modal, and a gaze and posture modal; a step of determining the stability of each of the plurality of modal inputs and calculating the usage reliability of each of the plurality of modal inputs according to the judgment result; and a step of inferring the state of the user for the plurality of modals based on the calculated usage reliability information. Since the user state can be inferred comprehensively from the perspective of multiple modals rather than relying on a single modal, the reliability and accuracy of user state inference can be improved by more precisely reflecting subtle changes or mutually contradictory information inherent in user responses. Furthermore, by reducing the influence of modals with low input quality and selectively utilizing information from high-reliability modals, stable user state inference can be achieved even in the event of problems such as noise, temporary occlusion, and sensor errors.

[0020] Here, the step of inferring the state of the user for multiple modals includes the step of identifying the user state regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense; this is desirable because it enables a multidimensional interpretation of the user state in the method invention and allows for precise analysis of the user response.

[0021] Furthermore, the step of analyzing the aforementioned multiple modal inputs using an artificial intelligence model is desirable if it includes a step of extracting features from each modal input, such as facial expression parameters, gaze vectors, speech rate, tone, energy, emotion keywords, intention expressions, posture angles, and movement frequencies, which are used to determine the user state. This provides rich input features to the learning and inference models, thereby improving the expressiveness of the user state inference model and enabling more precise classification of complex emotional and cognitive states.

[0022] And the step of analyzing the above-mentioned multiple modal inputs using an artificial intelligence model is desirable if it further includes a step of determining the stability of the language modal, including sentence omission and utterance overlap; the stability of the speech modal, including noise ratio and silent intervals; the stability of the visual modal, including face tracking stability and occlusion; and the stability of the gaze and posture modal, including gaze retention rate and posture stability, so that the quality of the modal input can be evaluated more accurately and the misrecognition of the user state caused by incorrect input can be effectively prevented.

[0023] Here, the step of analyzing the multiple modal inputs using an artificial intelligence model is desirable if it further includes the step of generating contribution weights for each modal in proportion to the usage reliability and performing multimodal fusion by assigning the generated contribution weights to the usage reliability. This enables the implementation of a reliability-based dynamic fusion strategy at the method level, thereby minimizing distortion of the overall fusion result even in situations of temporary instability or information loss of a specific modal and maintaining the accuracy of user state inference.

[0024] Furthermore, the step of inferring the state of the user's multiple modals is desirable because inferring the user state regarding neutrality, focus, tension, cognitive load, avoidance, frustration, and defense based on the performed multimodal fusion allows for a clear definition of the causal relationship between the fusion step and the state inference step, thereby enabling high-reliability state classification that directly utilizes the multimodal fusion results, which can improve the explainability of the entire algorithm and the clarity of the system design. Effects of the invention

[0026] According to the present invention, the user's state can be comprehensively inferred from a multi-modal perspective without relying on a single modal, thereby more precisely reflecting subtle changes or mutually contradictory information inherent in user responses, which can improve the reliability and accuracy of user state inference. Furthermore, by reducing the influence of modals with low input quality and selectively utilizing information from high-reliability modals, stable user state inference is possible even in the face of problems such as noise, transient occlusion, and sensor errors.

[0027] Furthermore, by enabling high-dimensional user state inference that extends beyond simple emotion classification to include cognitive and behavioral states, it is possible to precisely provide customized responses in various application fields such as user interface adaptation, learning systems, and counseling systems. Additionally, since the expressiveness of information used for user state judgment is enhanced and quantitative and qualitative information of individual modals can be richly reflected during user state inference, there is an effect of improving the sensitivity and discriminability of state classification.

[0028] In addition, since it is possible to calculate reliability by reflecting modal characteristics, it can more accurately reflect the quality of actual user responses compared to simple statistical reliability calculation methods, thereby having the effect of improving the reliability of overall user state inference.

[0029] Furthermore, adaptive fusion is possible, which gives greater weight to the most reliable modal information. This allows for flexible responses to quality variations per modal caused by environmental changes or user movements, thereby improving the consistency and accuracy of user state inference results. Additionally, since it enables user state determination that comprehensively reflects the reliability and characteristics of each modal, it has the effect of providing state inference results that have a high correlation with actual user behavior.

[0030] In addition, it facilitates the application of the same algorithm across various hardware and software environments, improves the modularization and reusability of user state inference functions, and enables multidimensional interpretation of user states in method inventions, thereby allowing for precise analysis of user responses.

[0031] Furthermore, by providing rich input features to the learning and inference models, the expressiveness of the user state inference model is enhanced, and complex emotional and cognitive states can be classified more precisely. At the methodological level, variable reliability information based on input quality can be generated, allowing for flexible adjustment of modal weights and selection of fusion strategies in subsequent steps, thereby improving the fault tolerance and adaptability of the entire method.

[0032] In addition, it is possible to more accurately evaluate the quality of modal inputs, thereby effectively preventing misrecognition of user states caused by incorrect inputs. Furthermore, by implementing a reliability-based dynamic fusion strategy at the method level, it is possible to minimize distortion of the overall fusion result even in situations of temporary instability or information loss of specific modals, thus maintaining the accuracy of user state inference.

[0033] In addition, since the causal relationship between the fusion stage and the state inference stage can be clearly defined, high-reliability state classification that directly utilizes the multimodal fusion results becomes possible, which has the effect of improving the explainability of the entire algorithm and the clarity of the system design. Brief explanation of the drawing

[0035] Figure 1 is a configuration diagram of a multimodal user response recognition-based user state inference system according to the present invention. FIG. 2 is a first exemplary diagram of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system according to the present invention. FIG. 3 is a second exemplary diagram of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system. FIG. 4 is a third exemplary diagram of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system. FIG. 5 is a diagram illustrating a fourth embodiment of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system. Specific details for implementing the invention

[0036] Hereinafter, a multimodal user response recognition-based user state inference system (1) according to a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

[0037] FIG. 1 is a configuration diagram of a multimodal user response recognition-based user state inference system (1) according to the present invention.

[0038] Referring to FIG. 1, the configuration of a multimodal user response recognition-based user state inference system (1) is described.

[0039] A multimodal user response recognition-based user state inference system (1) includes an input unit (10), a display unit (20), a speaker (30), and a control unit (40).

[0040] The input unit (10) receives a user's response signal in a multiple modal form. The input unit (10) includes a communication unit (11), a camera (12), a microphone (13), and a user input unit (14).

[0041] The communication unit (11) can perform wireless communication, and the wireless communication includes at least one of infrared communication, RF, Zigbee, and Bluetooth. The communication unit (11) receives a video signal and transmits it to the control unit (40) to be described later, and can be implemented in various ways corresponding to the specifications of the received video signal and the implementation form of the user terminal. For example, the communication unit (11) can wirelessly receive an RF (radio frequency) signal transmitted from a broadcasting station (not shown), or receive a video signal according to composite video, component video, super video, SCART, HDMI (high definition multimedia interface) specifications, etc. via a wired connection. If the video signal is a broadcast signal, the communication unit (11) may include a tuner that tunes the broadcast signal by channel.

[0042] The camera (12) can capture the outside and the front, including the user.

[0043] The microphone (13) can receive all voices including the voice of the video, the voice of the application, and the voice of the user.

[0044] The user input unit (14) may be composed of an input means through which a user can input user commands. The user input unit (14) may receive a user's touch input or a user's remote input using a remote controller and transmit it to the corresponding control unit (40). Additionally, the user input unit (14) may receive voice input spoken by the user and transmit the voice signal to the control unit (40). In that case, the user input unit (14) may be implemented, for example, as a microphone. The user input unit (14) may also perform signal processing on the received voice signal itself. However, the form of user input that the user input unit (14) can receive is not limited to this, and user input such as motion recognition may also be received.

[0045] The display unit (20) displays an image based on an image signal processed by image processing. The implementation method of the display unit (20) is not limited and can be implemented in various display methods such as liquid crystal, plasma, light-emitting diode, organic light-emitting diode, surface conduction electron-emitter, carbon nanotube, nano-crystal, etc.

[0046] The display unit (20) may additionally include additional configurations depending on the implementation method. For example, if the display unit (20) is a liquid crystal type, the display unit (20) includes a liquid crystal display panel (not shown), a backlight unit (not shown) that supplies light thereto, and a panel driving board (not shown) that drives the panel (not shown). The display unit (20) may display a voice recognition result as information regarding the recognized voice. Here, the voice recognition result can be displayed in various forms such as text, graphics, and icons, and the text includes characters and numbers. The display unit (20) may further display candidate commands and application information based on the voice recognition result. The user can check whether the voice has been correctly recognized by the voice recognition result displayed on the display unit (20), and can select a command corresponding to the voice spoken by the user from among the displayed candidate commands by operating the user input unit (14) provided on the remote control, or select and check information related to the voice recognition result.

[0047] The speaker (30) can output a total of voices including the voice of the video, the voice of the application, and the voice of the user.

[0048] The control unit (40) stores multiple judgment information for determining each user state of a plurality of modals including a language modal, a voice modal, a visual modal, and a gaze and posture modal, determines the stability of each of the plurality of modal inputs input to the input unit (10), calculates the usage reliability of each of the plurality of modal inputs according to the judgment result, and infers the state of the user's plurality of modals based on the calculated usage reliability.

[0049] The control unit (40) can identify the user's state regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense.

[0050] The control unit (40) can extract features including facial expression parameters, gaze vectors, speech rate, tone, energy, emotion keywords, intention expressions, posture angles, and movement frequency from each modal input that are used to determine the user state.

[0051] The control unit (40) can determine the stability of the language modal including sentence omission and speech overlap, the stability of the voice modal including noise ratio and silent interval, the stability of the visual modal including face tracking stability and occlusion, and the stability of the gaze and posture modal including gaze retention rate and posture stability.

[0052] The control unit (40) generates a contribution weight for each modal in proportion to the reliability of use, and can perform multimodal fusion by assigning the generated contribution weight to the reliability of use.

[0053] The control unit (40) can infer user states regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense based on the multimodal fusion performed.

[0054] FIG. 2 is a first exemplary diagram of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system (1) according to the present invention.

[0055] Multiple judgment information is stored to determine each user state of a plurality of modals including language modals, voice modals, visual modals, and gaze and posture modals (S1).

[0056] The stability of each of the multiple modal inputs is determined, and the reliability of use of each of the multiple modal inputs is calculated based on the determination result (S2).

[0057] Based on the calculated usage reliability, the state of the user's multiple modals is inferred (S3).

[0058] FIG. 3 is a second exemplary diagram of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system (1).

[0059] Multiple judgment information is stored to determine each user state of a plurality of modals, including language modals, voice modals, visual modals, and gaze and posture modals (S11).

[0060] The stability of each of the multiple modal inputs is determined, and the reliability of use of each of the multiple modal inputs is calculated based on the determination result (S12).

[0061] Based on the calculated user reliability, user states regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense are identified and inferred (S13).

[0062] FIG. 4 is a third exemplary diagram of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system (1).

[0063] Multiple judgment information is stored to determine each user state of a plurality of modals, including language modals, voice modals, visual modals, and gaze and posture modals (S21).

[0064] Features including facial expression parameters, gaze vector, speech rate, tone, energy, emotion keywords, intention expression, posture angle and movement frequency are extracted from each modal input to be used for determining user state (S22).

[0065] The stability of each of the multiple modal inputs is determined, and the reliability of use of each of the multiple modal inputs is calculated based on the determination result (S23).

[0066] A contribution weight for each modal is generated in proportion to the usage reliability, and multimodal fusion is performed by assigning the generated contribution weight to the usage reliability (S24).

[0067] Based on the multimodal fusion performed, user states regarding neutrality, focus, tension, cognitive load, avoidance, frustration, and defense are inferred (S25).

[0068] FIG. 5 is a diagram illustrating a fourth embodiment of a multimodal user response recognition-based user state inference method using a multimodal user response recognition-based user state inference system (1).

[0069] Multiple judgment information is stored to determine each user state of a plurality of modals, including language modals, voice modals, visual modals, and gaze and posture modals (S31).

[0070] Features including facial expression parameters, gaze vector, speech rate, tone, energy, emotion keywords, intention expression, posture angle and movement frequency are extracted from each modal input to determine the user state (S32).

[0071] The stability of each of the multiple modal inputs is determined, and the reliability of use of each of the multiple modal inputs is calculated based on the determination result (S33).

[0072] The stability of the language modal, including sentence omission and speech overlap, the stability of the speech modal, including noise ratio and silent interval, the stability of the visual modal, including face tracking stability and occlusion, and the stability of the gaze and posture modal, including gaze retention rate and posture stability, are determined (S34).

[0073] A contribution weight for each modal is generated in proportion to the usage reliability, and multimodal fusion is performed by assigning the generated contribution weight to the usage reliability (S35).

[0074] Based on the multimodal fusion performed, user states regarding neutrality, focus, tension, cognitive load, avoidance, frustration, and defense are inferred (S36).

[0075] This section provides an overview of the technology for a multimodal user response recognition-based user state inference system.

[0076] 1) Recognition of verbal responses

[0077] - Real-time STT text conversion of user utterances → Extraction of linguistic features such as utterance length, frequency of repeated expressions, and proportion of emotion keywords at the utterance level, statistical and semantic hesitation expressions ("um...", "uh...").

[0078] 2) Speech response recognition technology

[0079] - Application of a Prosody Feature Extraction (SER) model to speech signals, including pitch, energy, speech rate, vibrato, and silence intervals

[0080] Normalize negative prosody features to convert them into relative tension and stability indicators.

[0081] 3) Facial expression recognition technology

[0082] - Tracks eye, eyebrow, mouth, and jaw movements using facial landmarks (FaceMesh)

[0083] - Parameter calculation based on tracked facial expressions → Calculation of facial expression state indicators such as tension, discomfort (grimacing), defensive reaction, and neutrality

[0084] 4) Posture behavior, gaze recognition technology

[0085] - Analyze upper body tilt, movement frequency, and inter-frame displacement; define behavioral indicators such as postural stability (movement distribution), frequent head movement, and shrinking.

[0086] - Estimate gaze direction based on eye landmarks and head pose

[0087] - Indicates whether the subject looks at the camera, frequency of gaze deviation, and duration of fixation.

[0088] 5) Development of Emotion and State Inference Technology Based on Multimodal Signal Fusion

[0089] - Integrates and analyzes speech, language, and visual response signals based on utterance units, and performs input quality-based reliability evaluation.

[0090] - Reliability is calculated per modal based on sentence omissions and breakage in STT results for Language, noise and silent interval ratios for Speech, and face tracking stability and occlusion for Visual → Underestimation of reliability is excluded

[0091] (Reliability calculation criteria based on modal input quality) Modal metewand Cases of reduced credibility language STT quality, sentence completeness Sentence omission, utterance overlap voice Noise-to-silence ratio, voice stability Ambient noise, speech interruptions Vision (face) Face tracking stability, occlusion Face displacement, occlusion Gaze and posture Eye gaze retention, postural stability Head bowing, body shaking

[0092] - Reflects reliability evaluation results and applies weighted multimodal fusion - Automatically reduces the influence of low-reliability modals and 7 types of states based on fusion results

[0093] User Response Metric Inference

[0094] - Inferred response indicators are provided as input information for the AI ​​Avatar Human response decision stage.

[0095] (Definition of classification criteria and meaning of user response states) User response status (Object of Recognition) Observation criteria (Multimodal basis) meaning neutrality Stable speech, maintaining eye contact No specific reaction Focus (Listen) Sustain speech, maintain gaze Immerse yourself in the question nervous Voice tremor, pitch rise Early stages of rising anxiety Cognitive Load Silence increase, hesitation Thought Identity evasion Eye averted, speech shortening Pressure avoidance Frustration (needs relief) Repetitive errors, facial stiffness Accumulation of performance failures defense Short answers, rigid posture Assessment Threat Recognition

[0096] Modifiable embodiments other than the above embodiments are described.

[0097] In a multimodal user response recognition-based user state inference system, the multiple modals are not limited to language modals, voice modals, visual modals, and gaze and posture modals, but may be configured to further include at least one of a biosignal modal, a gesture modal, a touch input modal, a wearable sensor modal, or an environment sensor modal. Additionally, user state inference based on the stability judgment and usage reliability calculation of each modal input can be performed in the same way. This allows for the collection of user responses from a more diverse range of perspectives, thereby improving the expressiveness and precision of user state inference. Furthermore, by maintaining an inference structure based on the stability judgment and usage reliability calculation of each modal input, it has the effect of enabling consistent user state inference without performance degradation due to modal expansion.

[0098] Artificial intelligence models can be implemented as deep learning-based neural networks, transformer-based models, recurrent neural networks, graph neural networks, or combinations thereof, and can be configured to apply a pre-trained model through fine-tuning or to be updated via online learning based on real-time user responses. This enables user state inference that can flexibly respond to changes in the application environment or user characteristics, and prevents degradation of inference performance and enables continuous performance improvement even during long-term use.

[0099] The feature extraction method for each modal can also be modified to utilize predefined rule-based feature extraction, statistical feature extraction, or automatic feature learning methods using artificial intelligence models, and different feature extraction models can be applied depending on the characteristics of each modal. Consequently, feature information optimized for the characteristics and data types of each modal can be secured, which has the effect of improving the reliability and discriminability of the information used for determining user status.

[0100] The method for determining the stability of modal input and calculating usage reliability can be implemented using calculation methods based on quantitative indicators, probability-based calculation methods, or prediction methods utilizing separate artificial intelligence models, and can be dynamically adjusted according to the usage environment, sensor performance, or user characteristics. This allows for the effective reflection of input quality variations caused by changes in the usage environment or deviations in sensor performance, thereby enabling stable and highly reliable user state inference even in actual usage environments.

[0101] Multimodal fusion can be performed using a weighted sum method, an attention-based fusion method, a hierarchical fusion method, or a sequential fusion method that considers chronological order, and the contribution weights for each modal can be configured to be updated statically or dynamically. This enables user state inference that effectively reflects temporal flow or correlations between modals, and by updating the contribution weights for each modal statically or dynamically, modals with high confidence are preferentially reflected in the inference results.

[0102] User states are not limited to neutral, focused, tense, cognitive load, avoidance, frustration, and defensive states, but can be expanded to include at least one additional state among fatigue, confusion, interest, immersion, stress, and anxiety. This enables a multidimensional interpretation of user responses, thereby improving the precision of user-customized interface control and adaptive service provision.

[0103] The user state inference system can be implemented independently on a single electronic device, as a distributed processing structure linked with a cloud server, or as a hybrid structure where edge devices and a central server cooperate. Accordingly, this enables efficient user state inference in various operating environments.

[0104] It can be implemented as a software program, a computer-readable recording medium, or a combination of hardware and software, and can be applied to various application fields such as educational systems, counseling systems, user interface control systems, robot control systems, and virtual or augmented reality systems. Accordingly, it can be easily applied to various systems and platforms, and its applicability to diverse fields significantly enhances industrial applicability.

[0105] The control unit can calculate the user state in the form of a probability distribution, and instead of determining the multimodal fusion result as a single state, it outputs a probability value vector for each of neutral / focus / tension / cognitive load / avoidance / frustration / defense, and the UI or subsequent control logic can operate based on the probability values.

[0106] The control unit can determine the trend of user state change, accumulate state inference results over a certain time window (e.g., 5 seconds to 30 seconds), and calculate the trend of increase / decrease / maintenance of the user state.

[0107] The control unit stores the user's average speech rate, gaze retention rate, and posture fluctuation range during the initial interaction period, and can infer the user's state based on the relative change amount relative to the individual rather than the absolute value when determining the state thereafter.

[0108] The control unit can determine a modal mismatch state when the language modal shows a positive signal and the visual / postural modal shows an avoidance signal, and can utilize the mismatch itself as an auxiliary basis for the defense or avoidance state.

[0109] The control unit can generate and provide explanatory data such as 40% voice modal contribution and 35% gaze modal contribution, and allow an administrator or higher-level AI to verify the basis for judgment.

[0110] The control unit can switch the modal weight calculation method according to the user state, and in normal times, all modals are considered equally, and when noise increases, it switches to decreasing speech modal weights and increasing visual / posture weights to utilize a situation-adaptive fusion strategy.

[0111] The control unit can subdivide the tension state into mild tension and high tension, and can output the result as a numeric scale or step code by reflecting the difference in intensity even within the same state.

[0112] The control unit can provide user state inference results to external systems by linking with a counseling system, an education platform, and a robot control system, and can output the status results in the form of an API or a message.

[0113] The control unit controls whether to collect additional modal samples based on the user state, reduces visual modal sampling in low-power mode when maintaining a focused state, and can increase the frequency of gaze and posture sampling when tension / avoidance is detected.

[0114] The control unit can separate and infer individual user states in a multi-user environment, and can infer states by separating modal streams for each user after visual modal-based user identification (ID). Accordingly, it can be applied to meeting, lecture, and consultation environments.

[0115] Due to the above-described multimodal user response recognition-based user state inference system (1) and multimodal user response recognition-based user state inference method, the user's state can be inferred comprehensively from a multimodal perspective without relying on a single modal, thereby more precisely reflecting subtle changes or mutually contradictory information inherent in the user response, and thus the reliability and accuracy of user state inference can be improved.

[0116] Furthermore, by enabling high-dimensional user state inference that goes beyond simple emotion classification to include cognitive and behavioral states, it is possible to precisely provide customized responses in various application fields such as user interface adaptation, learning systems, and counseling systems. Additionally, the expressiveness of information used for judging user states is enhanced, and quantitative and qualitative information of individual modals can be richly reflected during user state inference, thereby improving the sensitivity and discriminability of state classification.

[0117] In addition, by reducing the impact of modals with low input quality and selectively utilizing information from high-reliability modals, stable user state inference can be achieved even in the face of issues such as noise, transient occlusion, and sensor errors. Furthermore, since reliability can be calculated by reflecting the characteristics of each modal, it can more accurately reflect the quality of actual user responses compared to simple statistical reliability calculation methods, thereby improving the reliability of overall user state inference.

[0118] In addition, adaptive fusion that places greater weight on the most reliable modal information is possible, allowing for flexible responses to quality variations per modal caused by environmental changes or user movements, thereby improving the consistency and accuracy of user state inference results. Furthermore, since user state judgment that comprehensively reflects the reliability and characteristics of each modal is possible, it is possible to provide state inference results that have a high correlation with actual user behavior.

[0119] In addition, it facilitates the application of the same algorithm across various hardware and software environments, improves the modularization and reusability of user state inference functions, and enables multidimensional interpretation of user states in method inventions, thereby allowing for precise analysis of user responses.

[0120] In addition, rich input features can be provided to the learning and inference models, which enhances the expressiveness of the user state inference model and enables more precise classification of complex emotional and cognitive states. Furthermore, at the methodological level, variable reliability information based on input quality can be generated, allowing for flexible adjustment of modal weights and selection of fusion strategies in subsequent steps, thereby improving the fault tolerance and adaptability of the entire method.

[0121] In addition, the quality of modal input can be evaluated more accurately, effectively preventing misrecognition of user state caused by incorrect input, and reliability-based dynamic fusion strategies can be implemented at the method level, thereby minimizing distortion of the overall fusion result even in situations of temporary instability or information loss of specific modals and maintaining the accuracy of user state inference.

[0122] In addition, since the causal relationship between the fusion stage and the state inference stage can be clearly defined, high-reliability state classification that directly utilizes the multimodal fusion results becomes possible, which can improve the explainability of the entire algorithm and the clarity of the system design. Explanation of the symbols

[0124] 1: Multimodal User Response Recognition-Based User State Inference System 10: Input section 11: Communications Unit 12: Camera 13: Microphone 14: User input section 20: Display section 30: Speaker 40: Control unit

Claims

Claim 1 In a multimodal user response recognition-based user state inference system, an input unit that receives a user's response signal in a multimodal form; The system includes a control unit that stores multiple judgment information for determining each user state of a plurality of modals, including a language modal, a voice modal, a visual modal, and a gaze and posture modal; determines the stability of each of the plurality of modal inputs input through the input unit; calculates the usage reliability of each of the plurality of modal inputs according to the judgment result; and infers the state of the user for the plurality of modals based on the calculated usage reliability. The control unit identifies user states regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense; extracts features from each modal input that include facial expression parameters, gaze vectors, speech rate, tone, energy, emotion keywords, intention expressions, posture angles, and movement frequency used to determine the user state; determines the stability of the language modal, including sentence omission and speech overlap; the stability of the voice modal, including noise ratio and silent intervals; the stability of the visual modal, including face tracking stability and occlusion; and the stability of the gaze and posture modal, including gaze retention rate and posture stability. The system stores feature judgment information and stability judgment information corresponding to each user state of the plurality of modals. For the language modal, a language stability index including sentence omission and utterance overlap is calculated; for the speech modal, a speech stability index including noise ratio and silent intervals is calculated; for the visual modal, a visual stability index including face tracking stability and occlusion status is calculated; and for the gaze and posture modal, a gaze and posture stability index including gaze retention rate and posture stability is calculated; based on each stability index, the usability reliability of each modal input is calculated; multimodal fusion is performed by generating contribution weights for each modal according to the usability reliability; and based on the multimodal fusion result, neutral, attention, tension, cognitive load, avoidance,A multimodal user response recognition-based user state inference system characterized by inferring at least one user state among frustration and defense, calculating the user state in the form of a probability distribution, not determining the multimodal fusion result as a single state, outputting a probability value vector for each of neutral, focus, tension, cognitive load, avoidance, frustration, and defense, having the UI or subsequent control logic operate based on the probability value, determining the trend of user state change, accumulating the state inference results over a certain period of time, calculating the increase, decrease, and maintenance trends of the user state, storing the user's average speech rate, gaze retention rate, and posture fluctuation range during the initial interaction period, and inferring the user state based on the relative change amount relative to the individual rather than the absolute value when determining the state thereafter. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 A user state inference method based on multimodal user response recognition comprises: a step of storing multiple judgment information for determining each user state of a plurality of modals including a language modal, a voice modal, a visual modal, and a gaze and posture modal; a step of determining the stability of each of the plurality of modal inputs and calculating the usage reliability of each of the plurality of modal inputs according to the judgment result; The method includes a step of inferring the state of a user's multiple modals based on the usage reliability information calculated above, wherein the step of inferring the state of the user's multiple modals includes a step of identifying user states regarding neutrality, concentration, tension, cognitive load, avoidance, frustration, and defense, and the step of analyzing the input multiple modal inputs using an artificial intelligence model includes a step of extracting features from each modal input that include facial expression parameters, gaze vectors, speech rate, tone, energy, emotion keywords, intention expressions, posture angles, and movement frequency used for judging the user state, and the step of analyzing the input multiple modal inputs using an artificial intelligence model further includes a step of judging the stability of a language modal including sentence omissions and utterance overlaps, the stability of a speech modal including noise ratios and silent intervals, the stability of a visual modal including face tracking stability and occlusion status, and the stability of a gaze and posture modal including gaze retention rates and posture stability, wherein feature judgment information and stability judgment information corresponding to each of the user states of the multiple modals are stored, and for the language modal, a language stability index including sentence omissions and utterance overlaps Calculate a voice stability index including a noise ratio and a silent interval for the voice modal, calculate a visual stability index including face tracking stability and occlusion status for the visual modal, calculate a gaze and posture stability index including a gaze retention rate and posture stability for the gaze and posture modal, and calculate the reliability of use of each modal input based on each stability index.A multimodal user response recognition-based user state inference method characterized by generating contribution weights for each modal according to the above usage reliability to perform multimodal fusion, inferring at least one user state among neutral, focused, tense, cognitive load, avoidance, frustration, and defense based on the multimodal fusion result, calculating the user state in the form of a probability distribution, not determining the multimodal fusion result as a single state but outputting it as a probability value vector for each of neutral, focused, tense, cognitive load, avoidance, frustration, and defense, wherein the UI or subsequent control logic operates based on the probability value, determines the trend of user state change, accumulates the state inference result over a certain period of time, calculates the increase, decrease, and maintenance trends of the user state, stores the user's average speech rate, gaze retention rate, and posture fluctuation range during the initial interaction period, and infers the user state based on the relative change amount relative to the individual rather than the absolute value when determining the state thereafter. Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete

Images