Video-Based AI Interactive Human Response Generation System

Abstract

A video-based AI interactive human response generation system includes a state input unit for inputting user state information from the outside, and a control unit that stores multiple response type information for each of multiple user states and multiple response videos including facial expressions and attitudes of AI humans corresponding to the multiple response type information, determines a response type of an AI human for each of the user state information received by the state input unit based on the stored multiple response type information, and generates a response video of an AI human corresponding to the determined response type of an AI human.

Description

Technology Field

[0001] The present invention relates to a video-based AI interactive human response generation system, and more specifically, to a video-based AI interactive human response generation system that generates a response video with facial expressions, attitudes, and lip-syncing of an AI human applied based on user state information and response type information. Background Technology

[0003] With the recent advancement of artificial intelligence technology, user interaction technologies utilizing virtual characters, digital humans, and AI humans are being applied across various industries. In particular, there is a continuously increasing demand for video-based AI humans capable of naturally interacting with users in fields such as counseling, education, customer service, entertainment, and virtual reality environments.

[0004] Conventional video-based AI human technology primarily relies on responding to user input through pre-produced reaction videos or simple rule-based controls. While this approach offers the advantage of being relatively easy to implement, it has limitations in providing sophisticated and flexible responses to changes in the user's state or situation. In particular, when a user's emotions or response intensity change rapidly, there have been issues where the AI ​​human's facial expressions or attitudes shift unnaturally or repetitive responses appear.

[0005] In addition, existing technologies have limited definitions of AI human response types, making it difficult to accurately reflect various user states. Furthermore, during the generation of response videos, facial expressions, lip-syncing, and posture expressions were processed in a fragmented manner, leading to a decrease in the realism and immersion of the video. Moreover, generating high-quality response videos in real-time requires high computational resources, making it difficult to apply in real-time streaming environments or environments with limited hardware.

[0006] Meanwhile, technology for continuously improving AI human responses by reflecting user feedback or satisfaction has been relatively lacking, leading to concerns that user experience may stagnate or satisfaction may decline over long-term use.

[0007] Therefore, there is a need for the development of new technology that can systematically classify appropriate response types according to user status, generate high-quality AI human response videos in real-time environments through natural transitions of facial expressions and attitudes and efficient control of lip-syncing, and further improve response quality by reflecting user evaluations. Prior art literature

[0009] Korean Patent No. 10-2170563 Human-Machine Interactive Method and Apparatus Based on Artificial Intelligence (Registration Date: Oct. 21, 2020) The problem to be solved

[0010] Therefore, the objective of the present invention is to provide a video-based AI interactive human response generation system that can generate natural and immersive response videos even in a real-time environment by systematically determining the response type of an AI human according to user state information and efficiently controlling facial expression transitions and lip-syncing. means of solving the problem

[0012] A video-based AI interactive human response generation system according to the present invention for achieving the above objective comprises: a state input unit for inputting user state information from an external source; and a control unit that stores a plurality of response type information for each of a plurality of user states and a plurality of response images including facial expressions and attitudes of an AI human corresponding to the plurality of response type information, determines a response type of the AI ​​human for each of the user state information received by the state input unit based on the stored plurality of response type information, and generates a response image of the AI ​​human corresponding to the determined response type of the AI ​​human. Since it is possible to provide natural and consistent responses of the AI ​​human suitable for the user state, user immersion and interaction satisfaction can be improved compared to existing simple script-based responses.

[0013] Here, the system further includes a communication unit that communicates with an external device, and the control unit controls the communication unit to collect multiple situational images through a network. It is desirable that by using an artificial intelligence model to extract response images corresponding to multiple response type information from the multiple situational images collected, the data diversity and expressiveness for each response type are improved, and the cost and time for data construction can be reduced compared to the conventional method that relied on manual editing.

[0014] Furthermore, the control unit is desirable in that, by generating a plurality of similar response images based on response images corresponding to the extracted plurality of response type information, it can provide various modified responses even for the same response type, thereby preventing monotony in AI human responses and maintaining natural interaction even during repeated use.

[0015] Here, the control unit classifies the response types of the AI ​​human into seven basic states and three high-emotion responses, wherein the seven basic states are neutral, listening, calm, encouragement, empathy (low intensity), guidance and smile (low intensity) in calm, and the three high-emotion responses are joy, fear, and surprise when the user's response state changes rapidly. This allows for distinguishing and processing between everyday interactions and situations of rapid emotional change, thereby enabling the selective provision of emotional and behavioral responses of appropriate intensity according to the user's state, reducing over- or under-response, and making sophisticated emotional expressions appropriate to the context possible, which is desirable.

[0016] Furthermore, the control unit applies facial landmark-based expression intensity interpolation only during the transition interval when switching between basic states. In the case of rapid facial expression changes of high-emotional responses, it is desirable to interpolate parameters of major facial landmarks, including the eyes, eyebrows, mouth, and chin, in stages to simultaneously ensure naturalness and stability of the expression change, thereby minimizing rapid facial expression distortion or unnaturalness.

[0017] Here, the control unit applies viseme-based lip sync to achieve lightweight voice-mouth synchronization in the basic response state, selectively applies wav2Lip-based high-precision lip sync only in sections requiring high emotional response or emphasis, and constructs a real-time streaming output pipeline by synthesizing pre-produced video, transition section facial expression change results, and generative video in a layered manner, which is desirable as it can provide a response video with high visual and auditory naturalness in real time throughout the entire section while efficiently managing system resources and latency.

[0018] Furthermore, the control unit collects evaluation information regarding user satisfaction and reactions to the AI ​​human's reaction video, and by updating a plurality of reaction videos including the AI ​​human's facial expressions and attitudes corresponding to the plurality of reaction type information based on the evaluation information, continuous improvement of reaction quality based on user feedback becomes possible, which is desirable as it enables the implementation of an AI human reaction system that adapts to user preferences even during long-term use.

[0019] A method for generating a video-based AI interactive human response according to the present invention for achieving the above objective comprises: a step of storing a plurality of response images including a plurality of response type information for each of a plurality of user states and an expression and attitude of an AI human corresponding to the plurality of response type information; a step of determining a response type of the AI ​​human for each of the user state information received by the state input unit based on the stored plurality of response type information; and a step of generating a response image of the AI ​​human corresponding to the determined response type of the AI ​​human.

[0020] The process of generating video-based AI human responses can be implemented systematically and reproducibly, which can improve the consistency and reliability of the system implementation.

[0021] Here, the step of determining the response type of the AI ​​human includes: a step of collecting multiple situational videos through a network; and a step of extracting a response video corresponding to the multiple response type information from the collected multiple situational videos using an artificial intelligence model. This is desirable because it enables the determination of a response type that reflects various real-world situations, thereby allowing for a flexible and scalable determination of the response type that does not rely on predefined limited videos.

[0022] Furthermore, if the step of generating the reaction video of the AI ​​human includes the step of generating a plurality of similar reaction videos based on reaction videos corresponding to the extracted plurality of reaction type information, it is desirable to be able to derive various reaction results from limited original data, thereby enabling the construction of a rich reaction repertoire even with limited training and collection data, increasing data efficiency, and reducing monotony caused by repeated exposure in actual services.

[0023] Here, the step of determining the reaction type of the AI ​​human includes the step of classifying the reaction types of the AI ​​human into seven basic states and three high-emotion reactions, wherein the seven basic states are neutral, listening, calm, encouragement, empathy (low intensity), guidance from calm, and smile (low intensity), and the three high-emotion reactions are characterized as joy, fear, and surprise when the user's reaction state changes rapidly. This is desirable because it allows for a clearly defined reaction category system to provide explicit criteria for selecting which category of reaction to use in each situation, thereby preventing an excess or deficiency of emotional expression.

[0024] Furthermore, the step of generating the reaction video of the AI ​​human described above is desirable because it includes: a step of applying facial landmark-based expression intensity interpolation only in the transition section when transitioning between basic states; and a step of interpolating parameters of major facial landmarks, including the eyes, eyebrows, mouth, and chin, in stages in the case of rapid facial expression changes of high-emotion responses, thereby effectively controlling the continuity and naturalness of the facial expression changes and reducing the unnaturalness of the AI ​​human facial expression.

[0025] Here, the step of generating the reaction video of the AI ​​human includes: a step of synchronizing voice and lip movements by applying viseme-based lip sync in the basic reaction state; a step of selectively applying wav2Lip-based high-precision lip sync only in sections requiring high-emotional reactions or emphasis; a step of synthesizing the pre-produced video, the result of facial expression conversion in the transition section, and the generative video in a layered manner; and a step of constructing a real-time streaming output pipeline, which is desirable as it enables stable service provision even in various device and network environments.

[0026] Furthermore, the step of storing the plurality of reaction videos is desirable if it includes: a step of collecting evaluation information regarding user satisfaction and reactions to the AI ​​human's reaction videos; and a step of updating the plurality of reaction videos, including the facial expressions and attitudes of the AI ​​human corresponding to the plurality of reaction type information, based on the evaluation information, so that the AI ​​human's reactions can be continuously improved according to user experience, thereby enhancing user satisfaction and the long-term utility value of the system. Effects of the invention

[0028] According to the present invention, natural and consistent responses of an AI human suitable for the user's state can be provided, which has the effect of improving user immersion and interaction satisfaction compared to existing simple script-based responses.

[0029] In addition, data diversity and expressiveness by response type are enhanced, and data construction costs and time can be reduced compared to conventional methods that relied on manual editing. Furthermore, by providing various variations of responses even for the same response type, it is possible to prevent monotony in AI human responses and maintain natural interactions even during repeated use.

[0030] In addition, by distinguishing between everyday interactions and situations involving sudden emotional changes, it is possible to selectively provide emotional and behavioral responses of appropriate intensity according to the user's state, reduce over- or under-responses, enable sophisticated emotional expressions that are context-appropriate, and simultaneously ensure the naturalness and stability of facial expression changes, thereby minimizing sudden facial distortion or a sense of unnaturalness.

[0031] Furthermore, it is possible to provide real-time responsive videos with high visual and auditory naturalness across the entire section while efficiently managing system resources and latency, and continuous improvement of response quality based on user feedback enables the implementation of an AI human response system that adapts to user preferences even during long-term use.

[0032] According to the present invention, the process of generating image-based AI human responses can be implemented systematically and reproducibly, thereby improving the consistency and reliability of the system implementation.

[0033] Furthermore, it becomes possible to determine response types that reflect various real-world situations, enabling flexible and scalable response type determination that does not rely on predefined, limited images. Additionally, since diverse response results can be derived from limited original data, a rich response repertoire can be built even with limited training and collection data, thereby increasing data efficiency and reducing the monotony caused by repetitive exposure in actual services.

[0034] In addition, since the response category system is clearly defined, it can provide explicit criteria for selecting which category of response to use in different situations, thereby preventing excessive or insufficient emotional expression and effectively controlling the continuity and naturalness of facial expression changes, which has the effect of reducing the unnaturalness of AI human facial expressions.

[0035] Furthermore, stable service provision becomes possible across various devices and network environments, and since AI human responses can be continuously improved based on user experience, this has the effect of enhancing user satisfaction and the long-term utility value of the system. Brief explanation of the drawing

[0037] Figure 1 is a configuration diagram of an image-based AI interactive human response generation system according to the present invention. FIG. 2 is a first exemplary diagram of a method for generating an image-based AI interactive human response using an image-based AI interactive human response generation system according to the present invention. FIG. 3 is a second exemplary diagram of a method for generating video-based AI interactive human responses using a video-based AI interactive human response generation system. FIG. 4 is a third exemplary illustration of a method for generating video-based AI interactive human responses using a video-based AI interactive human response generation system. FIG. 5 is a diagram illustrating the fourth embodiment of a method for generating video-based AI interactive human responses using a video-based AI interactive human response generation system. Specific details for implementing the invention

[0038] Hereinafter, an image-based AI interactive human response generation system (1) according to a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

[0039] FIG. 1 is a configuration diagram of an image-based AI interactive human response generation system (1) according to the present invention.

[0040] Referring to FIG. 1, the configuration of the image-based AI interactive human response generation system (1) is explained.

[0041] The video-based AI interactive human response generation system (1) includes a state input unit (10), a communication unit (20), a camera (30), a microphone (40), a user input unit (50), a display unit (60), a speaker (70), and a control unit (80).

[0042] The status input unit (10) is configured to input user status information from the outside.

[0043] The communication unit (20) can communicate with an external device. The communication unit (20) can perform wireless communication, and the wireless communication includes at least one of infrared communication, RF, Zigbee, and Bluetooth. The communication unit (20) receives a video signal and transmits it to the control unit (80) to be described later, and can be implemented in various ways corresponding to the specifications of the received video signal and the implementation form of the user terminal. For example, the communication unit (20) can wirelessly receive an RF (radio frequency) signal transmitted from a broadcasting station (not shown), or receive a video signal according to composite video, component video, super video, SCART, HDMI (high definition multimedia interface) specifications, etc. via a wired connection. If the video signal is a broadcast signal, the communication unit (20) may include a tuner that tunes the broadcast signal by channel.

[0044] The camera (30) can photograph the outside and the front, including the user.

[0045] The microphone (40) can receive a whole of voices including the voice of the video, the voice of the application, and the voice of the user.

[0046] The user input unit (50) may be composed of an input means that allows a user to input user commands. The user input unit (50) may receive a user's touch input or a user's remote input using a remote controller and transmit it to the corresponding control unit (80). Additionally, the user input unit (50) may receive voice input spoken by the user and transmit the voice signal to the control unit (80). In that case, the user input unit (50) may be implemented, for example, as a microphone. The user input unit (50) may also perform signal processing on the received voice signal itself. However, the form of user input that the user input unit (50) can receive is not limited to this, and user input such as motion recognition may also be received.

[0047] The display unit (60) displays an image based on an image signal processed by image processing. The implementation method of the display unit (60) is not limited and can be implemented in various display methods such as liquid crystal, plasma, light-emitting diode, organic light-emitting diode, surface conduction electron-emitter, carbon nanotube, nanocrystal, etc.

[0048] The display unit (60) may additionally include additional configurations depending on the implementation method. For example, if the display unit (60) is a liquid crystal type, the display unit (60) includes a liquid crystal display panel (not shown), a backlight unit (not shown) that supplies light thereto, and a panel driving board (not shown) that drives the panel (not shown). The display unit (60) may display a voice recognition result as information regarding the recognized voice. Here, the voice recognition result can be displayed in various forms such as text, graphics, and icons, and the text includes characters and numbers. The display unit (60) may further display candidate commands and application information based on the voice recognition result. The user can check whether the voice has been correctly recognized by the voice recognition result displayed on the display unit (60), and can select a command corresponding to the voice spoken by the user from among the displayed candidate commands by operating the user input unit (50) provided on the remote control, or select and check information related to the voice recognition result.

[0049] The speaker (70) can output a total of voices including the voice of the video, the voice of the application, and the voice of the user.

[0050] The control unit (80) stores multiple reaction images including multiple reaction type information for each of multiple user states and facial expressions and attitudes of AI humans corresponding to the multiple reaction type information, determines the reaction type of an AI human for each user state information received by the state input unit based on the stored multiple reaction type information, and generates a reaction image of an AI human corresponding to the determined reaction type of an AI human.

[0051] The control unit (80) controls the communication unit (20) to collect multiple situational images through a network, and can extract reaction images corresponding to multiple reaction type information from the collected multiple situational images using an artificial intelligence model.

[0052] The control unit (80) can generate a plurality of similar reaction images based on reaction images corresponding to a plurality of extracted reaction type information.

[0053] The control unit (80) classifies the reaction types of the AI ​​human into 7 basic states and 3 high-emotion reactions, and the 7 basic states are neutral, listening, calm, encouragement, empathy (low intensity), guidance and smile (low intensity) from calm, and the 3 high-emotion reactions are cases where the user's reaction state changes rapidly, and can be characterized as joy, fear, and surprise.

[0054] The control unit (80) applies facial landmark-based expression intensity interpolation only during the transition interval when transitioning between basic states, and in the case of rapid facial expression changes of high emotional response, it can interpolate parameters of major facial landmarks including eyes, eyebrows, mouth and chin in stages.

[0055] The control unit (80) can apply a viseme-based lip sync in the basic response state to achieve lightweight voice-mouth synchronization, selectively apply a wav2Lip-based high-precision lip sync only in sections requiring high emotional response or emphasis, and can construct a real-time streaming output pipeline by synthesizing a pre-produced video, a transition section facial expression change result, and a generative video in a layered manner.

[0056] The control unit (80) collects evaluation information regarding user satisfaction and reactions to the AI ​​human's reaction video, and can update multiple reaction videos including the AI ​​human's facial expressions and attitudes corresponding to multiple reaction type information based on the evaluation information.

[0057] FIG. 2 is a first exemplary illustration of a method for generating an image-based AI interactive human response using an image-based AI interactive human response generation system (1) according to the present invention.

[0058] Multiple reaction videos are stored, each containing multiple reaction type information for each of the multiple user states and the facial expressions and attitudes of the AI ​​human corresponding to the multiple reaction type information (S1).

[0059] Based on the stored multiple response type information, the response type of the AI ​​human for each user state information received by the state input unit (10) is determined (S2).

[0060] A reaction video of the AI ​​human corresponding to the determined AI human reaction type is generated (S3).

[0061] FIG. 3 is a second exemplary illustration of a method for generating a video-based AI interactive human response using a video-based AI interactive human response generation system (1).

[0062] Multiple reaction videos are stored, each containing multiple reaction type information for each of the multiple user states and the facial expressions and attitudes of the AI ​​human corresponding to the multiple reaction type information (S11).

[0063] Multiple situational videos are collected through the network (S12).

[0064] Reaction videos corresponding to multiple reaction type information are extracted from multiple situation videos collected using an artificial intelligence model (S13).

[0065] Based on the stored multiple reaction type information, the reaction type of the AI ​​human for each extracted user state information is determined (S14).

[0066] A reaction video of the AI ​​human corresponding to the determined AI human reaction type is generated (S15).

[0067] Multiple similar reaction images are generated based on reaction images corresponding to the extracted multiple reaction type information (S16).

[0068] FIG. 4 is a third example of an AI interactive human response generation method using an AI interactive human response generation system (1).

[0069] Multiple reaction videos are stored, each containing multiple reaction type information for each of the multiple user states and the facial expressions and attitudes of the AI ​​human corresponding to the multiple reaction type information (S21).

[0070] Multiple situational videos are collected through the network (S22).

[0071] Reaction videos corresponding to multiple reaction type information are extracted from multiple situation videos collected using an artificial intelligence model (S23).

[0072] AI Human's response types are classified into 7 basic states, which are neutral, listening, calm, encouragement, empathy (low intensity), calm, guidance, and smile (low intensity), and 3 high-emotional responses, which are cases where the user's response state changes rapidly, which are joy, fear, and surprise (S24).

[0073] Based on the stored multiple reaction type information, the reaction type of the AI ​​human for each extracted user state information is determined (S25).

[0074] When transitioning between basic states, face landmark-based expression intensity interpolation is applied only in the transition section (S26).

[0075] In the case of rapid facial expression changes of high emotional response, parameters of major facial landmarks including eyes, eyebrows, mouth and jaw are interpolated stepwise (S27).

[0076] A reaction video of the AI ​​human corresponding to the determined AI human reaction type is generated (S28).

[0077] Multiple similar reaction images are generated based on reaction images corresponding to the extracted multiple reaction type information (S29).

[0078] FIG. 5 is a diagram illustrating a fourth embodiment of a method for generating a video-based AI interactive human response using a video-based AI interactive human response generation system (1).

[0079] Multiple reaction videos are stored, each containing multiple reaction type information for each of the multiple user states and the facial expressions and attitudes of the AI ​​human corresponding to the multiple reaction type information (S31).

[0080] User satisfaction and evaluation information regarding the response video of the AI ​​human are collected (S32).

[0081] Based on evaluation information, multiple reaction videos including the facial expressions and attitudes of the AI ​​human corresponding to multiple reaction type information are updated (S33).

[0082] Multiple situational videos are collected through the network (S34).

[0083] Reaction videos corresponding to multiple reaction type information are extracted from multiple situation videos collected using an artificial intelligence model (S35).

[0084] AI Human's response types are classified into 7 basic states, which are neutral, listening, calm, encouragement, empathy (low intensity), calm, guidance, and smile (low intensity), and 3 high-emotional responses, which are cases where the user's response state changes rapidly, which are joy, fear, and surprise (S36).

[0085] Based on the stored multiple reaction type information, the reaction type of the AI ​​human for each extracted user state information is determined (S37).

[0086] When transitioning between basic states, face landmark-based expression intensity interpolation is applied only in the transition section (S38).

[0087] In the case of rapid facial expression changes of high emotional response, parameters of major facial landmarks including eyes, eyebrows, mouth and jaw are interpolated stepwise (S39).

[0088] In the default response state, lightweight voice-lip shape synchronization is performed by applying viseme-based lip sync (S40).

[0089] wav2Lip-based high-precision lip-sync is selectively applied only to sections requiring high emotional response or emphasis (S41).

[0090] Pre-produced video, transition section expression change result, and generative video are composited in a layered manner (S42).

[0091] Build a real-time streaming output pipeline (S43).

[0092] A reaction video of the AI ​​human corresponding to the determined AI human reaction type is generated (S44).

[0093] Multiple similar reaction images are generated based on reaction images corresponding to the extracted multiple reaction type information (S45).

[0094] I will explain the overall technology of the video-based AI interactive human response generation system (1).

[0095] The video-based AI interactive human response generation system (1) is a technology for generating human responses centered on facial expressions and attitudes of an AI avatar interacting with a user, using user response state indicators derived from the multimodal response recognition and state inference stages as input. Avatar responses are defined as 7 basic states and 3 high-emotion responses, and can be output in real time by combining pre-produced video and limited generative video.

[0096] 1) Logic for determining response type

[0097] - Infers the user's response state through multimodal response recognition, and based on this, maps and generates 7 types of basic facial expression responses of the AI ​​avatar based on rules.

[0098] (User Response State-Based AI Avatar Default Response State Mapping Logic (Example)) User response status Avatar default response state Avatar facial expression explanation neutrality neutrality Staring at the user with a blank expression, without showing any emotion Focus (Listen) ▶ ear Expressing that you are listening by fixing your gaze and tilting your head slightly. nervous stability The purpose is to relieve tension with a soft facial expression where facial muscles are relaxed. Cognitive Load ▶ encouragement A positive reaction with a slight upward turn of the corners of the mouth and eyes evasion Empathy (low intensity) Expressing understanding and empathy through subtle changes around the eyebrows and mouth Frustration (needs relief) ▶ Stability → Guidance Maintain a calm expression, then transition to a guiding attitude to guide the next action. defense Smile (low intensity) An unexaggerated, natural, and subtle smile

[0099] - High-emotion responses (joy, fear, surprise) are restricted to event-based, one-off types only in cases where the user's response state changes abruptly based on the results of multimodal response recognition and state inference.

[0100] (Criteria for applying high-emotion responses and output conditions when user reactions change rapidly) Usage situation (context) User Response Key observed signals (multimodal) AI Reaction use Assignment success, correct answer to question, and positive feedback provided Focus (Listening) → Increased Positive Response Smile, positive keywords, stable voice tone delight Enhanced sense of accomplishment and increased immersion in interaction Immediately after presenting a difficult question Tension escalates Increased voice tremor, pitch rise, increased silence awe Induce a calming response after providing a signal recognizing the state of tension. Unexpected question / situation change Rapid increase in load Speechless speech, gaze averted, repetitive speech surprise Recognition of and Attention to Yellow Transition

[0101] 2) Construction of a library of pre-made videos of basic states (total of 56 types) To express the basic response states of the AI ​​avatars, a library of 7 types of pre-made facial expressions and attitudes, including neutral, listening, empathy, calm, clarification, guidance, and alertness, was constructed.

[0102] Eight low-intensity loop videos were produced for each state (a total of 56 videos) to maintain a natural response even during repeated playback.

[0103] To minimize visual discontinuity during state transitions, video transition metadata is tagged and linked with a real-time facial expression conversion module for the transition section.

[0104] 3) Application of high-emotion response generating videos (3 types)

[0105] Three types of high-emotion responses of AI avatars—joy (strong laughter), fear, and surprise—are implemented as short videos using deep learning-based generative video technology (GAN / Diffusion series).

[0106] By applying face landmark-based control and identity constraints, the facial contours and eye / mouth structures are maintained, and the range of facial expression deformation for fear responses is limited.

[0107] High emotional response is limited to event-based output within 30 to 90 frames at 30fps, and is designed to return to the default response state immediately after output.

[0108] 4) Development of a Real-time Facial Expression Transformation Module for Transition Sections

[0109] Developed a real-time expression transformation module that applies face landmark-based expression intensity interpolation only during the transition interval when switching between basic response states.

[0110] Minimizes the sense of unnaturalness caused by abrupt changes in facial expressions by progressively interpolating parameters of major facial landmarks (eyes, eyebrows, mouth, chin).

[0111] By limiting facial expression transformation operations to the transition interval, natural visual effects and real-time processing efficiency are simultaneously secured.

[0112] 5) Lip-sync, video compositing, and output

[0113] In the default response state, lightweight voice-lip synchronization is implemented by applying viseme-based lip sync.

[0114] Selectively apply wav2Lip-based high-precision lip-sync only to sections requiring high emotional response or emphasis.

[0115] Modifiable embodiments other than the above embodiments are described.

[0116] In a video-based AI interactive human response generation system, the state input unit may be configured to receive user state information indirectly by being linked with a user state inference system, an emotion recognition system, a biosignal analysis system, or a user behavior analysis system. This has the effect of enabling the generation of more precise and reliable user state-based responses without relying on a single recognition method.

[0117] Multiple response type information is not limited to predefined static classification information but can be stored separately to be configured differently by user group, service purpose, or usage environment, and can also be configured to be dynamically updated over time. This enables the configuration of response types suitable for the situation and user characteristics, thereby allowing for flexible system operation while maintaining response quality even during service expansion.

[0118] The artificial intelligence model can be implemented as a deep learning-based image generation model, a transformer-based time series model, a diffusion model, or a combination thereof, and can be configured to apply a pre-trained model by fine-tuning it or to continuously learn based on user response data. As a result, it can effectively adapt to changes in user response patterns or new situations, thereby improving the realism and adaptability of AI human responses even during long-term operation.

[0119] The method of generating the reaction video may be implemented as a generative video method that synthesizes facial expressions, gaze, posture, and lip-syncing in real time. In this case as well, the control of facial expression intensity and the selection and application of lip-syncing according to the reaction type can be performed in the same way. As a result, flexible reaction video generation becomes possible without relying on pre-produced videos, thereby enhancing real-time interaction effects that can immediately respond to changes in the user's state.

[0120] The reaction video generated by the control unit is not limited to photorealistic AI humans and can be implemented in various visual forms, such as 2D animated characters, 3D cartoon-style avatars, or custom characters set by the user. This has the effect of enhancing content diversity and user immersion.

[0121] The generated response video can be transformed to provide three-dimensional interaction by being output through a holographic device, augmented reality (AR) glasses, or virtual reality (VR) headset, in addition to a general display unit. This allows for a three-dimensional user experience that goes beyond flat interaction, thereby significantly increasing the potential for use in virtual and mixed reality environments.

[0122] The facial expression interpolation method can be replaced with muscle-based face models, blend-shape-based models, or neural network-based expression generation models, and the interpolation resolution or application range can be adjusted depending on whether a high-emotion response is present. This allows for situational control of the naturalness and stability of facial expressions, effectively minimizing facial distortion or unnatural transitions even during high-emotion responses.

[0123] The lip-sync method can be modified into a phoneme-based lip-sync, speech feature-based lip-sync, or generative lip-sync model, and can be configured to automatically switch depending on network conditions or computational resource status. This allows for simultaneous satisfaction of computational efficiency and expression quality, thereby enabling the provision of stable responsive video regardless of the network environment or device performance.

[0124] The system of the present invention may be implemented independently on a single device, as a distributed processing structure linked with a cloud server, or as a hybrid structure in which an edge device and a central server cooperate. Accordingly, flexible system design based on computational resources, latency, and security requirements becomes possible, thereby enhancing applicability to various service environments.

[0125] It can be implemented as a software program, a computer-readable recording medium, or a combination of hardware and software, and can be applied to various fields such as AI humans for counseling, virtual teachers for education, avatars for customer service, metaverse interaction characters, and virtual or augmented reality interfaces. Consequently, the versatility and industrial applicability of the technology are significantly enhanced, allowing it to be expanded into diverse service models rather than being limited to single-purpose systems.

[0126] In addition to the 7 basic states and 3 high-emotion responses, response types can be determined by adding specialized emotion categories such as 'educational immersion state', 'counseling empathy state', and 'medical pain perception state' depending on the application field. This allows for more precise reflection of the specific user states in each application field, thereby enabling the implementation of domain-specific AI human interaction.

[0127] The state input unit can be linked with a separate multimodal recognition system that goes beyond simple state information reception to infer the state by analyzing the user's language, voice, visual, and posture information in real time. This has the effect of improving the understanding of user responses compared to a simple state transmission method.

[0128] The state input unit can be configured to directly receive raw data acquired through a camera and microphone and analyze the user state in real time within the control unit. This reduces dependence on external systems and minimizes latency, thereby enabling the generation of fast and consistent user state-based responses in a real-time interaction environment.

[0129] Here, the control unit stores multiple judgment information for determining the user state of each of multiple modals, including language modals, voice modals, visual modals, and gaze and posture modals; analyzes multiple modal inputs received through the input unit using an artificial intelligence model; and can infer the user's state regarding the multiple modals based on the multiple judgment information. Since the user's state can be inferred comprehensively from the perspective of multiple modals rather than relying on a single modal, the reliability and accuracy of user state inference can be improved by more precisely reflecting subtle changes or mutually contradictory information inherent in user responses.

[0130] It is desirable for the control unit to identify user states regarding neutrality, focus, tension, cognitive load, avoidance, frustration, and defense, as this enables high-level user state inference that includes cognitive and behavioral states beyond simple emotion classification, thereby allowing for the precise provision of user-customized responses in various application fields such as user interface adaptation, learning systems, and counseling systems.

[0131] It is desirable for the control unit to extract features from each modal input that include facial expression parameters, gaze vectors, speech rate, tone, energy, emotion keywords, intention expressions, posture angles, and movement frequencies used for user state determination. This enhances the expressiveness of the information used for user state determination and allows for the rich reflection of quantitative and qualitative information from individual modals during user state inference, thereby improving the sensitivity and discriminability of state classification.

[0132] The control unit determines the stability of each of the multiple modal inputs and calculates the usage reliability of each based on the determination result. This allows for the reduction of the impact of modals with low input quality and the selective utilization of information from high-reliability modals, which is desirable as it enables stable user state inference even in the face of issues such as noise, transient occlusion, and sensor errors.

[0133] The control unit determines the stability of the language modal, including sentence omission and utterance overlap; the stability of the speech modal, including noise ratio and silent interval; the stability of the visual modal, including face tracking stability and occlusion; and the stability of the gaze and posture modal, including gaze retention rate and posture stability. This enables the calculation of reliability that reflects the characteristics of each modal, which can more accurately reflect the quality of actual user responses compared to a simple statistical reliability calculation method, thereby improving the reliability of overall user state inference, which is desirable.

[0134] The control unit generates contribution weights for each modal in proportion to the usage reliability. By applying these generated contribution weights to the usage reliability and performing multimodal fusion, adaptive fusion is possible that gives greater weight to the most reliable modal information. This allows for flexible response to quality deviations by modal due to environmental changes or user movements, thereby improving the consistency and accuracy of user state inference results, which is desirable.

[0135] It is desirable that the control unit, by inferring user states regarding neutrality, focus, tension, cognitive load, avoidance, frustration, and defense based on the performed multimodal fusion, enables a user state determination that comprehensively reflects the reliability and characteristics of each modal, thereby providing state inference results that have a high correlation with actual user behavior.

[0136] Due to the above-described video-based AI interactive human response generation system (1) and video-based AI interactive human response generation method, natural and consistent responses of AI humans suitable for the user's state can be provided, which has the effect of improving user immersion and interaction satisfaction compared to existing simple script-based responses.

[0137] In addition, data diversity and expressiveness by response type are enhanced, and data construction costs and time can be reduced compared to conventional methods that relied on manual editing. Furthermore, by providing various variations of responses even for the same response type, it is possible to prevent monotony in AI human responses and maintain natural interactions even during repeated use.

[0138] In addition, by distinguishing between everyday interactions and situations involving sudden emotional changes, it is possible to selectively provide emotional and behavioral responses of appropriate intensity according to the user's state, reduce over- or under-responses, enable sophisticated emotional expressions that are context-appropriate, and simultaneously ensure the naturalness and stability of facial expression changes, thereby minimizing sudden facial distortion or a sense of unnaturalness.

[0139] Furthermore, it is possible to provide real-time responsive videos with high visual and auditory naturalness across the entire section while efficiently managing system resources and latency, and continuous improvement of response quality based on user feedback enables the implementation of an AI human response system that adapts to user preferences even during long-term use.

[0140] According to the present invention, the process of generating image-based AI human responses can be implemented systematically and reproducibly, thereby improving the consistency and reliability of the system implementation.

[0141] Furthermore, it becomes possible to determine response types that reflect various real-world situations, enabling flexible and scalable response type determination that does not rely on predefined, limited images. Additionally, since diverse response results can be derived from limited original data, a rich response repertoire can be built even with limited training and collection data, thereby increasing data efficiency and reducing the monotony caused by repetitive exposure in actual services.

[0142] In addition, since the response category system is clearly defined, it can provide explicit criteria for selecting which category of response to use in different situations, thereby preventing excessive or insufficient emotional expression and effectively controlling the continuity and naturalness of facial expression changes, which has the effect of reducing the unnaturalness of AI human facial expressions.

[0143] Furthermore, stable service provision becomes possible across various devices and network environments, and since AI human responses can be continuously improved based on user experience, this has the effect of enhancing user satisfaction and the long-term utility value of the system. Explanation of the symbols

[0145] 1: Video-based AI Interactive Human Response Generation System 10: Status input section 20: Communications Department 30: Camera 40: Mike 50: User Input Section 60: Display section 70: Speaker 80: Control unit

Claims

Claim 1 A video-based AI interactive human response generation system comprises: a state input unit for inputting user state information from the outside; and a control unit that stores a plurality of response type information for each of a plurality of user states and a plurality of response videos including facial expressions and attitudes of an AI human corresponding to the plurality of response type information, determines a response type of the AI ​​human for each of the user state information received by the state input unit based on the stored plurality of response type information, and generates a response video of the AI ​​human corresponding to the determined response type of the AI ​​human, wherein the control unit applies viseme-based lip sync to achieve lightweight voice-lip shape synchronization in a basic response state, selectively applies wav2Lip-based high-precision lip sync only in sections requiring high-emotional responses or emphasis, and constructs a real-time streaming output pipeline by synthesizing a pre-produced video, a transition section facial expression conversion result, and a generative video in a layered manner. Claim 2 An image-based AI interactive human response generation system according to claim 1, further comprising a communication unit that communicates with an external device, wherein the control unit controls the communication unit to collect a plurality of situational images through a network, and extracts a response image corresponding to a plurality of response type information from the collected plurality of situational images using an artificial intelligence model. Claim 3 In paragraph 2, the image-based AI interactive human response generation system is characterized in that the control unit generates a plurality of similar response images based on a response image corresponding to the extracted plurality of response type information. Claim 4 In paragraph 3, the control unit classifies the reaction types of the AI ​​human into seven basic states and three high-emotion reactions, wherein the seven basic states are neutral, listening, calm, encouragement, empathy (low intensity), guidance from calm, and smile (low intensity), and the three high-emotion reactions are cases where the user's reaction state changes rapidly, and are characterized as joy, fear, and surprise, in a video-based AI interactive human reaction generation system. Claim 5 In claim 4, the control unit is characterized by applying face landmark-based expression intensity interpolation only during the transition interval when transitioning between basic states, and in the case of rapid expression changes of high-emotional responses, interpolating parameters of major face landmarks including eyes, eyebrows, mouth, and chin in stages, in an image-based AI interactive human response generation system. Claim 6 delete Claim 7 A video-based AI interactive human response generation system according to claim 1, wherein the control unit collects evaluation information regarding user satisfaction and response to the response video of the AI ​​human, and updates a plurality of response videos including facial expressions and attitudes of the AI ​​human corresponding to the plurality of response type information based on the evaluation information. Claim 8 A method for generating a video-based AI interactive human response, comprising: a step of storing a plurality of response videos including a plurality of response type information for each of a plurality of user states and an expression and attitude of the AI ​​human corresponding to the plurality of response type information; a step of determining a response type of the AI ​​human for each of the user state information received by a state input unit based on the stored plurality of response type information; and a step of generating a response video of the AI ​​human corresponding to the determined response type of the AI ​​human, wherein the step of generating the response video of the AI ​​human comprises: a step of synchronizing voice and lip shape by applying viseme-based lip sync in a basic response state to achieve lightweight voice-lip shape synchronization; a step of selectively applying wav2Lip-based high-precision lip sync only in sections requiring high-emotional response or emphasis; a step of synthesizing a pre-produced video, a transition section expression conversion result, and a generative video in a layered manner; and a step of constructing a real-time streaming output pipeline. Claim 9 A method for generating video-based AI interactive human responses, characterized in that, in claim 8, the step of determining the response type of the AI ​​human comprises: a step of collecting a plurality of situational videos through a network; and a step of extracting a response video corresponding to the plurality of response type information from the collected plurality of situational videos using an artificial intelligence model. Claim 10 A method for generating video-based AI interactive human responses, characterized in that, in claim 9, the step of generating a response video of the AI ​​human comprises the step of generating a plurality of similar response videos based on a response video corresponding to the extracted plurality of response type information. Claim 11 In claim 10, the step of determining the reaction type of the AI ​​human includes the step of classifying the reaction type of the AI ​​human into seven basic states and three high-emotion reactions, wherein the seven basic states are neutral, listening, calm, encouragement, empathy (low intensity), guidance from calm, and smile (low intensity), and the three high-emotion reactions are cases where the user's reaction state changes rapidly, characterized as joy, fear, and surprise, a video-based AI interactive human reaction generation method. Claim 12 A method for generating an image-based AI interactive human response according to claim 11, wherein the step of generating a response video of the AI ​​human comprises: a step of applying facial landmark-based expression intensity interpolation only in the transition section when transitioning between basic states; and a step of interpolating parameters of major facial landmarks including eyes, eyebrows, mouth, and chin in stages in the case of rapid facial expression changes of high-emotional responses. Claim 13 delete Claim 14 A method for generating video-based AI interactive human responses, characterized in that, in claim 8, the step of storing the plurality of response videos comprises: a step of collecting evaluation information regarding user satisfaction and response to the AI ​​human’s response videos; and a step of updating the plurality of response videos, including the facial expressions and attitudes of the AI ​​human corresponding to the plurality of response type information, based on the evaluation information.

Images