Customer consultation support system and method using customized virtual consultation avatars based on user profiles

Abstract

The present invention comprises: a profile input unit for collecting static profile data and dynamic emotion data of a user; a customer consultation avatar generation unit for generating a user-customized customer consultation avatar by applying the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library; a consultation agent unit for generating a consultation response to a user's query using LLM and reflecting real-time data provided from an external data source and a domain-specific API into the consultation response; an emotion recognition and classification unit for recognizing and classifying the user's emotional state by analyzing the user's dynamic emotion data obtained during the consultation process; and an avatar behavior control unit for controlling the avatar's voice synthesizer, facial expression renderer, and gesture animator so that the consultation response is expressed with the voice tone, facial expression, and gesture movements of the customer consultation avatar suitable for the user's emotional state.

Description

Technology Field

[0001] The present invention relates to a customer consultation support system and method using a user profile-based customized customer consultation avatar. Background Technology

[0002] Conventional banking consultation systems rely primarily on simple text-based chatbots or standardized consultation scenarios that follow predefined conversation flows. These systems can only perform fixed FAQ answers or a limited scope of tasks (e.g., checking account balances, providing simple product information), which limits their ability to provide personalized consultations that reflect the individual characteristics, situational context, and emotional state of each customer.

[0003] For example, elderly customers require age-friendly interfaces such as enlarged font sizes, slow explanation speeds, and a friendly tone, but existing systems are unable to flexibly support this. Conversely, digitally savvy young customers prefer quick and intuitive guidance and modernly designed interfaces, but current consultation systems are limited to providing a single UI and a uniform conversational tone.

[0004] Furthermore, while Customer Experience (CEAT) has emerged as a core element of service competitiveness in the recent financial environment, existing chatbot-based consultations fail to recognize non-verbal cues such as voice tone, facial expressions, eye contact, and gestures. Consequently, it is virtually impossible to detect and respond to customer complaints or confusion in real time. This leads to issues such as decreased customer satisfaction during the consultation process or unnecessary delays in connecting to actual agents.

[0005] Furthermore, although banking consultation services must satisfy personal information protection, financial transaction security, and international compliance (GDPR, PCI-DSS, domestic electronic financial transaction laws, etc.), existing technologies lack sufficient built-in features for real-time anonymization, sentiment analysis, and conversation log management to ensure data protection and regulatory compliance. This acts as a significant constraint in the process of financial institutions driving digital transformation.

[0006] Therefore, the limitations of existing counseling systems can be summarized as a lack of customer individualization, failure to support emotional responses, and vulnerabilities in security and compliance, requiring new technological approaches to improve them. Prior art literature

[0007] Registered Patent Publication No. 10-1951761 The problem to be solved

[0008] The objective of the present invention is to provide a customer consultation support system and method using a user profile-based customized customer consultation avatar that can solve conventional problems. means of solving the problem

[0009] A customer consultation support system using a user profile-based customized customer consultation avatar according to an embodiment of the present invention for solving the above problem comprises: a profile input unit that collects static profile data and dynamic emotion data of a user; a customer consultation avatar generation unit that generates a customized customer consultation avatar by applying the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library; a consultation agent unit that generates a consultation response to a user's query using LLM and reflects real-time data provided from an external data source and a domain-specific API into the consultation response; an emotion recognition and classification unit that analyzes the user's dynamic emotion data obtained during the consultation process to recognize and classify the user's emotional state; and an avatar behavior control unit that controls the avatar's voice synthesizer, facial expression renderer, and gesture animator so that the consultation response is expressed with the voice tone, facial expression, and gesture movements of the customer consultation avatar suitable for the user's emotional state. and includes a connection assistance mode control unit that controls whether to connect a call between an actual agent and a user according to the level of user's requirements, the complexity of the query, or the difficulty of the business scenario during the consultation process, switches the customer consultation avatar to a standby mode or an assistance mode after the call connection is completed, and supports the actual agent in performing visual consultation assistance when the customer consultation avatar is switched to the standby mode.

[0010] A customer consultation support method using a user profile-based customized customer consultation avatar according to an embodiment of the present invention for solving the above problem comprises: a step of collecting static profile data and dynamic emotion data of a user in a profile input unit; a step of generating a customer consultation avatar customized to the user by applying the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library in a customer consultation avatar generation unit; a step of generating a consultation response to the user's query content using a large-scale language model (LLM) in a consultation agent unit and reflecting real-time data provided from an external data source and a domain-specific API in the consultation response; a step of recognizing and classifying the user's emotional state by analyzing the user's dynamic emotion data obtained during the consultation process in an emotion recognition and classification unit; and a step of controlling a speech synthesizer, an expression renderer, and a gesture animator of the customer consultation avatar in an avatar behavior control unit so that the consultation response is expressed in the customer consultation avatar with a voice tone, facial expression, and gesture movement suitable for the classified emotional state. The method includes the step of controlling whether to connect a call between an actual agent and a user based on the level of user's requirements, the complexity of the query, or the difficulty of the business scenario during the consultation process, switching the customer consultation avatar to a standby mode or an auxiliary mode after the call connection is completed, and, when the customer consultation avatar is switched to the standby mode, supporting the actual agent to perform visual consultation assistance. Effects of the invention

[0011] According to the present invention, there is an advantage in that a user-customized customer consultation avatar can be automatically generated by comprehensively reflecting not only static profile data such as the user's age, gender, cultural background, and preferences, but also dynamic emotional data such as voice tone, facial expressions, and eye gaze patterns that change in real time during the consultation process.

[0012] Through this, users can receive a personalized counseling experience that reflects human empathy and consideration, rather than the existing mechanical and uniform conversational agent services. For instance, avatars with clear pronunciation and slow explanation speeds are provided to the elderly, while avatars with modern appearances and fast response speeds are provided to young users, thereby maximizing satisfaction by generation and group.

[0013] In addition, the present invention has the advantage of allowing a customer consultation avatar to handle repetitive and standardized preliminary tasks, such as providing information, verifying documents, and answering basic questions at the beginning of the consultation, while enabling immediate connection to an actual agent or manager in complex or specialized situations.

[0014] Even upon connection, the customer service avatar switches to standby mode to provide visual and non-verbal assistance on-screen, such as pointing at documents, highlighting items, and guiding procedures. This allows actual agents to focus on core decision-making or specialized tasks, while enabling users to experience a seamless and natural service flow. Consequently, this reduces consultation processing times across various service environments and enhances user satisfaction even in non-face-to-face interactions.

[0015] Furthermore, the present invention has the advantage of providing a real-time emotion-behavior mapping algorithm that combines a large-scale language model (LLM)-based counseling engine and a deep learning-based emotion classification model to immediately reflect conversational context and gesture responses associated with the user's emotional state by adjusting LLM prompts in real time.

[0016] For example, if a user voices dissatisfaction, the customer service avatar immediately responds with a smiling expression and a calm tone, avoiding unnecessary arguments. This human-friendly interaction provides a distinctive technological effect that is difficult to achieve with existing simple text-based consultation systems. Furthermore, this technology has the potential for expanded application in various fields, including finance, customer service, healthcare, education, public services, and metaverse-based remote support.

[0017] In addition, the present invention has the advantage of satisfying international data protection standards, such as the Personal Information Protection Act, industry-specific security regulations, and GDPR, by anonymizing and encrypting user data (profile, voice, facial expressions, consultation records, etc.) collected during the consultation process in real time.

[0018] In addition, all consultation processes are stored along with log records and metadata for auditing, which can meet the compliance requirements of relevant supervisory authorities or operational managers, thereby ensuring a level of reliability and legal compliance applicable in actual service environments.

[0019] The present invention has the potential to be applied in various industrial sectors, such as the financial sector, service industry, healthcare, and public services.

[0020] For example, in the field of financial consulting, customized financial avatars that reflect customers' profile data and real-time emotional states can provide specialized advice on loans, investments, and account management, thereby ensuring both regulatory compliance and customer satisfaction.

[0021] In the service industry and customer center response sectors, the customer experience can be improved by reflecting customers' verbal and non-verbal signals, providing response avatars focused on reassurance and apology to dissatisfied customers and avatars focused on guidance and recommendations to new customers.

[0022] In addition, in the healthcare field, virtual health coordinator avatars reflecting the patient's age, health status, and emotional changes can be created to perform functions such as medication guidance, lifestyle management, and psychological counseling.

[0023] Furthermore, in the public service sector, administrative efficiency and public satisfaction can be enhanced by providing avatars optimized for the user's background and emotional state in areas such as civil complaint processing, administrative consultation, and education and training environments.

[0024] Consequently, since the present invention enables the implementation of an intelligent consultation support platform that is regulation-friendly and applicable across various industries, it has the advantage of significantly enhancing technical and commercial value and has a very high potential for commercialization in diverse industrial fields. Brief explanation of the drawing

[0025] FIG. 1 is a device configuration diagram of a customer consultation support system using a user profile-based customized customer consultation avatar according to one embodiment of the present invention. Figure 2 is a detailed configuration diagram of the consultation agent unit shown in Figure 1. FIG. 3 is a flowchart illustrating a customer consultation support method using a user profile-based customized customer consultation avatar according to an embodiment of the present invention. Figure 4 is a detailed flowchart of the S750 process illustrated in Figure 3. Figure 5 is an example diagram illustrating the process of supporting financial business (loan contract procedure) for a customer through a customer consultation avatar. Specific details for implementing the invention

[0026] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0027] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them. Furthermore, when a part is described as "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components, and it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0028] Terms such as “about,” “substantially,” etc., used throughout the specification, are used to mean at or near the stated value when inherent manufacturing and material tolerances are presented in the stated meaning, and are used to prevent unscrupulous infringers from unfairly exploiting the disclosure in which precise or absolute values ​​are mentioned to aid in understanding the invention. Terms such as “step” or “step of” used throughout the specification of the invention do not mean “step for”.

[0029] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, "part" is not limited to software or hardware, and "part" may be configured to reside in an addressable storage medium or configured to run on one or more processors. Accordingly, as an example, "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." In addition, the components and '~parts' may be implemented to play one or more CPUs within the device or secure multimedia card.

[0030] Some of the operations or functions described herein as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server. In this specification, some of the operations or functions described as mapping or matching with a terminal may be interpreted to mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data.

[0032] Hereinafter, a customer consultation support system and method using a user profile-based customized customer consultation avatar according to an embodiment of the present invention will be described in more detail based on the attached drawings.

[0033] FIG. 1 is a device configuration diagram of a customer consultation support system using a user profile-based customized customer consultation avatar according to one embodiment of the present invention, and FIG. 2 is a detailed configuration diagram of the consultation agent unit shown in FIG. 1.

[0034] As illustrated in FIG. 1, a customer consultation support system (100) using a user profile-based customized customer consultation avatar according to one embodiment of the present invention includes a profile input unit (110), a customer consultation avatar creation unit (120), a customer consultation agent unit (130), an emotion recognition and classification unit (140), an avatar behavior control unit (150), a connection / assistance mode control unit (160), and a compliance management unit (170).

[0035] The profile input section (110) may be configured to collect data regarding the user's demographic information (e.g., age, gender, cultural background, linguistic characteristics) and service usage preferences (e.g., consultation method, visual interface preference, conversation speed, interaction channel).

[0036] The above profile input unit (110) can be processed through encryption and anonymization procedures during the data collection process to ensure the privacy of personal information and sensitive data.

[0037] Additionally, the profile input section (110) can be linked with data provided by a user terminal, a corporate or institutional server, or an external trusted institution, and may be configured to extensively collect and integrate digital behavioral data of the user (e.g., online service usage patterns, access time zones, preferred service / product types, learning / consumption records) and bio-based auxiliary data (e.g., voice characteristics, fine changes in facial expressions, gestures, bio-signals through wearable devices) as needed.

[0038] This expanded data can go beyond a simple basic profile level and be reflected in the visual representation (appearance, facial expressions, attire) of customer consultation avatars or agents, conversation scenarios (explanation style, level of information provided, language choice), and emotional response modeling (expressions of empathy, calming and encouraging gestures), thereby further enhancing the level of personalization.

[0039] Next, the customer consultation avatar generation unit (120) may be configured to generate a user-customized customer consultation avatar by applying the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library.

[0040] More specifically, the customer consultation avatar generation unit (120) receives user profile data collected and preprocessed from the profile input unit (110) as input and dynamically generates a user-customized virtual consultation avatar through an avatar model generation engine. The generated customer consultation avatar may be configured to express the voice tone, facial expression, gesture, clothing style, and interface language of the customer consultation avatar in a personalized manner by reflecting not only static data such as the user's age, gender, cultural background, linguistic characteristics, and service usage preference, but also dynamic emotional data such as the customer's voice tone, facial expression, eye gaze pattern, and biosignal that change in real time during the customer consultation process.

[0041] The customer consultation avatar generation unit (120) can be linked with a pre-trained 3D human model library and texture mapping engine, and can integrate a face synthesis module (Face Rendering Engine), a speech synthesizer (Text-to-Speech Engine with Emotion Control), an expression renderer (Expression Renderer), and a gesture animator (Motion Capture-based Animator) as needed to implement a multimodal avatar suitable for consultation scenarios and user characteristics in real time.

[0042] Furthermore, the generated customer consultation avatars can be customized to meet the requirements of various domains, such as finance, services, healthcare, and public services, and can be designed to maintain consistency with the brand identity (BI) of the organization or service provider.

[0043] In addition, the customer consultation avatar generation unit (120) can periodically update and improve the voice, facial expression, and motion expression methods of the avatar based on user feedback and consultation logs, thereby having a self-adaptive and continuous evolutionary structure.

[0044] Next, the consultation agent unit (130) may be configured to generate a consultation response to a user's query using LLM and to reflect real-time data provided from an external data source and a domain-specific API in the consultation response.

[0045] That is, the consultation agent unit (130) is operated based on a large-scale language model (LLM) and is configured to handle consultation, explanation, and guidance tasks in various fields such as finance, services, healthcare, education, and public administration according to the user's input query, and may include an LLM consultation unit (131), a data linkage unit (132), a regulation and policy verification unit (133), a knowledge graph and domain-specific RAG unit (134), a multilingual and multimodal processing unit (135), and a speech recognition and synthesis unit (136).

[0046] The LLM consultation unit (131) is configured to receive a user's natural language query and generate a consultation response. It does not stop at generating a simple text-based response, but rather integrates real-time data provided by the data linkage unit (132), regulatory conditions and policy rules provided by the regulation and policy verification unit (133), and knowledge graph query results provided by the knowledge graph and domain-specific RAG unit (134) to produce a consultation response that ensures contextual consistency and regulatory compliance.

[0047] More specifically, the LLM consultation unit (131) parses and semantically interprets the input query sentence to extract the core intent and constraints of the query, and then maps them to real-time attribute values ​​provided by the data linkage unit (132) (e.g., account status, service usage history, equipment sensor data), a set of rules provided by the regulation and policy verification unit (133) (e.g., loan limit rules, medical personal information processing standards, service terms restriction conditions), and relational knowledge structures returned by the knowledge graph and domain-specific RAG unit (134) (e.g., causal relationships between regulation nodes and policy nodes, product / service metadata).

[0048] Subsequently, the above mapping result is provided as prompt input to a large-scale language model, and the LLM generates response candidates. It then undergoes a re-verification process to ensure that the candidates do not conflict with real-time data values ​​and regularization conditions, thereby producing the final response.

[0049] Therefore, the LLM consultation unit (131) can be implemented not as a simple natural language generation module, but as an advanced response generation engine that integrates real-time data fusion, regulation and policy verification, and knowledge graph-based inference, thereby achieving the technical effect of providing user-customized, regulation-friendly, and domain context-optimized responses.

[0050] The data linkage unit (132) may be configured to collect and process real-time data required for the consultation process by linking with an external data source specialized in a specific industry domain.

[0051] More specifically, the data linkage unit (132) can collect financial data such as account balance, transaction history, loan repayment conditions, credit rating, interest rate fluctuations, and foreign exchange rates by linking with a financial API, and when linked with a healthcare database, it can receive the patient's biometric indicators (e.g., heart rate, blood pressure, blood sugar, oxygen saturation), diagnosis history, medication records, etc. in real time and reflect them in the consultation response.

[0052] In addition, the data linkage unit (132) can be linked with a public administration API to obtain data such as the status of civil complaint processing, tax payment history, and administrative service application status, and when linked with a service industry CRM (Customer Relationship Management) system, data such as customer profile, purchase history, service usage patterns, and past complaint processing records can be retrieved in real time and reflected in the consultation scenario.

[0053] The above data linkage unit (132) is linked simultaneously with a plurality of heterogeneous data sources, and the collected data is synchronized with the data format, unit, and time through a standardization module, and then integrated and reflected in the LLM prompt input of the consultation agent unit (130).

[0054] Furthermore, the data linkage unit (132) uses a secure channel (e.g., TLS / SSL-based encrypted transmission) when collecting data, and performs masking, pseudonymization, and anonymization in the case of sensitive information to support processing that complies with personal information protection regulations.

[0055] Therefore, the data linkage unit (132) is not a simple data collection module, but can function as a data hub that integrates, normalizes, and securely processes domain-specific data sources in real time and reflects them in the consultation response.

[0056] The regulation and policy verification unit (133) may be configured to store and search for regulatory conditions and policy rules for each domain in the form of a knowledge graph in order to ensure the legality and policy suitability of the response generated during the consultation process.

[0057] The regulation and policy verification unit (133) stores regulatory conditions, such as loan limit restrictions, debt-to-income ratio rules, restrictions on risk transactions for customers with credit ratings below a certain level, and anti-money laundering (AML) obligations, in the form of nodes and edges in a graph database for the financial sector, and verifies in real time whether the LLM response generated by the consultation agent unit (130) violates the relevant regulatory conditions.

[0058] In the healthcare sector, guidelines for protecting patient personal information, rules for managing medical records, and restrictions on drug prescriptions can be reflected in the knowledge graph to control medical consultation responses so that they do not violate relevant regulations.

[0059] In addition, in the education sector, evaluation criteria, rules for utilizing learning data, and personal information consent policies can be verified, while in the public administration sector, regulations on civil complaint processing, the Administrative Procedure Act, and e-government security guidelines can be included as subjects for verification regarding compliance.

[0060] The regulation and policy verification unit (133) parses the meaning of the query when a user's query is entered, searches for related regulation nodes and policy rules on the knowledge graph, and compares and contrasts them with response candidates generated by the LLM.

[0061] If the response violates or is likely to violate a regulation or policy, the regulation / policy verification unit (133) may block the response or automatically generate a modified response within the scope of regulatory compliance, and if necessary, may trigger an administrator approval procedure or an actual agent connection procedure.

[0062] In addition, the above-mentioned regulation and policy verification unit (133) is not limited to simple rule-based verification, but is linked with a machine learning-based policy conformity classification model to probabilistically evaluate the possibility of regulatory compliance even for ambiguous or borderline cases, and performs a prior warning or blocking for high-risk responses.

[0063] Accordingly, the regulation and policy verification unit (133) can function as a conformity verification engine that blocks the possibility of violating regulations and policies in real time by combining and verifying the regulation and policy knowledge graph and LLM response during the process of generating the consultation response.

[0064] The knowledge graph and domain-specific RAG section (134) may be configured to systematically collect and analyze knowledge specialized in a specific industrial field, structure and store it in the form of a graph database, and use it to reinforce LLM-based consultation responses.

[0065] The above knowledge graph is designed to represent key industry-specific data and rules in the form of nodes and edges, enabling the reflection of semantic connectivity beyond simple keyword-based search.

[0066] For example, in the financial domain, financial product metadata (e.g., product type, interest rate, repayment terms, credit rating criteria), regulatory rules (e.g., loan limits, debt-to-income ratio, anti-money laundering requirements), and customer profile-linked data (e.g., age group, income level, transaction patterns) can be structured and stored.

[0067] In the healthcare domain, the appropriateness of medical consultations can be ensured by representing medical protocols, clinical guidelines, patient management rules, and privacy guidelines as nodes and relationships, while in the service industry, it may include customer response manuals, CRM (Customer Relationship Management) data, and standard service procedures.

[0068] Furthermore, in the legal and administrative domains, administrative processing rules, civil complaint procedures, regulatory statute provisions, and agency-specific policy standards are reflected in the knowledge graph to enable regulation-friendly consultation in the public service sector.

[0069] When a customer query is input, the knowledge graph and domain-specific RAG unit (134) parses the meaning of the query and searches for and extracts semantically related subgraphs within the graph DB.

[0070] The retrieved subgraphs are subsequently combined with the RAG (Retrieval-Augmented Generation) framework and reflected in the LLM prompt, thereby enabling the LLM to generate augmented responses that include domain-specific knowledge, regulatory conditions, and industry-specific policy contexts, rather than simply producing responses based on statistical probability.

[0071] Furthermore, the knowledge graph and domain-specific RAG section (134) is linked with the real-time data linkage section (132) and the regulation / policy verification section (133) to provide a response that simultaneously reflects the latest data and regulatory compliance.

[0072] In addition, by combining with a multilingual processing unit (135), the same knowledge graph can be utilized in various language environments, and industry-specific regulations and policy standards are consistently applied even when providing global services.

[0073] Therefore, the knowledge graph and domain-specific RAG section (134) can function as a core intelligent module that goes beyond a simple information retrieval module and encompasses knowledge structuring, semantic retrieval, LLM prompt enhancement, and regulatory compliant response generation.

[0075] The multilingual / multimodal processing unit (135) may be configured to recognize multiple natural language inputs, such as Korean, English, Chinese, Japanese, and Spanish, and to convert and output consultation results in a target language selected by the user or automatically determined by the system.

[0076] The aforementioned module goes beyond simple machine translation and includes a language model that reflects the grammatical characteristics, cultural context, and differences in expression styles of each language, thereby providing consultation results that are natural and free from meaning distortion to users of each language group.

[0077] In addition, the multilingual / multimodal processing unit (135) may include an expandable structure capable of simultaneously processing not only text-based queries but also various forms of input such as voice, images, and videos.

[0078] For example, in the case of voice input, the utterance is converted into text through a STT (Speech-to-Text) engine and reflected in the LLM prompt to generate a response.

[0079] In the case of image input, complex visual data such as documents, tables, graphs, and even identification cards or medical images can be recognized and analyzed through an OCR (Optical Character Recognition) module or a Vision Transformer-based analysis engine.

[0080] For video input, object recognition, scene classification, and gesture recognition technologies can be utilized to identify the context of the user's behavior and reflect this information in the consultation response generation process.

[0081] The above multilingual / multimodal processing unit (135) supports conversion and expression in various formats, such as text, voice, and avatar video, even during the process of outputting the consultation response.

[0082] For example, if a user asks a question in English and wants a response in Chinese, the system recognizes and interprets the English input, generates a text response translated and optimized into Chinese, and simultaneously synthesizes the response into Chinese speech or reflects it in the avatar's spoken words and gestures to provide it in multiple formats.

[0083] Furthermore, the multilingual / multimodal processing unit (135) is linked with the knowledge graph and domain-specific RAG unit (134) and the regulation / policy verification unit (133) to control that the same regulatory conditions, industry-specific policy rules, and domain context are consistently reflected in the response even if the language changes.

[0084] This enables the securing of industry-specific regulatory compliance and policy-friendly consultation quality even in a global environment, while significantly improving scalability and reliability compared to single-language-based systems.

[0085] Accordingly, the multilingual / multimodal processing unit (135) can be implemented as an advanced integrated module that goes beyond a simple translation module or input conversion device and encompasses multi-language processing, multi-input format analysis, multi-output format expression, and maintenance of regulatory consistency.

[0087] The voice recognition and synthesis unit (136) may be configured to include a voice recognition module (STT: Speech-to-Text) that receives the user's spoken voice in real time during the consultation process and converts it into text data, and a voice synthesis module (TTS: Text-to-Speech) that synthesizes the text response produced by the consultation agent unit (130) into a voice suitable for the user's emotional state.

[0088] The above speech recognition module can remove noise by analyzing acoustic features such as fundamental frequency (F0), formant, syllable length, and speech rate, and perform precise text conversion that reflects the user's unique speaker characteristics such as intonation, repression, and speech habits.

[0089] In addition, to enable operation in a multilingual environment, it can recognize speech in multiple languages ​​such as Korean, English, Chinese, and Japanese, and may include language-specific speech recognition models to handle linguistic specificities such as dialect, intonation, and pronunciation variations.

[0090] The above-mentioned speech synthesis module goes beyond simply converting text into speech and is linked with the emotion recognition and classification unit (140) to reflect the user's emotional state that changes in real time during consultation.

[0091] Specifically, the emotion classification result (e.g., joy, anger, anxiety, embarrassment, neutral) output from the emotion recognition and classification unit (140) is dynamically input into the parameters of the voice synthesis module, and the voice tone, intonation, stress, speech rate, pause duration, etc. are adjusted to suit the emotional state.

[0092] For example, if the user is classified as being in an angry state, the speech synthesis module relieves the tension of the conversation by synthesizing LLM responses with a low tone, calm intonation, and slow speaking speed.

[0093] Conversely, when the user is perceived as being in a positive and active state, the synthesized voice reflects a brighter tone, faster rhythm, and emphasized stress to enhance intimacy.

[0094] In addition, if the user is in an anxious state, the synthesized voice maintains a soft intonation and a slow speech rate to induce a sense of calm.

[0095] The above voice recognition and synthesis unit (136) is linked with consultation scenarios, user profile data, and internal policy conditions so as to reflect situational context, and through this, even for the same query, differentiated voice output can be provided according to the characteristics of user groups (e.g., elderly people, youth, workers in specific industries).

[0096] Accordingly, the speech recognition and synthesis unit (136) can be implemented as an advanced voice interface module that goes beyond a simple combination of STT / TTS engines and realizes multilingual recognition, speaker characteristic preservation, emotion-adaptive synthesis, and context-based speech optimization.

[0097] Therefore, the counseling agent unit (130) can operate as a general-purpose counseling support platform applicable to different industry domains such as finance, healthcare, service industry, and public service.

[0098] Specifically, the consultation agent department (130) is designed to simultaneously reflect data sources and regulatory conditions specialized for each industry domain, so that in the financial sector, data and rules such as account balances, transaction history, and loan limit regulations; in the healthcare sector, patient health indicators, diagnostic protocols, and medical privacy protection guidelines; in the service industry, customer CRM data, service quality regulations, and contract performance standards; and in the public service sector, administrative civil complaint data, statute-based processing guidelines, and public institution regulations can be integrated and reflected in real time.

[0099] In particular, the consulting agent unit (130) adopts an architecture that allows for the replacement and addition of industry-specific knowledge graphs and policy rules in a modular fashion within the same system structure, making it easy to expand into new industry domains.

[0100] For example, the regulatory verification unit and financial knowledge graph module used in the financial domain can be configured by replacing them with a healthcare guideline verification unit and medical protocol knowledge graph, or by adding them in parallel. This structure ensures both universality and scalability by maintaining common modules across domains (e.g., LLM consultation unit, multilingual / multimodal processing unit, speech recognition / synthesis unit) while replacing only the modules that reflect industry-specific characteristics.

[0101] Therefore, the counseling agent unit (130) of the present invention is not merely a counseling system limited to a specific industry group, but can be implemented as an advanced counseling support platform that satisfies all of the following: ① real-time data linkage by domain, ② compliance with industry-specific regulations and policies, ③ knowledge graph-based semantic inference, and ④ a modular replaceable expansion architecture. This provides significant technical advantages, such as the ability to apply it quickly in various industrial fields and to respond with minimal structural changes even when expanding to new domains.

[0102] Next, the emotion recognition and classification unit (140) may be configured to classify and determine the emotional state of the customer by integrating and analyzing multimodal input signals obtained from the customer during the consultation process.

[0103] The emotion recognition and classification unit (140) first extracts acoustic features, such as fundamental frequency (F0), formant, voice intensity, speaking speed, and Mel-frequency cepstrum coefficient (MFCC), from voice input to quantitatively evaluate the customer's speech pattern, intonation, and tension level. In addition, it detects changes in facial expressions, micro-expressions, eye tracking results, head posture, and movements from video to produce non-verbal emotional signals.

[0104] Additionally, as needed, biosignals such as heart rate, skin conductivity (GSR), and breathing patterns are synchronized and input, thereby allowing for multidimensional reflection of the customer's psychological and physiological responses. The analyzed multimodal features are input into a machine learning-based emotion classifier, which may be, for example, a CNN-LSTM mixed model, a Transformer-based emotion analysis model, or a deep learning model including a Multi-head Attention structure.

[0105] Through this process, the customer's state is classified into multiple emotion classes such as joy, anger, anxiety, embarrassment, and neutrality, and a probability distribution for each emotional state is calculated. The results of this emotion classification are reflected in real-time in the voice tone, facial expression rendering, and gesture animation control of the virtual bank teller avatar, providing a consultation experience optimized for the customer's psychological state.

[0106] For example, if a customer is perceived to be in a state of anger, the virtual bank teller performs calming gestures, such as spreading both hands or nodding, accompanied by calm, low-toned speech.

[0107] Conversely, if the customer is perceived as anxious, the avatar uses a trustworthy smile and gentle explanatory gestures to induce emotional stability. Additionally, in a positive emotional state, active hand movements and bright facial expressions are enhanced, and movements are controlled to increase customer intimacy.

[0108] Furthermore, the emotion recognition and classification unit (140) is designed to learn multilingual voice patterns and differences in emotional expressions by culture, so that even if the utterance content is the same, emotional expressions according to cultural context can be distinguished.

[0109] For example, by separately learning the laughter and intonation patterns of Western customers and the differences in silence and subtle facial expressions of East Asian customers, high-accuracy emotion recognition performance can be maintained even in a global environment. Therefore, the emotion recognition and classification unit of the present invention can be implemented as an advanced emotion recognition engine that goes beyond the level of simple voice analysis and includes multimodal fusion, real-time classification, and culture-adaptive learning.

[0110] Next, the avatar behavior control unit (150) may be configured to control a speech synthesizer, an expression renderer, and a gesture animator so that the selected counseling response is not merely output, but the avatar's voice tone, facial expression, and gesture movements are expressed in harmony with the meaning of the counseling response and the user's emotional state, by reflecting the emotion classification result output from the emotion recognition and classification unit (140) into the LLM prompt.

[0111] The above avatar behavior control unit (150) may be configured to control multiple expression modules, such as a voice synthesizer, facial expression renderer, and gesture animator of the customer consultation avatar generation unit (120), so that even if the consultation response content is the same, visual and auditory expressions are made in a manner optimized for the user's current state and consultation context.

[0112] More specifically, the avatar behavior control unit (150) applies a hybrid mapping algorithm that combines the output probability distribution of an LLM prompt adjustment module and a deep learning-based emotion classifier.

[0113] Through this, even with the same text response, if, for example, the user is perceived to be in a state of anger, the avatar's utterance is converted to a calm and low tone, and visual expressions are controlled to perform calming gestures such as spreading both hands, nodding, and a soft facial expression. Conversely, if the user displays positive emotions, the response is converted to a more cheerful tone and accompanied by lively hand movements and a friendly smile to increase the intimacy of the interaction.

[0114] In addition, the avatar behavior control unit (150) does not rely solely on the emotion classification results, but optimizes the combination of voice, facial expressions, and gestures for each situation by considering the context of the consultation scenario, user profile information (age, cultural background, service preference, etc.), and domain-specific internal policy conditions. To this end, it may include a self-learning structure that continuously generates and updates an adaptive mapping table and gradually improves mapping accuracy by reflecting consultation log data and user feedback.

[0115] Accordingly, the avatar behavior control unit (150) can be applied in various domains such as financial counseling, healthcare guidance, customer service in the service industry, and public complaint processing, and can implement advanced human-friendly interactions beyond simple emotion-based response modules through ① the combination of LLM and emotion models, ② context-based behavior optimization, ③ self-learning mapping rule updates, and ④ multi-domain adaptability, thereby supporting users to receive an empathetic and situation-adaptive counseling experience similar to that of a human counselor.

[0116] Next, the connection / assistance mode control unit (160) may be configured to automatically control whether to connect with an actual agent or expert according to the level of user's requirements, the complexity of the inquiry, or the difficulty of the work scenario during the consultation process, and to switch the virtual consultation avatar to standby mode or assistance mode after the connection is completed.

[0117] The above connection / assistance mode control unit (160) operates to continuously provide visual and non-verbal support within the interaction screen even after the customer consultation avatar has been switched to standby mode.

[0118] For example, it can perform functions such as gestures pointing to electronic documents, visual highlighting to emphasize specific items, animations simulating document delivery, and visually guiding step-by-step procedures.

[0119] In addition, the connection / assistance mode control unit (160) synchronizes the voice / video interface of the actual agent (or expert) with the visual interface of the customer consultation avatar to support the user in continuing the consultation without confusion. Furthermore, by monitoring the user's emotional state and concentration level in real time, it controls the customer consultation avatar so that it can perform roles such as assisting with explanations, repeating questions, displaying visual materials, and providing educational simulations even after the actual agent is connected.

[0120] Furthermore, at the end of a consultation, it can include features to visually display a summary of the key points or guide users through follow-up procedures (e.g., electronic signature, appointment registration, payment processing, and follow-up notification settings). Through this scalable structure, the same functions can be utilized across various industries—not only in the financial sector but also in healthcare consultation guidance, customer support for the service industry, education and training simulations, and public complaint processing—thereby simultaneously improving consultation efficiency, user convenience, and service continuity in multi-domain environments.

[0121] Next, the compliance management department (170) may be configured to manage customer-related data acquired and processed during the consultation process in accordance with international and domestic financial regulatory standards and personal information protection laws.

[0122] The compliance management unit (170) performs real-time anonymization, pseudonymization, and encryption procedures on sensitive data such as customer profile data, consultation conversation logs, and financial transaction history, and all data processing processes include functions for blocking unauthorized access, end-to-end encryption, and database encryption.

[0123] In addition, consultation records are stored in an immutable log format to support audits and internal compliance monitoring by financial supervisory authorities, and access rights are managed by multi-factor authentication and role-based access control (RBAC).

[0124] Furthermore, the Compliance Management Department (170) can be linked with the bank's internal compliance policy engine to control whether a specific consultation response or data provision may violate regulations, by blocking it in advance or requiring an administrator approval process.

[0125] In addition, the compliance management department (170) supports compliance with global security standards such as GDPR, CCPA, ISO / IEC 27001, PCI-DSS, and may include scalable functions such as customer consent-based data usage management, automatic data deletion cycle control, and blockchain-based anti-tampering audit log storage.

[0127] FIG. 3 is a flowchart illustrating a customer consultation support method using a user profile-based customized customer consultation avatar according to an embodiment of the present invention, FIG. 4 is a detailed flowchart of the S750 process shown in FIG. 3, and FIG. 5 is an example diagram illustrating a process in which a virtual banker avatar and a real counselor cooperate to support banking business (loan contract procedure) according to an embodiment of the present invention.

[0128] Below, based on the system configuration described above, a user profile-based customized virtual consultation support method according to an embodiment of the present invention is described in detail step by step.

[0129] This method is applicable to various domains such as finance, healthcare, service industries, and public services, and some steps may be optionally omitted, added, or their order changed.

[0130] Referring to FIGS. 3 to 5, a customer consultation support method (S700) using a user profile-based customized customer consultation avatar according to an embodiment of the present invention collects static profile data and dynamic emotion data of a user (S710), and applies the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library to generate a customer consultation avatar customized to the user (S720). Subsequently, a consultation response to the user's query content is generated using a large-scale language model (LLM), real-time data provided from an external data source and a domain-specific API is reflected in the consultation response (S730), and the user's emotional state is recognized and classified by analyzing the user's dynamic emotion data obtained during the consultation process (S740). Then, the method may include a series of processes such as controlling a voice synthesizer, an expression renderer, and a gesture animator of the customer consultation avatar so that the consultation response is expressed in the customer consultation avatar with a voice tone, facial expression, and gesture movement suitable for the classified emotional state (S750).

[0131] More specifically, the S710 process is a process of collecting static profile data and dynamic emotional data of a user from a profile input unit (110). The static profile data may include the user's age, gender, cultural background, linguistic characteristics, service usage preferences, etc., and the dynamic emotional data may include real-time emotional and behavioral data such as voice tone, facial expression changes, eye gaze patterns, and biosignals (heart rate, skin conductivity, etc.) during consultation.

[0132] Collected data is processed securely through encryption and anonymization in accordance with personal information protection regulations.

[0133] The S720 process is a process of generating a user-customized virtual customer service avatar by applying static and dynamic data collected from the customer service avatar generation unit (120) to a pre-trained customer service avatar library, and the generated avatar reflects the user's profile and current emotional state, so that the appearance, facial expressions, voice tone, gestures, clothing style, and interface language are dynamically adjusted.

[0134] The S730 process is a step in which the counseling agent unit (130) interprets and analyzes the user's query based on a large-scale language model (LLM) to generate a counseling response. In this process, the counseling agent unit (130) does not merely produce a statistical and linguistic response to the input query, but integrates real-time data by linking with external data sources and domain-specific APIs.

[0135] Specifically, in the financial domain, data such as account balances, transaction history, loan repayment conditions, credit ratings, interest rate fluctuations, and foreign exchange rates are retrieved and analyzed in real time and immediately reflected in consultation responses, thereby enabling precise consultation based on the customer's actual financial situation; in the healthcare domain, key patient health indicators (e.g., heart rate, blood pressure, blood sugar, oxygen saturation), diagnostic history, prescription history, and clinical protocols are reflected in real time to provide customized medical consultations that meet medical regulations and safety standards; and in the public service domain, data such as the status of administrative civil complaint processing, tax payment history, administrative procedure status, and e-government service request results are linked in real time to provide administrative guidance that reflects compliance with policies and regulations.

[0136] In addition, the consultation agent unit (130) controls the consultation response so that it is not a simple natural language generation result, but a precise response that satisfies three requirements: ① based on the latest data, ② guarantee of compliance with regulations and policies, and ③ optimization of the domain context, by verifying and reflecting not only the real-time data but also domain-specific rules and policy conditions.

[0137] The S740 process is a process of recognizing and classifying the user's emotional state by analyzing the user's dynamic emotional data obtained during the counseling process in the emotion recognition and classification unit (140). The analyzed data is input into a deep learning-based emotion classification model (CNN-LSTM, Transformer, etc.) and classified into emotional states such as joy, anger, anxiety, and neutrality, and the result can be produced in the form of a probability distribution.

[0138] The S750 process is a step in which the avatar behavior control unit (150) reflects the emotion classification result output from the emotion recognition and classification unit (140) into the LLM prompt so that the generated counseling response is not limited to simple text output but is expressed in a multimodal manner that matches the user's emotional state and emotional context. To this end, the avatar behavior control unit (150) integrally controls the avatar's speech synthesizer, facial expression renderer, and gesture animator, thereby producing a differentiated expression style in real time according to the user's current psychological state and counseling context, even if the response content is the same.

[0139] Specifically, when the user is classified as being in an angry state, the avatar behavior control unit (150) converts the LLM’s counseling response into a calm and low-toned speech, and controls the visual expression to perform calming gestures such as spreading both hands, nodding, and a soft expression to induce relaxation of the conversation.

[0140] If the user is perceived to be in a state of anxiety, the response is synthesized in a soft and stable tone, and the avatar supports the user's psychological stability by performing gentle hand gestures to aid explanation, along with a smile that conveys trust.

[0141] When the user is in a positive and active state, the avatar behavior control unit (150) converts the response into a cheerful intonation and bright tone, and maximizes the interaction intimacy by enhancing active hand movements and active facial expressions.

[0142] Additionally, the avatar behavior control unit (150) does not rely solely on the emotion classification results, but comprehensively considers the context of the counseling scenario, user profile information (e.g., age, cultural background, service preference), and domain-specific policy conditions to optimize the combination of voice, facial expressions, and gestures for each situation. To this end, the avatar behavior control unit (150) may include a self-learning structure that continuously generates and updates an adaptive mapping table and learns counseling logs and user feedback data to gradually improve mapping accuracy.

[0143] Therefore, the S750 process of the present invention goes beyond the level of visual counseling assistance using a static avatar and combines ① real-time emotion recognition reflection, ② multimodal expression control, ③ context-based behavior optimization, and ④ a self-learning improvement structure to implement empathetic and situation-adaptive interaction comparable to a human counselor, thereby allowing the user to enjoy the effect of receiving a human-friendly and reliable counseling experience in real time that mechanical counseling systems could not provide.

[0145] According to the present invention, there is an advantage in that a user-customized customer consultation avatar can be automatically generated by comprehensively reflecting not only static profile data such as the user's age, gender, cultural background, and preferences, but also dynamic emotional data such as voice tone, facial expressions, and eye gaze patterns that change in real time during the consultation process.

[0146] Through this, users can receive a personalized counseling experience that reflects human empathy and consideration, rather than the existing mechanical and uniform conversational agent services. For instance, avatars with clear pronunciation and slow explanation speeds are provided to the elderly, while avatars with modern appearances and fast response speeds are provided to young users, thereby maximizing satisfaction by generation and group.

[0147] In addition, the present invention has the advantage of allowing a customer consultation avatar to handle repetitive and standardized preliminary tasks, such as providing information, verifying documents, and answering basic questions at the beginning of the consultation, while enabling immediate connection to an actual agent or manager in complex or specialized situations.

[0148] Even upon connection, the customer service avatar switches to standby mode to provide visual and non-verbal assistance on-screen, such as pointing at documents, highlighting items, and guiding procedures. This allows actual agents to focus on core decision-making or specialized tasks, while enabling users to experience a seamless and natural service flow. Consequently, this reduces consultation processing times across various service environments and enhances user satisfaction even in non-face-to-face interactions.

[0149] Furthermore, the present invention has the advantage of providing a real-time emotion-behavior mapping algorithm that combines a large-scale language model (LLM)-based counseling engine and a deep learning-based emotion classification model to immediately reflect conversational context and gesture responses associated with the user's emotional state by adjusting LLM prompts in real time.

[0150] For example, if a user voices dissatisfaction, the customer service avatar immediately responds with a smiling expression and a calm tone, avoiding unnecessary arguments. This human-friendly interaction provides a distinctive technological effect that is difficult to achieve with existing simple text-based consultation systems. Furthermore, this technology has the potential for expanded application in various fields, including finance, customer service, healthcare, education, public services, and metaverse-based remote support.

[0151] In addition, the present invention has the advantage of satisfying international data protection standards, such as the Personal Information Protection Act, industry-specific security regulations, and GDPR, by anonymizing and encrypting user data (profile, voice, facial expressions, consultation records, etc.) collected during the consultation process in real time.

[0152] In addition, all consultation processes are stored along with log records and metadata for auditing, which can meet the compliance requirements of relevant supervisory authorities or operational managers, thereby ensuring a level of reliability and legal compliance applicable in actual service environments.

[0153] The present invention has the potential to be applied in various industrial sectors, such as the financial sector, service industry, healthcare, and public services.

[0154] For example, in the field of financial consulting, customized financial avatars that reflect customers' profile data and real-time emotional states can provide specialized advice on loans, investments, and account management, thereby ensuring both regulatory compliance and customer satisfaction.

[0155] In the service industry and customer center response sectors, the customer experience can be improved by reflecting customers' verbal and non-verbal signals, providing response avatars focused on reassurance and apology to dissatisfied customers and avatars focused on guidance and recommendations to new customers.

[0156] In addition, in the healthcare field, virtual health coordinator avatars reflecting the patient's age, health status, and emotional changes can be created to perform functions such as medication guidance, lifestyle management, and psychological counseling.

[0157] Furthermore, in the public service sector, administrative efficiency and public satisfaction can be enhanced by providing avatars optimized for the user's background and emotional state in areas such as civil complaint processing, administrative consultation, and education and training environments.

[0158] Consequently, since the present invention enables the implementation of an intelligent consultation support platform that is regulation-friendly and applicable across various industries, it has the advantage of significantly enhancing technical and commercial value and has a very high potential for commercialization in diverse industrial fields.

[0160] Meanwhile, the components, units, modules, components, etc. described as "~parts" in this specification may be implemented together or individually as interoperable logic devices.

[0161] The description of different features of modules, units, etc. is intended to highlight different functional embodiments and does not necessarily imply that they must be realized by individual hardware or software components. Rather, functions associated with one or more modules or units may be performed by individual hardware or software components or integrated within common or individual hardware or software components.

[0162] Although operations are depicted in a specific order in the drawings, it should not be understood that these operations must be performed in the specific order depicted or in a sequential order to achieve the desired result, or that all depicted operations must be performed. In any environment, multitasking and parallel processing may be advantageous. Furthermore, the distinction between the various components in the above-described embodiments should not be understood as requiring such distinction in all embodiments, and it should be understood that the described components may generally be integrated together into a single software product or packaged into multiple software products.

[0163] A terminal device program (also known as a program, software, software application, script, or code) may be written in any form of a programming language, including a compiled or interpreted language or an a priori or procedural language, and may be deployed in any form, including a standalone program or a module, component, subroutine, or other unit suitable for use in a terminal device environment.

[0164] Additionally, the logical flow and structural block diagrams described in this patent document describe corresponding actions and / or specific methods supported by corresponding functions and steps supported by the disclosed structural means, and can also be used to construct corresponding software structures and algorithms and equivalents thereof. The processes and logical flow described in this specification can be executed by one or more programmable processors that execute one or more terminal device programs to perform functions by operating on input data and generating outputs.

[0165] The description provided herein presents the best mode of the invention and offers examples to explain the invention and to enable those skilled in the art to manufacture and use the invention. The specification thus written is not intended to limit the invention to the specific terms presented.

[0166] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art or those with ordinary knowledge in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and technical scope of the invention as described in the claims set forth below.

[0167] Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims. Explanation of the symbols

[0168] 100: Customer support system using user profile-based customized customer support avatars 110: Profile Input Section 120: Customer Consultation Avatar Creation Section 130: Consultation Agent Department 131: LLM Counseling Department 132: Data Integration Section 133: Regulatory Policy Verification Department 134: Knowledge Graph and Domain-Specific RAG Section 135: Multilingual Multimodal Processing Unit 136: Speech Recognition and Synthesis Unit 140: Emotion Recognition and Classification Unit 150: Avatar Behavior Control Unit 160: Connection / Auxiliary Mode Control Unit 170: Compliance Management Department

Claims

Claim 1 A profile input unit that collects static profile data and dynamic emotion data of a user; a customer consultation avatar generation unit that generates a user-customized customer consultation avatar by applying the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library; a consultation agent unit that generates a consultation response to a user's query using LLM and reflects real-time data provided from external data sources and domain-specific APIs into the consultation response; an emotion recognition and classification unit that analyzes the user's dynamic emotion data acquired during the consultation process to recognize and classify the user's emotional state; and an avatar behavior control unit that controls the avatar's voice synthesizer, facial expression renderer, and gesture animator so that the consultation response is expressed with the voice tone, facial expression, and gesture movements of the customer consultation avatar suitable for the user's emotional state. A customer consultation support system using a user profile-based customized customer consultation avatar, comprising a connection assistance mode control unit that controls whether to connect a call between an actual agent and a user according to the level of user's requirements, complexity of the query, or difficulty of the business scenario during the consultation process, switches the customer consultation avatar to a standby mode or an assistance mode after the call connection is completed, and supports the actual agent in performing visual consultation assistance when the customer consultation avatar is switched to the standby mode. Claim 2 delete Claim 3 A customer consultation support system using a user profile-based customized customer consultation avatar, characterized in that, in claim 1, the consultation agent unit structures and stores domain-specific regulatory conditions as nodes and relationships of a knowledge graph, and when a user query is input, maps the user query to a regulatory condition retrieved from the knowledge graph, combines it with a consultation response generated by the LLM, and automatically verifies that the consultation response does not violate the relevant domain regulation or internal policy. Claim 4 A customer consultation support system using a user profile-based customized customer consultation avatar, wherein, in the case of financial consultation, the consultation agent unit reflects the user's account balance, transaction history, interest rate fluctuations, loan repayment conditions, and credit rating data obtained by linking with an external financial data API in real time during the consultation response generation process, and automatically verifies whether the generated consultation response is compliant with financial regulatory conditions through verification rules. Claim 5 A customer consultation support system using a user profile-based customized customer consultation avatar, wherein, in claim 1, the avatar behavior control unit considers consultation scenarios, user profiles, and internal policy conditions together to generate and update an adaptive mapping table that optimizes the mapping relationship between the user's emotional state and the voice tone, facial expressions, and gesture movements of the customer consultation avatar on a situational basis, and learns consultation logs and user feedback data to continuously improve the accuracy of the emotion-behavior mapping relationship. Claim 6 A step of collecting static profile data and dynamic emotion data of a user in a profile input section; a step of generating a user-customized customer consultation avatar in a customer consultation avatar generation section by applying the collected static profile data and dynamic emotion data to a pre-trained customer consultation avatar library; a step of generating a consultation response to a user's query using a Large Language Model (LLM) in a consultation agent section and reflecting real-time data provided from external data sources and domain-specific APIs into the consultation response; a step of recognizing and classifying the user's emotional state by analyzing the user's dynamic emotion data obtained during the consultation process in an emotion recognition and classification section. A method for supporting customer consultation using a user profile-based customized customer consultation avatar, comprising: a step of controlling a voice synthesizer, an expression renderer, and a gesture animator of the customer consultation avatar so that the consultation response is expressed in the customer consultation avatar with a voice tone, an expression, and a gesture action suitable for an emotional state classified by an avatar behavior control unit; a step of controlling whether to connect a call between an actual agent and a user according to the level of user's request, the complexity of the query, or the difficulty of the work scenario during the consultation process, and after the call connection is completed, switching the customer consultation avatar to a standby mode or an auxiliary mode, and when the customer consultation avatar is switched to the standby mode, supporting the actual agent to perform visual consultation assistance. Claim 7 delete Claim 8 In claim 6, the step of reflecting the real-time data in the consultation response comprises, in the consultation agent unit, structuring and storing domain-specific regulatory conditions as nodes and relationships of a knowledge graph, and when a user query is input, mapping the user query to a regulatory condition retrieved from the knowledge graph, and then combining it with a consultation response generated by the LLM to automatically verify that the consultation response does not violate the relevant domain regulation or internal policy, thereby providing a customer consultation support method using a user profile-based customized customer consultation avatar. Claim 9 In claim 6, the step of controlling the voice synthesizer, facial expression renderer, and gesture animator of the customer consultation avatar is characterized by including the step of generating and updating an adaptive mapping table that optimizes the mapping relationship between the user's emotional state and the voice tone, facial expression, and gesture movements of the customer consultation avatar on a situational basis by considering the consultation scenario, user profile, and internal policy conditions together in the avatar behavior control unit. Claim 10 In claim 8, the step of reflecting the real-time data in the consultation response is characterized by including the step of reflecting the user's account balance, transaction history, interest rate fluctuations, loan repayment conditions, and credit rating data obtained by linking with an external financial data API in the consultation agent unit in real-time during the consultation response generation process, and automatically verifying through a verification rule whether the generated consultation response complies with financial regulatory conditions, in a customer consultation support method using a user profile-based customized customer consultation avatar.

Images