Method for intelligent digital human service using seamless real and virtual operator transition and apparatus for the same

Abstract

The present disclosure relates to an intelligent digital human service method utilizing seamless switching between real and virtual operators and an apparatus for the same. A method for providing interaction between a service user and a virtual model for a specific service domain according to one embodiment of the present disclosure may include: a step of outputting a virtual model based on a real model in a three-dimensional virtual space; a step of outputting an interaction of the virtual model based at least on the physical information of the service user; and a step of outputting video and audio of the real model switched from the virtual model based on a specific event.

Description

Technology Field

[0001] The present disclosure relates to intelligent digital human services, and specifically to an intelligent digital human service method utilizing seamless real and virtual operator switching and an apparatus for the same. Background Technology

[0002] Conversational AI services are primarily implemented as text-based services, but recently, they are evolving into multimodal services utilizing vision and voice. For example, services are being introduced where users interact with a digital human—that is, a virtual operator—displayed on a screen using voice.

[0003] Conventional digital humans exhibit visual and auditory disparities from real people, so users can clearly recognize that the operator displayed on the screen is not a real person. Furthermore, even if a digital human displayed in a virtual 3D space is visually and auditorily indistinguishable from a real person, users may perceive the operator displayed on the screen as not a real person if it does not physically interact with other entities in the space, moves awkwardly, or responds inappropriately during conversations involving unlearned patterns. Therefore, to support the smooth use of the service by users, a new method is required to ensure homogeneity between the virtual operator and the real operator and to support mutual switching. The problem to be solved

[0004] The present disclosure aims to provide a method and apparatus that support seamless switching between a real and a virtual operator.

[0005] The present disclosure aims to provide a method and apparatus for a digital human service using a virtual operator that has visual, auditory, physical, and contextual homogeneity with an actual operator.

[0006] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below. means of solving the problem

[0007] A method for providing interaction between a service user and a virtual model for a specific service domain according to one embodiment of the present disclosure may include: a step of outputting a virtual model based on a real model in a three-dimensional virtual space; a step of outputting an interaction of the virtual model based at least on the physical information of the service user; and a step of outputting an image and audio of the real model that are switched from the virtual model based on a specific event.

[0008] An apparatus for providing interaction between a service user and a virtual model for a specific service domain according to an additional embodiment of the present disclosure may include one or more user interfaces; one or more transceivers; one or more memories; and one or more processors. The one or more processors may be configured to output a virtual model based on a real model within a three-dimensional virtual space through the one or more user interfaces; to output an interaction of the virtual model through the one or more user interfaces based at least on the physical information of the service user; and to output video and audio of the real model switched from the virtual model based on a specific event through the one or more user interfaces.

[0009] In some embodiments of the present disclosure, the physical information may include one or more of information regarding the actions of the service user or the physical environment of the service user.

[0010] In some embodiments of the present disclosure, the interaction of the virtual model may include the action of the virtual model in response to the action of the service user.

[0011] In some embodiments of the present disclosure, the interaction of the virtual model may include the operation of the virtual model presented to the service user with respect to the service user's physical environment.

[0012] In some embodiments of the present disclosure, the interaction of the virtual model may further be based on one or more of the service user's visual information, vocal information, or contextual information.

[0013] In some embodiments of the present disclosure, the contextual information may include service flow information based on a pattern of events occurring within the specific service domain.

[0014] In some embodiments of the present disclosure, the service flow information may be predicted by an artificial intelligence model-based prediction algorithm.

[0015] In some embodiments of the present disclosure, the interaction of the virtual model may be updated by reflecting one or more of the service user's needs, emotional state, behavioral pattern, or preference derived based on one or more of the service user's visual information, the voice information, the physical information, or the contextual information.

[0016] In some embodiments of the present disclosure, the interaction of the virtual model may be dynamically changed based on information that changes in real time for the service user.

[0017] In some embodiments of the present disclosure, the interaction of the virtual model may include a response optimized in real time for the service user and the specific service domain based on the service user's data and a knowledge graph for the specific service domain.

[0018] A method for providing a virtual model based on a real model according to an additional embodiment of the present disclosure may include: generating or updating a visual representation and a voice representation of the virtual model based on visual information and voice information of the real model; outputting the visual representation and the voice representation into a three-dimensional virtual space; outputting an autonomous interaction of the virtual model with one or more virtual objects within the three-dimensional virtual space; and outputting an image and voice of the real model that are switched from the virtual model based on a specific event.

[0019] An apparatus for providing a virtual model based on a real model according to an additional embodiment of the present disclosure may include one or more user interfaces; one or more transceivers; one or more memories; and one or more processors. The one or more processors may be configured to generate or update a visual representation and a voice representation of the virtual model based on visual information and voice information of the real model; output the visual representation and the voice representation in a three-dimensional virtual space through the one or more user interfaces; output autonomous interaction of the virtual model with one or more virtual objects in the three-dimensional virtual space through the one or more user interfaces; and output an image and voice of the real model switched from the virtual model based on a specific event through the one or more user interfaces.

[0020] In some embodiments of the present disclosure, in the operation switching of the virtual model, a transition from a first operation to a second operation according to the flow from a first operation to a second operation can be derived through an artificial intelligence model that has learned operations for a plurality of operations.

[0021] In some embodiments of the present disclosure, the autonomous interaction of the virtual model may be derived based on learning through an artificial intelligence model of the interaction with one or more real objects in the real space of the real model.

[0022] In some embodiments of the present disclosure, the autonomous interaction of the virtual model may include behavior in accordance with physical laws for one or more virtual objects in the three-dimensional virtual space.

[0023] In some embodiments of the present disclosure, the specific event may correspond to an event that is not included in an event pattern pre-learned for a specific service domain.

[0024] In some embodiments of the present disclosure, based on the homogeneity between the virtual model and the actual model, the transition between the virtual model and the actual model can be performed seamlessly.

[0025] In some embodiments of the present disclosure, the virtual model can provide a real-time optimized response for the service user and the specific service domain based on the service user's data and a knowledge graph for the specific service domain.

[0026] In some embodiments of the present disclosure, the virtual model that is switched from the actual model may be output based on a command or other event of the actual model.

[0027] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure. Effects of the invention

[0028] According to the present disclosure, a method and apparatus for a digital human service that can significantly reduce the workload of an actual operator may be provided.

[0029] According to the present disclosure, a method and apparatus for a digital human service capable of improving user experience through interaction with a user based on multimodal information such as visual, auditory, and spatial information may be provided.

[0030] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below. Brief explanation of the drawing

[0031] FIG. 1 is a diagram illustrating an example of a method for providing interaction between a service user and a virtual model for a specific service domain according to the present disclosure. FIG. 2 is a drawing for illustrating an example of a method for providing a virtual model based on an actual model according to the present disclosure. FIG. 3 shows an example of a block diagram of an apparatus according to the present disclosure. Specific details for implementing the invention

[0032] The present disclosure is subject to various modifications and may have various embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of components in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0033] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0034] Where it is stated that any component of the present disclosure is "connected" or "connected" to another component, it should be understood that it may be directly connected to or connected to the other component, or that there may be other components in between. On the other hand, where it is stated that any component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0035] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0036] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration "comprising" does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0037] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0039] The present disclosure describes various examples of methods and devices for intelligent digital human services that support seamless switching between a real operator and a virtual operator.

[0040] Conversational artificial intelligence (AI) services are undergoing rapid development based on the knowledge accumulated by humanity. Currently, text-based conversational services are the mainstream, but future development is expected to evolve into multimodal AI services that analyze visual and auditory information alongside text content.

[0041] For example, as a domestic case, Naver’s Clova Care Call, a welfare call service utilizing HyperClova—a massive AI—contributes to the operational efficiency of welfare institutions and local governments by providing natural interactions through context-aware conversations and appropriate responses, utilizing records of previous services for subsequent services, and offering purposeful welfare conversations that check user status and naturally guide users on safety guidelines. However, the current level of service is limited to voice-based conversational services, and there are limitations in expanding it to various service domains due to the restriction that it relies solely on user voice data for basic information analysis of user situations. Alternatively, in the example of Saltlux’s kiosk-based digital human service, users can receive services more familiarly through voice interaction with a digital human displayed on the screen, but its utilization remains low to date.

[0042] As such, in the case of existing digital human services, service users may perceive the operator displayed on the screen as not being a real person, which can increase a sense of unease and lower trust. Therefore, to support seamless service usage by users, a new method is described below that ensures homogeneity between virtual and real operators, supports mutual switching, and enables virtual operators to interact with service users to the same extent as real operators.

[0043] To this end, the present disclosure describes examples of providing more efficient customer service by reflecting intelligent agents optimized for knowledge specialized for each service domain into digital humans, thereby enabling a natural and seamless transition between a real operator (or real model) and a virtual operator (or virtual model) to a level where the user cannot distinguish between them.

[0044] Furthermore, the present disclosure allows for the generation of a virtual model (i.e., a digital human) that is homogeneous with the actual model by learning multimodal information, such as visual and auditory information, of the actual model. Moreover, the present disclosure allows for the systematization of knowledge and functions specialized for a specific service domain into a knowledge graph (e.g., a set of descriptions regarding an interconnected network of entities such as objects, events, situations, and concepts, and the relationships between them) using information acquired in the real space, thereby enabling the provision of responses optimized for constraints appropriate to the service context. Additionally, by utilizing reinforcement learning to develop an intelligent agent and reflecting it in the digital human, the diversity, efficiency, and quality of the service can be maintained or improved.

[0045] To expand beyond existing conversational user services focused on voice information to metaverse-based conversational services that utilize visual and physical (or spatial) information, it is necessary to enhance the diversity, efficiency, and quality of intelligent digital humans. To improve service quality, one can consider providing more efficient customer service by transitioning naturally between real and virtual models—to a level where users cannot perceive the transition, while offering a relatively higher degree of freedom compared to existing services. Given that visual information accounts for 70% to 80% of human reliance on senses, one can consider providing virtual models that incorporate visual information analysis and spatial interaction to allow users to feel and react as if they were interacting with a real model, in order to offer a more convenient and natural interface.

[0046] To this end, the present disclosure aims to provide a natural intelligent digital human service at a level where the user cannot perceive the transition between a real model and a virtual model by comprehensively utilizing user data and service domain knowledge to provide a response optimized for constraints while maintaining visual, auditory, physical, and contextual homogeneity.

[0047] FIG. 1 is a diagram illustrating an example of a method for providing interaction between a service user and a virtual model for a specific service domain according to the present disclosure.

[0048] In step S110, the device can output a virtual model based on a real model in a three-dimensional virtual space.

[0049] In the present disclosure, the output may include providing information to a user so that the user can perceive it visually and / or audibly, for example, through a user interface of a device described below.

[0050] For example, a virtual model based on a real model can be provided as in the example of Fig. 2 described below, and based on this, the virtual model can be output in a three-dimensional virtual space.

[0051] In step S120, the device can output the interaction of the virtual model based at least on the physical information of the service user.

[0052] For example, the physical information of the service user may include information about the service user's actions and / or the service user's physical environment. Accordingly, an interaction of the virtual model may be output, including the actions of the virtual model responding to the service user's actions and / or the actions of the virtual model presented (or guided) to the service user regarding the service user's physical environment.

[0053] For example, in addition to the service user's physical information, virtual model interactions may be output based on the service user's visual information (e.g., facial expressions, gestures, environment, gaze, etc.), vocal information (e.g., pronunciation, intonation, utterance content, etc.), and / or contextual information. Here, contextual information may include service flow information based on patterns of events occurring within the service domain. For example, service flow information may be predicted by a prediction algorithm based on an artificial intelligence model. Accordingly, the service user's needs, emotional state, behavioral patterns, and / or preferences may be derived based on the service user's visual information, vocal information, physical information, and / or contextual information. Virtual model interactions may be updated to reflect the service user's needs, emotional state, behavioral patterns, and / or preferences derived in this way.

[0054] Furthermore, various information regarding service users can change in real time, and accordingly, the interactions of the virtual model can also change dynamically. Based on service user data and a knowledge graph regarding a specific service domain, these virtual model interactions may include responses optimized in real time for the service user and the specific service domain.

[0055] In step S130, the device can output video and audio of the actual model transitioning from the virtual model based on a specific event.

[0056] For example, a specific event may correspond to an event that is not included in the event patterns pre-trained for a specific service domain—that is, an event outside the prediction range. In this case, the virtual model can be converted into a real model to prevent the output of inappropriate interactions from the virtual model and to provide the service smoothly to the user. As mentioned above, since the virtual model possesses visual, auditory, physical, and contextual homogeneity with the real model, the user may not be aware that the virtual model has been converted into a real model.

[0057] Meanwhile, a transition from the actual model to the virtual model may be performed based on commands from the actual model or other events (e.g., the occurrence of a service flow within the prediction range).

[0058] FIG. 2 is a drawing for illustrating an example of a method for providing a virtual model based on an actual model according to the present disclosure.

[0059] In step S210, the device can generate or update the visual and vocal representations of the virtual model based on the visual and vocal information of the actual model.

[0060] For example, the visual information of the actual model may include physical characteristics such as appearance, gender, and facial expressions, as well as information regarding the surrounding environment and object(s) of the actual space where the actual model is located. The auditory information of the actual model may include information regarding language, volume, tone, pronunciation, intonation, and ambient noise. After being generated based on the information about the actual model, the virtual model may be updated to reflect the changed visual / auditory information of the actual model. Accordingly, information about a virtual model that is homogeneous with the actual model can be generated and updated.

[0061] In step S220, the device can output visual and vocal representations within a three-dimensional virtual space.

[0062] Accordingly, a virtual model created and updated to have homogeneity with the actual model can be output in a three-dimensional virtual space.

[0063] In step S230, the device can output autonomous interaction with one or more virtual objects within the three-dimensional virtual space of the virtual model.

[0064] Autonomous interaction of a virtual model means that the virtual model acts autonomously in response to the surrounding environment within the virtual space, and can be distinguished from the interaction between the virtual model and the service user in the example of Fig. 1.

[0065] For example, a generated or updated virtual model can reflect the visual or auditory information of the actual model in real time. Additionally, in some situations, once a virtual model is generated or updated and output to a virtual space, it can autonomously operate or speak as a virtual model identical to the actual model (i.e., without mimicking the actual model).

[0066] For example, a virtual model can sequentially perform various tasks suitable for a specific service domain. Accordingly, the behavior of the virtual model can be naturally perceived by service users as if it were a real model. For example, through an artificial intelligence model that has learned the actions for multiple tasks, a transition from a first action to a second action is derived according to the flow from a first task to a second task, and accordingly, the virtual model can perform a second action after performing a first action.

[0067] For example, autonomous interactions of a virtual model can be derived based on learning through an artificial intelligence model regarding the interaction of the virtual model with one or more real objects within the real space. Additionally, autonomous interactions of the virtual model may include behaviors that follow physical laws regarding one or more virtual objects within a three-dimensional virtual space. Accordingly, to service users, the interaction of the virtual model with its surrounding environment can be perceived as natural, just like that of the real model.

[0068] In step S240, the device can output video and audio of the actual model transitioning from the virtual model based on a specific event.

[0069] For example, a specific event may correspond to an event that is not included in the event patterns pre-trained for a specific service domain—that is, an event outside the prediction range. In this case, the virtual model can be converted into a real model to prevent the output of inappropriate interactions from the virtual model and to provide the service smoothly to the user. As mentioned above, since the virtual model possesses visual, auditory, physical, and contextual homogeneity with the real model, the user may not be aware that the virtual model has been converted into a real model.

[0070] Meanwhile, a transition from the actual model to the virtual model may be performed based on commands from the actual model or other events (e.g., the occurrence of a service flow within the prediction range).

[0071] The examples of FIGS. 1 and FIGS. 2 can be performed or applied together or in parallel. For example, a virtual model based on a real model according to the example of FIGS. 2 can provide interaction with a user according to the example of FIGS. 1.

[0072] More specific features of the present disclosure will be described below with reference to the examples of FIGS. 1 and FIGS. 2.

[0073] First, we will explain the visual homogenization between the virtual model and the real model.

[0074] To achieve this, a realistic digital human is generated by utilizing video footage of a real human operator (i.e., a real model), and the real model's facial expressions, gestures, and environment are precisely analyzed to enable the digital human (i.e., the virtual model) to reproduce them. Through this, the digital human becomes able to display a natural appearance, just like a real person.

[0075] It is crucial to accurately match the movements of the digital human with those of the real model to suit the service scenario. To achieve this, appropriate blending processing may be applied as the digital human mimics the real model's gestures, facial expressions, and gaze in real time to maintain a consistent video flow. This blurs the boundaries between the real model and the digital human, enabling highly realistic interactions.

[0076] In addition, virtual models must be optimized to operate naturally on various devices, and this must be considered to provide perfect visual integration even when used on smartphones, VR, and AR devices. Typically, movements are reproduced through animation using rigging, which involves creating a digital human skeleton and embedding the bones; blending may also be applied to ensure natural transitions between different tasks. This can be implemented by training AI with various task movements to minimize errors during transitions between different workflows.

[0077] AI-based photorealistic correction (e.g., adding expressions such as facial expressions and wrinkles) may be additionally applied to the images generated through the presented process. Through this, facial expressions and wrinkles of the virtual model (i.e., digital human) can be rendered more realistically to match the actual model, thereby further enhancing visual consistency.

[0078] Next, we will explain the phonetic homogenization between the virtual model and the real model.

[0079] Speech homogenization may be necessary to utilize speech recognition and natural language processing technologies to accurately recognize service user requests and enable digital humans (i.e., virtual models) to respond naturally. By applying AI-powered personalized speech utterance (e.g., Text-to-Speech) technology to the voice of a real model to provide identical utterance patterns, digital humans can be implemented to look and sound like real models (i.e., lip movements and facial expressions during speech). Recent speech processing technologies have advanced to a level where personalized speech utterance learning is possible using only short segments of user voice, typically a few minutes; consequently, this allows digital humans to provide speech characteristics that are homogeneous to those of real models.

[0080] A system that utilizes speech recognition and processing technology to understand the service user's voice in real time and generate appropriate responses can maintain a natural conversational flow between the digital human and the service user. In this process, the digital human acquires information regarding the service user's pronunciation and intonation, and can analyze this information (combined with other information if necessary) and the user's emotional state in real time. By reflecting these analysis results, it becomes possible to provide more natural and human-like interactions between the digital human and the service user. Through this, the service user feels as if they are interacting with a real operator and does not feel any awkwardness or resistance during the conversation with the digital human, thereby significantly enhancing service immersion.

[0081] Next, we will explain the physical homogenization between the virtual model and the real model.

[0082] Physical homogenization may be necessary to implement natural physical behavior in a space where the real and virtual are fused, by analyzing how digital humans adapt to the physical environment in real time and interact autonomously with it. To this end, the movements of real models can be tracked in video, and they can learn to interact with objects. For example, by developing a reinforcement learning-based interaction model for a robot applying agent technology within a 3D virtual space, the virtual model can be enabled to perform tasks autonomously. Once a digital human learns to interact autonomously within a virtual environment, it communicates with its surroundings in real time and performs actions that reflect physical laws. This can be further enhanced through simulations in which the digital human performs learned tasks within a diverse virtual environment specialized for a specific service domain and accumulates knowledge through reinforcement learning. For instance, when a digital human picks up a virtual object, opens a door, or interacts with other objects in a virtual environment, the more natural its physical behavior patterns are, the more it appears like a real person.

[0083] This enables natural interaction between digital humans and service users within a virtual environment such as the metaverse, grants service users a sense of physical reality, and can provide a new form of user experience.

[0084] Furthermore, physical assimilation can also be provided in a way that guides service users, along with digital humans, in their ability to perceive and adapt to the physical environment. For example, when a user is in a specific space or takes a specific action, the digital human can respond accordingly, organically linking physical behavior.

[0085] Next, contextual homogenization between the virtual model and the real model is explained.

[0086] Contextual homogenization may be necessary to learn the patterns of events occurring within the service domain and, based on this, provide a flow suitable for planned service scenarios.

[0087] For contextual homogenization, it is necessary to identify common processes in the actual operator workflow and systematize the service scenario flow in the form of a knowledge graph. To achieve this, a knowledge graph can be obtained by summarizing the characteristics of multimodal data in the training data and dividing it into multiple segments.

[0088] The service flow according to each service domain proceeds as a flow with various branching points, and when responding to actual services, stable services can be provided by finding and connecting the most appropriate node in a service scenario in the form of a pre-built knowledge graph through the given multimodal input.

[0089] The process of accurately predicting various situations that arise as the service progresses and providing appropriate responses to service users is crucial for a stable service.

[0090] For example, even if a service user makes a special request or performs unexpected actions during the service, the digital human must be able to grasp the context and provide an appropriate response.

[0091] If a service user's requests or behaviors fall outside the predicted range, it may be necessary to switch to an actual operator to continue the service, rather than having the digital human respond with unnatural actions or utterances unsuitable for the service domain. For example, if a significant event occurs or an unlearned situation arises, processing is required to seamlessly transition the digital human into an actual operator to ensure a smooth connection and allow the user to receive the service without interruption.

[0092] Alternatively, if the actual operator is not in a state to respond, there are inevitably limitations in ensuring homogeneity solely through technical means. In cases where switching between a digital human and a real operator is difficult, the service user can be made aware of the intervention of another operator through methods such as simulating a conversation with a different virtual operator. Consequently, by switching the virtual operator to a different real operator, it is possible to address situations where an immediate transition between the existing virtual and real operators is difficult. Such appropriate situational judgment and proactive operator switching can be provided by utilizing a knowledge graph flow system that has previously understood and learned domain characteristics.

[0093] As mentioned above, since the digital human (i.e., virtual model) possesses visual, auditory, physical, and contextual homogeneity with the actual operator (i.e., actual model), the service user may not recognize that the digital human has been switched to the actual operator.

[0094] In this process, AI-based predictive algorithms can be utilized to forecast service flow and continuously improve service quality. While services such as responses and consultations generally follow fixed patterns, even when the service is modified or adaptively changed due to users behaving or speaking in unpredictable ways, a consistent user experience can be guaranteed through contextual homogenization by adding the additional service flow as knowledge graph nodes.

[0095] In addition, the user experience can be continuously improved by utilizing real-time data analysis and reinforcement learning. By analyzing data generated in real time, digital humans can quickly recognize situations involving interaction with service users and respond appropriately. For example, reinforcement learning can be used to progressively improve the behavioral patterns of digital humans and continuously refine the system to provide services optimized for the user experience. To achieve this, real-time data analysis can identify users' emotional states, behavioral patterns, and preferences, and based on this, these findings can be reflected in the autonomous interactions of digital humans and / or interactions with service users to provide better services. Through the aforementioned examples, intelligent digital humans can provide increasingly personalized services and deliver a continuously improving user experience.

[0096] Specific application examples of the present disclosure are described below.

[0097] By learning the video and audio of hospital operators (i.e., real models), a smart medical consultation digital human (i.e., a virtual model) can be generated, enabling rapid consultation and guidance without waiting for simple patient health status checks and medical appointment scheduling. For this purpose, the consultation digital human agent can provide consultation and guidance optimized for the patient's condition based on data obtained from medical examinations conducted by the user at a medical institution. The digital human agent continuously improves service quality by systematizing medical knowledge into a knowledge graph and analyzing conversations with patients using reinforcement learning. It contributes to reducing the administrative burden on hospitals and maximizing patient convenience through 24-hour consultation capabilities, while also facilitating efficient and reliable medical consultations across various service contexts specialized in the medical service domain. Furthermore, utilizing digital humans allows for expansion into multilingual support services for foreign patients seeking medical tourism, as well as services that recommend appropriate insurance products based on patient status and hospital data.

[0098] FIG. 3 shows an example of a block diagram of an apparatus according to the present disclosure.

[0099] The device (300) may include one or more processors (310), one or more memories (320), one or more transceivers (330), one or more user interfaces (340), etc. The memory (320) may be included in the processor (310) or configured separately. The memory (320) may store instructions that cause the device (300) to perform an operation when executed by the processor (310). The transceiver (330) may transmit and / or receive signals, data, etc. that the device (300) exchanges with other entities. The user interface (340) may receive user input regarding the device (300) or provide the output of the device (300) to the user. Among the components of the device (300), components other than the processor (310) and memory (320) may not be included depending on the case, and other components not shown in FIG. 3 may be included in the device (300).

[0100] The processor (310) may be configured to enable the device (300) to perform the operation of the device according to various examples of the present disclosure. Although not illustrated in FIG. 3, the processor (310) may be configured as a set of modules that perform each function. The modules may be configured in the form of hardware and / or software.

[0101] For example, the processor (310) may be configured to output a virtual model based on a real model through a user interface (340) in a three-dimensional virtual space, output interactions of the virtual model through the user interface (340) based on visual / audio / physical / contextual information of the service user, and output images and audio of the real model switched from the virtual model through the user interface (340) based on a specific event.

[0102] Furthermore, the processor (310) may be configured to generate or update visual and auditory representations of a virtual model based on visual and auditory information of a real model and output them through a user interface (340) in a three-dimensional virtual space, output autonomous interaction with one or more virtual objects in the three-dimensional virtual space of the virtual model through the user interface (340), and output images and audio of a real model switched from a virtual model based on a specific event through the user interface (340).

[0103] According to the various examples of the aforementioned disclosure, it is possible to expand beyond existing voice-based conversational user services to metaverse-based conversational services that utilize video and space. Through this, the efficiency of the service can be enhanced by facilitating a seamless transition with intelligent digital humans, thereby reducing the workload of actual operators and enabling the provision of more detailed customer service.

[0104] In addition, according to various examples of the present disclosure, interactions that sustain the flow of the service can be provided through a digital human, with the same degree as the online / offline service process in a real environment, by grasping the response of the service user and obtaining important clues from multimodal information such as conversations or non-verbal interactions.

[0105] In addition, according to various examples of the present disclosure, through pre-learned service scenarios, it is possible to provide a user experience that is free from long waiting times occurring in conventional voice-based consultations, is more user-friendly, and minimizes time constraints.

[0106] In addition, according to various examples of the present disclosure, by extending the use of digital humans to a metaverse where the real and virtual are fused, services based on user and spatial interaction can be provided in an XR environment.

[0107] The components described in the exemplary embodiments of the present disclosure may be implemented by hardware elements. For example, the hardware elements may include at least one of a digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element such as an FPGA, a GPU, other electronic devices, or a combination thereof. At least some of the functions or processes described in the exemplary embodiments of the present disclosure may be implemented in software, and the software may be recorded on a recording medium. The components, functions, and processes described in the exemplary embodiments may be implemented by a combination of hardware and software.

[0108] A method according to one embodiment of the present disclosure may be implemented as a program that can be executed by a computer, and said computer program may be recorded on various recording media such as magnetic storage media, optical reading media, digital storage media, etc.

[0109] The various technologies described in this disclosure may be implemented as digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The technologies may be implemented as computer program products, namely, computer programs tangibly implemented on information media or computer programs (e.g., machine-readable storage devices (e.g., computer-readable media) or data processing devices), or as computer programs implemented as signals processed by or propagated to perform operations of data processing devices (e.g., programmable processors, computers, or a plurality of computers).

[0110] Computer program(s) may be written in any form of programming language, including compiled or interpreted languages, and may be distributed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may be executed by a single computer, or by multiple computers separated at one site or multiple sites and interconnected by a communication network.

[0111] Examples of processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and one or more processors of a digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. Components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Additionally, the computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or may be connected to said mass storage devices to receive and / or transmit data. Examples of information media suitable for implementing computer program instructions and data include semiconductor memory devices (magnetic media such as hard disks, floppy disks, and magnetic tapes), optical media such as compact disc read-only memory (CD-ROM) and digital video discs (DVD), magneto-optical media such as floptical disks, and Read Only Memory (ROM), Random Access Memory (RAM), flash memory, Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), and other known computer-readable media. Processors and memory may be complemented or integrated by special-purpose logic circuits.

[0112] A processor may execute an operating system (OS) and one or more software applications running on the OS. A processor unit may also access, store, manipulate, process, and generate data in response to software execution. For simplification, a processor unit has been described in the singular; however, a person of ordinary skill in the art will understand that a processor unit may include multiple processing elements and / or various types of processing elements. For example, a processor unit may include multiple processors or a processor and a controller. It may also constitute different processing structures, such as parallel processors. Furthermore, a computer-readable medium means any medium accessible by a computer and may include both computer storage media and transmission media.

[0113] The present disclosure includes detailed descriptions of various detailed embodiments, but such details are not intended to limit the invention or claims proposed in the present disclosure and should be understood as describing the features of specific exemplary embodiments.

[0114] Features individually described in exemplary embodiments in this disclosure may be implemented by a single exemplary embodiment. Conversely, various features described with respect to a single exemplary embodiment in this disclosure may be implemented by a combination of multiple exemplary embodiments or a suitable sub-combination. Furthermore, in this disclosure, said features may operate by a specific combination and may be described as said combination first claimed, but in some cases, one or more features may be excluded from the claimed combination, or the claimed combination may be changed into a sub-combination or a modified form of a sub-combination.

[0115] Likewise, even if operations are described in a specific order in the drawings, it should not be understood that it is necessary to execute the operations in a specific sequence or order, or that all operations must be performed, in order to obtain the desired result. In certain cases, multitasking and parallel processing may be useful. Furthermore, it should not be understood that the various device components in the exemplary embodiments of all embodiments must be separated, and the aforementioned program components and devices may be packaged into a single software product or multiple software products.

[0116] The exemplary embodiments disclosed in this specification are merely illustrative and are not intended to limit the scope of this disclosure. Those skilled in the art will recognize that various modifications to the exemplary embodiments may be made without departing from the spirit and scope of the claims and their equivalents.

[0117] Accordingly, the present disclosure shall be deemed to include all other substitutions, modifications, and changes falling within the scope of the following claims.

Claims

Claim 1 A method for providing interaction between a service user and a virtual model for a specific service domain, the method comprising: a step of outputting a virtual model based on a real model in a three-dimensional virtual space; a step of outputting an interaction of the virtual model based at least on the physical information of the service user; and a step of outputting an image and audio of the real model switched from the virtual model based on a specific event. Claim 2 A method according to claim 1, wherein the physical information comprises one or more of information regarding the actions of the service user or the physical environment of the service user. Claim 3 In claim 2, the interaction of the virtual model comprises an action of the virtual model responding to an action of the service user. Claim 4 In claim 2, the interaction of the virtual model comprises the operation of the virtual model presented to the service user with respect to the service user's physical environment. Claim 5 A method according to claim 1, wherein the interaction of the virtual model is further based on one or more of the service user's visual information, auditory information, or contextual information. Claim 6 In claim 5, the method comprises contextual information including service flow information based on a pattern of events occurring within the specific service domain. Claim 7 In claim 6, the method wherein the service flow information is predicted by an artificial intelligence model-based prediction algorithm. Claim 8 A method in which the interaction of the virtual model is updated by reflecting one or more of the service user's needs, emotional state, behavioral patterns, or preferences derived based on one or more of the service user's visual information, the voice information, the physical information, or the contextual information. Claim 9 A method according to claim 1, wherein the interaction of the virtual model is dynamically changed based on real-time changing information regarding the service user. Claim 10 A method according to claim 1, wherein the interaction of the virtual model comprises a response optimized in real time for the service user and the specific service domain based on the service user's data and a knowledge graph for the specific service domain. Claim 11 A device for providing interaction between a service user and a virtual model for a specific service domain, wherein the device comprises: one or more user interfaces; one or more transceivers; one or more memories; and one or more processors, wherein the one or more processors are: Outputting a virtual model based on a real model within a three-dimensional virtual space through one or more user interfaces; Based at least on the physical information of the service user, the interaction of the virtual model is output through the one or more user interfaces; and A device configured to output video and audio of the actual model, which is switched from the virtual model based on a specific event, through the one or more user interfaces. Claim 12 A method for providing a virtual model based on a real model, wherein the method comprises: generating or updating a visual representation and a voice representation of the virtual model based on visual information and voice information of the real model; outputting the visual representation and the voice representation into a three-dimensional virtual space; outputting an autonomous interaction of the virtual model with one or more virtual objects within the three-dimensional virtual space; and outputting an image and voice of the real model that are switched from the virtual model based on a specific event. Claim 13 In claim 12, a method wherein, in the operation switching of the virtual model, a transition from a first operation to a second operation according to the flow from a first operation to a second operation is derived through an artificial intelligence model that has learned operations for a plurality of operations. Claim 14 In claim 12, the autonomous interaction of the virtual model is derived based on learning through an artificial intelligence model of the interaction with one or more real objects in the real space of the real model. Claim 15 In claim 12, the autonomous interaction of the virtual model comprises behavior in accordance with physical laws for one or more virtual objects within the three-dimensional virtual space. Claim 16 In claim 12, the method wherein the specific event corresponds to an event not included in an event pattern pre-learned for a specific service domain. Claim 17 A method according to claim 12, wherein the transition between the virtual model and the actual model is performed seamlessly based on the homogeneity of the virtual model and the actual model. Claim 18 In claim 12, the virtual model provides a real-time optimized response to the service user and the specific service domain based on the service user's data and a knowledge graph of the specific service domain. Claim 19 A method according to claim 12, wherein the virtual model switched from the actual model is output based on a command or other event of the actual model. Claim 20 In a device for providing a virtual model based on a real model, the device comprises: one or more user interfaces; one or more transceivers; one or more memories; and one or more processors, wherein the one or more processors are: Based on the visual and auditory information of the actual model, generate or update the visual and auditory representations of the virtual model; Outputting the above visual representation and the above vocal representation within a three-dimensional virtual space through the one or more user interfaces; Outputting autonomous interaction with one or more virtual objects within the three-dimensional virtual space of the above virtual model through the one or more user interfaces; and A device configured to output video and audio of the actual model, which is switched from the virtual model based on a specific event, through the one or more user interfaces.

Images