Experience-support wearable terminal and dialogue support system by ai character

Abstract

To present appropriate support information for experiential activity of a user.SOLUTION: An experiential support system automatically identifies a target scene on the basis of a gaze position, an object of fixation, or voice input, and presents corresponding support information through an interactive AI character by image or voice. The system can be applied to childcare, occupational experience, spatial design, emotional response, multilingual navigation, translation support, and the like, and adaptively controls the content according to behavior or emotion of a user to enhance the quality of experience.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an experience support system that automatically identifies an object of a user's attention based on gaze and voice input, and presents support information corresponding to that object via an interactive AI character. In particular, the present invention relates to technology that can be used in a wide range of applications, such as childcare experiences, work experiences, space design support, navigation, interpretation, and educational support, using virtual spaces and augmented reality environments in conjunction with wearable information terminals. [Background technology]

[0002] In recent years, with the development of virtual reality (VR) and augmented reality (AR) technologies, there has been a growing need for interactive experience environments that provide support according to the user's behavior and psychological state.In addition, guidance and support technologies using conversational AI characters that use natural language have been widely put into practical use, but conventional technologies have limited functionality in identifying the target of the user's attention in real time and providing support information that is optimal for that situation. Summary of the Invention [Problem to be solved by the invention]

[0003] The present invention relates to an experience support system that can guide or assist a user's experience activities by automatically identifying an object of interest to the user based on gaze position, gaze target, or voice input, and presenting support information related to that object using an AI character.

[0004] The experience support system of the present invention includes a wearable information terminal with visual information acquisition and audio input functions, and is characterized by automatically recognizing a target scene in a real space or a virtual space based on the user's line of sight, gaze direction, audio content, etc., and presenting support information corresponding to the scene via video or audio.

[0005] The experience support system may be configured to provide virtual or simulated experience content related to childcare, for example. An AI character is generated as a virtual child character and dynamically provides childcare support, suggestions, and empathetic dialogue based on the user's speech, gaze, behavior, or emotional responses.

[0006] The experience support system may also be configured to present a simulated environment for a specific occupation. In this case, the AI ​​character presents work procedures, options, conversation flow, etc. in response to the user's actions, providing an experience in which the scenario unfolds differently depending on the user's choices.

[0007] Furthermore, virtual furniture, building components, interior elements, structures, etc. may be superimposed on the user's field of view as augmented reality. The AI ​​character will make suggestions and simulations based on the layout, traffic flow, landscape design, and sensory evaluation of the displayed objects, and provide support for spatial design choices.

[0008] This experience support system estimates the user's emotional state or psychological tendency based on information such as facial expressions, voice intonation, speech pauses, eye movements, and gestures, and dynamically adjusts the AI ​​character's speech tempo, voice type, content, facial expressions, output timing, etc. according to that state, thereby realizing shared experience support that is suited to the user's psychological state.

[0009] Furthermore, the system can be configured to include a review function that records the user's experiences, comments, choices, emotional responses, gaze history, behavioral logs, etc. in chronological order on the cloud, allowing the data to be shared with the user themselves, their guardians, educational institutions, or third parties, and that allows them to view, play, compare, analyze, or check editing history.

[0010] In addition, the AI ​​character can support multiple languages, and by switching the language for voice output, screen display, option presentation, navigation, etc. depending on the user's selection or usage situation, it is possible to provide educational support, travel support, interview support, or cross-cultural support in multiple languages.

[0011] Furthermore, map information, location information, and route guidance information can be acquired, and an AI character can present a route based on the user's current location and destination, displaying virtual guidance displays or guidance actions within the user's field of view, and notifying the user of the direction of travel, estimated arrival time, or points of caution by voice. The system can also be equipped with an interpretation function that translates conversations in a foreign language and responds based on the user's input or reaction. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the overall configuration of an experience support system according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram showing a processing flow for presenting assistance information based on a user's gaze target. [Figure 3] FIG. 10 is an explanatory diagram showing an example of support provided in a virtual space for childcare experience, work experience, or the like. [Figure 4] Examples of adaptive processing of character responses based on emotional states and multilingual translation navigation. DETAILED DESCRIPTION OF THE INVENTION

[0013] An experience support system according to the present invention includes a wearable information terminal (e.g., smart glasses or a head-mounted display) that can acquire visual information and voice input information from a user.

[0014] The wearable device has a camera, microphone, speaker, gaze sensor, acceleration sensor, etc., and acquires information in real time about the user's gaze position, gaze target, speech content, facial expressions, movements, environmental sounds, etc. The acquired information is analyzed by a processing device internally or on the cloud, and the target scene in real space or virtual space that the user is focusing on is automatically identified.

[0015] Depending on the target scene, the system selects support information and presents it through an interactive AI character. The support information is presented in the form of video (e.g., guide images, overlaid information, virtual objects, etc.) or audio (e.g., explanations, suggestions, dialogue, etc.) to guide or assist the user's actions.

[0016] For example, in the childcare experience support, a virtual child character is displayed and responds to the user's voice and actions. The AI ​​character dynamically controls responses such as praise, encouragement, suggestions, and empathy according to the user's gaze, voice, and emotions, providing a simulated childcare experience.

[0017] In addition, in the simulated experience of a specific occupation, an AI character presents work procedures and options in a virtual space that mimics the workplace. The story branches depending on the user's choices and utterances, and feedback is provided according to the results, making interactive learning possible.

[0018] In the space design support mode, virtual furniture, structures, interiors, etc. are superimposed on the real space, and an AI character presents layout proposals, traffic flow, aesthetic evaluation of the landscape, etc. This allows users to simulate space design and design selection even before the actual objects are present.

[0019] Furthermore, it is also capable of adapting to the user's emotions and psychological state. For example, it can estimate whether the user is anxious, confused, or focused based on speech pauses, tone of voice, eye movements, gestures, etc., and automatically adjust the AI ​​character's facial expression, speech content, presentation timing, and amount of information.

[0020] This series of experience logs (comments, choices, emotional responses, gaze history, etc.) are recorded in chronological order on the cloud for each user, and can be used for later review, sharing, and analysis by Japanese people or related parties. The logs can also be shared with third parties in an anonymous or permission-controlled state, and can be used for collaboration with educational institutions and families.

[0021] The system also supports multilingual environments, with the AI ​​character's voice, subtitles, and display content changing depending on the language selected by the user. It also has an interpretation function, providing real-time translation of conversations in foreign languages, suggesting reply suggestions, and assisting users in speaking.

[0022] As a navigation aid, the route to the destination is presented using virtual arrows and character guidance based on map information and current location information, and the appropriate direction to travel is displayed within the line of sight while walking or moving. This allows for intuitive guidance without having to look away, and audio notifications are also provided regarding the required travel time and important points to note.

[0023] According to this invention, by combining various input means such as vision, speech, and emotion with complex output means such as virtual space, augmented reality, and interactive characters, it is possible to provide a next-generation experience support interface that goes beyond simply presenting information and allows users to actively and naturally deepen their experience.

Claims

1. An experiential support system that uses a wearable information terminal with visual information acquisition and voice input functions to automatically identify a target scene in real or virtual space that the user is focusing on based on the user's gaze position, gaze target, or voice input, and presents support information corresponding to the target scene through a visual display or voice output via an interactive AI character, thereby guiding or assisting the user's experiential activities.

2. 2. The experience support system of claim 1, wherein the support information includes virtual or simulated experience content related to child-rearing, and the AI ​​character generates and controls a virtual child character, and has the function of dynamically providing child-rearing support, child-rearing suggestions, or empathetic dialogue in response to the user's speech, gaze, behavior, or emotional response.

3. 3. The experience support system according to claim 1 or 2, wherein the experience activity includes a virtual environment or an augmented reality environment that simulates a specific occupation, and the AI ​​character presents work procedures, options, or conversation scenarios, and has the function of branching and changing the experience scenario depending on the user's actions or selection results.

4. An experience support system according to any one of claims 1 to 3, wherein virtual furniture, building components, interior elements or structures are superimposed on the user's field of vision as augmented reality, and the AI ​​character provides suggestions, comparisons, simulations or emotional comments regarding the placement, traffic flow, landscape or flow design of the displayed objects, thereby supporting the user in spatial design and sensory selection.

5. An experience support system according to any one of claims 1 to 4, which estimates the user's emotional state or psychological tendency based on their facial expressions, intonation of voice, speech pauses, eye movements or gestures, and adaptively adjusts the AI ​​character's speech tempo, type of voice, content, tone, facial expressions or difficulty of presented information, thereby realizing shared experience support that is suited to the user's psychological state.

6. An experience support system according to any one of claims 1 to 5, which records a user's experience content, comments, selections, emotional responses, gaze history and behavior log in chronological order on the cloud, enables the recorded data to be shared with the user, their family, educational institutions or third parties, and is equipped with a review function that enables viewing, playback, comparison, analysis or confirmation of editing history.

7. 7. An experience support system according to any one of claims 1 to 6, wherein the AI ​​character supports multiple languages, and voice output, screen display, option presentation, and dialogue navigation are executed in different languages ​​depending on the user's selection or usage situation, thereby enabling education support, travel support, interview support, or cross-cultural support to be provided in multiple languages.

8. An experience support system according to any one of claims 1 to 7, which acquires map information, location information, and route guidance information, and has the AI ​​character present a route based on the user's current location and destination, displays a virtual guide display or the character's guiding action within the user's field of view, and notifies the user of the direction of travel, estimated arrival time, or points to note by voice, and has an interpretation function that translates the content of conversations in a foreign language and responds based on the user's input or reaction.

Images