Multisensory human-computer interaction system and multisensory human-computer interaction method

Abstract

This invention provides a multisensory human-computer interaction system and a multisensory human-computer interaction method. [Solution] A multisensory human-computer interaction system comprising an interaction device, a plurality of sensors, and a server, wherein each of the plurality of sensors performs the operation of receiving a plurality of sensing signals from the real environment; the server performs the operation of generating event data corresponding to each of the plurality of sensing signals and representing the situation of the real environment based on the plurality of sensing signals; the server performs the operation of generating a plurality of multisensory relevance data corresponding to the situation based on the plurality of event data; the server performs the operation of determining a sensory feedback result based on the plurality of multisensory relevance data; and the interaction device performs the operation of generating an interaction message for interacting with the user based on the sensory feedback result.

Claims

1. In a multisensory human-computer interaction system including an interaction device, a plurality of sensors, and a server coupled to the sensors and for controlling the interaction device, Each of the aforementioned sensors operates to receive multiple sensing signals from the real environment, The server performs the operation of generating event data corresponding to each of the sensing signals and representing the status of the actual environment, based on the sensing signals. The server performs the operation of generating multiple multisensory relationship data corresponding to the situation based on the event data, The server performs an operation to determine the sensory feedback result based on the multisensory relationship data, The interaction device performs the operation of generating an interaction message for interacting with the user based on the sensory feedback result, This is what does the following: The server performs a marking operation on each of the event data to generate a plurality of marks, including at least one of the situation meaning marks, distance marks, and sensory evaluation value / intensity marks. The server analyzes the marks between each of the event data and performs an operation to generate multisensory relationship data that represents at least one relationship between the event data. The server performs a scoring operation on each of the multisensory relevance data to generate a relevance score, The server performs the operation of comparing the correlation scores based on a plurality of pre-configured values ​​and generating a plurality of comparison results, The server, based on the comparison results, determines the operation feedback of multiple senses of the interaction device, including at least one of visual, auditory, olfactory, gustatory, and tactile senses, as the sensory feedback result. A multi-sensory human-computer interaction system characterized by its ability to perform the following actions.

2. The interaction device is a humanoid robot capable of understanding the conditions of the real environment, and the multisensory human-computer interaction system according to claim 1 includes a plurality of sensory output devices, including at least one of a visual output device, an auditory output device, a tactile output device, an olfactory output device, a gustatory output device, and a brain-machine interconnection device.

3. The multisensory human-computer interaction system according to claim 1, wherein the interaction device is a virtual robot for projecting virtual reality, and the virtual robot is coupled to a plurality of sensory output devices, and the sensory output devices generate motion feedback for a plurality of senses, including at least one of the visual sense, auditory sense, olfactory sense, gustatory sense and tactile sense.

4. The multisensory human-computer interaction system according to claim 1, wherein the sensor comprises at least one of a visual sensor, an auditory sensor, a tactile sensor, an olfactory sensor, a gustatory sensor, and a brain-machine interconnection device.

5. The aforementioned multi-sensory human-computer interaction system is The interaction device performs an action that generates motion feedback for multiple senses based on the sensory feedback result, The multisensory human-computer interaction system according to claim 1, wherein the interaction device performs an action that assists the user in responding to the situation in the actual environment based on the motion feedback.

6. In a multisensory human-computer interaction method that performs the following steps using a multisensory human-computer interaction system, A process of receiving multiple sensing signals from the real environment, A step of generating event data that corresponds to each of the aforementioned sensing signals and represents the state of the actual environment, based on the aforementioned sensing signals. A step of generating multiple multisensory relationship data corresponding to the situation based on the event data, A step of determining the sensory feedback result based on the aforementioned multisensory relationship data, Based on the aforementioned sensory feedback results, the process involves generating interaction messages for interacting with the user and controlling the interaction device. Equipped with, The step of generating event data corresponding to each of the sensing signals based on the sensing signals is: A step of performing a marking operation on each of the aforementioned event data to generate a plurality of marks, including at least one of semantic marks, distance marks, and sensory evaluation value / intensity marks, The process further includes analyzing the marks between each of the event data to generate multisensory relationship data representing at least one relationship between the event data, The process involves performing a scoring operation on each of the aforementioned multisensory relevance data to generate a relevance score. A step of comparing the correlation scores based on multiple pre-set values ​​to generate multiple comparison results, Based on the comparison results, the process involves determining the operation feedback of multiple senses of the interaction device, including at least one of visual, auditory, olfactory, gustatory, and tactile senses, as the sensory feedback result. A multi-sensory human-computer interaction method characterized by further comprising the following features.

7. The multisensory human-computer interaction method according to claim 6, wherein the interaction device is a humanoid robot capable of understanding the conditions of the real environment, and includes a plurality of sensory output devices, including at least one of a visual output device, an auditory output device, a tactile output device, an olfactory output device, a gustatory output device, and a brain-machine interconnection device.

8. The multisensory human-computer interaction method according to claim 6, wherein the interaction device is a virtual robot for projecting virtual reality, and the virtual robot is coupled to a plurality of sensory output devices, and the sensory output devices generate operational feedback for a plurality of senses, including at least one of the visual sense, auditory sense, olfactory sense, gustatory sense and tactile sense.

9. A step of generating motion feedback for multiple senses based on the aforementioned sensory feedback results, The process of assisting the user in responding to the situation in the actual environment through the aforementioned operational feedback, The multisensory human-computer interaction method according to claim 6, further comprising the above.

Images