Information processing system, information processing method, and program

Abstract

[Problem] To provide, for example, an information processing system capable of offering a novel user experience with a highly soothing effect. [Solution] One aspect of the present invention provides an information processing system. This information processing system includes at least one processor, and the processor is configured to execute a program so that each of the following steps is performed. In a setting step, theme information indicating at least one of a plurality of pre-managed themes is set. In a first presentation step, multimodal information corresponding to the theme information is presented to a user located in a closed space. The multimodal information is a combination of information comprising at least two of visual information, auditory information, tactile information, and olfactory information.

Description

Information processing system, information processing method and program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] For example, Patent Document 1 discloses an environmental control system for a private space that allows users to easily relieve stress without acquiring special knowledge and has a low barrier to implementation. The environmental control system includes a lighting device installed in the private space and capable of changing the illuminance or color temperature within the private space, a video output device that outputs video within the private space, and a control device equipped with a communication device that can send control instructions to the lighting device and the video output device. The control device controls the lighting device and the video output device to perform a lighting relaxation control that reduces the illuminance or color temperature within the private space and a video relaxation control that outputs a first video within the private space, and then a lighting awakening control that increases the illuminance or color temperature within the private space and a video awakening control that outputs a second video different from the first video within the private space.

[0003] Japanese Patent Application Laid-Open No. 2020-053346

[0004] However, there is a demand for a new user experience that has a soothing effect, different from the technology disclosed in Patent Document 1.

[0005] In view of the above circumstances, the present invention provides an information processing system and the like that can realize a novel user experience that has a highly soothing effect.

[0006] According to one aspect of the present invention, there is provided an information processing system. The information processing system includes at least one processor configured to execute a program to perform the following steps: In a setting step, theme information indicating at least one of a plurality of themes managed in advance is set; In a first presentation step, multimodal information corresponding to the theme information is presented to a user present in a closed space; The multimodal information is a combination of at least two of visual information, auditory information, tactile information, and olfactory information.

[0007] According to this aspect, a novel user experience with a highly soothing effect can be realized.

[0008] FIG. 5 is a configuration diagram showing the hardware configuration of an information processing system 1 according to a first embodiment. FIG. 6 is a block diagram showing the functional configuration of an information processing device 2 according to an embodiment. FIG. 7 is an overview diagram showing a specific configuration of the information processing system 1. FIG. 8 is an activity diagram showing an overview of information processing. FIG. 9 shows examples of screens presented to a user U, with FIG. 5A illustrating a screen 8a corresponding to activity A104 and FIG. 5B illustrating a screen 8b corresponding to activity A109. FIG. 10 is an overview diagram showing an example of a method for presenting visual information VI. FIG. 11 shows an experimental protocol of an example for verifying effectiveness. FIG. 12 shows experimental results of an example. FIG. 13 shows experimental results of an example.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0010] Incidentally, a program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] In one embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In one embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage or current, high or low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations can be performed on the circuit in the broad sense.

[0012] Furthermore, a circuit in the broad sense is a circuit realized by at least an appropriate combination of a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] [Embodiment] 1. Hardware Configuration In this section, the hardware configuration of one embodiment will be described.

[0014] 1.1 Information Processing System 1 FIG. 1 is a diagram illustrating the hardware configuration of an information processing system 1 according to a first embodiment. The information processing system 1 includes, for example, an information processing device 2, an image output device 3, an audio output device 4, a tactile presentation device 5, an olfactory presentation device 6, and a wearable device 7, all of which are electrically connected. These components will be further described. Here, the system exemplified as the information processing system 1 is composed of one or more devices or components. Therefore, it should be noted that the information processing system 1 includes both the information processing device 2 alone and a combination of the information processing device 2 and at least two of the image output device 3, the audio output device 4, the tactile presentation device 5, and the olfactory presentation device 6. More specifically, the information processing system 1 may include elements selected from the group consisting of the information processing device 2 and the components 3 to 7. Furthermore, multiple information processing devices 2 may be used corresponding to the respective components 3 to 7. The unselected elements are not included in the information processing system 1, but are electrically connected to the selected elements as external elements.

[0015] 1.2 Information Processing Device 2 The information processing device 2 has a communication unit 21, a storage unit 22, a control unit 23, and an input unit 24, and these components are electrically connected via a communication bus 20 inside the information processing device 2. Each component will be further described.

[0016] The communication unit 21 is preferably a wired communication means such as USB, IEEE 1394, Thunderbolt (registered trademark), or wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 5G, Bluetooth (registered trademark), or the like, as necessary. That is, it is more preferable to implement it as a collection of multiple communication means. That is, the information processing device 2 may communicate various information from the outside via the communication unit 21 and a network. Specifically, the information processing device 2 is connected to the image output device 3, the audio output device 4, the tactile presentation device 5, and the olfactory presentation device 6 via the communication unit 21 and each transmission path. Each of the components 3 to 6 will be described later.

[0017] The memory unit 22 stores various pieces of information defined above. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the information processing device 2 executed by the control unit 23, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to program calculations. The memory unit 22 stores various programs, variables, etc. related to the information processing device 2 executed by the control unit 23.

[0018] The control unit 23 (processor) processes and controls the overall operation of the information processing device 2. The control unit 23 is, for example, a central processing unit (CPU) (not shown). The control unit 23 realizes various functions related to the information processing device 2 by reading predetermined programs stored in the storage unit 22. In other words, information processing by software stored in the storage unit 22 is specifically realized by the control unit 23, which is an example of hardware, and can be executed as each functional unit included in the control unit 23. This will be described in more detail in the next section. Note that the control unit 23 is not limited to a single unit, and multiple control units 23 may be implemented for each function. A combination of these may also be used. In particular, the control unit 23 can generate control signals for controlling the operation of components 3 to 6, which will be described later. In other words, components 3 to 6 can be controlled by the control unit 23 (processor) of the information processing device 2.

[0019] The input unit 24 may be included in the housing of the information processing device 2 or may be externally attached. For example, the input unit 24 may be implemented as a touch panel integrated with a display (which may be the display 31 described below). The touch panel allows the user to input tapping, swiping, and the like. Of course, instead of a touch panel, a switch button, a mouse, a QWERTY keyboard, a microphone for voice input, and the like may also be used. In other words, the input unit 24 accepts an operation input made by the user. The input is transferred as a command signal to the control unit 23 via the communication bus 20, and the control unit 23 can execute predetermined control or calculation as necessary.

[0020] 1.3 Image Output Device 3 The image output device 3 is a device that can present visual information (hereinafter also referred to as visual information VI) to a user and may display a screen of a graphical user interface (GUI) that can be operated by the user. The image output device 3 may be, for example, a display 31 described below or a projector 32 (see FIG. 3). When the image output device 3 is the display 31, it is preferable to use a display device such as a CRT display, a liquid crystal display, an organic EL display, or a plasma display.

[0021] 1.4 Audio Output Device 4 The audio output device 4 is a device that can present auditory information (hereinafter also referred to as auditory information AI) to the user. For example, it may be headphones 41 (described later) or a speaker 42 (see FIG. 3 ). The audio output device 4 is preferably capable of outputting audio in a frequency band above the human audible range. The headphones 41 are preferably open-air type headphones that do not significantly block external sounds even when worn by the user. Furthermore, if the audio output device 4 is a speaker 42, the speaker 42 may be connected to the information processing device 2 via an amplifier (not shown), or may be an active speaker with a built-in amplifier. Furthermore, the audio output device 4 may be an ultrasonic generator (not shown).

[0022] 1.5 Tactile Presentation Device 5 The tactile presentation device 5 is a device that can present tactile information (hereinafter also referred to as tactile information TI) to a user, and may be, for example, an electric fan 51 described below or an air conditioner 52 (see FIG. 3 ). Alternatively, the tactile presentation device 5 may be an ultrasonic generator (not shown) or a glove-type device.

[0023] 1.6 Olfactory Presentation Device 6 The olfactory presentation device 6 is a device that can present olfactory information (hereinafter also referred to as olfactory information OI) to the user, and may be, for example, an aroma diffuser 61 described below.

[0024] To summarize the above, the information processing system 1 includes at least one image output device 3 that provides visual information VI, at least one audio output device 4 that provides audio information AI, at least one tactile presentation device 5 that provides tactile information TI, and at least one olfactory presentation device 6 that provides olfactory information OI. A processor (e.g., a control unit 23) is configured to control at least two of the image output device 3, audio output device 4, tactile presentation device 5, and olfactory presentation device 6, which act on different senses, thereby presenting multimodal information IF2. The multimodal information IF2 will be described in more detail later. This aspect allows for a novel user experience as a combined device. Furthermore, as described above, the audio output device 4 or the tactile presentation device 5 may include at least an ultrasonic generator. This aspect allows for a more novel user experience.

[0025] 1.7 Wearable Device 7 The wearable device 7 is worn by the user U and is configured to be able to continuously measure the user U's biological information via sensors (not shown). The sensors (not shown) may include temperature sensors, infrared sensors, light sensors, pressure sensors, potential sensors, chemical sensors, etc. The wearable device 7 may include, for example, a smart watch, a smart ring, smart glasses, smart earphones, etc. The wearable device 7 can transmit and receive information to and from the communication unit 21 in the information processing device 2 via wireless communication.

[0026] 2. Functional Configuration Next, referring to FIG. 2, the functional configuration of the information processing device 2 of the information processing system 1 will be described. FIG. 2 is a block diagram showing the functional configuration of the information processing device 2 according to one embodiment. As shown in FIG. 2, the control unit 23 (processor) executes various programs stored in the storage unit 22 to function as a reception unit 231, an acquisition unit 232, a setting unit 233, and a presentation unit 234. That is, information processing by software stored in the storage unit 22 is specifically realized by the control unit 23, which is an example of hardware, and can be executed as each functional unit included in the control unit 23. In other words, the information processing system 1 includes at least one processor (e.g., the control unit 23), and the processor executes a program to function as each of the following units. From another perspective, the program causes a computer to execute each step of the information processing system 1. Each step includes, for example, a reception step, an acquisition step, a setting step, and a presentation step.

[0027] The receiving unit 231 is configured to receive various types of information. For example, the receiving unit 231 receives various types of information based on an operation performed by a user via the input unit 24.

[0028] The acquisition unit 232 is configured to acquire various pieces of information as the first acquisition step or the second acquisition step. Preferably, the acquisition unit 232 acquires biometric information of the user U. Details will be described later.

[0029] The setting unit 233 is configured to execute a setting step. The setting unit 233 sets theme information IF1 indicating at least one of a plurality of themes managed in advance. Details will be described later.

[0030] The presentation unit 234 is configured to present various information including visual information VI, auditory information AI, tactile information TI, and olfactory information OI as the first presentation step or the second presentation step. "Presenting" refers to making various information comprehensible to the user, and may include, for example, the control unit 23 generating control signals for controlling the image output device 3, the audio output device 4, the tactile presentation device 5, and the olfactory presentation device 6. For example, when controlling the image output device 3, the presentation unit 234 is configured to output display information to the image output device 3. The display information may be the visual information VI itself, such as a screen, an image, an icon, or text, generated in a manner visible to the user, or may be rendering information for displaying the visual information VI on the image output device 3, for example.

[0031] 3. Overview of Information Processing This section provides an overview of the information processing executed by the information processing system 1. FIG. 3 is a schematic diagram illustrating a specific configuration of the information processing system 1. When using the information processing system 1, a user U is assumed to be in a closed space 11 while wearing a wearable device 7. The closed space 11 is, for example, a small room with white interior walls. The closed space 11 is provided with a display 31 and a projector 32 as examples of image output devices 3, headphones 41 and speakers 42 as examples of audio output devices 4, an electric fan 51 and an air conditioner 52 as examples of tactile presentation devices 5, and an aroma diffuser 61 as an example of an olfactory presentation device 6. Visual information VI1 is presented by the display 31, and visual information VI2 is presented by the projector 32. Auditory information AI1 is presented by the headphones 41, and auditory information AI2 is presented by the speaker 42.

[0032] These are just examples, and at least some of the components may be provided multiple times. The specific layout of the components 3 to 6 is not particularly limited, and they may be arranged in a manner that is favorable for the physical structure of the closed space 11. By receiving the multimodal information IF2 within the closed space 11, the user U can have a new user experience.

[0033] 4 is an activity diagram illustrating an overview of information processing. While a user U is resting in a closed space 11 that is closed off from the outside world, a wearable device 7 acquires biometric information of the user U (activity A101). The biometric information may include, for example, a sleep score that quantifies the quality and quantity of sleep, blood pressure, pulse rate, heart rate, blood oxygen concentration, respiratory rate, body temperature, electrocardiogram, electroencephalogram, blood glucose, alcohol, lactic acid, level of consciousness, number of steps taken, walking distance, amount of exercise, and a biometric score calculated from at least one of these pieces of information.

[0034] Next, the biometric information is shared from the wearable device 7 to the information processing device 2 via wireless communication (activity A102). Next, the acquisition unit 232 acquires (receives) the biometric information of the user U via the communication unit 21 of the information processing device 2 as a first acquisition step (activity A103). Note that thereafter, even as time passes, the wearable device 7 continues to acquire the biometric information of the user U and share it with the information processing device 2.

[0035] Next, the user U's fatigue level estimated based on the biometric information is displayed on the display 31 (activity A104). Next, the user U operates the GUI displayed on the display 31 to select a desired theme (activity A105). At this time, the accepting unit 231 accepts the user's operation. Note that multiple themes may be prepared in advance, and when selecting a theme, a recommended theme may be visualized for the user based on the user's current fatigue level. Thereafter, the setting unit 233 sets the selected theme as theme information IF1 (activity A106). In other words, the setting unit 233 sets theme information IF1 indicating at least one of multiple themes managed in advance.

[0036] Next, at least two of the image output device 3, the audio output device 4, the tactile presentation device 5, and the olfactory presentation device 6 are controlled based on the set theme information IF1 (activity A107). As a result, multimodal information IF2, which is a combination of at least two of visual information VI, auditory information AI, tactile information TI, and olfactory information OI, is presented to the user U (activity A108). In other words, as a first presentation step, the presentation unit 234 presents multimodal information IF2 corresponding to the theme information IF1 to the user U located in the closed space 11. According to this aspect, a novel user experience with a high soothing effect can be realized.

[0037] Furthermore, while the user U is in the closed space 11, the information processing related to activities A107 and A108 may be repeatedly executed. That is, the presentation unit 234 continuously presents the multimodal information IF2. Thereafter, when the repeated information processing related to activities A107 and A108 satisfies a termination condition, the fatigue level of the user U estimated based on the biometric information is again displayed on the display 31 (activity A109). Such termination condition is not particularly limited, and may be, for example, the passage of a certain amount of time, the fatigue level falling below or equal to a threshold value, or the user U performing a predetermined operation via the input unit 24.

[0038] The presented visual information VI, auditory information AI, tactile information TI, and olfactory information OI are presented as unified multimodal information IF2 that gives the user U a sense of a specific theme. More specifically, the multimodal information IF2 is information configured to give the user U the illusion that he or she is in a location consistent with the theme information IF1. For example, the theme selected by the user U may evoke an image of a place that can actually be visited, such as "forest," "beach," or "countryside," or it may represent a place or concept that cannot be reached by humans in their natural state, such as "fire," "undersea," "moon," "space," or "stars." For example, if the set theme is "forest," an image of lush trees is displayed on the display 31 as visual information VI1, and a faint image of sunlight filtering through the trees is projected from the projector 32 onto the inner wall of the closed space 11 as visual information VI2. Furthermore, the sound of a faint breeze is played from the headphones 41 as auditory information AI1, and the chirping of distant birds is played from the speaker 42 as auditory information AI2. Furthermore, the electric fan 51 is operated to provide a gentle breeze, the air conditioner 52 is set to a comfortable temperature range, and the aroma diffuser 61 fills the closed space 11 with the scent of plants. In other words, it is preferable that the control methods for the various components 3 to 6 according to the theme are managed in advance in a table and stored in the storage unit 22. Then, based on the selected theme information IF1, the presentation unit 234 generates and controls control signals for each of the components 3 to 6, thereby presenting multimodal information IF2 according to the theme. This aspect makes it possible to give the user U the illusion of location, thereby realizing a novel user experience.

[0039] Preferably, the presented multimodal information IF2 is configured so that its content or intensity changes over time. For example, if the set theme is "forest," the wind direction may change over time, birds may be heard chirping, or the lighting from the projector 32 may change. This configuration makes it possible to present multimodal information IF2 that the user U will not tire of, thereby providing a novel user experience.

[0040] More preferably, the presentation unit 234 variably presents the multimodal information IF2 in accordance with the biometric information. For example, the content or intensity may vary in accordance with changes in the biometric information. For example, if it is expected that the user U will recover from fatigue, it may be possible to weaken the intensity of the multimodal information IF2. According to this aspect, it is possible to present preferable multimodal information IF2 that is tailored to the user U's physical condition, etc., and ultimately to realize a novel user experience.

[0041] To summarize the above, an information processing method according to one embodiment includes steps executed by the information processing system 1. In the setting step, theme information IF1 indicating at least one of a plurality of themes managed in advance is set. In the first presentation step, multimodal information IF2 corresponding to the theme information IF1 is presented to a user U present in the closed space 11. The multimodal information IF2 is a combination of information consisting of at least two of visual information VI, auditory information AI, tactile information TI, and olfactory information OI. This aspect can provide a novel user experience with a high soothing effect.

[0042] 4. Details of Information Processing In this section, the details of the information processing outlined in the previous section will be explained using additional diagrams, etc.

[0043] 5A shows an example of a screen 8a corresponding to activity A 104, and FIG. 5B shows an example of a screen 8b corresponding to activity A 109. Screens 8a and 8b are displayed on, for example, the display 31. Screen 8a includes areas 81 to 83 and a button 84. Screen 8b also includes areas 81 to 83.

[0044] Focusing on the screen 8a, an area 81 displays text information saying "Your energy level," and five star marks are displayed below it. Of the five star marks, only the leftmost star is displayed in black. This indicates that the user U is not currently in good spirits (high level of fatigue). An area 82 displays text information saying "You seem very tired, so please take a good rest in this space." An area 83 displays an avatar Ab with a facial expression corresponding to the energy level (fatigue level) displayed in the area 81. A button 84 displays text information saying "Select a theme." When the user U presses the button 84 (by clicking, tapping, or other operations), the screen transitions to a screen (not shown) for selecting a theme (theme information IF1). For examples of selectable themes, please refer to the description in the previous section.

[0045] On the other hand, focusing on screen 8b, text information "Your energy level" is drawn in area 81, and five star marks are drawn below it. All of the five star marks are drawn in black. In other words, this indicates that user U is currently in good health (low level of fatigue or no fatigue). Text information "I'm glad you're full of energy" is drawn in area 82. An avatar Ab with a facial expression corresponding to the energy level (fatigue level) presented in area 81 is drawn in area 83.

[0046] In this way, support information IF3 that allows the user U to grasp whether or not, or to what extent, he or she has recovered from fatigue is depicted on the screens 8a and 8b. For example, the information presented in areas 81 to 83 is an example of support information IF3. In other words, as a second presentation step, the presentation unit 234 further presents support information IF3 that allows the user U to grasp whether or not, or to what extent, he or she has recovered from fatigue. According to this aspect, the user U can grasp his or her own physical condition, etc., via the information processing system 1.

[0047] Furthermore, the user U's energy level increases (their fatigue level decreases or they no longer feel fatigued) before and after using the information processing system 1 (i.e., corresponding to the timing when the screens 8a and 8b are displayed). In other words, the presentation unit 234 presents the multimodal information IF2 to the user U who is resting in the closed space 11, thereby promoting recovery from fatigue in the user U. According to this aspect, it is possible to promote recovery from fatigue in the user U.

[0048] 4.2 Presentation of Visual Information VI FIG. 6 is a schematic diagram illustrating an example of a method for presenting visual information VI. As also shown in FIG. 3 , the information processing system 1 according to an embodiment includes a display 31 and a projector 32 as the image output device 3. The display 31 can directly display visual information VI1. The projector 32 can project visual information VI2 onto the inner wall of the closed space 11. As shown in FIG. 3 , while using the information processing system 1, the user U primarily looks directly at the display 31, so the visual information VI1 is primarily presented to the user U. It is also preferable that the visual information VI2 be projected over the entire surface of the inner wall of the closed space 11, which is an area not being gazed upon. In other words, the visual information VI includes visual information VI1 (an example of first visual information) and visual information VI2 (an example of second visual information). The visual information VI1 is presented in the closed space 11 at a position where the user U looks directly. The visual information VI2 is presented around the visual information VI1 so as to cover the field of view of the user U. According to this aspect, contrast can be added to the visual information VI, thereby realizing a novel user experience.

[0049] Although not shown, contrast may also be added to information other than the visual information VI. For example, the information processing system 1 according to one embodiment includes headphones 41 and a speaker 42 as the audio output device 4. The headphones 41 can directly output the auditory information AI1 close to the user U's ears. The speaker 42 can output the auditory information AI2 from a distant position in the closed space 11, creating the illusion of, for example, a faint sound. In other words, the auditory information AI includes auditory information AI1 (an example of first auditory information) and auditory information AI2 (an example of second auditory information). The auditory information AI1 is presented close to the user U's ears. The auditory information AI2 is presented in the closed space 11 so that the user U can recognize it as ambient sound. This aspect adds contrast to the auditory information AI, thereby providing a novel user experience.

[0050] [Others] The information processing system 1 according to an embodiment can also be implemented in the following manner.

[0051] In one embodiment, the display 31 presents the visual information VI1 and the projector 32 presents the visual information VI2, but multiple displays 31 presenting the visual information VI2 may be used, or a projector 32 presenting the visual information VI1 may be used.

[0052] In one embodiment, the headphones 41 present the auditory information AI1 and the speakers 42 present the auditory information AI2, but the speakers 42 may be used to present the auditory information AI1, or the headphones 41 may be used to present the auditory information AI2. When the auditory information AI is presented using only the headphones 41, it is preferable to use a closed-type headphone with a noise-canceling function rather than the open-air type exemplified in the embodiment, in order to improve the sense of immersion. Furthermore, when the speakers 42 are used, it is preferable to use a surround sound system.

[0053] In one embodiment, it is assumed that the user U is resting in the closed space 11, but the user U may also be engaged in an activity other than resting. For example, the user U may eat or drink in the closed space 11. In such a case, the presentation unit 234 presents the multimodal information IF2 to the user U who is eating or drinking in the closed space 11, thereby inducing a taste change in the user U. According to this aspect, it is possible to induce a taste change in the user U, thereby realizing a novel user experience.

[0054] In the embodiment, the closed space 11 is described as a small room, but it may be the interior of a vehicle such as an automobile. In other words, the information processing system 1 is an in-vehicle system, and the closed space 11 is the interior of the vehicle. This aspect can contribute to providing a vehicle that can realize a novel user experience.

[0055] Biometric information may be acquired by other devices instead of the wearable device 7. For example, the other devices may include at least a smartphone, a body composition monitor, a thermometer, medical equipment such as an MRI, a CT scanner, or an X-ray machine, a thermograph, a blood pressure monitor, a skin age measurement device, a blood glucose measurement device, and an electrocardiograph. The presentation state (intensity, on / off, etc.) of various sensory information included in the multimodal information IF2 may be changed based on the biometric information acquired from the wearable device 7 or the other devices. Alternatively, a configuration may be adopted in which biometric information is not acquired. For example, instead of biometric information from the wearable device 7, the user U's level of fatigue may be determined from facial expression. Specifically, the information processing system 1 includes a camera (not shown), which captures an image of the user U's face. The user U's level of fatigue can be determined by analyzing the facial expression included in the captured image data.

[0056] In one embodiment, a table is used in which the theme information IF1 and the control methods for the components 3 to 6 are associated with each other, but instead of or in addition to the table, a trained model that has undergone deep learning or the like may be employed. For example, when the multimodal information IF2 is made variable over time, the preferences and fatigue recovery tendencies of the user U who has used it in the past may be learned, and the like, so that the device operates in a more desirable manner each time it is used.

[0057] In one embodiment, the multimodal information IF2 is variably presented in accordance with the biometric information acquired via the wearable device 7. However, the multimodal information IF2 may be variably presented in accordance with designated information desired by the user U, together with or instead of the biometric information. More specifically, the designated information may include information regarding the user's state after using the information processing system 1. For example, the designated information may include information regarding a state such as feeling energetic, wanting to calm down, or wanting to relax. The designated information may be recommended by the information processing system 1 or may be selected by the user U. In other words, the acquisition unit 232 acquires the designated information designated by the user U as a second acquisition step. Then, the presentation unit 234 variably presents the multimodal information IF2 in accordance with the designated information as a first presentation step. This aspect allows the user U to be presented with preferred multimodal information IF2, thereby realizing a novel user experience.

[0058] When implementing the information processing system 1 as a business, a program may be implemented, an information processing method may be implemented, an information processing device 2 may be implemented, the information processing device 2 and at least two of the components 3 to 6 may be implemented, or a wearable device 7 may be implemented.

[0059] The information processing device 2 and each of the components 3 to 6 may be remotely connected via a network. In such a case, for example, a user U who wants to use the information processing device for personal use may purchase each of the components 3 to 6 for a small room in his or her home and access the information processing device 2, which is used commercially, to perform the information processing described above (a so-called cloud-based service). When provided as such a service, at least one of the devices included in the information processing system 1 may be installed outside Japan. For example, the information processing device 2 may be installed outside Japan, and the user U may access the information processing device 2 from within Japan using each of the components 3 to 6 in his or her small room. This type of configuration allows for a more interesting experience to be provided to the user through various management configurations.

[0060] In this embodiment, the reception unit 231, acquisition unit 232, setting unit 233, and presentation unit 234 are described as functional units realized by the control unit 23 of the information processing device 2, but at least some of these may be implemented as functional units realized by components other than the information processing device 2. Furthermore, the various information described in the above example may be stored in a distributed manner not only in the storage unit 22 of the information processing device 2 but also in other external devices. In such cases, distributed ledger management based on a blockchain or the like may be performed. Alternatively, the components 3 to 6 may each have a processor and be able to communicate with each other, thereby performing information processing without going through the information processing device 2.

[0061] Furthermore, it may be provided in the following aspects.

[0062] (1) An information processing system comprising at least one processor, the processor configured to execute a program to perform the following steps: in a setting step, setting theme information indicating at least one of a plurality of themes managed in advance; and in a first presentation step, presenting multimodal information corresponding to the theme information to a user located in a closed space, wherein the multimodal information is a combination of information consisting of at least two of visual information, auditory information, tactile information, and olfactory information.

[0063] According to this aspect, a novel user experience with a highly soothing effect can be realized.

[0064] (2) The information processing system described in (1) above further includes at least one image output device that provides the visual information, at least one audio output device that provides the auditory information, at least one tactile presentation device that provides the tactile information, and at least one olfactory presentation device that provides the olfactory information, and the processor is configured to control at least two devices that act on different senses among the image output device, the audio output device, the tactile presentation device, and the olfactory presentation device, thereby presenting the multimodal information.

[0065] According to this aspect, a novel user experience can be realized as a composite device.

[0066] (3) In the information processing system described in (2) above, the audio output device or the tactile presentation device includes at least an ultrasonic generator.

[0067] According to this aspect, a more novel user experience can be realized.

[0068] (4) In the information processing system described in any one of (1) to (3) above, the visual information includes first visual information and second visual information, the first visual information is presented at a position in the closed space where the user looks directly, and the second visual information is presented so as to cover the user's field of view around the first visual information.

[0069] According to this aspect, contrast can be added to visual information, thereby realizing a novel user experience.

[0070] (5) In the information processing system described in any one of (1) to (4) above, the auditory information includes first auditory information and second auditory information, the first auditory information is presented at a position close to the user's ear, and the second auditory information is presented in the closed space so that the user can recognize it as ambient sound.

[0071] According to this aspect, contrast can be added to the auditory information, thereby realizing a novel user experience.

[0072] (6) In the information processing system described in any one of (1) to (5) above, in the first presentation step, the multimodal information is continuously presented, and the presented multimodal information is configured to vary in content or intensity over time.

[0073] According to this aspect, it is possible to present multimodal information that the user will not tire of, and it is therefore possible to realize a novel user experience.

[0074] (7) In the information processing system described in any one of (1) to (6) above, the multimodal information is information configured to give the user the illusion that they are in a location that is consistent with the theme information.

[0075] According to this aspect, it is possible to give the user the illusion of location, thereby realizing a novel user experience.

[0076] (8) In the information processing system described in any one of (1) to (7) above, further, in the first acquisition step, biometric information of the user is acquired, and in the first presentation step, the multimodal information is variably presented according to the biometric information.

[0077] According to this aspect, it is possible to present desirable multimodal information that is tailored to the user's physical condition, etc., and ultimately to realize a novel user experience.

[0078] (9) In the information processing system described in any one of (1) to (8) above, further, in the second acquisition step, specified information specified by the user is acquired, wherein the specified information includes information regarding the state of the information processing system after use as desired by the user, and in the first presentation step, the multimodal information is variably presented according to the specified information.

[0079] According to this aspect, it is possible to present the multimodal information desired and preferred by the user, thereby realizing a novel user experience.

[0080] (10) In the information processing system described in any one of (1) to (9) above, in the second presentation step, support information that allows the user to understand whether or not fatigue has been recovered and to what extent is further presented.

[0081] According to this aspect, the user can grasp his / her own physical condition and the like via the information processing system.

[0082] (11) In the information processing system described in any one of (1) to (10) above, in the first presentation step, the multimodal information is presented to the user who is resting in the closed space, thereby promoting recovery from fatigue in the user.

[0083] According to this embodiment, it is possible to promote recovery from fatigue in the user.

[0084] (12) In the information processing system described in any one of (1) to (11) above, in the first presentation step, the multimodal information is presented to the user who is eating or drinking in the closed space, thereby causing a change in the user's sense of taste.

[0085] According to this embodiment, it is possible to induce a change in the user's sense of taste, thereby realizing a novel user experience.

[0086] (13) In the information processing system described in any one of (1) to (12) above, the information processing system is an in-vehicle system, and the closed space is the interior of the vehicle.

[0087] This aspect can contribute to providing a car that can provide a new user experience.

[0088] (14) An information processing method, comprising steps executed by the information processing system according to any one of (1) to (13) above.

[0089] According to this aspect, a novel user experience with a highly soothing effect can be realized.

[0090] (15) A program that causes a computer to execute each step of the information processing system described in any one of (1) to (13) above.

[0091] According to this embodiment, it is possible to realize a novel user experience that has a highly soothing effect. Of course, this is not a limitation.

[0092] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the accompanying claims.

[0093] The following describes the effectiveness verification of the information processing method described in the embodiment using an example. Figure 7 shows the experimental protocol of the example used for the effectiveness verification. Figures 8 to 10 show the experimental results of the example.

[0094] In this experiment, we quantified the extent to which the stimulation of this device could relieve fatigue caused by work loads comparable to office work. The experimental protocol for this purpose is shown in Figure 7. This experimental protocol consisted of three blocks: an evaluation task to measure information processing ability, a control task to accumulate mental fatigue, and a rest period. First, information processing ability was measured in the initial state at the start of the experiment using the evaluation task. Next, after accumulating mental fatigue using the control task, an evaluation task was used to confirm whether information processing ability had declined. After a break, the evaluation task was repeated, and the degree of recovery was confirmed by the change in score compared to before the break. Heart rate variability was also recorded during the experiment to assess the degree of recovery using physiological indicators.

[0095] Three types of assessment tasks were used: two-back calculation (medium difficulty), two-back calculation (high difficulty), and a cognitive ability test (CogEvo). The cognitive ability test consists of five tasks (e.g., visual search) and is capable of measuring four types of cognitive ability: spatial recognition, attention, memory, and planning. However, the five CogEvo tasks were low in difficulty, and there was a possibility that a decline in performance due to mental fatigue or an improvement in performance due to rest would not be observed. Therefore, two-back calculation tasks of medium and high difficulty were used as more difficult tasks. The stimulation conditions during rest periods are shown in Table 1.

[0096] Three samples were prepared: "full," "no vision," and "control condition." The full condition contained four composite stimuli for the senses of sight, smell, and touch, evoking a forest. The no vision condition contained three composite stimuli, excluding the visual stimuli from the full condition. The control condition contained only visual stimuli, presenting only an image of an office computer. The experimental subjects were 47 people (25 men, 22 women, age 30.3±4.4 years), and each subject underwent all three conditions in random order.

[0097] Figure 8A shows a moderately difficult two-back task (N ａ The graph shows the progress of scores by stimulus during the break before and after the break. In the full and no-visual conditions with compound stimuli, there was a statistically significant improvement in the score rate before and after the break, whereas in the control condition without compound stimuli, there was no improvement in the score due to the break. In other tasks, no statistically significant differences were observed between conditions.

[0098] Furthermore, Figure 8B shows a comparison of the workload of everyday work (office work) and the tasks in this experiment using NASA-TLX. Since both showed roughly similar trends, it can be said that the workload in this experiment was equivalent to that of office work. Furthermore, Figure 9A shows the average heart rate trends for each subject during the experiment. While the trends were generally similar across all conditions until the break, differences in heart rate during the break were observed depending on the stimulation conditions. In particular, when comparing the difference in each subject's heart rate immediately before the break and two minutes after the start of the break between conditions, a statistically significant difference was observed in the decrease in heart rate compared to the control experiment in both full and no-visual stimulation (see Figure 9B). In other words, the decrease in heart rate was greater under complex stimulation than under the control condition, suggesting a greater relaxation effect. These results demonstrate that, in terms of both information processing ability and physiological indicators, complex stimulation provides a greater recovery effect from fatigue caused by workloads equivalent to office work. Furthermore, the effects of stimulation can be quantified based on increases or decreases in parasympathetic nervous activity indicators, including heart rate.

[0099] Based on the above results, the Japanese version of the Perceived Restorativeness Scale (PRS) was administered under each condition to identify the factors that contributed to the high restorative effects of the stimulus. The PRS is a scale that evaluates the restorative properties of an environment as perceived by humans, and has a track record of being used indoors. Table 2 shows a list of questions categorized by sub-item of the PRS.

[0100] Furthermore, Figure 10A shows the PRS results for each stimulus in this experiment (47 subjects). For the three PRS sub-items of "escape," "fascination," and "preference," scores were higher in the full and no-visual conditions than in the control condition. However, familiarity showed the same trend, but was excluded because previous research has shown that it has not contributed to the identification of environments with high recovery characteristics. In other words, recovery may be promoted in environments where scores for the PRS sub-items of "escape," "fascination," and "preference" are higher than in the control condition. Figure 10B shows the PRS results when only one stimulus is removed from the "full" condition (the legend indicates the removed stimulus). The "main video" condition in Figure 10B is closest to the "no-visual" condition in Figure 9B, and the average scores for escape, fascination, and preference were all the smallest in Figure 10B. Despite this, recovery was observed in this experiment, suggesting that when three stimuli, excluding one from the full stimulus shown in Figure 10B, are presented, a higher recovery effect can be expected compared to when no compound stimuli are presented.

[0101] 1: Information processing system, 11: Closed space, 2: Information processing device, 20: Communication bus, 21: Communication unit, 22: Memory unit, 23: Control unit, 231: Reception unit, 232: Acquisition unit, 233: Setting unit, 234: Presentation unit, 24: Input unit, 3: Image output device, 31: Display, 32: Projector, 4: Audio output device, 41: Headphones, 42: Speaker, 5: Tactile presentation device, 51: Electric fan, 52: Air conditioner, 6: Olfactory presentation device, 61: Aroma diffuser, 7: Wearable device, 8a: Screen, 8b: Screen, 81: Area, 82: Area, 83: Area, 84: Button, AI: Auditory information, Ab: Avatar, IF1: Theme information, IF2: Multimodal information, IF3: Support information, OI: Olfactory information, TI: Tactile information, U: User, VI: Visual information

Claims

1. An information processing system comprising at least one processor, the processor being configured to execute a program to perform the following steps: in a setting step, setting theme information indicating at least one of a plurality of themes managed in advance; and in a first presentation step, presenting multimodal information corresponding to the theme information to a user located in a closed space, the multimodal information being a combination of at least two of visual information, auditory information, tactile information, and olfactory information.

2. An information processing system as described in claim 1, further comprising at least one image output device for providing the visual information, at least one audio output device for providing the auditory information, at least one tactile presentation device for providing the tactile information, and at least one olfactory presentation device for providing the olfactory information, wherein the processor is configured to control at least two of the image output device, the audio output device, the tactile presentation device, and the olfactory presentation device that act on different senses, thereby presenting the multimodal information.

3. An information processing system according to claim 2, wherein the audio output device or the tactile presentation device includes at least an ultrasonic generator.

4. An information processing system according to any one of claims 1 to 3, wherein the visual information includes first visual information and second visual information, the first visual information is presented at a position in the closed space where the user looks directly at it, and the second visual information is presented so as to cover the user's field of vision around the first visual information.

5. An information processing system according to any one of claims 1 to 4, wherein the auditory information includes first auditory information and second auditory information, the first auditory information is presented at a position close to the user's ear, and the second auditory information is presented in the closed space so as to be recognizable to the user as an ambient sound.

6. An information processing system according to any one of claims 1 to 5, wherein in the first presentation step, the multimodal information is presented continuously, and the presented multimodal information is configured so that the content or intensity of the information varies over time.

7. An information processing system according to any one of claims 1 to 6, wherein the multimodal information is information configured to give a user the illusion of being in a location consistent with the theme information.

8. An information processing system according to any one of claims 1 to 7, further comprising: a first acquisition step for acquiring biometric information of the user; and a first presentation step for variably presenting the multimodal information in accordance with the biometric information.

9. An information processing system according to any one of claims 1 to 8, further comprising: a second acquisition step of acquiring specified information specified by the user, the specified information including information regarding the state of the information processing system after use as desired by the user; and a first presentation step of variably presenting the multimodal information in accordance with the specified information.

10. An information processing system according to any one of claims 1 to 9, further comprising, in a second presentation step, presenting support information that enables the user to grasp the presence or absence, or the degree of, recovery from fatigue.

11. An information processing system according to any one of claims 1 to 10, wherein in the first presentation step, the multimodal information is presented to the user who is resting in the closed space, thereby promoting recovery from fatigue in the user.

12. An information processing system according to any one of claims 1 to 11, wherein in the first presentation step, the multimodal information is presented to the user who is eating or drinking in the closed space, thereby causing a change in the user's sense of taste.

13. An information processing system according to any one of claims 1 to 12, wherein the information processing system is an in-vehicle system, and the closed space is the inside of a vehicle.

14. An information processing method, comprising the steps executed by an information processing system according to any one of claims 1 to 13.

15. A program causing a computer to execute each step of the information processing system according to any one of claims 1 to 13.

Images