Information processing system, information processing method and recording medium
By using sensor data to estimate the user status and control information presentation when the user is in a state where information can be easily accepted, the problem of information being ignored in the prior art is solved, and a more effective information presentation effect is achieved.
Patent Information
- Application Number
- CN201980029807.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-09
- Filing Date
- 2019-02-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-02-18
AI Technical Summary
The prior art fails to effectively consider the status of the user receiving information, resulting in the presentation of information when the user is unwell and cannot be fully identified.
When the user is in a situation where information can be easily accepted, the user's status is used to infer the user's status and control the information presentation, including biosensors, motion sensors and environmental sensors, determine the user's mental state, environmental noise, etc., and present the information in a timely manner.
It realizes more effective information presentation when users accept information in a timely manner, reduces the possibility of information being ignored, and improves the recognizability and acceptance of information.
Smart Images

Figure CN112074804B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing system, an information processing method, and a recording medium. Background Art
[0002] In recent years, mobile device technology and communication technology represented by smartphones and wearable devices such as smart watches have developed, and it is now possible to obtain necessary information at any time. In addition, technology related to display devices such as large-screen liquid crystal displays and single-focus projectors has also been greatly developed, so now there are few restrictions on the location where information is displayed, and it has become possible to display information anywhere.
[0003] In such a technical background, technologies for displaying information in various scenes have been proposed. For example, in Patent Document 1 described below, it is disclosed that an icon is displayed immediately before information such as an e-mail is read out by voice in a vehicle, and the icon reflects the content of the information.
[0004] Furthermore, Patent Document 2 listed below discloses a technology in which a projector is installed at an entrance or the like and a family member who has returned home turns on a light switch at the entrance to start the projector, thereby causing the projector to perform a lighting function and project a message.
[0005] Furthermore, Patent Document 3 listed below discloses a technology in which, when a passenger gets on or off a vehicle, information useful to the passenger or image information required by the passenger is projected onto a road surface by means of a projector attached to a side mirror of the vehicle.
[0006] Furthermore, Patent Document 4 listed below discloses projection navigation that predicts user behavior, generates a guide image, and displays the guide image on a wall or a corridor or the like using a projector.
[0007] Furthermore, in Patent Document 5 listed below, a technology is disclosed for displaying a user interface (UI) image such as a mouse pad area to a user wearing a transparent head mounted display (HMD) that enables any real object to be used as a UI.
[0008] Reference List
[0009] Patent Literature
[0010] Patent Document 1: Japanese Patent Publication No. 2005-352645
[0011] Patent Document 2: Japanese Patent Publication No. 2014-021428
[0012] Patent Document 3: Japanese Patent Publication No. 2017-087876
[0013] Patent Document 4: Japanese Patent Publication No. 2014-123277
[0014] Patent Document 5: International Publication No. WO 2014 / 045683 Summary of the invention
[0015] Technical issues
[0016] However, according to the above conventional technology, the situation in which the user receives the presented information is not considered. Therefore, if the user is not in a state in which the user can easily accept the information, such as when the user is in a panic, presenting information such as messages at various locations may not allow sufficient recognition of the presented information and the presented information may be ignored.
[0017] Therefore, according to the present disclosure, an information processing system, an information processing method, and a recording medium are proposed, which realize effective information presentation by presenting information when the user is in a situation where he can easily accept the information.
[0018] Solution to the problem
[0019] According to the present disclosure, there is provided an information processing system including a control unit that controls a presentation unit to present a message to a user when it is presumed based on sensing data related to the user that the user is in a condition that can easily accept the message.
[0020] According to the present disclosure, there is provided an information processing method in which a processor performs processing including controlling a presentation unit to present a message to a user when it is presumed based on sensing data related to the user that the user is in a condition that can easily accept the message.
[0021] According to the present disclosure, there is provided a recording medium including a program for causing a computer to function as a control unit that controls a presentation unit to present a message to a user when it is presumed based on sensing data related to the user that the user is in a condition that can easily accept the message.
[0022] Beneficial effects of the present invention
[0023] As described above, according to the present disclosure, more effective information presentation is achieved by presenting information when the user is in a situation where the user can easily accept the information.
[0024] The above-described effect is not necessarily limitative, and together with or instead of this effect, any effect disclosed in the present specification or any other effect that can be grasped from the present specification can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] [Figure 1 ] Figure 1 is a diagram for explaining an overview of an information processing system according to one embodiment of the present disclosure.
[0026] [ Figure 2 ] Figure 2 is a diagram showing an example of the overall configuration of an information processing system according to an embodiment.
[0027] [ Figure 3 ] Figure 3 is a block diagram showing an example of a configuration of an output device according to an embodiment.
[0028] [ Figure 4 ] Figure 4 is a block diagram showing an example of a configuration of a management server according to an embodiment.
[0029] [ Figure 5 ] Figure 5 is a sequence diagram showing an example of the flow of the information presentation process according to the first example.
[0030] [ Figure 6 ] Figure 6 is a flowchart showing an example of the flow of an output control process for a message according to the first example.
[0031] [ Figure 7 ] Figure 7 is a sequence diagram showing an example of the flow of recognition processing of a user's situation based on sensing data according to the first example.
[0032] [ Figure 8 ] Figure 8 is a sequence diagram showing an example of a flow of accumulation processing for messages according to the second example.
[0033] [ Fig. 9 ] Fig. 9 is a sequence diagram showing an example of the flow of the message presentation process according to the second example.
[0034] [ Fig.10 ] Fig.10 is a diagram showing an example of guidance content that changes a user's situation according to a third example.
[0035] [ Fig.11 ] Fig.11 is a sequence diagram showing an example of the flow of the message presentation process according to the third example.
[0036] [ Fig.12 ] Fig.12 is a sequence diagram showing an example of the flow of the message presentation process according to the third example.
[0037] [ Fig.13] Fig.13 is a sequence diagram showing an example of the flow of the message presentation process according to the third example.
[0038] [ Fig.14 ] Fig.14 1 is a diagram for explaining projection control on a location that can be easily recognized visually by a user according to a fourth example.
[0039] [ Fig.15 ] Fig.15 is a flowchart showing an example of the flow of the projection control process according to the fourth example.
[0040] [ Fig.16 ] Fig.16 : is a sequence diagram showing an example of the flow of a process for learning determination regarding any situation that can easily accept a message according to the fifth example.
[0041] [ Fig.17 ] Fig.17 : is a sequence diagram showing an example of the flow of a process for learning determination regarding any situation that can easily accept a message according to the fifth example.
[0042] [ Fig.18 ] Fig.18 is a flowchart illustrating an example of the flow of a process for determining whether a user accepts a message according to a fifth example. DETAILED DESCRIPTION
[0043] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the accompanying drawings, redundant descriptions will be omitted by assigning the same reference numerals to any components having substantially the same functional configuration.
[0044] The description will be made in the following order.
[0045] 1. Overview of an information processing system according to an embodiment of the present disclosure
[0046] 2. Configuration Example
[0047] 2-1. System configuration example
[0048] 2-2. Example of Configuration of Output Device 1
[0049] 2-3. Example of configuration of management server 2
[0050] 3. Example
[0051] 3-1. First Example (Presentation of Message)
[0052] 3-2. Second example (presentation based on message content and user status)
[0053] 3-3. Third Example (Change in User Status)
[0054] 3-4. Fourth Example (Presentation at a Location Visually Recognizable to the User)
[0055] 3-5. Fifth example (Learning "a situation where a message can be easily received")
[0056] 4. Conclusion
[0057] 1. Overview of an information processing system according to an embodiment of the present disclosure
[0058] Figure 1 1 is a diagram for explaining an overview of an information processing system according to an embodiment of the present disclosure. Figure 1 As shown, in the information processing system according to the present embodiment, information presentation is performed in various scenes of daily life using the output device 1. Assumed examples of the output device 1 include a projector that projects images onto various locations such as a wall and a floor. The projector may be a portable wearable projector that is held or worn by a user, or a fixed projector that is fixed at any of various locations such as a room, an entrance, a kitchen, and a bathroom. The wearable projector may be, for example, Figure 1 The shoulder-mounted output device 1A shown. (For example, the wearable projector can be formed by a tight and compact shape of a character. In addition, there is no particular limitation on the method of wearing the shoulder-mounted projector, but, for example, the shoulder-mounted projector can be fixed to its body with a clip or a strap.) In addition, the wearable projector can be attached to (integrated with) a hat, a helmet, glasses, etc., can be worn by hanging around the neck, or can be attached to a strap wrapped around the chest or waist. In addition, the projector can be a driven projector that can be driven in horizontal and tilted directions. The output device 1 can control the angle of the projector, etc. to project the image to a predetermined projection location. In addition, the projector can be a mobile projector that can move forward, backward, left, right, etc. in addition to being able to be driven in horizontal and tilted directions.
[0059] By using any of such various projectors, the information processing system according to the embodiment allows information to be presented to the user in various scenes in daily life.
[0060] As described above, if the user is not in a state where the user can accept information according to the situation, for example, if the user is chatting with friends, if the user is in a noisy environment, if the user is in a hurry, if the user is in a panic, if the user is excited, or if the user is exercising vigorously; presenting information such as messages at various locations may not be able to fully recognize the presented information, and the presented information may be ignored. In addition, even if the user can recognize the message in such a situation, the user may be in a state where the user cannot mentally accept the message (for example, the user may ignore the message or may feel unhappy).
[0061] Therefore, according to an embodiment, by performing information presentation when the user is in a condition where he or she can easily accept the information, more effective information presentation is achieved in view of the condition in which the user accepts the information presentation. The output device 1 according to an embodiment is provided with various sensors for detecting the condition of the user or the surrounding environment (for example, a microphone for detecting voice or ambient sound; a position measurement unit for measuring indoor / outdoor position; a biosensor for detecting body temperature, heart rate, etc.; a motion sensor for detecting movement; and a distance measurement sensor for detecting the distance to an object). The output device 1 determines whether the user is in a condition where he or she can easily accept the information based on various sensing data, and performs information presentation if the user is in a condition where he or she can easily accept the information. For example, Figure 1 The output device 1A shown has a biosensor 122a, a cord having the biosensor 122a at the end of the cord is drawn out from the output device 1A and attached to the user's body, and continuously detects the user's heart rate, body temperature, and the like.
[0062] In this specification, "a condition in which information can be easily accepted" may be, for example, a condition in which the user's mental state is calm, a condition in which the user's emotions are calm, a condition in which the user is relaxed, or a condition in which the user is unaware. Such a mental state of the user can be identified by, for example: biosensor data including heart rate, body temperature, and breathing; and / or facial expressions, mumbling, and movement. In addition to such a mental state, a condition in which the user is not talking to other people, a condition in which the user is not moving around in a hurry, a condition in which the user is stationary, a condition in which the user is in a quiet environment, and a condition in which the user is alone can be inferred as a "condition in which information can be easily accepted". These conditions can be identified by, for example, analyzing voice data collected by a microphone or analyzing acceleration information detected from motion sensor data.
[0063] Furthermore, “a situation in which information can be easily accepted” can be determined based on, for example, the following basic behavior of the user.
[0064] When there is no keyboard input for a given period of time while the user is using the PC (the user is unlikely to be in a working state, and therefore the user can accept information presentation without disturbing the user).
[0065] A case where the user speaks the first sentence to the agent terminal (eg, AI speaker) when the user wakes up in the morning (the user is in a state of receiving information).
[0066] The information presented to the user by the output device 1 may be a response from an agent system that recommends content or behavior based on context, or may be information presented by an information presentation system that presents various types of information, such as emails, messages, notification information, recommendation information, advertising information, and health information. The information processing system according to the embodiment may be used as a common basis (platform) for controlling the timing of presenting various types of information output from such various systems to the user.
[0067] In addition, the information presented to the user from the output device 1 may be, for example, any one of text, voice data, or an image (moving image or still image), or a combination including at least any one of text, voice data, or an image (moving image or still image). If the information is a response from an agent system, an image (character image) of the agent may be displayed, and the message may be read out in the voice of the agent, thereby making it easier for the user to accept the information.
[0068] Furthermore, the output device 1 may present information by projecting the information onto a wall, corridor, ceiling, or piece of furniture (e.g., a table), or as Figure 1 As shown in FIG, information is presented by projecting it onto a mirror, steam (white smoke-like substance), water surface (hot water or oil in a pot), etc. For example, in Figure 1In the example shown, since the user is in a calm state when looking at the mirror and brushing his teeth, the output device 1A projects the agent onto the surface of the mirror and outputs the message from the agent system in voice. In addition, when the user soaks himself in the bathtub and relaxes, the output device 1A projects the agent onto the mirror in the steam or bathroom, and outputs the message from the agent system in voice. In addition, when the user is in an idle moment while waiting for the water to boil during cooking, the fixed output device 1B set in the kitchen projects the agent onto the water surface in the pot, and outputs the information from the agent system in voice. In each of these examples, the agent is projected in the user's field of vision to attract the user's attention, and information is presented through voice output, text display, etc., and the user is able to accept the information without pressure. The voice of the agent is output, for example, from a speaker provided in the output device 1. The voice of the agent can be output in voice using a directional speaker or mosquito sound so that only the user can hear the voice. In addition, the voice of the agent can be sound-localized (such as audio augmented reality (AR)) so that the voice can be heard from the direction of the projected agent.
[0069] In addition, the information processing system according to the embodiment can output information differently by comparing the content of the presented information with the user's situation. Therefore, it is possible to present an easily acceptable message that matches the user's situation at an easily acceptable time.
[0070] Furthermore, when the user is not in a condition to easily accept information, the information processing system according to the embodiment may present information prompting a change in the user's condition, put the user in a condition to easily accept information, and thereafter perform information presentation.
[0071] An overview of the information processing system according to the embodiment has been described above. The output device 1 is not necessarily a projector that displays an image by projecting the image onto a real object, but may also be: a display device with a display screen, such as a smart phone, a smart bracelet, a smart watch, a tablet terminal, a PC, a TV device, or a gaming device; or a transparent or non-transparent head-mounted display (HMD). When a transparent HMD is used, the proxy image can be displayed in the user's field of view through an AR display in which the image is superimposed on the real space. In addition, when the user is watching a virtual space video using a non-transparent HMD, the proxy image can be displayed on the virtual space video. In addition, the output device 1 may be a voice output device that does not have a display device, such as a smart headset or a smart necklace.
[0072] Next, the configuration of the information processing system according to the embodiment will be described in detail with reference to the drawings.
[0073] 2. Configuration Example
[0074] 2-1. System configuration example
[0075] Figure 2 1 is a diagram showing an example of the overall configuration of an information processing system according to an embodiment. Figure 2 As shown, the information processing system according to the embodiment includes an output device 1 and a management server 2. The output device 1 and the management server 2 may be connected to each other via a network 3. The output device 1 may be a wearable projector carried by a user, or a fixed projector.
[0076] For example, when the management server 2 receives a message from the agent system server 4, if the user is in a condition that can easily accept the message, the management server 2 controls the output device 1 to output the message according to the user's condition based on the user's sensed data, which is obtained from the output device 1. The case of receiving a message from the agent system is described herein as an example, but the embodiment is not limited to this example. For example, the management server 2 can receive information from a general information presentation system, such as a content recommendation system (recommending various types of content information such as music, movies, events, and games) or a health care system (presenting health management information), or can receive a message from a mail server or a social network server.
[0077] In addition, the management server 2 does not necessarily receive messages from a general agent system that responds to a direct and short-term conversation with the user, and can receive messages from, for example, a new system called a host system, which prompts the user to change behavior so that the user gradually solves the problem from a long-term perspective. The host is an example of a virtual agent that prompts the user to change behavior spontaneously, and the host system can, for example, automatically generate a predetermined code of conduct for a specific community or for each user, indirectly prompt a change in behavior based on the code of conduct, and thereby solve problems in the community. When the user acts according to the words of the host (like a master) without knowing the code of conduct, the problem in the community can be solved or the user can act according to the values in the community, and thus the situation of the community can be improved without the user knowing such improvement.
[0078] 2-2. Example of Configuration of Output Device 1
[0079] Next, we will refer to Figure 3 The configuration of the output device 1 is described. Figure 3 1 is a block diagram showing an example of the configuration of the output device 1 according to the embodiment. Figure 3 As shown, the control device 1 has a control unit 100 , a communication unit 110 , an input unit 120 , an output unit 130 , and a storage unit 140 .
[0080] The control unit 100 functions as an arithmetic processing device and a control device, and controls the overall operation in the output device 1 according to various programs. The control unit 100 is implemented by, for example, an electronic circuit such as a central processing unit (CPU) or a microprocessor. In addition, the control unit 100 may include: a read-only memory (ROM) in which programs to be used, arithmetic parameters, etc. are stored; and a random access memory (RAM) in which parameters that are appropriately changed, etc. are temporarily stored.
[0081] In addition, the control unit 100 according to the embodiment may perform: control to transmit the sensing data acquired from the sensor 122 to the management server 2 from the communication unit 110; and control to output the information received from the management server 2 via the communication unit 110 from the output unit 130. In addition, the control unit 100 may perform various analyses of the sensing data (e.g., voice recognition, analysis of biological data, object recognition based on camera images and / or distance measurement data, recognition of a motion state, expression analysis, motion analysis, gesture recognition, face recognition, or line of sight recognition).
[0082] Communication unit 110
[0083] The communication unit 110 is connected to the network 3 by wire or wirelessly, and transmits and receives data to and from the management server 2 on the network. The communication unit 110 is communicably connected to the network 3 via, for example, a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a mobile communication network (Long Term Evolution (LTE) or third generation (3G) mobile communication system).
[0084] Input unit 120
[0085] The input unit 120 has an operation input unit 121 and a sensor 122, and inputs information acquired from the operation input unit 121 or the sensor 122 into the control unit 100. The operation input unit 121 detects input of a user's operation on the output device 1. The operation input unit 121 may be, for example, a touch sensor, a pressure sensor, or a proximity sensor; or may be physically configured as any one of a button, a switch, and a lever.
[0086] The sensor 122 includes various sensors that sense the user's condition or the surrounding environment and input the sensed data into the control unit 100. The sensor 122 may be, for example: a position measurement unit (using a global positioning system (GPS) or the like to measure an outdoor position, or using Bluetooth, Wi-Fi, or the like to measure an indoor position); a camera; a microphone (hereinafter, referred to as a microphone); a distance measurement sensor; a thermal sensor (for example, provided in a fixed projector and measuring the user's body temperature); an ultrasonic sensor; a motion sensor (for example, an acceleration sensor, a gyro sensor, or a magnetic field sensor); a biosensor (detecting, for example, body temperature, veins, pulse, heart rate, respiration, perspiration, blood pressure, brain waves, eye potential, myoelectric value, eye movement, sight, etc.); or an environmental sensor (for example, for illumination, air temperature, and / or humidity).
[0087] Output unit 130
[0088] The output unit 130 is a presentation unit that presents information to the user according to the control of the control unit 100. The output unit 130 has a projector 131 and a speaker 132. The projector 131 is an example of an information display device that can project an image onto various locations such as a wall, a floor, a ceiling, a mirror, and steam. The projector 131 is used as an information display device in this article, but the information display device is not limited to the projector 131, and the information display device according to the embodiment may be a display unit implemented by a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, etc.
[0089] The speaker 132 reproduces the voice signal according to the control of the control unit 100. The speaker 132 may be, for example, a directional speaker. Using a directional speaker allows the user to hear only the voice of the agent. In addition, by using mosquito sound in consideration of the age of the user, etc., the speaker 132 can enable the user to hear only the voice of the agent. In addition, the speaker 132 can realize voice AR by reproducing the voice signal of sound localization by the control unit 100, so that the voice of the agent is heard from the direction of the image of the projected agent.
[0090] Storage unit 140
[0091] The storage unit 140 is realized by a read-only memory (ROM) which stores therein programs and arithmetic parameters and the like used in processing performed by the control unit 100 , and a random access memory (RAM) which temporarily stores therein parameters and the like which are changed appropriately.
[0092] An example of the configuration of the output device 1 according to the embodiment is described above in detail. The configuration of the output device 1 is not limited to Figure 3 The example shown, and for example, the output device 1 may be formed of a plurality of devices, or the output unit 130 may also have a display device.
[0093] 2-3. Example of configuration of management server 2
[0094] Next, we will refer to Figure 4 The configuration of the management server 2 is described. Figure 4 2 is a block diagram showing an example of the configuration of the management server 2 according to the embodiment. Figure 4 As shown, the management server 2 (information processing device) has a control unit 200 , a communication unit 210 , and a storage unit 220 .
[0095] Control unit 200
[0096] The control unit 200 functions as an arithmetic processing device and a control device, and controls the overall operation in the management server 2 according to various programs. The control unit 200 is implemented by, for example, an electronic circuit such as a central processing unit (CPU) or a microprocessor. In addition, the control unit 200 may include: a read-only memory (ROM) in which programs to be used, arithmetic parameters, etc. are stored; and a random access memory (RAM) in which parameters that are appropriately changed, etc. are temporarily stored.
[0097] Furthermore, the control unit 200 according to the embodiment also functions as a user management unit 201 , a situation recognition unit 202 , a message management unit 203 , and a message presentation control unit 204 .
[0098] The user management unit 201 manages information related to each user operating the output device 1 (eg, identification information; attribute information including age, gender, etc.; preference information and / or terminal information of the output device 1 ). Information about each user may be stored in the storage unit 220 .
[0099] The situation recognition unit 202 performs recognition of the situation of the user himself or the environment around the user based on the sensing data or the analysis result of the sensing data that has been received from the output device 1. For example, the situation recognition unit 202 performs recognition of whether the user is talking to someone, whether the user is in a panic or a hurry, whether the user is in a noisy environment, whether the user is alone, whether the user is still and calm, whether the user is excited, etc. In addition, the situation recognition unit 202 can infer whether the user is in a situation that can easily accept a message.
[0100] The message management unit 203 manages information that has been received from the agent system server 4 and is to be transmitted to the user in association with the user's identification information, etc. As described above, the information to be transmitted to the user may be: a response message from the agent to the user; an email addressed to the user; a message addressed to the user via a social network; notification information, recommendation information, advertising information, or health information to the user, etc. In addition, the information to be transmitted to the user may be a message received from a host system (a host indicating a user's behavioral guidelines) and prompting the user to spontaneously change the user's behavior. The message management unit 203 accumulates these messages associated with the user into the storage unit 220.
[0101] In the case where the situation recognition unit 202 presumes that the user is in a situation where the message can be easily accepted, the message presentation control unit 204 extracts the message addressed to the user from the storage unit 220 and performs control so that the message is presented to the user from the output device 1. Upon receiving the message from the agent system server 4, the message presentation control unit 204 may determine whether the message can be presented (in real time) based on the recognition result of the current situation of the user by the situation recognition unit 202. In addition, if the message cannot be presented, the message presentation control unit 204 may perform control so that the message is presented when the situation recognition unit 202 presumes that the user has entered a situation where the message can be easily accepted thereafter.
[0102] In addition, the message presentation control unit 204 can compare the user status with the content of the message, and if the content of the message matches the user status, the message presentation control unit 204 can perform control so that the message is presented when the user is in a condition that can easily accept the message.
[0103] Furthermore, if the situation identification unit 202 has inferred that the user is not in a situation where he can easily accept a message, the message presentation control unit 204 may send a message to change the user's situation (eg, the user's own situation such as his mood, or the environment around the user).
[0104] Communication unit 210
[0105] The communication unit 210 is connected to the network 3 by wire or wirelessly, and transmits and receives data to and from each output device 1 via the network 3. The communication unit 210 is communicably connected to the network 3 through, for example, a wired / wireless local area network (LAN), Bluetooth, or Wireless Fidelity (Wi-Fi) (registered trademark).
[0106] Storage unit 220
[0107] The storage unit 220 is implemented by a ROM in which programs, arithmetic parameters, etc. to be used in the processing performed by the control unit 200 are stored, and a RAM in which parameters that are changed appropriately are temporarily stored, etc. For example, the storage unit 220 according to the embodiment can store therein a message received from the agent system server 4.
[0108] The configuration of the management server 2 according to the embodiment has been specifically described above. Figure 4 The configuration of the management server 2 shown is only an example, and the embodiment is not limited to this example. For example, at least a part of the configuration of the management server 2 may be in an external device, or at least a part of the function of the control unit 200 may be implemented by the output device 1, an edge server, etc. In addition, Figure 4 All the components of the storage unit 140 and the control unit 200 shown may be provided in the output device 1 , and control of presentation of a message received from the agent system server 4 according to a user condition may be performed by means of an application in the output device 1 .
[0109] 3. Example
[0110] Next, an example of the information processing system according to the embodiment will be described in detail using the drawings.
[0111] 3-1. First Example (Presentation of Message)
[0112] First, refer to Figures 5 to 7 A first example of an embodiment is described. In the first example, presenting a message to a user when the user is in a situation where the user can easily accept the message enables more efficient presentation of information.
[0113] Figure 5 1 is a sequence diagram showing an example of the flow of the information presentation process according to the first example. Figure 5 As shown, first, the output device 1 checks the management server 2 whether there is a message (step S103).
[0114] Subsequently, the user management unit 201 in the management server 2 performs a search for messages (step S106). The user management unit 201 can search for messages to be transmitted to the user by querying the agent system server 4, or search for messages addressed to the user from messages received in advance from the agent system server 4 and accumulated in the storage unit 220.
[0115] Then, the management server 2 transmits the retrieved message to the output device 1 (step S109).
[0116] Subsequently, the output device 1 acquires the sensing data from the sensor 122 (step S112 ), and transmits the acquired sensing data to the management server 2 (step S115 ). The output device 1 may transmit the analysis result of the sensing data to the management server 2 in real time.
[0117] Subsequently, the situation recognition unit 202 in the management server 2 performs recognition of the user's situation based on the information (eg, sensing data or analysis results of sensing data) transmitted from the output device 1 (step S118). Figure 7 The details of the process of identifying the user status are described.
[0118] Subsequently, if it has been presumed that the user is in a situation where he can easily accept the message (step S121 / Yes), the management server 2 notifies the output device 1 of the fact that the user is in a situation where he can easily accept the message (step S124).
[0119] The output device 1 performs control of outputting a message to the user based on the notification from the management server 2 (step S127). Figure 6 Describes the details of the control that outputs messages to the user.
[0120] Thus, the first example enables output of a message to the user when the user is in a situation where the user can easily accept the message.
[0121] Output control processing for messages
[0122] Will refer to Figure 6 An example of the output control process for the message represented by step S127 described above is described. Figure 6 1 is a flowchart showing an example of the flow of the output control process for a message according to the first example. Here, a case where a message is output using the projector 131 will be described.
[0123] like Figure 6 As shown, if the user is in a condition where he can easily accept the message (step S130 / Yes), the control unit 100 in the output device 1 determines whether there is a monochrome plane at a distance where the projector 131 can project based on the sensing data of the sensor 122 (step S133). Specifically, the control unit 100 analyzes the camera image and the distance measurement data acquired by the sensor 122, and determines whether there is an object having a shape that can be used for projection (herein, a monochrome plane) at a distance where projection can be performed from the projector 131.
[0124] Alternatively, the control unit 100 similarly analyzes the camera image and the distance measurement data, and determines whether a mirror surface, a still water surface, or white smoke-like matter exists at a distance at which projection can be performed from the projector 131 (steps S136 and S139).
[0125] Subsequently, if there is a monochrome plane, a mirror, a still water surface, or a white smoke-like substance at a distance where the projector 131 can project (step S133 / Yes, step S136 / Yes, or step S139 / Yes), the control unit 100 displays (projects) the message using characters (agent image) or multimedia content (message text, moving image, still image, etc.) with the aid of the projector 131 (step S142). In addition, the control unit 100 can output the voice data (agent voice data) of the message from the speaker 132 as multimedia content together with the display of the agent image.
[0126] As described above, when presenting a message using the projector 131, the output device 1 appropriately determines an appropriate projection location (a projection location that satisfies a given condition) from the user's surroundings, and performs control so that the message is projected to the projection location (by controlling the drive of the projector 131). Therefore, in various scenes of daily life, when the user is in a situation where the message can be easily accepted, the message can be presented anywhere.
[0127] User status identification and processing
[0128] Next, we will refer to Figure 7 Description by Figure 5 The above-mentioned steps S112 to S121 show details of the process of identifying the user status.
[0129] Figure 7 1 is a sequence diagram showing an example of the flow of a recognition process of a user's condition based on sensing data according to the first example. Figure 7 As shown, first, the output device 1 acquires voice information through the microphone, analyzes the voice data (step S153), and sends the analysis result (feature value data) to the management server 2 (step S156).
[0130] Subsequently, based on the analysis result, the situation recognition unit 202 of the management server 2 determines whether there is a speech voice of the user (step S159 ) and whether the environment around the user is noisy (step S162 ).
[0131] Furthermore, the output device 1 measures the user's physical condition using a biosensor (eg, a heart rate sensor and a body temperature sensor) (step S165 ), and transmits the measurement result (physical condition data) to the management server 2 (step S168 ).
[0132] Subsequently, the situation recognition unit 202 of the management server 2 determines whether to maintain a calm state in which the user's heart rate is not too high and the user's body temperature is normal based on the measurement result (step S171). Specifically, the situation recognition unit 202 determines whether the user's heart rate does not exceed a predetermined value and whether the user's body temperature is normal.
[0133] Furthermore, the output device 1 detects the motion of the user using a motion sensor (acceleration sensor, gyro sensor, etc.) (step S174 ), and transmits the detection result (motion information) to the management server 2 (step S177 ).
[0134] Subsequently, the situation recognition unit 202 of the management server 2 determines whether the user is exercising vigorously based on the detection result (step S180). Whether the user is exercising vigorously can be determined, for example, based on a feature value obtained by analyzing acceleration data or the like, or by comparing with a predetermined threshold.
[0135] Then, if there is no speaking voice of the user (step S159 / No), the environment around the user is not noisy (step S162 / No), the user's heart rate is not too high and the user's body temperature is normal, and the user is in a calm state (step S171 / Yes), and the user is not exercising vigorously (step S180 / No), the situation recognition unit 202 of the management server 2 infers that the user is in a situation where he can easily receive messages (step S183).
[0136] An example of the recognition processing of the user status according to the first example has been described above. This example enables the presentation of messages when the user is in a state where the user can easily accept the message, and enables the presentation of messages more effectively. For messages from the host system and (indirectly) prompting a change in behavior, the user is more preferably in a state where the message can be easily accepted. The message presented by the host system can, for example (for example, by projecting a camera image of the childcare status) let a father who does not help share the housework and childcare know the state of his wife struggling alone or the state of a crying baby. For example, when the user (father) soaks himself in a bathtub and feels relaxed, the presentation of such a message is more expected to make the father feel naturally to help with housework and childcare (that is, accept the message and make it spontaneously change his behavior). In addition, the presentation of a notification of the fact that a colleague is not very happy about the messy desk of the user in the office (for example, by projecting a camera image of how people view the messy desk) is more expected to make the user feel naturally to tidy up the desk (that is, make the user accept the message and spontaneously change his behavior).
[0137] Figure 7 The determination criteria shown are merely examples, and the first example is not limited to this example.
[0138] also, Figure 7The order of steps S153 to S162 (determination processing based on voice data), steps S165 to S171 (determination processing based on biological data), and steps S174 to S180 (determination processing based on motion data) shown in FIG. 1 is not limited to Figure 7 The order shown is not intended, and sets of these steps may be performed in parallel with each other or in a different order.
[0139] 3-2. Second example (presentation based on message content and user status)
[0140] Next, a second example of the implementation will be described. In the second example, in addition to the timing according to the user status described with respect to the first example, when the user status matches the content of the message presented by comparing the content of the message with the user status, the presentation of the message enables the effect of the presentation of the message to be further enhanced. That is, even if the user is in a state that can easily accept the message, if the content of the message does not match the user's status (e.g., physical condition or emotion), the presentation of the message is not performed.
[0141] For example, if the content of the message includes keywords related to sports or physical exercise such as exercise or going out, it is desirable to present the message when the user is in a state where the user's physical condition is good and the user is full of energy. On the other hand, if the content of the message includes keywords related to conserving physical energy such as rest or go home, it is desirable to present the message when the user is in a state where the user's physical condition is poor and the user is tired. Therefore, for example, considering the user's physical condition, when the user is tired, a message recommending exercise is not presented.
[0142] The following will refer to Figure 8 and Fig. 9 The operation and processing according to the second example are described in detail.
[0143] Figure 8 : is a sequence diagram showing an example of a flow for accumulation processing of messages according to the second example. Figure 8 As shown, first, the host system server generates a message prompting the user to spontaneously change his behavior (step S193), and sends the message to the management server 2 (step S196). The message prompting the user to spontaneously change his behavior and presented by the host system is described as an example of a message, but is not limited to this example. According to a second example, the message can be a response message presented by a general agent system, information presented from any information presentation system, etc.
[0144] Subsequently, the message management unit 203 of the management server 2 performs semantic analysis on the message received from the host system server (step S199).
[0145] Subsequently, the message management unit 203 of the management server 2 determines whether the message has content that should be considered when displaying the message to the user (step S202). For example, if the message includes a predetermined keyword related to the user's state (for example, a keyword related to something that requires physical strength, such as "exercise", or a keyword related to saving physical strength, such as "rest"), the message management unit 203 determines that the message has content that should be considered.
[0146] Subsequently, if it is determined that the message has content of the user's state that should be considered (step S202 / Yes), the message management unit 203 associates the condition of the user's state that matches the message content with the message, and the condition is used as supplementary information (step S205). The condition of the user's state may be, for example, "good / unwell", "energetic / no energy", or "good / fatigued".
[0147] Then, the message management unit 203 accumulates the received messages in the storage unit 220 (step S208).
[0148] As described above, when the message management unit 203 of the management server 2 obtains messages addressed to a user from the system, the message management unit 203 performs semantic analysis on the message content, and for messages whose user status should be considered, the message management unit 203 is able to accumulate messages in advance in association with supplementary information that is a condition of the user status.
[0149] Such message accumulation processing can be performed continuously. Fig. 9 The operation and processing when the already accumulated messages are presented from the output device 1 to the user are described.
[0150] Fig. 9 is a sequence diagram showing an example of the flow of the message presentation process according to the second example. Fig. 9 The processing at steps S103 to S124 shown in FIG. Figure 5 The same processing as described with respect to the first example is the same as that of the same reference numerals. That is, if there is a message to be presented to the user, processing for inferring whether the current user status is a status that can easily accept the message is performed.
[0151] Subsequently, although in the first example, output control processing for the message is performed when a notification is received from the management server 2, which notification is the fact that the user is in a condition where he can easily accept the message, in the second example, the output device 1 checks whether the message sent from the management server 2 at the above-mentioned step S109 has supplementary information (condition of the user status) associated with it (step S213).
[0152] Subsequently, if supplementary information has been associated with the message transmitted from the management server 2 (step S213 / Yes), the output device 1 analyzes the sensing data acquired from the sensor 122 (step S216 ), and transmits the analysis result to the management server 2 (step S219 ).
[0153] Subsequently, the situation recognition unit 202 of the management server 2 performs recognition of the user status based on the analysis result of the sensing data (step S222 ), and transmits the recognition result to the output device 1 (step S225 ).
[0154] Identifying the user status can be, for example, identifying "good physical condition" when the body temperature is normal, there is no sighing, the voice is loud and / or the user is smiling, or identifying "poor physical condition" when the body temperature is higher than normal body temperature, the number of sighs is greater than a predetermined value, the voice is small, the user is silent and / or the user is not smiling.
[0155] Subsequently, the output device 1 determines whether the user status indicated by the recognition result from the management server 2 matches the supplementary information (condition of the user status) that has been associated with the message (step S228). The condition of the user status as the supplementary information may be, for example, "good / bad physical condition", "energetic / no energy", or "good / tired" as described above, and the output device 1 determines whether the current state of the user matches the condition.
[0156] If the user's state matches the supplementary information (step S228 / Yes), the output device 1 performs output control of the message for the user (step S127). The details of the output control for the message are the same as those in the first example (see Figure 6 ). If no supplementary information has been associated with the message at the above step S213 (step S213 / No), output control of the message for the user may be performed without considering any match between the user status and the content (step S127).
[0157] Therefore, when the user is in a situation that can easily accept a message and the content of the message to be presented matches the state of the user, the message can be presented.
[0158] In addition, the output device 1 can present a message having content matching the state of the user among the multiple messages received from the management server 2. For example, in the case where the behavior change message for increasing the time for family gatherings includes a "Why don't you go to the batting center and do moderate exercise there?" message and a "Let's go home early" message, if the user's physical condition is not good and the user is tired, the latter message can be presented, and if the user's physical condition is good and the user is full of energy, the former message can be presented. The former message from the host system and recommending exercise enables the user to obtain time for family gatherings in the following ways: relieve stress for the user through exercise; reduce alcohol intake according to the amount of stress on the user; thereby improve the user's sleep quality; make the wake-up time depend on the length of early bedtime; and enable the user to have breakfast with his family. In addition, the latter message from the host system enables the user to obtain time for family gatherings in the following ways: enable the user to go home early; thereby match his life rhythm with the life rhythm of other family members; and enable the user to have dinner with his family.
[0159] 3-3. Third Example (Change in User Status)
[0160] Next, a third example of the implementation will be described. In the third example, when the user is in a situation where it is difficult for the user to accept a message, a control guide (hereinafter, also referred to as guide content) is presented to change the user's situation to a situation where the user can easily accept the message. Therefore, the chance that the message can be presented in a situation where the message can be easily accepted can be increased.
[0161] In addition, if the content of the message does not match the user's state, some guidance content that changes the user's state to a state that matches the content of the message is presented. Therefore, the opportunity of being able to present a message with content that matches the user's state can be increased.
[0162] For example, Fig.10 An example of guidance content that changes the user's situation or status is shown. Fig.10 is a diagram showing an example of guidance content that changes a user's situation according to a third example.
[0163] like Fig.10As shown, for example, if the user is having a conversation, a guidance display image 500 (projected by the output device 1) requesting the user to be quiet can be presented to guide the user to a quiet state where the message can be easily accepted. In addition, if the user is in a state of fatigue and discomfort, a photographic image 501 of the user's children or family, or an image 502 of the user's favorite character that cheers the user, etc. can be presented to more effectively present a message with content matching the good state. In addition, if the user is excited, hurried or panicked, a direct message 504, etc., such as "Calm down!" or "Take a deep breath." can be presented to guide the user to a calm state where the message can be easily accepted.
[0164] Will refer to Figures 11 to 13 Such operations and processes in the third example are described.
[0165] Figures 11 to 13 is a sequence diagram showing an example of the flow of the message presentation process according to the third example.
[0166] Fig.11 The processing of steps S103 to S121 shown in FIG. Figure 5 The same processing as described with respect to the first example is the same as that of the same reference numerals. That is, if there is a message to be presented to the user, processing for inferring whether the current user status is a status that can easily accept the message is performed.
[0167] Subsequently, if it has been inferred that the user is not in a condition that can easily accept the message (step S121 / No), the management server 2 selects a guide image that controls the user's condition (i.e., changes the user's condition to a condition that can easily accept the message) (step S253), and sends the guide image to the output device 1 (step S256). For example, the guide image has been previously accumulated in the storage unit 220, and the management server 2 selects an appropriate guide image based on the current user condition.
[0168] Then, the output device 1 controls the output of the received guide image (step S259). In the control of the output of the guide image, for example, the guide image is projected to an appropriate projection location around by means of the projector 131. Figure 6 The output control process for the message described can be applied to the details of the projection control for the guide image. That is, if there is a monochrome plane, a mirror, a still water surface, a white smoke-like substance, etc. at a distance that can be projected, the output device 1 projects the guide image to that place.
[0169] It is desirable to present a guidance image to change the user's situation to one in which the user can easily accept the message. After the guidance image has been presented, Fig.11As shown, the above-mentioned processing starting from step S112 is repeated again.
[0170] On the contrary, if it is estimated that the user is in a state where he can easily receive the message (step S121 / yes), Fig.12 As shown, the management server 2 notifies the output device 1 of the fact that the user is in a condition where he can easily accept the message (step S124).
[0171] Subsequently, the output device 1 checks whether supplementary information (condition of the user status) has been associated with the message sent from the management server 2 (step S213), and if no supplementary information is associated with the message sent from the management server 2 (step S213 / No), the output device 1 performs output control of the message for the user (step S127). Processing at steps S124, S213, and S127 and reference Fig. 9 The processes of the same reference numerals described with respect to the second example are the same.
[0172] Then, if the supplementary information has been associated with the message sent from the management server 2 (step S213 / yes), Fig.13 As shown, the output device 1 acquires the sensing data and transmits the sensing data to the management server 2 (steps S216 and S219), and acquires the recognition result of the user's state from the management server 2 (steps S222 and S225).
[0173] Subsequently, the output device 1 determines whether the user status indicated by the recognition result from the management server 2 matches the supplementary information (condition of the user status) that has been associated with the message (step S228). The above processing at steps S216 to S228 is the same as that described with respect to the second example. Fig. 9 The same processes are shown with the same reference numerals.
[0174] Subsequently, if the user status does not match the supplementary information of the message (condition of the user status) (step S228 / No), the output device 1 notifies the management server 2 of the fact that the user status does not match the supplementary information of the message (step S263).
[0175] Subsequently, based on the supplementary information, the management server 2 selects a guidance image that controls (changes) the user's state (eg, emotion) to match the condition of the user's state indicated by the supplementary information (step S266), and sends the guidance image to the output device (step S269).
[0176] Then, the output device 1 performs output control on the received guide image (step S272). In the output control for the guide image, for example, the guide image is projected to an appropriate surrounding projection location by means of the projector 131. Figure 6The output control process for the message described can be applied to the details of the projection control for the guide image. That is, if there is a monochrome plane, a mirror, a still water surface, a white smoke-like substance, etc. at a distance that can be projected, the output device 1 projects the guide image to that place.
[0177] It is desirable to present a guidance image to change the user state to a state that matches the message content. After the guidance image has been presented, Fig.13 As shown, the above-mentioned processing starting from step S216 is repeated again.
[0178] If the user status matches the supplementary information of the message (condition of the user status) (step S228 / Yes), the output device 1 performs output control of the message for the user ( Fig.12 .
[0179] 3-4. Fourth Example (Presentation at a Location Visually Recognizable to the User)
[0180] Next, a fourth example of the implementation will be described. In each of the first to third examples described above, when a message or a guide image is presented to the user, for example, by appropriately projecting the message or guide image to an appropriate projection location around the user with the aid of the projector 131, the message or guide image can be displayed anywhere in various scenes of daily life. Even if the projection location (projection target) is an appropriate location (object), if the projection location is behind the user, the user is looking at a different location, or the focus is not on the projection location, the projection location may actually be a location that is difficult for the user to visually recognize.
[0181] Therefore, in the fourth example, if Fig.14 As shown, when there is an appropriate projection location in a visually recognizable area determined based on the user's head or facial orientation, line of sight, etc., control is performed so that a message or a guide image to the user is projected to the projection location so that the message can be presented to the user more reliably in various scenes of daily life. For example, the output device 1 can perform projection control according to whether the user's line of sight is directed to the projection location, or perform projection control according to whether the user's focus is on the projection location.
[0182] The following will refer to Fig.15 Such projection control according to the fourth example is described.
[0183] Fig.15 is a flowchart showing an example of the flow of the projection control process according to the fourth example. Fig.15 The processing at steps S130 to S139 in the embodiment is the same as that described in relation to the first example. Figure 6That is, when a message is displayed in a situation where the user is in a condition where the message can be easily accepted, the output device 1 performs a process of determining an appropriate projection location around the user (e.g., a monochrome plane, a mirror, a still water surface, a white smoke-like substance, etc. at a distance where projection is possible).
[0184] Subsequently, the output device 1 detects the user's line of sight using a line of sight detection sensor (step S303), and determines whether the user's line of sight is directed to the direction of the appropriate projection location determined at the above-mentioned steps S133 to S139 (step S306). The line of sight detection sensor is an example of the sensor 122, and the method of detecting the line of sight used may be, for example, a method of performing detection based on the detection result from the electrooculogram sensor, a method of performing detection by emitting infrared (IR) light to the eyeball and detecting the light reflected from the eyeball (corneal reflection method, sclera reflection method, etc.), or a method of performing detection based on the position of a reference point of the eye (e.g., the inner corner of the eye or corneal reflection) and a moving point of the eye (e.g., the iris or pupil) using a visible light camera. In addition, based on the camera video in which the user's face has been captured or the sensing data of the motion sensor, the output device 1 can determine the user's facial orientation or head direction, and regard the user's facial orientation or head direction as the line of sight direction.
[0185] Subsequently, if the user's line of sight is directed in the direction of the projection location (step S306 / yes), the output device 1 measures, for example, whether the user's focus is on the projection location (i.e., whether the image has been formed in front of the retina) by means of an eye refractometer sensor (e.g., an autorefractometer) (step S309).
[0186] Then, if the focus is on the projection location (step S312 / Yes), the output device 1 performs projection of a message or a guidance image (step S315 ).
[0187] The output device 1 may determine a plurality of appropriate projection locations and select a location that is visually easy for the user to recognize from among these projection locations. Fig.15 In the flowchart shown, it is determined whether the projection location is a location that is visually easy for the user to recognize based on the line of sight direction and focus, but the fourth example is not limited to this determination. For example, the determination may be performed using either the line of sight direction or the focus, or may be performed based on another factor (e.g., whether the projection location is in front of the user, whether the projection location is in the direction of travel, whether the projection location is within the range of the field of view, or whether the projection location is predicted to be a location that will immediately point to the line of sight).
[0188] 3-5. Fifth example (Learning "a situation where a message can be easily received")
[0189] Finally, the fifth example of the implementation will be described. Whether the user is in a condition where he can easily accept a message can be determined by comparing sensing data acquired from any of various sensors such as a microphone, a camera, a biosensor, and a motion sensor with a predetermined standard (pre-set predetermined standard), but the determination standard may change depending on the environment around the user, etc., and accurate determination may not be achieved by using only a predetermined given rule.
[0190] Therefore, in the fifth example, based on the reaction (feedback) of the user after the message has been presented, it is known whether the user is in a "condition that can easily accept the message", and thus the condition can be determined more accurately. For example, in the case where the agent image is projected and the message is output in voice, if the user looks at the agent and says one or more words of acceptance of the message, such as "Understood." or "OK.", it can be known that the user is in a "condition that can easily accept the message". In addition, if the user's behavior after the message presentation is observed, and the behavior after the message presentation is changed compared to the behavior before the message presentation, it can be known that the user is in a "condition that can easily accept the message".
[0191] The following will refer to Figures 16 to 18 Such projection control according to the fifth example is described.
[0192] Fig.16 and Fig.17 is a sequence diagram showing an example of the flow of a learning process for determining whether a user is in a situation in which he or she can easily accept a message according to the fifth example.
[0193] Fig.16 The processing at steps S103 to S118 shown in FIG. Figure 5 The same processing as described with respect to the first example is the same. That is, if there is a message to be transmitted to the user, recognition of the user's situation by the management server 2 is performed based on the sensing data acquired at the output device 1.
[0194] Subsequently, the management server 2 sends the recognition result of the user status to the output device 1 (step S330). In the case where the management server 2 has performed the learning of the user status recognition described later and has recorded the data of the "condition that can be easily accepted", the management server 2 sends the data to the output device 1 together with the recognition result of the user status.
[0195] Then, if the data on the “condition that can be easily accepted” cannot be acquired from the management server 2 (i.e., if the learning of the “condition that can be easily accepted” is not performed) (step S333 / No), the output device 1 performs output control of the message (step S127). The details of the output control of the message are the same as those in the first example (see Figure 6 ).
[0196] Then, based on the user's reaction after the message has been presented, the output device 1 determines whether the user has accepted the message (step S339). Fig.18 Details of the process used for this determination are described.
[0197] Subsequently, if it is determined that the user has accepted the message (step S342 / Yes), the output device 1 notifies the management server 2 of the fact that the message has been accepted (step S345).
[0198] Then, the management server 2 stores the recognition result of the user's situation in the above step S118 as “a situation that (the user can accept the message) easily” (step S348 ).
[0199] In contrast, if it is determined that the user has not accepted the message (step S342 / No), the output device 1 does not notify the management server 2.
[0200] As described above, the management server 2 can understand the situation of each user who can easily accept the message. Therefore, for example, some people may be in a situation where they can easily accept the message when they are exercising vigorously (for example, when they are running on a treadmill in a gym), and thus the situation that can easily accept the message can be learned that varies from person to person.
[0201] At the above step S333, if data on "conditions that can be easily accepted" can be obtained from the management server 2 (that is, learning of "conditions that can be easily accepted" has been performed) (step S333 / Yes), then Fig.17 The output device 1 determines whether the recognized user situation matches the "situation that can be easily accepted" (step S353).
[0202] Then, if the recognized user status matches the "situation that can be easily accepted" (step S353 / Yes), the output device 1 performs output control of the message (step S127). The details of the output control of the message are the same as those in the first example (see Figure 6 ).
[0203] Then, based on the user's reaction after the message has been presented, the output device 1 determines whether the user has accepted the message (step S339). Fig.18Describe the details of the processing used for this determination. In the fifth example, the user's condition in which he or she can easily accept a message may change depending on the environment around the user, etc., and therefore even if it has been determined based on the learning result that the user is in a condition in which he or she can easily accept a message and the message has been presented, the user's reaction thereafter is observed again, and it is determined whether the message has actually been accepted.
[0204] Subsequently, if it is determined that the user has not accepted the message (step S362 / No), the output device 1 notifies the management server 2 of the fact that the message has not been accepted (step S365).
[0205] The management server 2 deletes the user status (status of not accepting the message) from the data that has been recorded about the "status that can be easily accepted" (step S368). Therefore, the situation that the status that can easily accept the message has changed is learned, and the "status that can be easily accepted" can be updated.
[0206] If the user has accepted the message (step S362 / Yes), the output device 1 does not notify the management server 2.
[0207] Determine whether the message has been received
[0208] Next, we will refer to Fig.18 An example of the determination process of whether the message has been accepted represented by the above-mentioned step S339 is described.
[0209] Fig.18 is a flowchart illustrating an example of a process flow for determining whether a user has accepted a message according to a fifth example.
[0210] like Fig.18 As shown, first, the output device 1 performs facial recognition and line of sight analysis on the user based on a camera image in which the user's face is captured by means of the sensor 122 (camera) (step S363), and determines whether the user is facing the proxy image projected as a message presentation (i.e., whether the user is directing his line of sight or face toward the proxy image) (step S366).
[0211] Then, if the user is not facing the agent image (step S366 / No), the output device 1 determines that the message is not accepted (step S378). For example, if the user looks away immediately after just viewing the agent image, if the user's focus is not on the agent image, if the user does not view the agent image carefully, or if the user has viewed the agent image but does not seem satisfied, the output device 1 may determine that the message is not accepted.
[0212] Subsequently, based on the voice data collected by the sensor 122 (microphone), the output device 1 performs voice recognition (step S369), and determines whether a voice indicating acceptance of a message has been detected (step S372).
[0213] Subsequently, if a voice indicating acceptance of the message ("Got it.", "OK.", "Sounds good.", "Got it.", "Sounds interesting", etc.) is not detected (step S372 / No), the output device 1 determines that the message has not been accepted (step S378). Without limitation to the case when a voice indicating acceptance of the message is not detected, for example, when the user sighs, the user taps his tongue, or a voice indicating that the user feels that the presentation of the message is obstructed has been detected, the output device 1 may determine that the message has not been accepted.
[0214] If the user is facing the agent image (step S366 / Yes) and a voice indicating acceptance of the message has been detected (step S372 / Yes), the output device 1 determines that the message has been accepted (step S375).
[0215] As an example in this article, the determination of whether the message has been accepted is performed based on sight and voice, but the fifth example is not limited to this example. For example, the determination can be performed using either sight and voice, or can be performed based on another element (such as the user's behavior or gesture after the message is presented).
[0216] 4. Conclusion
[0217] As described above, the information processing system according to the embodiment of the present disclosure realizes more effective information presentation by performing information presentation when the user is in a situation where the user can easily accept information.
[0218] The preferred embodiment of the present disclosure is described in detail above with reference to the accompanying drawings, but the present technology is not limited to this example. It is obvious that a person skilled in the art of the present disclosure can obtain various modifications or revisions within the scope of the technical idea recorded in the claims of this patent, and it should be understood that these modifications or revisions also reasonably belong to the technical scope of the present disclosure.
[0219] For example, a computer program may be generated for causing hardware such as a CPU, ROM, and RAM provided internally in the output device 1 or the management server 2 to function as the above-described output device 1 or the management server 2. In addition, a computer-readable storage medium having the computer program stored therein may also be provided.
[0220] In addition, the effects described in this specification are only illustrative or exemplary, not restrictive. That is, in addition to or instead of the above effects, the technology according to the present disclosure can achieve other effects that are obvious to those skilled in the art based on the description in this specification.
[0221] The present technology may also include the following configurations. (1)
[0223] An information processing system, comprising:
[0224] A control unit controls a presentation unit to present a message to the user when it is estimated based on the sensing data related to the user that the user is in a condition where the user is likely to accept the message. (2)
[0226] An information processing system according to (1), wherein the control unit infers whether the user is in a condition that is easy to accept the message based on at least any one of the noise conditions around the user, the user's biological information, the user's expression, and the user's movement information. (3)
[0228] The information processing system according to (1) or (2), wherein:
[0229] The presentation unit is a projector, and
[0230] When there is a projection location satisfying a given condition around the user, the control unit controls the projector to project the message to the projection location. (4)
[0232] The information processing system according to (3), wherein the control unit determines the projection location that satisfies the given condition based on sensing data around the user. (5)
[0234] The information processing system according to (3), wherein, when there is a projection location that the user can visually recognize, the control unit controls the projector to project the message to the projection location. (6)
[0236] The information processing system according to (5), wherein the control unit performs a line of sight analysis on the user based on sensing data related to the user, and determines the projection location that the user can visually recognize. (7)
[0238] The information processing system according to any one of (1) to (6), wherein the control unit controls the presentation unit to present a message that can be perceived by the user. (8)
[0240] The information processing system according to (7), wherein the control unit presents the message by means of at least any one of an image or a sound. (9)
[0242] The information processing system according to any one of (1) to (8), wherein even if it is inferred that the user is in a condition where he or she is likely to accept a message, the control unit controls not to perform message presentation to the user according to the content of the message. (10)
[0244] The information processing system according to (9), wherein the control unit performs control to:
[0245] When the content of the message matches the state of the user, performing message presentation to the user, and
[0246] When the content of the message does not match the state of the user, the message presentation to the user is not performed. (11)
[0248] An information processing system according to any one of (1) to (10), wherein, when it is inferred that the user is not in a condition to be easily receptive to a message, the control unit presents control guidance for making the user's condition become a condition to be more easily receptive to a message. (12)
[0250] An information processing system according to (11), wherein the control guidance is an image or sound intended to change at least any one of the noise conditions around the user, the user's biological information, the user's expression, and the user's movement information. (13)
[0252] The information processing system according to (10), wherein, when it is inferred that the state of the user does not match the content of the message, the control unit presents control guidance for matching the state of the user with the content of the message. (14)
[0254] The information processing system according to any one of (1) to (13), wherein the control unit performs the following operations:
[0255] detecting a reaction of the user to the message presented by the presentation unit based on sensory data associated with the user, and
[0256] The user's condition of being receptive to messages is updated by learning the relationship between the detection result and the user's condition. (15)
[0258] The information processing system according to any one of (1) to (14), wherein the control unit controls the presentation unit to present a message provided from an external agent system to the user. (16)
[0260] An information processing method, wherein a processor performs the following processing:
[0261] When it is estimated based on the sensing data related to the user that the user is in a condition that is easy to accept a message, the presentation unit is controlled to present a message to the user. (17)
[0263] A recording medium having recorded therein a program for causing a computer to function as:
[0264] A control unit controls a presentation unit to present a message to the user when it is estimated based on the sensing data related to the user that the user is in a condition where the user is likely to accept the message.
[0265] Reference numerals list
[0266] 1 Output Device
[0267] 2 Management Server
[0268] 3 Network
[0269] 4 Proxy System Server
[0270] 100 Control Unit
[0271] 110 Communication unit
[0272] 120 Input Units
[0273] 121 Operation input unit
[0274] 122 Sensors
[0275] 122a Biosensor
[0276] 130 Output unit
[0277] 131 Projector
[0278] 132 Speakers
[0279] 140 storage units
[0280] 200 Control Unit
[0281] 201 User Management Unit
[0282] 202 Situation Recognition Unit
[0283] 203 Message Management Unit
[0284] 204 Message presentation control unit
[0285] 210 Communication Unit
[0286] 220 storage units
Claims
1. An information processing system, include: Control unit, determining whether the user is in a condition to be easily receptive to a message based on at least one of the user's biometric information and the user's facial expression; When it is determined that the user is in a condition that is receptive to a message, determining whether to present the message to the user by comparing (1) the user's physical state determined by the sensed data of the user's physical state with (2) the expected physical state of the user indicated by the content of the message; and Only when it is determined to present the message to the user, the projector is caused to project the message.
2. The information processing system according to claim 1, in, When there is a projection location satisfying a given condition around the user, the control unit controls the projector to project the message to the projection location.
3. The information processing system according to claim 2, in, The control unit determines a projection location that satisfies the given condition based on sensing data around the user.
4. The information processing system according to claim 2, in, When there is a projection location that the user can visually recognize, the control unit controls the projector to project the message to the projection location.
5. The information processing system according to claim 4, in, The control unit performs a line of sight analysis on the user based on the sensing data related to the user, and determines a projection location that the user can visually recognize.
6. The information processing system according to claim 1, in, The control unit controls the presentation unit to present a message that can be perceived by the user.
7. The information processing system according to claim 6, in, The control unit presents the message by means of at least any one of an image and a sound.
8. The information processing system according to claim 1, in, When it is estimated that the user is not in a condition that is easy to accept a message, the control unit presents control guidance for changing the condition of the user to a condition that is easy to accept a message.
9. The information processing system according to claim 8, in, The control guidance is an image or sound intended to change at least any one of the noise conditions around the user, the biological information of the user, the expression of the user, and the movement information of the user.
10. The information processing system according to claim 1, in, When it is inferred that the state of the user does not match the content of the message, the control unit presents a control guide for matching the state of the user with the content of the message.
11. The information processing system according to claim 1, in, The control unit: detecting a reaction of the user to the message presented by the presentation unit based on sensory data associated with the user, and The user's condition of being receptive to messages is updated by learning the relationship between the detection result and the user's condition.
12. The information processing system according to claim 1, in, The control unit controls the presentation unit to present a message provided from an external agent system to the user.
13. An information processing method, in, The processor performs the following processing: determining whether the user is in a condition to be easily receptive to a message based on at least one of the user's biometric information and the user's facial expression; When it is determined that the user is in a condition that is receptive to a message, determining whether to present the message to the user by comparing (1) the user's physical state determined by the sensed data of the user's physical state with (2) the expected physical state of the user indicated by the content of the message; and Only when it is determined to present the message to the user, the projector is caused to project the message.
14. A recording medium having a program recorded thereon, the program for causing a computer to function as: Control unit, determining whether the user is in a condition to be easily receptive to a message based on at least one of the user's biometric information and the user's facial expression; When it is determined that the user is in a condition that is susceptible to receiving a message, determining whether to present the message to the user by comparing (1) a physical state of the user determined by sensing data of the physical state of the user with (2) an expected physical state of the user indicated by content of the message; and Only when it is determined to present the message to the user, the projector is caused to project the message.
