Information notification device

Abstract

An information notification device according to one aspect of the present invention comprises a reception unit, an acquisition unit, a recognition unit, an expansion unit, and a transmission unit. The reception unit receives a notification destined for a user. The acquisition unit acquires user characteristic information indicating the superiority and inferiority of the user with respect to a plurality of modals. The recognition unit recognizes the situation of the user with respect to a terminal of the user. On the basis of the user characteristic information acquired by the acquisition unit and the situation of the user recognized by the recognition unit, the expansion unit expands the notification, received by the reception unit, to one or more pieces of modal information not including modals serving as distractor stimuli for the user among the plurality of modals. The transmission unit transmits, to the user, a notification including the one or more pieces of modal information expanded by the expansion unit.

Description

Information notification device

[0001] One aspect of the present invention relates to an information notification device.

[0002] When notification information such as news, email, or SNS (Social Networking Service) is received via a network including the Internet, other modal information such as text, voice, or facial expression that is not included in the received information is generated to expand the notification provided to the user.

[0003] For example, in Patent Document 1, when an email is received, an avatar's movement is generated from the language input information to provide an enhanced notification.

[0004] Regarding modal information, for example, as disclosed in Non-Patent Document 1, there is known an experiment by Mehrabian that investigated which modal takes priority in influencing impressions when visual, auditory, and linguistic information do not match.

[0005] Furthermore, Non-Patent Document 2 explains the cognitive characteristics of understanding, organizing, memorizing, and expressing information received through sensory organs such as the eyes and ears.

[0006] JP 2011-138464 A

[0007] Kurita, "How did emojis originate and spread around the world?", Institute of Electronics, Information and Communication Engineers, Communication Society Magazine 11 (3), 2017. Honda, Mami, "A Doctor's Intelligence Test", Kobunsha Shinsho, June 15, 2012.

[0008] However, when notifications are expanded to include modal information such as text, voice, and facial expressions, the notification may contain modal information that can be disruptive depending on the characteristics or circumstances of the user receiving the notification.

[0009] For example, regarding user characteristics, in an extreme example, if a user with hyperacusis is notified by audio, it is possible that the notification will cause distress. More generally, if a user with hyperacusis or hearing dominance is notified by audio, it is possible that the user will be too distracted and the notification will interfere with their activities.

[0010] Furthermore, with regard to the user's situation, for example, if the user is listening to music, providing the user with a voice notification will interfere with the user's music listening activity.

[0011] Notifications using not only voice but also other modal information may disrupt user activities, such as researching, viewing, and creating materials using a work terminal such as a personal computer (PC) or smartphone, watching and creating content, programming, remote meetings, phone calls, emails, chats, and social media.

[0012] The present invention has been made in view of the above circumstances, and has an object to provide a technique that enables extended notifications to be made without interfering with the user's activities.

[0013] In order to solve the above problem, an information notification device according to one embodiment of the present invention includes a receiving unit, an acquiring unit, a grasping unit, an expanding unit, and a delivering unit. The receiving unit receives a notification addressed to a user. The acquiring unit acquires user characteristic information indicating the user's superiority and inferiority with respect to multiple modals. The grasping unit grasps the user's situation with respect to the user's terminal. The expanding unit expands the notification received by the receiving unit to one or more pieces of modal information, of the multiple modals, that do not include any modal that constitutes a disruptive stimulus for the user, based on the user characteristic information acquired by the acquiring unit and the user's situation grasped by the grasping unit. The delivering unit delivers the notification including the one or more pieces of modal information expanded by the expanding unit to the user.

[0014] According to one aspect of the present invention, modal information that does not include modals that are disruptive stimuli for the user is delivered based on user characteristic information and the user's situation, thereby providing a technology that enables extended notifications to be made without interfering with the user's activities.

[0015] FIG. 1 is a block diagram showing an example of the hardware configuration of a user terminal to which an information notification device according to a first embodiment of the present invention is applied. FIG. 2 is a block diagram showing an example of the software configuration of the user terminal. FIG. 3 is a diagram showing an example of the contents stored in a modal characteristics storage unit. FIG. 4 is a diagram schematically showing each modal in the example of modal characteristics shown in FIG. 3. FIG. 5 is a diagram showing an example of the contents stored in a modal selection rule storage unit. FIG. 6 is a flowchart showing an example of the processing procedure and processing content of information notification processing executed by a control unit of a user terminal. FIG. 7 is a diagram showing an example of a received notification. FIG. 8 is a diagram showing an example of multiple pieces of modal information obtained by expanding a received notification. FIG. 9 is a diagram showing an example of a recognition status. FIG. 10 is a diagram schematically showing a modal selected in the example shown in FIG. 9. FIG. 11 is a diagram showing expanded modal information included in a notification delivered in the example shown in FIG. 9. FIG. 12 is a diagram showing another example of a recognition status. FIG. 13 is a diagram schematically showing a modal selected in the example shown in FIG. 12. FIG. 14 is a diagram showing expanded modal information included in a notification delivered in the example shown in FIG. 12. FIG. 15 is a diagram showing another example of the grasping status. FIG. 16 is a diagram schematically showing a modal selected in the example shown in FIG. 15. FIG. 17 is a diagram showing extended modal information included in the notification delivered in the example shown in FIG. 15. FIG. 18 is a block diagram showing an example of the software configuration of a user terminal to which an information notification device according to a second embodiment of the present invention is applied. FIG. 19 is a flowchart showing an example of the processing procedure and processing content of information notification processing executed by a control unit of a user terminal in the second embodiment. FIG. 20 is a block diagram showing an example of the hardware configuration of an information notification system to which an information notification device according to a third embodiment of the present invention is applied. FIG. 21 is a block diagram showing an example of the software configuration of an information notification system. FIG. 22 is a flowchart showing an example of the processing procedure and processing content of information notification processing executed by a control unit of a distribution server included in the information notification system. FIG. 23 is a flowchart showing an example of the processing procedure and processing content of information notification processing executed by a control unit of a user terminal included in the information notification system.

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] 1 and 2 are block diagrams showing an example of the hardware and software configurations of a user terminal 1 to which an information notification device according to a first embodiment of the present invention is applied. The user terminal 1 is an information processing device such as a PC, a smartphone, or a tablet terminal that can be connected to a network including the Internet.

[0018] The user terminal 1 comprises a control unit 11 using a hardware processor such as a CPU, and a storage unit having a program storage unit 12 and a data storage unit 13, a communication interface unit 14, and an input / output interface unit 15, all connected to the control unit 11 via a bus 16. In the figure, "interface" is abbreviated as "IF."

[0019] The control unit 11 is a hardware processor such as a CPU (Central Processing Unit). For example, the CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded CPU. The control unit 11 may include multiple hardware processors.

[0020] Under the control of the control unit 11, the communication interface unit 14 transmits and receives information to and from other devices using a communication protocol defined by the network.

[0021] An input device 171, an audio input device 172, a video input device 173, an audio output device 174, a video output device 175, etc. are connected to the input / output interface unit 15. Note that the user terminal 1 does not necessarily have to be equipped with all of these devices, and some of the devices may be connected as external devices.

[0022] The input device 171 is used by the user of the user terminal 1 to input instructions necessary for the operation of the user terminal 1. If the user terminal 1 is a personal computer, the input device 171 includes, for example, a keyboard, a pointing device such as a mouse or a touchpad, etc. If the user terminal 1 is a smartphone or a tablet terminal, the input device 171 includes, for example, a touch panel and operation buttons arranged on the display screen of the video output device 175. The input device 171 may also include a reader device for reading information from a recording medium that stores various types of information such as programs and data.

[0023] The voice input device 172 includes a microphone and captures voices uttered by the user.

[0024] The video input device 173 includes a camera such as a web camera or a general video camera, and captures video of the user.

[0025] The audio output device 174 includes a speaker and transmits various pieces of information to the user who is the operator of the user terminal 1 by sound.

[0026] The video output device 175 includes a display such as a liquid crystal monitor, an organic EL (Electro Luminescence) monitor, or a projector screen, and displays various information to the user who is the operator of the user terminal 1 .

[0027] The program storage unit 12 is configured by combining, for example, a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or an EEPROM (registered trademark) (electrically erasable programmable read-only memory) that can be written to and read from at any time, with a nonvolatile memory such as a read-only memory (ROM). The program storage unit 12 stores middleware such as an operating system (OS) as well as application programs required for operation as the information notification device according to the first embodiment of the present invention. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0028] The data storage unit 13 is, for example, a combination of a nonvolatile memory that can be written to and read from as needed, such as an HDD, SSD, EEPROM, or memory card, as a storage medium, and a volatile memory, such as a RAM (Random Access Memory). The data storage unit 13 includes, in its storage area, a modal characteristics storage unit 131 and a modal selection rule storage unit 132 as storage units necessary for implementing the first embodiment.

[0029] The modal characteristic storage unit 131 pre-stores user characteristic information indicating the dominance / inferiority of the user of the user terminal 1 for each of multiple modalities, such as text, voice, and facial expression. This user characteristic information for each modality can be determined, for example, with reference to Mehrabian's experiment, as described in Non-Patent Document 1. The user characteristic information may also be determined with reference to a cognitive characteristic test, as disclosed in Non-Patent Document 2. Alternatively, the user characteristic information may be determined based on the self-evaluation input of the user of the user terminal 1. FIG. 3 is a diagram illustrating an example of the contents stored in the modal characteristic storage unit 131. For example, as shown in FIG. 3, the modal characteristic storage unit 131 can store the dominance / inferiority of each modal as a numerical percentage. FIG. 4 is a diagram schematically illustrating each modal in the example of modal characteristics shown in FIG. 3. In other words, FIG. 4 is a diagram schematically illustrating the dominance / inferiority of each modal for easy understanding. FIGS. 3 and 4 illustrate an example in which facial expression is determined as the inferior modal and voice is determined as the dominant modal.

[0030] The modal selection rule storage unit 132 stores selection rules for determining which modal to use based on user characteristic information about modalities and the user's status relative to the user terminal 1. The user's status can include whether the user terminal 1 is operating, i.e., the operating status of the user terminal 1. The user's status can also include the usage status of the user terminal 1, which is determined from the user's facial orientation toward the user terminal 1, the input / output status of text, audio, and video at the user terminal 1, and the like. Here, "face orientation" refers to whether the user's face is facing the user terminal 1. The input / output status of text, audio, and video at the user terminal 1 can include the status of keyboard input, mouse operation, audio input / output, and video input / output. The "keyboard input status" refers to the extent to which input is being performed using a keyboard, which is one of the input devices 171. The "mouse operation status" refers to the extent to which operation is being performed using a mouse, which is one of the input devices 171. The "audio input / output status" refers to whether audio input / output is being used by the audio input device 172 or the audio output device 174. The "video input / output status" refers to whether the video input / output is being used by the video input device 173 or the video output device 175.

[0031] The modal selection rule storage unit 132 can store, as a selection rule, a rule related to the user's concentration level, such as, for example, "when the face is facing directly toward the user terminal 1," regarding "face orientation status."

[0032] Furthermore, the modal selection rule storage unit 132 can store a rule relating to language, such as "when keyboard input is used, text modal is not used" in relation to "keyboard input situation."

[0033] Furthermore, the modal selection rule storage unit 132 can store a rule relating to the degree of concentration, such as "in situations where mouse operation is involved, use the inferior modal whenever possible," in relation to the "mouse operation situation."

[0034] Furthermore, the modal selection rule storage unit 132 can store a rule relating to hearing, such as "when audio input / output is fully used, the audio modal is not used," in relation to the "audio input / output situation."

[0035] Furthermore, the modal selection rule storage unit 132 can store a rule relating to vision, such as "when video output is fully used, facial expression modal is not used," in relation to the "video input / output situation."

[0036] The modal selection rules stored in the modal selection rule storage unit 132 may be learned using a decision tree or a support vector machine (SVM).

[0037] FIG. 5 is a diagram showing an example of the contents stored in the modal selection rule storage unit 132. While FIG. 5 shows the information in table format, it is not limited to this. Furthermore, FIG. 5 illustrates an extracted portion of the user characteristic information for modals stored in the modal characteristic storage unit 131, corresponding to the case where facial expression is the inferior modal and speech is the dominant modal, as shown in FIGS. 3 and 4 . It goes without saying that the modal selection rule storage unit 132 stores rules for various combinations of modals. Furthermore, if the user terminal 1 is used only by a specific user and the dominance and inferiority of multiple modals are fixed, the modal selection rule storage unit 132 may store only rules appropriate for that user. In this case, the modal characteristic item in the example shown in FIG. 5 may be omitted. Alternatively, the modal characteristic storage unit 131 may be omitted by substituting this modal characteristic item. In the modal characteristic items in FIG. 5 , a circle indicates a dominant, a cross indicates a subordinate, and a triangle indicates a neutral. In addition, in the items of comprehension status in Fig. 5, a circle indicates a situation in which it is fully used, a cross indicates a situation in which it is not used, and a triangle indicates a situation in which it is not fully used. In the items of selection modal in Fig. 5, a circle indicates a selection, and a hyphen indicates a non-selection.

[0038] The control unit 11 includes, as processing function units according to the first embodiment of the present invention, a notification receiving unit 111, a notification expanding unit 112, a status grasping unit 113, a notification selecting unit 114, and a notification delivering unit 115. All of these processing function units 111 to 115 are realized by causing the hardware processor of the control unit 11 to execute an information notification program for the information notification device according to the first embodiment of the present invention, which is stored in the program storage unit 12.

[0039] The information notification program may be stored in advance in the program storage unit 12, or may be read out from a storage medium storing the information notification program when necessary and stored in the program storage unit 12. Alternatively, the information notification program may be downloaded from a program server (not shown) or the like and stored in the program storage unit 12. At least a part of the processing functions of at least one of the processing function units 111 to 115 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), instead of being realized by the information notification program and the hardware processor of the control unit 11.

[0040] The notification receiving unit 111 receives notifications addressed to the user of the user terminal 1 that are transmitted over the network via the communication interface unit 14. The notification receiving unit 111 can store the received notifications in a temporary storage area (not shown) provided in the data storage unit 13.

[0041] The notification expansion unit 112 generates modal information for all of a plurality of modalities, such as text, voice, and facial expression, for the notification received by the notification receiving unit 111. Specifically, the notification expansion unit 112 adds modal information and expands the notification by generating modal information such as the facial expression of the notification sender, the voice of the message, and the text of the message. The notification expansion unit 112 can store the generated plurality of modal information in a temporary storage area of ​​the data storage unit 13.

[0042] Regarding text modal information, if the notification received by the notification receiving unit 111 is text-based, the notification expanding unit 112 can use the text included in the notification. On the other hand, if the received notification is audio- or video-based, the notification expanding unit 112 generates text modal information using a known voice recognition function or image recognition function. Furthermore, the notification expanding unit 112 may use, for example, AI (artificial intelligence) to generate text of other content related to the notification content based on the text included in the received notification or the recognized text.

[0043] Regarding the voice modal information, if the notification received by the notification receiving unit 111 is voice-based, the notification expanding unit 112 can use the voice included in the notification. On the other hand, if the received notification is text-based, the notification expanding unit 112 generates the voice modal information using a known voice synthesis function. Furthermore, if the received notification is video-based, the notification expanding unit 112 extracts text included in the image using a known character recognition function or image recognition function, and generates the voice modal information from the extracted text using a known voice synthesis function. Note that when generating this voice modal information, voice of any voice tone may be synthesized, or a specific voice tone may be selected and used for each sender.

[0044] The facial expression modal information is visual information that can identify the sender of the notification. If the notification received by the notification receiving unit 111 is video-based, the notification expanding unit 112 generates the facial expression modal information using the video of the sender included in the notification. For example, AI can be used to extract the sender's facial expression from the video. If the received notification is text-based or audio-based, the notification expanding unit 112 can generate the facial expression modal information based on information about the sender of the notification. The facial expression may be generated for any facial image or avatar. Alternatively, for a specific sender, facial images with multiple facial expressions may be stored in advance in the data storage unit 13 and selected for use.

[0045] The situation grasping unit 113 acquires input information from the input device 171, the audio input device 172, and the video input device 173 via the input / output interface unit 15, and also acquires information indicating the output status from the audio output device 174 and the video output device 175. The input information and / or information indicating the output status may not be acquired from the input and / or output devices, but may be acquired from another process being executed by the control unit 11 that controls the input and / or output by those devices. Based on this acquired information, the situation grasping unit 113 determines the input / output status of text, audio, and video in the user terminal 1, the user's facial orientation toward the user terminal 1, and the like, and grasps the user's situation relative to the user terminal 1 from the determination result. The situation grasping unit 113 transmits this grasped user situation to the notification selection unit 114.

[0046] The notification selection unit 114 acquires user characteristic information by reading user characteristic information indicating the user's dominance and inferiority with respect to multiple modals from the modal characteristic storage unit 131. The notification selection unit 114 selects one or more pieces of modal information that do not include modals that are disruptive stimuli for the user from the multiple pieces of modal information generated by the notification expansion unit 112, based on the acquired user characteristic information and the user's situation grasped by the situation grasping unit 113. More specifically, the notification selection unit 114 selects one or more pieces of modal information that do not constitute disruptive stimuli for the user, in accordance with a selection rule based on the user characteristic information and the user's situation stored in the modal selection rule storage unit 132.

[0047] The notification distribution unit 115 distributes to the user an expanded notification including one or more pieces of modal information selected by the notification selection unit 114 based on both the user characteristic information and the user situation. Specifically, if text modal information or facial expression modal information is selected, the notification distribution unit 115 outputs the modal information from the video output device 175 via the input / output interface unit 15. Furthermore, if audio modal information is selected, the notification distribution unit 115 outputs the modal information from the audio output device 174 via the input / output interface unit 15.

[0048] (Operation Example) Next, an operation example of the user terminal 1 to which the information notification device configured as described above is applied will be described. Fig. 6 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit 11 of the user terminal 1. The information notification processing shown in this flowchart is executed while the user terminal 1 is operating.

[0049] 6, the control unit 11 of the user terminal 1 first operates as a notification receiving unit 111 and determines whether a notification transmitted from any sender via the network has been received via the communication interface unit 14 (step S101). If it is determined that a notification has not been received, the control unit 11 performs the process of step S101 again. In this way, the control unit 11 waits for the reception of a notification.

[0050] When it is determined that a notification has been received, the control unit 11 operates as the notification expansion unit 112 and expands the received notification in all of a plurality of modalities such as text, voice, and facial expression (step S102). That is, the control unit 11 generates a plurality of modal information corresponding to the received notification.

[0051] In addition, the control unit 11 operates as a situation grasping unit 113, determining the input / output status of text, audio, and video at the user terminal 1, the direction of the user's face toward the user terminal 1, etc., and grasping the user's situation relative to the user terminal 1 from the results of this determination (step S103).

[0052] Then, the control unit 11 operates as the notification selection unit 114 to select modal information to be used from the generated plurality of pieces of modal information (step S104). Specifically, the control unit 11 reads out user characteristic information indicating the user's superiority and inferiority with respect to the plurality of modals from the modal characteristic storage unit 131, and selects one or more pieces of modal information from the generated plurality of pieces of modal information that do not include any modal that may be a disturbing stimulus for the user, based on the user characteristic information and the grasped user situation.

[0053] The control unit 11 operates as the notification delivery unit 115 and delivers a notification to the user including one or more pieces of modal information selected based on both the user characteristic information and the user situation (step S105). Specifically, the control unit 11 outputs facial expression and text modal information from the video output device 175 and outputs audio modal information from the audio output device 174 via the input / output interface unit 15. Thereafter, the control unit 11 proceeds to the process of step S101.

[0054] A specific example will be described below. Here, the user characteristic information stored in the modal characteristic storage unit 131 is assumed to be superior / inferior as shown in Fig. 3, and the modal selection rule storage unit 132 stores selection rules as shown in Fig. 5.

[0055] 7 is a diagram showing an example of a received notification. Here, it is assumed that the user terminal 1 has received an email notification MN. In the example of FIG. 7, the email notification MN includes the sender "Labor Manager" indicated in "From:" and the email body indicated in "Message" as "Please check the work schedule...".

[0056] 8 is a diagram showing an example of multiple pieces of modal information that are extensions of a received notification. In response to the email notification MN shown in FIG. 7, the control unit 11 generates text modal information TX, voice modal information VO, and facial expression modal information FE, as shown in FIG.

[0057] FIG. 9 is a diagram illustrating an example of a grasp situation. In this example, the facial orientation, keyboard input, and mouse operation statuses are marked with circles, and the audio input / output and video input / output statuses are marked with crosses. This indicates a situation in which the user is concentrating on, for example, creating a document, without turning their face away from the screen of the user terminal 1, and is actively using keyboard input and mouse operation. In this situation, according to the selection rules shown in FIG. 5, facial expression modal information is selected, but audio and text modal information are not selected. FIG. 10 is a diagram schematically illustrating modals selected in the example shown in FIG. 9. In FIG. 10, dotted lines indicate non-selection. In the user situation shown in FIG. 9, because the user's face is not turned away and there is mouse operation, the control unit 11 selects facial expression, which is a subordinate modal. Furthermore, the control unit 11 does not select audio because it is a dominant modal, and does not select text modal because there is keyboard input. FIG. 11 illustrates extended modal information included in the notification delivered in the example shown in FIG. 9. In the user's situation as shown in FIG. 9, only the facial expression modal information FE selected based on both the user characteristic information and the user's situation is notified as shown in FIG.

[0058] FIG. 12 is a diagram showing another example of a recognition status. In this example, the video input / output status is indicated by a circle, the facial orientation status and audio input / output status are indicated by a triangle, and the keyboard input status and mouse operation status are indicated by a cross. This situation represents, for example, a situation in which a user is participating in a remote conference but not making any comments, is facing away from the screen of the work terminal, and is not using keyboard input or mouse operation. In this situation, according to the selection rule shown in FIG. 5, audio and text modal information is selected, but facial expression modal information is not selected. FIG. 13 is a diagram schematically showing modals selected in the example shown in FIG. 12. In the user situation shown in FIG. 12, video input / output is used, so the control unit 11 does not select the facial expression modal. Furthermore, because audio input / output is not used at all and the user's face is facing away from the screen of the user terminal 1, the control unit 11 selects the audio modal, which is the dominant modal. Furthermore, because keyboard input is not used, the control unit 11 selects the text modal. FIG. 14 is a diagram showing extended modal information included in the notification delivered in the example shown in FIG. 12. In the user situation as shown in FIG. 12, the voice modal information VO and the text modal information TX selected based on both the user characteristic information and the user situation are notified as shown in FIG.

[0059] FIG. 15 is a diagram showing another example of a grasp situation. In this example, only the facial orientation status is marked with a triangle, while the other statuses, such as keyboard input status, mouse operation status, audio input / output status, and video input / output status, are all marked with a cross. This situation represents a situation where, for example, a task such as document creation or literature research is stalled, the user's face tends to be turned away from the screen of the user terminal 1, and there is little keyboard input or mouse operation. In such a situation, according to the selection rule shown in FIG. 5, all of the facial expression, audio, and text modal information are selected. FIG. 16 is a diagram schematically showing the modals selected in the example shown in FIG. 15. In the user's situation shown in FIG. 12, video input / output is not fully used, so the control unit 11 selects the facial expression modal. Furthermore, audio input / output is not fully used, and the user's face tends to be turned away from the screen of the user terminal 1, so the control unit 11 selects the audio modal, which is the dominant modal. Furthermore, keyboard input is not used, so the control unit 11 selects the text modal. FIG. 17 is a diagram showing expanded modal information included in the notification delivered in the example shown in FIG. 15. In the user situation as shown in FIG. 15, facial expression modal information FE, voice modal information VO, and text modal information TX selected based on both the user characteristic information and the user situation are notified as shown in FIG. 17.

[0060] (Operations and Effects) As described above, in the user terminal 1 to which the information notification device according to the first embodiment is applied, the notification receiving unit 111 receives a notification addressed to the user, and the situation grasping unit 113 grasps the user's situation with respect to the user terminal 1. The notification expanding unit 112 and the notification selecting unit 114 then acquire user characteristic information indicating the user's dominance and inferiority with respect to multiple modals from the modal characteristic storage unit 131. Based on the acquired user characteristic information and the user's situation grasped by the situation grasping unit, the notification received by the notification receiving unit 111 is expanded to one or more pieces of modal information among the multiple modals, excluding any modal that constitutes a disturbing stimulus for the user. The notification delivering unit 115 delivers to the user a notification including one or more pieces of modal information expanded based on both the user characteristic information and the user's situation. In this way, the notification expanding unit 112 and the notification selecting unit 114 are examples of an expanding unit. The notification selecting unit 114 is an example of a selecting unit. As described above, the information notification device according to the first embodiment delivers modal information that does not include modals that may be disruptive to the user based on both user characteristic information and the user's situation, thereby providing a technology that enables enhanced notifications without interfering with the user's activities. Depending on the usage situation of the user terminal 1, the user terminal 1 to which the information notification device according to the first embodiment is applied enhances notifications so as not to include modals that may be disruptive due to specific sensory sensitivities or sensory dominance. This enables clear notifications to be provided without interfering with the activities of users with sensory sensitivities or dominance, such as document research, viewing, and creation, content viewing and production, programming, and various types of communication, using the user terminal 1, such as a PC or smartphone.

[0061] In the user terminal 1 to which the information notification device according to the first embodiment is applied, the notification expansion unit 112 generates a plurality of pieces of modal information by expanding the notification received by the notification reception unit 111 for all of the plurality of modals, the notification selection unit 114 selects one or more pieces of modal information that do not constitute a disruptive stimulus to the user from the plurality of pieces of modal information generated by the notification expansion unit 112 based on both the user characteristic information and the user's situation, and the notification distribution unit 115 distributes a notification including the one or more pieces of modal information selected by the notification selection unit 114. Thus, the information notification device according to the first embodiment can distribute a notification including one or more pieces of modal information that do not constitute a disruptive stimulus to the user by selecting the modal information to be distributed from the plurality of pieces of modal information expanded for all modals based on both the user characteristic information and the user's situation.

[0062] Furthermore, in the user terminal 1 to which the information notification device according to the first embodiment is applied, the modal selection rule storage unit 132 stores selection rules based on user characteristic information and the user's situation, and the notification selection unit 114 selects one or more pieces of modal information that do not constitute a disturbing stimulus for the user in accordance with the selection rules stored in the modal selection rule storage unit 132. Thus, the information notification device according to the first embodiment can easily select modal information by storing selection rules in advance.

[0063] [Second Embodiment] In the first embodiment, expanded modal information is generated for all modals, including text, voice, and facial expressions, in response to a received notification, and then modal information to be actually used is generated based on both user characteristic information and the user's situation. Alternatively, a modal to be used may be selected first based on both the user characteristic information and the user's situation, and then modal information may be generated only for the selected modal. The second embodiment is an example of such selective generation of modal information.

[0064] Hereinafter, the same configurations and processes as those in the first embodiment will be denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted, and only the parts that are different from the first embodiment will be described.

[0065] (Configuration Example) FIG. 18 is a block diagram showing an example of the software configuration of a user terminal 1 to which an information notification device according to the second embodiment of the present invention is applied.

[0066] The notification receiving unit 111 in the second embodiment transmits the received notification to the notification selecting unit 114 in addition to the notification expanding unit 112 .

[0067] The notification selection unit 114 in the second embodiment acquires user characteristic information from the modal characteristic storage unit 131, and selects one or more modals from among a plurality of modals, excluding any modalities that may be disruptive stimuli for the user, based on both the acquired user characteristic information and the user's situation grasped by the situation grasping unit 113. More specifically, the notification selection unit 114 selects one or more modals that do not constitute disruptive stimuli for the user, in accordance with a selection rule based on the user characteristic information and the user's situation stored in the modal selection rule storage unit 132. The notification selection unit 114 transmits the result of the selection of modals based on both the user characteristic information and the user's situation to the notification expansion unit 112.

[0068] In the second embodiment, the notification expansion unit 112 generates modal information by expanding only the modal selected by the notification selection unit 114 based on both the user characteristic information and the user's situation for the notification received by the notification reception unit 111. The notification expansion unit 112 transmits the modal information generated based on both the user characteristic information and the user's situation to the notification distribution unit 115.

[0069] 19 is a flowchart showing an example of the processing procedure and processing content of information notification processing executed by the control unit 11 of the user terminal 1 to which the information notification device according to the second embodiment is applied. In this embodiment, when it is determined in step S101 that a notification has been received, the control unit 11 operates as the situation grasping unit 113 and grasps the user's situation with respect to the user terminal 1 (step S103).

[0070] The control unit 11 also operates as the notification selection unit 114 to determine a modal to be used from among the multiple modals (step S111). That is, the control unit 11 reads user characteristic information indicating the user's superiority and inferiority with respect to the multiple modals from the modal characteristic storage unit 131, and selects one or more modals from among the multiple modals, excluding any modal that may be a disturbing stimulus for the user, based on the user characteristic information and the grasped user situation.

[0071] The control unit 11 then operates as the notification expansion unit 112 and expands the received notification only with respect to modals selected from a plurality of modals such as text, voice, and facial expression based on both the user characteristic information and the user's situation (step S112). As a result, the control unit 11 generates one or more pieces of modal information for the received notification that do not include modals that may be disruptive stimuli for the user.

[0072] In step S105, the control unit 11 operates as the notification delivery unit 115 and delivers to the user a notification including one or more pieces of modal information generated based on both the user characteristic information and the user situation. Thereafter, the control unit 11 proceeds to the process of step S101.

[0073] (Actions and Effects) As described above, in the user terminal 1 to which the information notification device of the second embodiment is applied, modal information that does not include modals that act as disruptive stimuli for the user can be delivered, similar to the first embodiment, based on both user characteristic information and the user's situation.

[0074] Furthermore, in a user terminal 1 to which the information notification device according to the second embodiment is applied, the notification selection unit 114 selects one or more modals that do not constitute a disruptive stimulus for the user based on both the user characteristic information and the user's situation, the notification expansion unit 112 generates one or more pieces of modal information by expanding the notification received by the notification reception unit 111 for the one or more modals selected by the notification selection unit 114 based on both the user characteristic information and the user's situation, and the notification distribution unit 115 distributes a notification including the one or more pieces of modal information based on both the user characteristic information and the user's situation generated by the notification expansion unit 112. Thus, in the user terminal 1 to which the information notification device according to the second embodiment is applied, a modal to be used from a plurality of expandable modals is selected based on both the user characteristic information and the user's situation, and expanded modal information is generated only for the selected modal, thereby enabling the distribution of a notification including one or more pieces of modal information that do not constitute a disruptive stimulus for the user.

[0075] [Third Embodiment] In the first and second embodiments, an example in which the information notification device is applied to one user terminal 1 has been described, but some functions may be provided in a server computer installed on the web or in the cloud. This will be described below as the third embodiment. Note that while the description will be made in correspondence with the first embodiment, it goes without saying that the description can also be made in correspondence with the second embodiment.

[0076] 20 and 21 are block diagrams showing an example of the hardware and software configurations of an information notification system to which an information notification device according to the third embodiment of the present invention is applied. The information notification system includes a plurality of user terminals 1, a distribution server 2, and a sender terminal 3, which are connected to each other so as to be able to communicate with each other via a network NW. Note that there may be a plurality of sender terminals 3, and FIGS. 20 and 21 show only one of them as a representative.

[0077] The user terminals 1 may belong to different users, or one user may be able to operate multiple user terminals 1. The hardware configuration of each user terminal 1 is the same as that of the first embodiment. Regarding the software configuration, as shown in Fig. 21 , the control unit 11 of the user terminal 1 includes a status grasping unit 113 and a notification distribution unit 115 as processing function units according to the third embodiment of the present invention. Note that, due to space limitations, the program storage unit 12 is omitted from Fig. 21 .

[0078] The situation grasping unit 113 grasps the user's situation as described in the first embodiment, and transmits the grasped user's situation to the distribution server 2 via the network NW using the communication interface unit 14. However, in this embodiment, the situation grasping unit 113 operates in response to reception, via the communication interface unit 14, of an inquiry transmitted from the distribution server 2 via the network NW. The notification distribution unit 115 receives, via the communication interface unit 14, a notification including extended modal information transmitted from the distribution server 2 via the network NW. Then, the notification distribution unit 115 outputs the extended modal information included in the received notification to the audio output device 174 and / or the video output device 175 via the input / output interface unit 15.

[0079] 20, the distribution server 2 includes a control unit 21 that uses a hardware processor such as a CPU, and a storage unit having a program storage unit 22 and a data storage unit 23, and a communication interface unit 24, which are connected to the control unit 21 via a bus 25. The control unit 21, program storage unit 22, data storage unit 23, communication interface unit 24, and bus 25 correspond to the control unit 11, program storage unit 12, data storage unit 13, communication interface unit 14, and bus 16 in the user terminal 1.

[0080] Regarding the software configuration of the distribution server 2, as shown in Fig. 21 , the control unit 11 includes a notification receiving unit 211, a notification expanding unit 212, and a notification selecting unit 214 as processing function units according to the third embodiment of the present invention. The data storage unit 23 also includes, in its storage area, a modal characteristics storage unit 231, a modal selection rule storage unit 232, and a user information storage unit 233 as storage units required to implement the third embodiment. Due to space limitations, the program storage unit 22 is omitted from Fig. 21 .

[0081] The modal characteristic storage unit 231 stores user characteristic information common to all users. Alternatively, the modal characteristic storage unit 231 may store different user characteristic information for each user. The modal selection rule storage unit 232 is similar to the modal selection rule storage unit 232 in the first embodiment. The user information storage unit 233 stores information for communicating with the user terminal 1 used by each user, such as an IP address, that is, information for identifying the user terminal 1 of each user. The user information storage unit 233 may also store information for identifying multiple user terminals 1 for one user. When the modal characteristic storage unit 231 stores user characteristic information for each user, the user information storage unit 233 can be omitted by storing information for identifying the user terminal 1 in association with each piece of user characteristic information in the modal characteristic storage unit 231.

[0082] The notification receiving unit 211 receives notifications addressed to each user, transmitted from any sender terminal 3 via the network NW, via the communication interface unit 24. The notification expanding unit 212 is similar to the notification expanding unit 112 in the first embodiment. Like the notification selecting unit 114 in the first embodiment, the notification selecting unit 214 selects one or more pieces of modal information that do not constitute a disturbing stimulus for the user, in accordance with selection rules based on user characteristic information and the user's status stored in the modal selection rule storage unit 232. In this case, the notification selecting unit 214 acquires the user's status from the corresponding user terminal 1 by transmitting a user status inquiry to the status grasping unit 113 of the user terminal 1 of the user to whom the notification received by the notification receiving unit 211 is addressed, via the communication interface unit 24, in accordance with information for identifying the user terminal 1 stored in the user information storage unit 233. Note that if a user can use multiple user terminals 1, some user terminals 1 may be inactive. User status information is not acquired from inactive user terminals 1.

[0083] (Example of operation) Figure 22 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit 21 of the distribution server 2 included in the information notification system, and Figure 23 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit 11 of the user terminal 1 included in the information notification system.

[0084] 22, the control unit 21 of the distribution server 2 first operates as the notification receiving unit 211 and determines whether a notification transmitted from any sender via the network NW has been received via the communication interface unit 24 (step S201). If it is determined that a notification has not been received, the control unit 21 performs the process of step S201 again. In this way, the control unit 21 waits for the reception of a notification.

[0085] If it is determined that a notification has been received, the control unit 21 operates as the notification expansion unit 212 and expands the received notification in all of a plurality of modalities such as text, voice, and facial expression (step S202). That is, the control unit 21 generates a plurality of modal information corresponding to the received notification.

[0086] The control unit 21 also inquires about the status of the destination user of the received notification (step S203). Specifically, the control unit 21 acquires information for identifying the user terminal 1 of the destination user from the user information storage unit 233, and inquires about the user status of the corresponding user terminal 1 via the network NW using the communication interface unit 24.

[0087] 23, the control unit 11 of the user terminal 1 first operates as the status grasping unit 113 and determines whether or not an inquiry transmitted from the distribution server 2 via the network NW has been received via the communication interface unit 14 (step S121). If it is determined that an inquiry has not been received, the control unit 11 proceeds to the processing of step S123.

[0088] If it is determined that an inquiry has been received, the control unit 11 operates as a situation assessment unit 113 to determine the input / output status of text, audio, and video at the user terminal 1, the direction of the user's face toward the user terminal 1, etc., and from the results of this assessment, grasps the user's situation relative to the user terminal 1 (step S103).

[0089] Then, the control unit 11 transmits the grasped user status to the distribution server 2 via the network NW through the communication interface unit 14 (step S122). Thereafter, the control unit 11 proceeds to the process of step S123.

[0090] 22, the control unit 21 operating as the notification selection unit 214 of the distribution server 2 acquires the user status information returned from the inquired user terminal 1 via the network NW via the communication interface unit 24 (step S204). Note that if the inquired user terminal 1 is not in operation, the user status information is not returned. Therefore, the status of the destination user acquired in step S204 includes the operating status of the user terminal 1 in addition to the user status.

[0091] Then, the control unit 21 selects modal information to be used from the generated plurality of pieces of modal information (step S205). Specifically, the control unit 21 reads out user characteristic information indicating the user's superiority and inferiority with respect to the plurality of modals from the modal characteristic storage unit 231, and selects one or more pieces of modal information from the generated plurality of pieces of modal information that do not include any modal that may be a disturbing stimulus for the user, based on both the user characteristic information and the grasped user situation.

[0092] Furthermore, based on the acquired operating status of the user terminal 1, the control unit 21 determines the user terminal 1 to which to send a notification containing one or more modal information selected based on both the user characteristic information and the user status (step S206).

[0093] Then, the control unit 21 transmits a notification including one or more pieces of modal information selected based on both the user characteristic information and the user situation to the determined destination user terminal 1 via the network NW through the communication interface unit 24 (step S207). Thereafter, the control unit 21 proceeds to the processing of step S201.

[0094] 23, if the control unit 11 of the user terminal 1 determines in step S121 that it has not received an inquiry, or after transmitting the user's status in step S122, it operates as the notification distribution unit 115 and determines whether it has received, via the communication interface unit 14, a notification including one or more pieces of modal information selected based on both the user characteristic information and the user's status, transmitted from the distribution server 2 via the network NW (step S123). If it determines that it has not received such a notification, the control unit 11 proceeds to the processing of step S121.

[0095] If it is determined that a notification has been received, the control unit 11 delivers to the user a notification including one or more pieces of modal information selected based on both the received user characteristic information and the user's situation (step S105). Specifically, the control unit 11 outputs facial expression modal information and text modal information from the video output device 175 and outputs audio modal information from the audio output device 174 via the input / output interface unit 15. Thereafter, the control unit 11 proceeds to the process of step S121.

[0096] (Actions and Effects) As described above, in an information notification system to which the information notification device of the third embodiment is applied, as in the first embodiment, modal information that does not include modals that act as disruptive stimuli for the user can be delivered based on both user characteristic information and the user's situation.

[0097] Furthermore, in the third embodiment, notifications addressed to multiple users can be processed collectively in the distribution server 2 .

[0098] Furthermore, in the third embodiment, compared to the first and second embodiments in which all functions of the information notification device are performed by the user terminal 1, the user terminal 1 only needs to execute some of the functions related to the information notification device, thereby reducing the amount of operating resources used by the user terminal 1 when performing some information processing task, and minimizing the reduction in processing power that affects that information processing task.

[0099] [Other Embodiments] The present invention is not limited to the above-described embodiments.

[0100] For example, although text, voice, and facial expressions have been described as examples of modals to be expanded, other modals may be used, or other modals may be further included.

[0101] Furthermore, the flow of each process described with reference to the flowchart is not limited to the procedure described. For example, the order of the process of step S102 and the process of step S103 shown in FIG. 6 may be reversed, or they may be performed simultaneously in parallel. In this way, the order of some steps may be changed, or some steps may be performed simultaneously in parallel. Furthermore, the process content of some steps may be modified.

[0102] The program may be transferred in a state where it is stored in an electronic device, or in a state where it is not stored in an electronic device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium that can store the program and is computer-readable, such as a CD-ROM or a memory card.

[0103] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0104] In short, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0105] DESCRIPTION OF SYMBOLS 1...User terminal 2...Distribution server 3...Sender terminal 11, 21...Control unit 12, 22...Program storage unit 13, 23...Data storage unit 14, 24...Communication interface unit 15...Input / output interface unit 16, 25...Bus 111, 211...Notification receiving unit 112, 212...Notification expansion unit 113...Situation grasping unit 114, 214...Notification selection unit 115...Notification distribution unit 131, 231...Modal characteristic storage unit 132, 232...Modal selection rule storage unit 171...Input device 172...Audio input device 173...Video input device 174...Audio output device 175...Video output device 233...User information storage unit FE...Facial expression modal information NW...Network TX...Text modal information VO...Audio modal information

Claims

1. An information notification device comprising: a receiving unit that receives a notification addressed to a user; an acquiring unit that acquires user characteristic information indicating the user's superiority and inferiority in multiple modals; a grasping unit that grasps the user's status with respect to the user's terminal; an expansion unit that expands the notification received by the receiving unit to one or more modal information of the multiple modals, excluding any modal that is a disruptive stimulus for the user, based on the user characteristic information acquired by the acquiring unit and the user's status grasped by the grasping unit; and a distribution unit that distributes a notification including the one or more modal information expanded by the expansion unit to the user.

2. The information notification device of claim 1, wherein the expansion unit comprises: a notification expansion unit that generates multiple modal information by expanding the notification received by the receiving unit for all of the multiple modals; and a selection unit that selects, from the multiple modal information generated by the notification expansion unit based on the user characteristic information and the user's situation, one or more modal information that does not constitute a disruptive stimulus for the user; and the distribution unit distributes a notification including the one or more modal information selected by the selection unit.

3. The information notification device described in claim 1, wherein the expansion unit comprises: a selection unit that selects one or more modals that do not pose a disturbing stimulus to the user based on the user characteristic information and the user's situation; and a notification expansion unit that generates one or more pieces of modal information by expanding the notification received by the receiving unit with respect to the one or more modals selected by the selection unit; and the delivery unit delivers a notification including the one or more pieces of modal information generated by the notification expansion unit.

4. An information notification device as described in claim 2 or 3, further comprising a memory unit that stores selection rules based on the user characteristic information and the user's situation, and the selection unit performs the selection in accordance with the selection rules stored in the memory unit.

Images