Information processing system, control method thereof, and program

Abstract

To provide an information processing system, a method of controlling the same, and a program therefor which realize smooth communication.SOLUTION: An information processing system according to the present invention has analysis means (CPU of a portable terminal 190a) for analyzing feelings of a first person to quantize a feeling level of the first person, identifying means (CPU of server 160) for identifying a second person whom the first person communicate with, and notifying means (CPU of the server 160 and a vibration function of the portable terminal 190b, a speaker of a sensor, a screen display of an operation part, and so on) for notifying the second person if the feeling level exceeds a set threshold value. The analysis means quantizes the feeling level based on information of at least one of biometric information, audio information and video information of the first person.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, a control method thereof, and a program. [Background technology]

[0002] As people's values ​​become more diverse, it can be difficult to communicate smoothly. Patent Document 1 shows a communication analysis device that evaluates communication between multiple participants based on voice data and image data, and provides the evaluation results to the multiple participants in real time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 174088 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 shows that the participation level of participants in communication is analyzed and participants with low participation levels are encouraged to participate and speak up. However, simply encouraging participants with low participation levels to participate and speak up may make it difficult to achieve smooth communication between participants who have different values.

[0005] An object of the present invention is to provide a technique that is advantageous for realizing smooth communication. [Means for solving the problem]

[0006] In view of the above problem, an information processing system according to an embodiment of the present invention includes an analysis unit that analyzes an emotion of a first person and quantifies an emotion level of the first person, and an identification unit that identifies a second person who is a communication target of the first person. ,before 1st person and the second person A means for identifying a scene of a communication space in which participants communicate, the scene including the scene; a notification means for not notifying the first person and notifying the second person during communication when the emotion level exceeds a set threshold while the participant is communicating in the communication space; and a threshold changing means for changing the setting of the threshold depending on the scene, wherein the notification means notifies the second person that the emotion level has exceeded the threshold, and after a predetermined time has elapsed, the analysis means analyzes the emotion of the first person again, and if the emotion level after the predetermined time has elapsed exceeds the threshold, the notification means notifies a third person who is not included in the participants. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that is advantageous for realizing smooth communication. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a processing sequence of the information processing system of FIG. 1. [Diagram 3] FIG. 2 is a diagram showing an example of a communication space in which the information processing system of FIG. 1 is used. [Figure 4] FIG. 2 is a diagram showing an example of a processing flow of the information processing system of FIG. 1. [Diagram 5] FIG. 2 is a diagram showing an example of a processing flow of the information processing system of FIG. 1. [Figure 6] FIG. 2 is a diagram showing an example of a processing flow of the information processing system of FIG. 1. [Figure 7] FIG. 2 is a diagram showing an example of a scene of the information processing system of FIG. 1. [Figure 8] FIG. 2 is a diagram showing a processing sequence of the information processing system of FIG. 1. [Figure 9] FIG. 2 is a diagram showing an example of a communication space in which the information processing system of FIG. 1 is used. [Figure 10] FIG. 2 is a diagram showing an example of a processing flow of the information processing system of FIG. 1. [Figure 11] FIG. 2 is a diagram showing an example of a processing flow of the information processing system of FIG. 1. [Figure 12] FIG. 2 is a diagram showing an example of a processing flow of the information processing system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] An information processing system according to an embodiment of the present disclosure will be described with reference to FIGS. 1 to 12. FIG. 1 is a diagram showing a configuration example of an information processing system 110 in this embodiment. As shown in FIG. 1, the information processing system 110 includes a mobile terminal 190, an information processing device 100, and a server 160. As shown in FIG. 1, a plurality of mobile terminals 190 may be arranged in one information processing system 110. Also, as shown in FIG. 1, a plurality of information processing devices 100 may be arranged in one information processing system 110. The server 160 may be, for example, a cloud server. As with the mobile terminal 190 and the information processing device 100, the server 160 may also be arranged such that a plurality of servers are distributed with respect to one information processing system 110. Here, in this specification, when indicating a specific mobile terminal among the plurality of mobile terminals 190 included in the information processing system 110, a suffix such as "a" or "b" is added to the end of the reference symbol, such as mobile terminal 190a. When it is not necessary to distinguish between the respective mobile terminals, it is simply indicated as "mobile terminal 190". The same applies to the other components.

[0011] The mobile terminal 190, the information processing device 100, and the server 160 are connected to each other so as to be able to communicate with each other via a network 170 such as a LAN (Local Area Network) or a WAN (Wide Area Network). The network 170 may be a wired network or a wireless network. The network 170 may also be a combination of a wired network and a wireless network.

[0012] Further, other devices may be connected to the information processing system 110. In this embodiment, as shown in FIG. 1, peripheral devices 120 such as a printing device 121, a projector 122, and a network camera 123 are connected to the information processing system 110. The printing device 121 may hold output data such as image information and document information of an output such as a printed matter. The printing device 121 may determine the content of the output data based on the form type and description content of the output. The determination of the content of the output data is not limited to being performed by the printing device 121. The printing device 121 may output the output data to the server 160, the information processing device 100, the mobile terminal 190, or the like, and the server 160, the information processing device 100, or the mobile terminal 190 may determine the content of the output data.

[0013] The projector 122 may hold display data such as projected image information and document information. The projector 122 may determine the contents of the display data based on the form type and written contents of the projected image, document, etc. The determination of the contents of the display data is not limited to being performed by the projector 122. The projector 122 may output the display data to the server 160, the information processing device 100, the mobile terminal 190, etc., and the server 160, the information processing device 100, or the mobile terminal 190 may determine the contents of the display data. The projector 122 is an example of a display device, and may be another device capable of displaying images and text, such as a display.

[0014] The network camera 123 can hold captured video data (video information, audio information) and behavioral data that identifies people, places, and behaviors based on the video data. Identification of behavioral data is not limited to being performed by the network camera 123. The network camera may output video data to the server 160, the information processing device 100, the mobile terminal 190, or the like, and the server 160, the information processing device 100, or the mobile terminal 190 may identify the behavioral data. The network camera 123 is an example of a photographing device, and may be another device such as a still camera, a video camera, or a television camera.

[0015] The peripheral device 120 is not limited to the printer 121, the projector 122, and the network camera 123. It is sufficient that the peripheral device 120 can be used to obtain information such as what is being discussed and behavioral data showing when, who, where, and what is being done based on information such as images and documents. For example, a conference room usage status management system that can keep track of entry and exit from a conference room using a tool that can identify an individual, such as an employee card, may be used as the peripheral device 120. Also, for example, a schedule management system that manages information such as when, who, and what is being discussed may be used as the peripheral device 120.

[0016] Mobile terminal 190 is configured to include an operation unit 195, a storage device 196, a sensor 197, and a control device 191. The control device 191 for controlling each component in mobile terminal 190 such as operation unit 195, storage device 196, and sensor 197 includes a CPU 192, a ROM 193, and a RAM 194. CPU 192 performs overall control of each component of mobile terminal 190 based on an operating system program (hereinafter, may be referred to as OS), which is a control program stored in ROM 193 or other storage media. RAM 194 is used as a working area for CPU 192.

[0017] The operation unit 195 receives input from the user and displays images. The operation unit 195 may include, for example, a keyboard or a pointing device, or may include a touch panel display. The operation unit 195 notifies the control device 191 of information input by the user. The storage device 196 can be used as a work area for the CPU 192. The storage device 196 can also store / preserve data. The sensor 197 is a component for acquiring the state of the user. The sensor 197 can include, for example, a biometric information sensor for acquiring biometric information such as the user's heart rate, respiratory rate, blood pressure, sweating, and body movement. The sensor 197 can also include, for example, a microphone for acquiring audio information. The sensor 197 can also include, for example, a camera for acquiring video information. The sensor 197 may further include a speaker for audio output. The sensor 197 outputs data for acquiring biometric information, audio information, video information, and the like to the control device 191. The sensor 197 may analyze the biometric information, audio information, and video information, or the control device 191 may analyze the biometric information, audio information, and video information based on the data obtained by the sensor 197.

[0018] The mobile terminal 190 may be, for example, in the form of a smart watch having an information processing function and a communication function. However, the present invention is not limited to this. For example, the mobile terminal 190 may have a sensor 197 for obtaining biological information, and audio information, video information, and the like may be obtained by the information processing device 100, the peripheral device 120, and the like. As described above, a plurality of mobile terminals 190 may be connected to one information processing system 110 via the network 170.

[0019] The information processing device 100 includes a control device 101, an operation unit 105, a storage device 106, an input / output unit 107, and a display unit 108. The control device 101 is a control board (controller) that comprehensively controls each component in the information processing device 100. The control device 101 includes a CPU 102, a ROM 103, and a RAM 104. The CPU 102 controls each component in the control device 101 via a system bus (not shown). For example, the CPU 192 executes the functions of the information processing device 100 by reading and executing a program stored in the ROM 193, the RAM 194, the storage device 196, or another storage medium. The ROM 103 stores, for example, various control programs such as an operating system program (OS), and tables and setting data required to execute the functions of the information processing device 100. The RAM 104 is used, for example, as a work memory for the CPU 102. The storage device 106 stores, for example, various application programs, data, user information, device information, and the like.

[0020] The operation unit 105 may include a keyboard, a pointing device, etc. for receiving input from a user, or may include a touch panel display. The operation unit 105 outputs information such as operations input by the user to the control device 101. The input / output unit 107 includes, for example, a microphone for voice input, a speaker for voice output, a camera for image input, etc., and outputs acquired information to the control device 101. The display unit 108 is, for example, a liquid crystal display, and displays various user interface screens and information.

[0021] The server 160 may have a configuration similar to that of the information processing device 100. Therefore, a detailed description will be omitted here.

[0022] FIG. 2 is a diagram showing a process flow of the information processing system 110 in this embodiment. The information processing system 110 performs the following three processes. The first process is a scene identification process P201 for identifying a scene of the communication space where participants communicate. The second process is an emotion analysis process P202 for analyzing the emotion of a person using the mobile terminal 190 in the communication space and quantifying the emotion level of the person. The third process is a notification process P203 for identifying a person who is a communication target of the person whose emotion level has been quantified, and notifying the person who is a communication target when the emotion level exceeds a set threshold. The following will explain in detail.

[0023] Fig. 3 is a diagram showing an example of a communication space in which the information processing system 110 is used. The communication space is a place where multiple participants including persons 141 and 142 communicate with each other. The communication space may be a real space such as a conference room 140 as shown in Fig. 3. The communication space may also be an online virtual space as described later.

[0024] 3, a printer 121 and a network camera 123 are installed in a conference room 140 as peripheral devices 120. Persons 141 and 142 are gathered in the conference room 140 as participants who will be communicating, for example, in a conference. Also, a person 143 exists outside the conference room 140. Here, person 143 is not a participant who will be communicating, for example, in a conference room 140, which is a communication space.

[0025] Person 141 is carrying mobile terminal 190a and has printed matter 130 output from printing device 121. Person 142 is carrying mobile terminal 190b and has printed matter 130 like person 141. Person 143 is using information processing device 100a. In the example shown in Fig. 3, a one-on-one meeting between person 141 and person 142 will be described as an example, but it may also be a conference with three or more people participating.

[0026] FIG. 4 is a flow diagram showing each step of the scene identification process P201. The scene identification process P201 shown in FIG. 4 is executed by, for example, the server 160. In the control device 101 of the server 160, the CPU 102 reads out a program stored in the ROM 103 into the RAM 104 and executes it to realize the scene identification process P201. Here, it is assumed that the scene identification process P201 is started at regular intervals, but the scene identification process P201 may also be started when communication between the person 141 and the person 142 starts. More specifically, for example, the server 160 (control device 101, CPU 102) may determine that the person 141 and the person 142 have entered the conference room 140 from video data captured by the network camera 123, and start the scene identification process P201. Also, for example, an entrance / exit management system for the conference room 140 may be provided as the peripheral device 120, and the scene identification process P201 may be started when the person 141 and the person 142 enter the conference room 140.

[0027] When the scene identification process P201 is started, in S401, the CPU 102 of the server 160 requests data required to identify a scene from the peripheral device 120 and acquires the data from the peripheral device 120. For example, the printing device 121 transmits output data including image information and document information of an output such as a printed matter 130 to the server 160. Also, for example, an image capturing device of the peripheral device 120 such as the network camera 123 transmits video data including captured video information and audio information to the server 160. In this case, the image capturing device of the peripheral device 120 such as the network camera 123 may analyze the video data and transmit to the server 160 behavior data that identifies the persons 141 and 142 and the behavior of the persons 141 and 142.

[0028] 3, a display device among peripheral devices 120 such as projector 122 or a display may be arranged in conference room 140. When a display device is arranged, display data including image information, document information, and the like displayed on the display device in conference room 140 may be transmitted to server 160 as data required to identify the scene.

[0029] Next, in S402, the CPU 102 of the server 160 identifies a scene of the conference room 140, which is a communication space, from the various data acquired in S401 from the peripheral device 120. The CPU 102 of the server 160 identifies the scene based on at least one of image information, text information, audio information, and video information in the above-mentioned communication space.

[0030] 7 shows an example of a scene identification result. The output 702 may be, for example, image information or text information of an output such as a printout 130 output by a printing device 121, or image information or text information displayed on a display device such as a projector 122 in a conference room 140. The person 703 may be information of a participant associated with the image information or text information of the output 702, or information of a participant of a communication space such as persons 141 and 142 identified from audio information or video information obtained by a photographing device such as a network camera 123. The person 703 may also be information of a participant of the communication space obtained by an entrance and exit management system of the conference room 140. The location 704 is information of the location of a communication space such as the conference room 140.

[0031] The CPU 102 of the server 160 identifies a scene based on at least one of the above-mentioned image information, text information, audio information, and video information in the communication space. As described above, the server 160 acquires any one of image information, text information, audio information, and video information in the communication space (conference room 140) from the peripheral device 120 that can communicate with the server 160. The CPU 102 of the server 160 determines the scene of the communication space including the scene identification result 700 and the importance 701 based on information such as the output object 702, the person 703, and the place 704. Thus, in this embodiment, the server 160 (CPU 102) functions as an identification means for identifying the scene of the communication space in which the participants communicate.

[0032] For example, if the output 702 is a performance report, the person 141 is a general employee who created the document, the person 142 is a manager with approver authority, and there are also people other than the persons 141 and 142 in the room, it is determined that the scene is an important report such as a performance report. In this case, the importance is determined to be high. The determination criteria may not only follow predetermined ones, but may also be customized as appropriate, such as by the user registering arbitrary criteria. As described above, in S402, the server 160 identifies a scene that includes at least one of information on the discussion in the conference room 140, which is the communication space, information on the participants, and information on the location of the communication space.

[0033] When the scene is identified, in S403, the CPU 102 of the server 160 may notify the persons 141 and 142 of scene information (scene identification result, importance) via the mobile terminals 190a and 190b.

[0034] Next, in S404, the CPU 102 of the server 160 saves the history of the scene identification result in the storage device 106, and ends the scene identification process P201. The history data saved in S404 can be used as a reference material when a user customizes the judgment criteria, for example, by visualizing the results such as what kind of communication is taking place in what kind of scene by using a graph or the like. Also, for example, the CPU 102 of the server 160 may perform machine learning using the history data, generate a trained model that machine-learns the relationship between data such as image information, text information, audio information, and video information in the above-mentioned communication space and information on the scene such as information on the discussion in the communication space, information on the participants, and information on the location of the communication space, and save the model in the storage device 106. Furthermore, the CPU 102 of the server 160 may use the trained model to identify the scene in S402. This can improve the accuracy of identifying the scene. Also, for example, the CPU 102 of the server 160 may customize the judgment criteria of the importance 701 when identifying the scene from the trained model.

[0035] Next, the emotion analysis process P202 will be described. Fig. 5 is a flow diagram showing each step of the emotion analysis process P202 in the mobile terminal 190a. The emotion analysis process P202 shown in Fig. 5 is realized, for example, in the control device 191 of the mobile terminal 190a by the CPU 192 reading a program stored in the ROM 193 into the RAM 194 and executing it. In this embodiment, the emotion analysis process P202 is executed at a predetermined interval. The emotion analysis process P202 can also be executed simultaneously in the mobile terminal 190b.

[0036] When the emotion analysis process P202 is started, in S501, the CPU 192 of the mobile terminal 190a acquires data such as biometric information, image information, and audio information of the person 141 from the sensor 197 as described above. Next, in S502, the CPU 192 of the mobile terminal 190a determines whether or not emotion analysis needs to be performed. Although the step of S502 is inserted to suppress the processing load on the CPU 192 of the mobile terminal 190a, the emotion analysis step of S503 may always be executed. For example, when the emotion analysis process P202 is not being performed, such as immediately after the startup of the mobile terminal 190a, or when a predetermined time has passed since the previous execution of the emotion analysis process P202, the CPU 192 of the mobile terminal 190a may determine that emotion analysis needs to be performed. In addition, for example, when the scene information received from the server 160 in the above-mentioned S403 has changed, or when the data acquired in S501 has changed significantly, the CPU 192 of the mobile terminal 190a may determine that emotion analysis needs to be performed. If it is determined that emotion analysis needs to be performed, the emotion analysis process P202 transitions to S503. If it is determined that emotion analysis does not need to be performed, the emotion analysis process P202 transitions to S505. If it is determined that emotion analysis does not need to be performed, the emotion analysis process P202 may end.

[0037] In S503, the CPU 192 of the mobile terminal 190a quantifies the emotion level of the person 141, such as calm (normal), positive emotion (joy, enjoyment, satisfaction), negative emotion (anger, sadness, dissatisfaction), and abnormal emotion (impatience, emptiness), based on the data acquired in S501. The above-mentioned biometric information such as the heart rate of the person 141, and audio information such as the content, strength, and intonation of the words of the person 141 himself or the person 142 who is the communication target, may be used for the emotion analysis. Furthermore, data on the facial expressions of the person 141 and the person 142 based on video information captured by the network camera 123 may be used for the emotion analysis. In this way, the CPU 192 of the mobile terminal 190a quantifies the emotion level of the person 141 based on at least one of the biometric information, audio information, and video information of the person 141. Furthermore, the emotion level of the person 141 may be quantified based on at least one of the biometric information, audio information, and video information of the person 142. Thus, in this embodiment, the mobile terminal 190a (CPU 192) functions as an analysis means for analyzing the emotion of the person 141 and quantifying the emotion level of the person 141. Similarly, the mobile terminal 190b can analyze the emotion of the person 142 and quantify the emotion level of the person 142.

[0038] When the emotion level of person 141 is quantified, in S504 CPU 192 of mobile terminal 190a transmits the emotion level to server 160. In the flow shown in Fig. 5, when emotion analysis is performed in S503, the emotion level is always transmitted to server 160, but if there is no or only a small difference from the previous processing result, the step of S504 may be omitted. Next, in S505, CPU 192 of mobile terminal 190a saves the data of each piece of information acquired in S501 and the emotion level determined in S503 as history in storage device 196, and ends emotion analysis processing P202.

[0039] Next, the notification process P203 will be described. FIG. 6 is a flow diagram showing each step of the notification process P203 in the server 160. The notification process P203 shown in FIG. 6 is executed by the server 160, for example. In the control device 101 of the server 160, the CPU 102 reads out a program stored in the ROM 103 into the RAM 104 and executes it, thereby realizing the notification process P203. In the flow shown in FIG. 6, the notification process P203 is started in response to receiving the emotion level transmitted from the mobile terminal 190a to the server 160 in S504, but is not limited to this. For example, the server 160 may start the notification process P203 at a predetermined interval. In this case, the server 160 may request the mobile terminals 190a and 190b for the data of the emotion levels of the people 141 and 142 when starting the notification process P203.

[0040] When the notification process P203 is started, in S601, the CPU 102 of the server 160 acquires the emotion level transmitted from the mobile terminal 190a in S504. In addition, in S602, the CPU 102 of the server 160 acquires the latest scene information stored in the storage device 106 of the server 160 in S404. S601 and S602 may be performed in parallel, or the latest scene information may always be acquired.

[0041] Next, in S603, the CPU 102 of the server 160 judges whether or not to perform the steps of S604 and on in the notification process P203, that is, whether or not to perform notification (notification judgment). For example, when the emotion level acquired in S601 has changed from the emotion level acquired last time and the emotion level has exceeded a set threshold, the steps of S604 and on are executed (YES in S603). For example, when the emotion level has changed to a negative emotion or an abnormal emotion, the steps of S604 and on may be executed. Also, when the emotion has increased regardless of whether it is positive, negative, or abnormal, the steps of S604 and on may be executed. In other words, the server 160 (CPU 102) functions as at least a part of a notification means for issuing a notification when the emotion level has exceeded a set threshold. When there is no change in the emotion level, the notification process P203 may be terminated (NO in S603).

[0042] Here, in order to absorb differences in the states of the scenes in the communication space, the threshold may be changed based on information such as the importance 701 of the scene and the person 703 acquired in S602. For example, in the case of a scene with high importance, a setting may be made so that an early notification is made in order to smoothly advance communication in a meeting or the like. Also, for example, in a case where the positions of participants are significantly different, a setting may be made so that an early notification is made so that a notification is made before a participant in a lower position changes significantly to a negative emotion. The threshold for each scene may be set in advance by a user, etc. In this way, the server 160 (CPU 102) can function as a threshold change means for changing the setting of the threshold for making a notification depending on the scene.

[0043] When the notification process P203 transitions from S603 to S604, the CPU 102 of the server 160 determines the target of notification based on the emotion level acquired in S601 and the scene information acquired in S602. For example, consider a case where the emotion level of the person 141 exceeds a set threshold. In this case, the CPU 102 of the server 160 first identifies the person 142 who is the communication target of the person 141. The CPU 102 of the server 160 may identify the person 142 who is in the same room as the person 141 in the conference room 140, which is the communication space, as the communication target based on the scene information acquired in S602. When the person 142 who is the communication target is identified, the CPU 102 of the server 160 determines the mobile terminal 190b carried by the person 142 as the notification target. In this way, in this embodiment, the server 160 (CPU 102) functions as an identification means for identifying the person 142 who is the communication target of the person 141.

[0044] In S605, the CPU 102 of the server 160 notifies the mobile terminal 190b, which is the notification target determined in S604, of the execution of the notification. In accordance with the notification of the execution of the notification received from the server 160, the mobile terminal 190b notifies the person 142 that the emotion level of the person 141 has exceeded a set threshold. For example, the mobile terminal 190b performs the notification using an appropriate method such as a vibration function, a speaker for audio output that the sensor 197 or the like may have, or a screen display of the operation unit 195 or the like. The method of performing the notification in the mobile terminal 190b may be selected by the CPU 102 of the server 160 according to the configuration of the mobile terminal 190b. Also, a user such as the person 142 may be able to set an appropriate notification method via the operation unit 195 or the like. Thus, in this embodiment, the server 160 (CPU 102) and the mobile terminal 190b (for example, a vibration function, a speaker of the sensor 197, a screen display of the operation unit 195, etc.) cooperate to function as a notification means for notifying the person 142 when the emotion level of the person 141 exceeds a set threshold.

[0045] For example, consider a case where person 141, who is having a meeting in conference room 140, which is a communication space, is a subordinate of person 142. If person 141 is intimidated by person 142, who is his superior, and is at an emotional level of impatience, communication may not proceed smoothly, such as person 141 being unable to give a good presentation. In this case, even if person 141 is notified that his emotional level has exceeded the threshold, he may become even more impatient, which may further impede communication. On the other hand, in this embodiment, person 142 is notified that the emotional level of person 141, who is his subordinate, has exceeded the threshold. This allows person 142 to take measures such as calming down person 141, who is his subordinate, and increases the possibility of realizing smooth communication.

[0046] Also, for example, consider a case where person 141, who is having a meeting in conference room 140, which is a communication space, is at an emotional level of dissatisfaction with the content of a presentation by person 142. In this case, even if person 141 is notified that the emotional level of person 141 has exceeded a threshold, it is unlikely that smooth communication can be realized. On the other hand, in this embodiment, by notifying person 142 that the emotional level of person 141, which is dissatisfaction, has exceeded a threshold, person 142 can provide a more detailed explanation or supplement the content, and the possibility of smooth communication being realized increases.

[0047] In this manner, in this embodiment, the fact that the emotion level of person 141 has exceeded the threshold is notified to person 142, who is the communication target of the person, but to person 141 who is the party whose emotion level has been quantified. This increases the possibility of realizing smoother communication between multiple participants with different values ​​who participate in a communication space such as conference room 140.

[0048] When the notification is performed in S605, the notification process P203 transitions to S606, and the effect of the notification in S605 is determined. More specifically, after the person 142 is notified that the emotion level of the person 141 exceeds the threshold and a predetermined time has elapsed, the CPU 192 of the mobile terminal 190a analyzes the emotion of the person 141 again. For example, the CPU 102 of the server 160 may transmit a signal to the CPU 192 of the mobile terminal 190a to execute the above-mentioned emotion analysis process P202 again. Also, for example, the CPU 192 of the mobile terminal 190a may start the emotion analysis process P202 in response to the passage of a predetermined time since the previous execution of the emotion analysis process P202.

[0049] The CPU 102 of the server 160 judges whether the emotion level of the person 141 has improved from the emotion level in S603. More specifically, similar to the judgment in S603, when the emotion level of the person 141 after a predetermined time has passed exceeds a threshold, steps similar to the steps from S604 onwards are performed again, and a notification is made similar to S605. In this case, the notification may be made again to the person 142. Also, for example, a notification may be made to the person 143 who is not included in the participants of the communication space. In other words, when the emotion level of the person 141 has not improved, the CPU 102 of the server 160 may judge that a third party's intervention is necessary, and may notify the information processing device 100a of the person 143 who is registered in advance as a notification destination. For example, the persons 141 and 142 may be subordinates of the person 143.

[0050] The step of S606 does not necessarily have to be executed, and the notification process P203 may end after the step of S605 is executed. However, by executing the step of S606, the step of S606 may function as a feedback step for smoothly advancing communication in the communication space. The step of S606 may be executed every predetermined time while the conference is being held in the conference room 140. For example, the step of S606 may be executed repeatedly until the person 141 and the person 142 leave the conference room 140.

[0051] After the steps of S605 and S606 are completed, in S607, the CPU 102 of the server 160 may store the history data in the storage device 106. The stored history data may be used as a reference material when a user sets a threshold value for notification for each scene, for example, by visualizing, for example, in what scene communication was performed by what participants and how the emotion level of the participants changed due to notification, using a graph or the like. In addition, for example, the CPU 102 of the server 160 may perform machine learning using the history data, generate a trained model that machine-learns the relationship between the threshold value for notification and the change in emotion level due to notification for each scene, and store the model in the storage device 106. The CPU 102 of the server 160 may use this trained model to automatically change the setting of the threshold value for notification of the emotion level for each scene. This may allow the person 142 to be notified at a more appropriate time, and allow smoother communication.

[0052] In the present embodiment, the emotion of the person 141 is analyzed, and when the emotion level exceeds a set threshold, the person 142 is notified. At the same time, the emotion of the person 142 may be analyzed (mobile terminal 190b), and when the emotion level of the person 142 exceeds a set threshold, the person 141 may be notified. That is, in each mobile terminal 190 included in the information processing system 110, the emotion analysis process P202 of the person wearing each mobile terminal 190 is performed. When the emotion level of the person wearing each mobile terminal 190 exceeds the threshold, the communication target person other than the participant whose emotion level exceeds the threshold may be notified that the emotion level has exceeded the threshold.

[0053] The case where the communication space is a real space such as the conference room 140 has been described above with reference to Figs. 2 to 7. However, the information processing system 110 of this embodiment is not limited to being applied to a case where the communication space is a real space. The information processing system 110 can also be applied to a communication space that is an online virtual space such as an online conference. Below, an online conference will be taken as an example and a description will be given with reference to Figs. 8 to 12, focusing on the differences from the case where the above-mentioned communication space is the conference room 140.

[0054] FIG. 8 is a diagram showing a process flow of the information processing system 110 in this embodiment. The information processing system 110 performs the following three processes. The first process is a scene identification process P801 for identifying a scene of the communication space in which participants communicate. The second process is an emotion analysis process P802 for analyzing the emotion of a person using the information processing device 100a in the communication space and quantifying the emotion level of the person. The third process is a notification process P803 for identifying a person who is a communication target of the person whose emotion level has been quantified, and notifying the person who is a communication target when the emotion level exceeds a set threshold.

[0055] Fig. 9 is a diagram showing an example of a communication space in which the information processing system 110 is used. Unlike the configuration shown in Fig. 3, an information processing device 100a used by a person 141 and an information processing device 100b used by a person 142 are connected by an online conference system 145 to form a communication space. In addition, an information processing device 100c of a person 143 who is not a participant of the communication space is not connected to the online conference system 145. In the example shown in Fig. 8, a one-to-one online conference between the person 141 and the person 142 is taken as an example for explanation, but an online conference in which three or more people participate may also be used.

[0056] Fig. 10 is a flow diagram showing each step of the scene identification process P801. The scene identification process P201 shown in Fig. 10 is executed by, for example, the server 160. In the control device 101 of the server 160, the CPU 102 reads out a program stored in the ROM 103 into the RAM 104 and executes the program, thereby realizing the scene identification process P801. The steps of S402 and S404 in the scene identification process P801 shown in Fig. 10 may be the same as the steps shown in Fig. 4 described above. Therefore, the explanation of the steps of S402 and S404 will be omitted here, and only the step of S411 will be explained.

[0057] In this embodiment, the scene identification process P801 is started at regular intervals, but the scene identification process P801 may be started when communication between the person 141 and the person 142 is started. For example, the person 141 and the person 142 may connect the information processing device 100b and the information processing device 100b to the online conference system 145, thereby starting the scene identification process P801.

[0058] When the scene identification process P801 is started, in S411, the CPU 102 of the server 160 requests data required to identify a scene from the information processing devices 100a and 100b via the online conference system 145, and acquires the data from the information processing devices 100a and 100b. For example, video data such as audio information acquired by a microphone for audio input of the input / output unit 107 of the information processing devices 100a and 100b and video information acquired by a camera for image input are transmitted to the server 160. In addition, for example, data such as text information such as chat comments input via the operation unit 105 of the information processing devices 100a and 100b are transmitted to the server 160. In addition, for example, data such as image information and text information displayed on the display unit 108 of the information processing devices 100a and 100b are transmitted to the server 160. Based on these data, in the next S402, a scene including at least one of information on the discussion in the online communication space via the online conference system 145, information on the participants, and information on the location of the communication space is identified.

[0059] 10, the step of S403 is omitted, but the step of S403 may be executed. In this case, the CPU 102 of the server 160 may notify the persons 141 and 142 of the scene information via the information processing devices 100a and 100b. For example, the scene information may be displayed on the display units 108 of the information processing devices 100a and 100b.

[0060] FIG. 11 is a flow diagram showing each step of the emotion analysis process P802 in the server 160. The emotion analysis process P802 shown in FIG. 11 is realized, for example, by the CPU 102 in the control device 101 of the server 160 reading a program stored in the ROM 103 into the RAM 104 and executing it. In this embodiment, the server 160 (CPU 102) functions as an analysis means for analyzing the emotion of the person 141 and quantifying the emotion level of the person 141. Here, the emotion analysis process P802 is configured to be performed comprehensively on the server 160 connected to the online conference system 145, but each process may be performed by the information processing devices 100a and 100b as in the case of performing the process on the mobile terminals 190a and 190b shown in FIG. 5. Also, the process of quantifying the emotion level of the person 141 will be described as above, but the quantification of the emotion level of the person 142 may be performed in parallel.

[0061] When the emotion analysis process P802 is started, in S511, the CPU 102 of the server 160 acquires data such as audio information and video information of the people 141 and 142 via the online conference system 145. For example, audio information acquired by a microphone for audio input of the input / output unit 107 of the information processing device 100a or 100b and video data such as video information acquired by a camera for image input are transmitted to the server 160.

[0062] Here, when the person 141 is communicating online, such as in an online conference, the camera of the input / output unit 107 of the information processing device 100a, which functions as an imaging unit for imaging the person 141, operates even if an image of the person 141 is not displayed online. In other words, even if the camera image display on the online conference system 145 is turned off, the camera of the input / output unit 107 of the information processing device 100a may operate to acquire an image for the emotion analysis process P802.

[0063] Next, in S512, the CPU 102 of the server 160 determines whether or not sentiment analysis needs to be performed. Since the process may be the same as the above-mentioned step of S502 except that the determination is performed in the server 160, detailed description will be omitted.

[0064] When it is determined in S512 that emotion analysis is necessary, in S513, the CPU 102 of the server 160 quantifies the emotion level of the person 141 based on the data acquired in S511. When the emotion level of the person 141 is quantified in S512, in S514, the CPU 102 of the server 160 saves the data of each piece of information acquired in S511 and the emotion level determined in S513 as history in the storage device 106, and ends the emotion analysis process P802.

[0065] Fig. 12 is a flow diagram showing each step of the notification process P803 in the server 160. The notification process P803 shown in Fig. 12 is executed by the server 160, for example. In the control device 101 of the server 160, the CPU 102 reads out a program stored in the ROM 103 into the RAM 104 and executes the program, thereby realizing the notification process P803. Here, it is assumed that the server 160 starts the notification process P803 at a predetermined interval. However, this is not limited to the above, and the notification process P803 may be started in response to the end of the emotion analysis process P802.

[0066] When notification process P803 is started, in S611, CPU 102 of server 160 acquires the emotion data stored in storage device 106 of server 160 in S514. The steps of S602 and S603 may be similar to the steps described above with reference to FIG. 6, and therefore description thereof will be omitted here.

[0067] When the notification process P803 transitions to S614, the CPU 102 of the server 160 determines the target of notification based on the emotion level acquired in S611 and the scene information acquired in S602. In the configuration shown in Fig. 9, the CPU 102 of the server 160 determines the information processing device 100b connected to the online conference system 145, which is the communication space, as the target of notification based on the scene information acquired in S602. It can also be said that the CPU 102 of the server 160 identifies the person 142 using the information processing device 100b who is the communication target of the person 141 using the information processing device 100.

[0068] Next, in S615, the CPU 102 of the server 160 notifies the information processing device 100b, which is the notification target determined in S614, of the execution of notification. In accordance with the notification of notification received from the server 160, the information processing device 100b notifies the person 142 that the emotion level of the person 141 has exceeded a set threshold. For example, the information processing device 100b performs notification using an appropriate method such as a speaker or a display unit 108 for audio output that the input / output unit 107 may have. The method of performing notification in the information processing device 100b may be selected by the CPU 102 of the server 160 according to the configuration of the information processing device 100b. In addition, a user such as the person 142 may be able to set an appropriate notification method via the operation unit 105 or the like. Thus, in this embodiment, the server 160 (CPU 102) and the information processing device 100b (for example, the speaker of the input / output unit 107 and the display unit 108) cooperate to function as a notification means for notifying the person 142 when the emotion level of the person 141 exceeds a set threshold.

[0069] When the notification is performed in S615, the notification process P803 transitions to S616, and the effect of the notification in S615 is determined. More specifically, the person 142 is notified that the emotion level of the person 141 has exceeded the threshold, and after a predetermined time has elapsed, the CPU 102 of the server 160 analyzes the emotion of the person 141 again. The CPU 102 of the server 160 determines whether the emotion level of the person 141 has improved from the emotion level in S603. More specifically, similar to the determination in S603, if the emotion level of the person 141 exceeds the threshold after the predetermined time has elapsed, the same steps as those in S614 and after are performed again, and notification is performed similar to the step in S615. In this case, the notification may be performed again to the person 142. Also, for example, a notification may be performed to a person 143 who is not included in the participants of the communication space and whose information processing device 100c is not connected to the online conference system 145. That is, if the emotion level of the person 141 does not improve, the CPU 102 of the server 160 may determine that intervention by a third party is necessary and notify the information processing device 100c of the person 143 who is registered in advance as a notification destination.

[0070] The step of S616 may not necessarily be executed, and the notification process P803 may end after the step of S615 is executed. However, by executing the step of S616, the step of S616 may function as a feedback step for smoothly advancing communication in the communication space. The step of S616 may be executed, for example, every time a predetermined time elapses while the information processing devices 100a and 100b are connected to the online conference system 145. For example, the step of S616 may be executed repeatedly until the information processing devices 100a and 100b are disconnected from the online conference system 145. After the steps of S605 and S606 are finished, a step similar to the step of S607 described above may be executed.

[0071] In this way, by applying this embodiment to online communication spaces such as online conferences, it is highly possible that smoother communication can be realized between participants with different values. In other words, the information processing system 110 of this embodiment enables smoother communication in communication spaces where multiple participants communicate, regardless of whether the communication space is a real space or a virtual space.

[0072] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0073] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0074] 110: Information processing systems, 141, 142, 143: People

Claims

1. analysis means for analyzing an emotion of a first person and quantifying an emotion level of the first person; an identification means for identifying a second person with whom the first person is to communicate; An identification means for identifying a scene of a communication space in which participants including the first person and the second person communicate with each other; a notification means for not notifying the first person and notifying the second person during communication when the emotion level exceeds a set threshold while the participant is communicating in the communication space; and a threshold changing means for changing a setting of the threshold according to the scene, the notifying means notifies the second person that the emotion level has exceeded the threshold, and after a predetermined time has elapsed, the analyzing means analyzes the emotion of the first person again; an information processing system, characterized in that, when the emotion level after the specified time has elapsed exceeds the threshold, the notification means notifies a third person who is not included in the participants.

2. 2. The information processing system according to claim 1, wherein the analysis means quantifies the emotion level based on at least one of biometric information, audio information, and video information of the first person.

3. 3. The information processing system according to claim 1, wherein the scene includes at least one of information on the discussion in the communication space, information on the participants, and information on the location of the communication space.

4. 4. The information processing system according to claim 1, wherein the identifying means identifies the scene based on at least one of image information, text information, audio information, and video information in the communication space.

5. 5. The information processing system according to claim 4, wherein the identifying means acquires any one of image information, text information, audio information and video information in the communication space from a peripheral device capable of communicating with the identifying means.

6. 6. The information processing system according to claim 5, wherein the peripheral device includes at least one of a printing device, a display device, and an image capturing device.

7. 7. The information processing system according to claim 1, wherein the identification means identifies the second person based on the scene.

8. The information processing system of any one of claims 1 to 7, characterized in that the identification means identifies the scene using a trained model that has been machine-learned to determine the relationship between at least one of image information, text information, audio information, and video information in the communication space and at least one of information on discussions in the communication space, information on the participants, and information on the location of the communication space.

9. The information processing system according to any one of claims 1 to 8, characterized in that the threshold change means changes the setting of the threshold for each scene using a trained model that has machine-learned the relationship between the threshold and the change in the emotion level due to the notification.

10. 10. The information processing system according to claim 1, wherein, when the emotion level after the predetermined time has elapsed exceeds the threshold, the notification means issues a notification to the second person.

11. the first person is communicating online; the analyzing means includes an imaging means for imaging the first person, 11. The information processing system according to claim 1, wherein the image capturing means operates even when an image of the first person is not displayed online.

12. A method for controlling an information processing system, comprising: analyzing an emotion of a first person and quantifying an emotion level of the first person; identifying a second person with whom the first person is to communicate; identifying a scene of a communication space in which participants including the first person and the second person communicate; when the emotion level exceeds a set threshold while the participant is communicating in the communication space, not notifying the first person and notifying the second person while the participant is communicating in the communication space; changing the threshold setting depending on the scene; Including, A control method comprising the steps of: analyzing the emotion of the first person again after a predetermined time has elapsed since the step of making the notification; and, if the emotion level after the predetermined time has elapsed exceeds the threshold, making a notification to a third person who is not included in the participants.

13. A program for causing a computer to execute each step of the control method according to claim 12.

Images