Information processing device, information processing system, communication support system, information processing method and program

Abstract

To improve quality of communication in telecommunication through a network.SOLUTION: An information processing apparatus that processes information associated with communication of a plurality of participants comprises: information acquisition means that acquires participant information associated with the participant acquired from the participants in the communication; attention amount analysis means that acquires attention amount information associated with an attention amount of the participant for information used in the communication; presentation information determination means that determines presentation information presented to the plurality of participants on the basis of the attention amount information; and transmission means that transmits the presentation information to presentation means which is provided for each of the plurality of participants, the transmission means adjusting timing in which the presentation information is transmitted to other participant included in the plurality of participants.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, a communication support system, an information processing method, and a program. [Background technology]

[0002] Telecommunication systems have traditionally been used for online conversations, meetings, presentations, interviews, etc. Recently, the use of telecommunication systems has been increasing, and there is a demand for further improvements in the quality of communication.

[0003] Patent Document 1 discloses a communication analysis device that analyzes the participation levels of participants and provides feedback in real time to participants with low participation levels using beeps and indicators. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the past, it was difficult for individual participants to convey what information the other party was paying attention to and to what extent during telecommunication, which resulted in a problem of a decline in the quality of communication.

[0005] The present invention has been made in consideration of the above, and aims to provide an information processing device, an information processing system, a communication support system, an information processing method, and a program that can improve the quality of communication in telecommunications via a network. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides an information processing device that processes information related to communication between multiple participants, comprising: an information acquisition means that acquires participant information related to the participants acquired from the participants in the communication; an attention amount analysis means that acquires attention amount information related to the amount of attention the participants pay to information used in the communication; a presentation information determination means that determines presentation information to be presented to the multiple participants based on the attention amount information; and a transmission means that transmits the presentation information to a presentation means provided for each of the multiple participants, wherein the transmission means adjusts the timing of transmitting the presentation information to other participants included in the multiple participants. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the quality of communication in telecommunications via a network. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a functional block configuration of an information processing device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of connections between information processing devices. [Figure 3] FIG. 3 is an explanatory diagram of the arbitrary timing selection mechanism. [Figure 4] FIG. 4 is a diagram showing an embodiment of a timing determination means of the arbitrary timing selection mechanism. [Figure 5] FIG. 5 is an explanatory diagram of timing determination by the automatic timing selection mechanism. [Figure 6] FIG. 6 is a diagram showing an example of a method for determining timing using an automatic timing selection mechanism. [Figure 7] FIG. 7 is an explanatory diagram of an embodiment of the attention amount analysis means. [Figure 8] FIG. 8 is an explanatory diagram of the analysis of the amount of attention based on the gaze information of the participants. [Figure 9]FIG. 9 is a diagram illustrating an example of the configuration of an information processing system. [Figure 10] FIG. 10 is a diagram showing an example of a flow performed by the timing selection mechanism. [Figure 11] FIG. 11 is a diagram illustrating an example of another embodiment of the information processing system. [Figure 12] FIG. 12 is a diagram illustrating an example of a configuration of hardware blocks of a computer. [Figure 13] FIG. 13 is a diagram showing a first example of a usage scene. [Figure 14] FIG. 14 is a diagram showing a second example of a usage scene. [Figure 15] FIG. 15 is a diagram showing a third example of a usage scene. [Figure 16] FIG. 16 is a diagram showing a fourth example of a usage scene. [Figure 17] FIG. 17 is a diagram showing a first example of the output information m1. [Figure 18] FIG. 18 is a diagram showing a second example of the output information m1. [Figure 19] FIG. 19 is a diagram showing a third example of the output information m1. [Figure 20] FIG. 20 is a diagram showing a fourth example of the output information m1. [Figure 21] FIG. 21 is a diagram showing a fifth example of the output information m1. [Figure 22] FIG. 22 is a diagram showing a sixth example of the output information m1. [Figure 23] FIG. 23 is a diagram showing an embodiment in which a timing selection mechanism is selected by detecting a body movement (gesture). [Figure 24] FIG. 24 is a diagram showing an example of an embodiment of the speech time threshold of the arbitrary timing selection mechanism. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an information processing device, an information processing system, a communication support system, an information processing method, and a program will be described in detail with reference to the accompanying drawings. The information processing device shown in the present embodiment can be applied to various systems for communicating with multiple people, such as conferences, seminars, presentations, interviews, meetings, sales, or classes. In particular, it can be applied to a communication support system that allows communication-related information to be transmitted and received between people.

[0010] (Embodiment) <Functional blocks of information processing device> 1 is a diagram illustrating an example of a functional block configuration of an information processing device according to an embodiment. As shown in FIG. 1, the information processing device 1 according to the embodiment includes conversation participant information 11, attention amount analysis means 12, output control means 13, and transmission means 14.

[0011] Conversation participant information 11 is participant information related to a participant acquired from a participant in a communication between multiple participants. The participant information is, for example, a camera image or biometric information. The conversation participant information 11 is acquired by information acquisition means.

[0012] The attention amount analysis means 12 performs attention amount analysis to acquire attention amount information relating to the amount of attention that participants pay to information used in communication from the conversation participant information 11. The information used in communication is speech, displayed materials, gestures, or the like.

[0013] The output control means 13 determines output information m1 based on the amount of attention. The output information m1 corresponds to "presentation information," and the output control means 13 corresponds to "presentation information determination means."

[0014] The transmitting means 14 has a timing selection mechanism 15 that selects the timing of transmission, and transmits the output information m1 to the presentation means at a predetermined timing. The presentation means is a display, a speaker, or the like.

[0015] <<Specific explanation of the attention amount analysis means 12>> The attention amount analysis means 12 estimates or quantifies the amount of attention that the participant is paying to regarding information around the participant. The information around the participant includes information that can generally be obtained by human sensory organs, such as visual information, auditory information, tactile information, olfactory information, or taste information.

[0016] <<Specific Description of Output Control Means 13>> The output control means 13 outputs information based on the amount of attention obtained by the amount of attention analysis means 12 .

[0017] <<Specific Description of Transmission Means 14>> The transmitting means 14 transmits the output information obtained by the output control means 13. The transmitting means 14 is, for example, a light source such as a light bulb, LED, laser, fluorescent lamp, or neon tube, a display element such as a cathode ray tube, plasma display, liquid crystal display, or organic EL, a device for changing temperature such as a Peltier element, heating wire, blower, or air conditioner, a vibration generating device such as a piezoelectric element or motor, or a pressure generating device such as a compressor.

[0018] <<Specific explanation of timing selection mechanism 15>> The timing selection mechanism 15 is a mechanism that presents, adjusts, regulates, or determines the timing of transmitting the output information m1, such as the time instant, interval, period, or frequency. Conventional communication analysis or support devices have a problem in that feedback for improving communication is not transmitted to participants at the appropriate time. For example, if feedback is transmitted when a participant is concentrating on communication, the participant's attention to the communication decreases, resulting in a decline in the quality of communication. The timing selection mechanism 15 relates to a mechanism that adjusts, regulates, or determines the timing of transmitting the output information m1 to the appropriate time.

[0019] <Terminology> <<Conversation>> A conversation refers to the transfer of information, including at least verbal information, from one person to another between at least two or more people. Examples include situations in which participants in a conversation talk with each other, or situations such as a lecture in which one participant gives an explanation to many other participants. In such situations, a medium for transmitting information between participants, such as a microphone, speaker, camera, or video display, may be used. Furthermore, the medium does not need to be present between all participants; it may be present only between some of the participants.

[0020] <<Participant information>> Participant information is a general term for information related to the participants in a conversation. For example, it refers to social attributes such as the organization they belong to, their position within the organization, and their role; physical attributes such as gender and height; biometric information such as heart rate, breathing, gaze, eye movement, skin potential, muscle movement, skin movement, sweat, brain waves, blood oxygen concentration, stress level, body temperature, and blinking; visual information such as body movements, gestures, posture, and habits; audio information such as emotional outbursts, nods, and speech; and subjective information such as the participants' emotions, preferences, tendencies, characteristics, and personality.

[0021] <<Output information m1>> The output information m1 refers to the amount of attention and information generated by the output control means 13. The output information m1 includes information such as the type of attention the participant is paying, the amount, frequency, interval, and duration of that attention, etc. The output information m1 includes information for spatially and temporally changing media that humans can perceive using their senses of sight, hearing, touch, smell, and taste, such as light, color, sound, temperature, odor, taste, vibration, and pressure. Examples of output information m1 include electrical signals for generating flashing lights or changes in the color of light, electrical signals or discrete numerical arrays for generating image information or video information, electrical signals for changing the pitch or volume of sound, electrical signals for cooling and heating the skin, the air around the person, and objects in contact with the person through heat exchange, changes in odor due to the diffusion of molecules into the air, vibrations of the skin caused by vibrators, and electrical signals for applying pressure to the skin using hydraulic, pneumatic, or water pressure.

[0022] <Connection example of information processing device 1> 2 shows a connection example of an information processing device 1. The information processing device 1 is connected to a network N and communicates with each information processing device 1 of a plurality of participants A1, A2, A3, ... via the network N. The number of information processing devices 1 of participants A1, A2, A3, ... is two or more, including the information processing device itself.

[0023] The network N refers to a facility that enables transmission and reception of information between the information processing devices 1 of the participants A1, A2, A3, etc. The network N for enabling conversations between the participants A1, A2, A3, etc. and the network N for transmitting and receiving output information may or may not be the same.

[0024] The functional block configuration of each information processing device 1 of participant A1, participant A2, participant A3, etc. is the same as the functional block configuration shown in Figure 1, and includes participant information 11, attention amount analysis means 12, output control device 13, transmission means 14, and timing selection mechanism 15.

[0025] In this configuration, the attention amount analysis means 12 inputs participant information 11 and outputs the amount of attention. When analyzing the amount of attention corresponding to a certain participant A1, the attention amount analysis means 12 may input not only the information of that participant A1, but also participant information 11 of participants A2, A3, ... who are participating in the conversation. The output control means 13 outputs a value based on the amount of attention obtained by the attention amount analysis means 12. The transmission means 14 transmits the output information m1 obtained by the output control means 13 to the participants via an intermediary.

[0026] The mediator is one that acts on the participant's senses: sight, hearing, taste, smell, or touch. For example, for sight, the mediator can be lighting, LEDs, laser light, a display, or the object itself. For hearing, the mediator can be speakers, earphones, headphones, or bone conduction earphones. For taste or smell, the mediator can be electrodes that act on the human tongue, or steam or air containing certain molecules. For touch, the mediator can be, for example, a rotor such as a motor, a piezoelectric element, hydraulic pressure, air pressure, water pressure, air pressure, a spring, a heater such as an electric heating wire or hot water, a Peltier element, or a cooler such as water. These mediators may also be combined.

[0027] The candidates for destinations from the information processing device 1 are all people participating in the conversation, and the actual destination may be, for example, all other participants except the participant himself, a specific person among the other participants, or all participants including the participant himself. Also, the destination may be switched.

[0028] The timing selection mechanism 15 includes at least one of an arbitrary timing selection mechanism 15-1 that determines the timing at the participant's discretion and an automatic timing selection mechanism 15-2 that determines the timing automatically. Note that the timing selection mechanism 15 may change the timing for each transmission target.

[0029] <<Specific example of the configuration of the timing selection mechanism 15>> 3 is an explanatory diagram of the arbitrary timing selection mechanism 15-1. As shown in FIG. 3, the arbitrary timing selection mechanism 15-1 has a timing determination means 151 and a switch 152.

[0030] The switch 152 switches whether or not to transmit the output information output from the output control means 13 to the participant A1. The switching by the switch 152 is performed based on the output from the timing determination means 151. The arbitrary timing selection mechanism 15-1 is implemented by a combination of software and hardware.

[0031] <<Example of Arbitrary Timing Selection Mechanism 15-1>> 4 is a diagram showing an example of the timing determination means 151 of the arbitrary timing selection mechanism 15-1. As shown in FIG. 4, the computer of each information processing device 1 of each participant A1, participant A2, participant A3, ... is equipped with interfaces such as a trackpad 1001, a keyboard 1002, a mouse 1003, a button 1004, and a software button 1005. The display 1006 is provided with a touch panel. The timing determination means 151 determines the timing by combining a signal generated when participant A1 operates and inputs one of the hardware interfaces with software that detects the signal and runs on the computer.

[0032] The timing determination means 151 determines the timing by combining signals obtained from video and audio coming from participants or objects other than participants using input means such as the camera 1007 and microphone 1008 with software that detects the signals and runs on a computer. For example, software that determines the timing is used according to gestures, specific movements, patterns, or audio uttered by participant A1.

[0033] It should be noted that the various interfaces, camera 1007, microphone 1008, etc. are not necessarily pre-installed in the computer, and may be connected to the computer as external devices as needed.

[0034] The switch 152 may be either a switch that operates as software or a switch that operates as hardware that physically blocks electrical signals or optical information, or a combination of both.

[0035] In this way, by providing the arbitrary timing selection mechanism 15-1, participant A2 can switch at any timing whether or not to transmit the output information m1 output from the output control means 13 to participant A1. By leaving the timing of switching up to each participant, a participant can transmit the output information m1 when they want to inform other participants of what they are paying attention to. For example, they can transmit the output information m1 when they want to have a conversation about the object of their attention or when they want to convey what behavior of other participants they are paying attention to.

[0036] This allows the participant to receive the output information m1 to accurately determine which movements or parts of the participant the other participants are looking at when observing the communication. This makes it easier for the participant to synchronize with the other participant's movements by emphasizing or increasing the frequency of movements depending on the other participant during communication. As a result, information exchange with the other participant becomes smoother, improving the quality of communication.

[0037] Furthermore, the optional timing selection mechanism 15-1 allows the user to clearly communicate their own intentions as information, making it suitable for building trust or for educational purposes.

[0038] <<Automatic Timing Selection Mechanism 15-2>> The effect of the arbitrary timing selection mechanism 15-1 can also be obtained in the automatic timing selection mechanism 15-2. FIG. 5 is an explanatory diagram of timing determination by the automatic timing selection mechanism 15-2. As shown in FIG. 5, the transmission means 14 includes the automatic timing selection mechanism 15-2. The automatic timing selection mechanism 15-2 similarly has timing determination means 151 and a switch 152. In the automatic timing selection mechanism 15-2, the timing determination means 151 determines timing based on one or more pieces of information, either the participant information 11b or the attention amount 11a, of the participant information 11.

[0039] 6 is a diagram showing an example of a method for determining timing by the automatic timing selection mechanism 15-2. In the following, the automatic timing selection mechanism 15-2 determines timing according to the amount of attention of the participant and outputs that timing to the switch 152, which turns on the switch 152 for the period specified by that timing. Outside that period, the switch 152 turns off.

[0040] Fig. 6 shows a graph of changes in the amount of attention of participants, with the horizontal axis representing time, and the relationship between this and the operation of switch 152. Fig. 6(a) to Fig. 6(d) show four patterns of methods for determining the timing as examples.

[0041] In the first method shown in FIG. 6(a), switch 152 is turned on during a time period in which the participant's attention exceeds a certain threshold, and output information is transmitted to the participant. In the second method shown in FIG. 6(b), switch 152 is turned on during a time period in which the participant's attention falls within a certain condition range. In the third method shown in FIG. 6(c), when the participant's attention falls within a certain condition range and its duration exceeds a certain threshold, switch 152 is turned on after the duration threshold is exceeded. FIG. 6(d) further includes a graph showing the cumulative number of occurrences, with time on the horizontal axis. In the fourth method shown in FIG. 6(d), the number of occurrences in which the participant's attention exceeds a certain threshold is counted. If the number of occurrences exceeds a threshold within a certain time range, switch 152 is turned on after the number threshold is exceeded. When counting the number of occurrences, the duration during which the attention threshold is exceeded may be added as a condition for counting the number of occurrences.

[0042] In the case of the automatic timing selection mechanism 15-2, the target and amount of attention of the participants are automatically collected using a sensor or the like. Therefore, it is possible to transmit to the target person without missing the target or amount of attention that the participant is not aware of. In addition to the method of turning on the switch 152, it is also possible to determine the optimal timing in more complex situations using machine learning or the like.

[0043] In this way, the arbitrary timing selection mechanism 15-1 transmits information based on the participants' will, but requires an operation to determine the timing. On the other hand, the automatic timing selection mechanism 15-2 does not require any operation by the participants, so in relatively long conversations such as presentations or lectures, attention can be shared without requiring effort from the participants, and the quality of communication can be improved efficiently.

[0044] <<Example of the attention amount analysis means 12>> Fig. 7 is an explanatory diagram of an embodiment of the attention amount analysis means 12. Fig. 7(a) shows a graph showing the change in the amount of attention of the participants to each of information 1 to 3, with the horizontal axis representing time, and a graph showing the effective information among them.

[0045] Information 1, information 2, and information 3 are different types of information, such as gaze information, video information, and audio information.

[0046] The number of valid pieces of information is the number of pieces of information that satisfy a certain condition corresponding to each piece of information at a certain moment in time. In other words, the number of valid pieces of information is the total number of types of information that exceed the thresholds of each type of information.

[0047] A time interval that satisfies a certain condition (in this example, a threshold or more) is defined as a valid interval. Since information 1, information 2, and information 3 are different pieces of information, different thresholds such as threshold t1, threshold t2, and threshold t3 are set. The settings of threshold t1, threshold t2, and threshold t3 may be determined appropriately depending on the type of information. For example, two or more of threshold t1, threshold t2, and threshold t3 may be the same.

[0048] Since information 1, information 2, and information 3 are the pieces of information that participants pay attention to, they each vary differently depending on the time. Therefore, information 1, information 2, and information 3 each have different temporal positions and durations that indicate their valid intervals. In the valid intervals of information 1, information 2, and information 3, the number of valid pieces of information per time is calculated. For example, if two types of information satisfy the corresponding conditions at a certain moment, then two valid intervals overlap at that moment, and the number of valid pieces of information is counted as two.

[0049] Next, as shown in Figure 7(b), the accurate amount of attention is calculated for each of the information 1 attention amounts to information 3 attention amounts calculated from multiple pieces of information by applying the number of valid pieces of information at each time to the time changes of the information 1 attention amount to information 3 attention amount calculated from multiple pieces of information.

[0050] If the amount of attention for information 1 at a certain moment before the number of valid pieces of information is applied is Att_1(t), and the number of valid pieces of information at that moment is N_sub(t), then the actual amount of attention for information 1 Att_1_sub(t) is, for example, given by the following equation (1):

[0051] Att_1_sub(t) = Att_1(t) / N_sub(t)…(1)

[0052] This formula (1) is based on the theory that the amount of attention that humans can devote is limited, a finite amount known as attentional resources. For example, if a participant continuously gazes at a point while speaking, and if they continuously gaze at a point without speaking, the amount of attention based on visual information will be greater in the latter case, even if the same amount of gaze time is achieved. This is because in the former case, finite gaze resources are allocated to both the speech content and gaze, resulting in a smaller amount of effective gaze resources being allocated.

[0053] In addition, there may be cases where the actual amount of attention is not simply inversely proportional as in the above formula. In such cases, for example, the effective n types of information are weighted respectively, and the following formula (2) is used.

[0054] Att_1_sub(t) = Att_1(t)*w_1 + Att_2(t)*w_2 + Att_3(t)*w_3 …(2) 0≦w_1, w_2, w_3≦1 Let w_1+w_2+w_3 = 1.

[0055] The weighting coefficients w_1, w_2, and w_3 may be determined based on the participant's information and the context of the conversation. For example, subjective information indicating that the participant prefers to focus on visual information may be acquired in advance, and the weighting coefficients corresponding to that information may be increased accordingly. Also, in situations where speech information is given more importance than in a lecture or interview, for example, the weighting coefficients corresponding to speech information may be increased.

[0056] <<Example of attention amount analysis>> 8A and 8B are explanatory diagrams of an attention amount analysis based on gaze information of a participant A1. As shown in Fig. 8A, the information processing device 1 of the participant A1 displays video information of another participant A2 who is participating in the conversation. The participant A1 carries out the conversation while looking at the video information of the participant A2.

[0057] In Figure 8(a), the attention amount analysis means 12 detects the gaze of participant A1 and detects the coordinates of the gaze point. The attention amount analysis means 12 also detects the facial area and facial feature points of participant A2. The attention amount analysis means 12 calculates the facial area and facial feature points, for example, the eye contact area, and compares the gaze point coordinates with the eye contact area along coordinate axes X1 and Y1. The eye contact area is assumed to be, for example, a certain range from the center of participant A2's nose.

[0058] As shown in FIG. 8(b), the attention amount analysis means 12 performs a process in which, when participant A1 gazes at the area corresponding to the eye contact range, the amount of attention due to gaze increases.

[0059] <<Functional blocks of information processing systems>> 9 is a diagram showing an example of the configuration of an information processing system 2. As shown in FIG.

[0060] By executing a program, the computer of the information processing device 1 realizes a participant attribute / subjective information acquisition unit 16, a third selection mechanism processing unit 15-3, an attention amount analysis means (attention amount analysis processing unit) 12, an output control processing unit (output control means) 13, and a transmission processing unit (transmission means) 14 as functional units, and operates as follows.

[0061] First, the participant attribute / subjective information acquisition unit 16 reads the participant attribute information or subjective information from the storage unit 1020. Next, the third selection mechanism processing unit 15-3 determines the timing selection mechanism 15 to be used by the transmission processing unit 14 to transmit output information based on the participant attribute information or subjective information. Here, the timing selection mechanism 15 determined by the third selection mechanism processing unit 15-3 is, for example, the arbitrary timing selection mechanism 15-1 shown in FIG. 3 or the automatic timing selection mechanism 15-2 shown in FIG. 5. The third selection mechanism processing unit 15-3 transmits information indicating the determined timing selection mechanism 15 to the transmission processing unit 14. The operation of the third selection mechanism processing unit 15-3 will be described later with reference to FIG. 10. The third selection mechanism processing unit 15-3 then requests the sensing unit 1010 to perform measurement. The sensing unit 1010 measures the participant information and transmits the measured information to the computer.

[0062] Next, the attention amount analysis processing unit 12 requests measurement from the sensing means 1010. The sensing means 1010 measures information about the participants and transmits the measured information to the attention amount analysis processing unit 12. The attention amount analysis processing unit 12 analyzes the amount of attention based on the measurement information and the participant attributes and subjective information that have been read out in advance. The amount of attention is stored in the storage unit 1020.

[0063] Next, the output control processing unit 13 reads the amount of attention from the storage unit 1020 , calculates output information based on the amount of attention, and stores the output information in the storage unit 1020 .

[0064] Then, the transmission processing unit 14 reads the output information and performs a transmission process to send the output information to the participants. The transmission processing unit 14 adjusts the timing of transmitting the output information to the participants using the timing selection mechanism 15 (e.g., arbitrary timing selection mechanism 15-1, automatic timing selection mechanism 15-2) determined by the third selection mechanism processing unit 15-3. If the calculation time required for attention amount analysis is short, the output information can be transmitted to the participants with as little delay as possible with respect to the time instant selected by the timing selection mechanism 15 in the transmission process by performing subsequent operations within the computer.

[0065] <<Processing Flow of the Third Selection Mechanism Processing Unit 15-3>> 10 is a diagram showing an example of a flow in which the third selection mechanism processing unit 15-3 determines the timing selection mechanism 15 of the transmission processing unit 14. Here, a flow in which the third selection mechanism processing unit 15-3 selects one of the timing mechanisms when the transmission processing unit 14 is equipped with an automatic timing selection mechanism 15-2 and an arbitrary timing selection mechanism 15-1 will be described.

[0066] First, the third selection mechanism processing unit 15-3 acquires the participant attribute information and subjective information acquired by the participant attribute / subjective information acquiring unit 16 (S1).

[0067] Next, the third selection mechanism processing unit 15-3 presents a plurality of options (S2). This presentation is made to the participants to associate the conversation in which the participants are participating with a certain situation. At this time, the presented situations may be ones prepared in advance or ones assumed from the attribute information and subjective information of the participants. In addition, an option may be provided for when the participant is unable to select a situation that seems to be relevant, such as "there is no relevant situation."

[0068] Next, the participant selects a situation that he or she thinks is equivalent to or close to a conversation, and this selection is accepted by the third selection mechanism processing unit 15-3 (S3).

[0069] The third selection mechanism processor 15-3 determines whether or not the use of the automatic timing selection mechanism 15-2 is optimal in the selected situation (S4).

[0070] If the third selection mechanism processing unit 15-3 determines that use of the automatic timing selection mechanism 15-2 is optimal (S4: Yes), it presents to the participants a recommendation to use the automatic timing selection mechanism 15-2 (S5).

[0071] Next, the participant uses the options to select whether to use the automatic timing selection mechanism 15-2 or the arbitrary timing selection mechanism 15-1, and the third selection mechanism processing unit 15-3 determines whether the automatic timing selection mechanism 15-2 has been selected (S6).

[0072] If the use of the automatic timing selection mechanism 15-2 is selected (S6: Yes), the third selection mechanism processing unit 15-3 sets the timing selection mechanism 15 to the automatic timing selection mechanism 15-2 (S7).

[0073] If the use of the automatic timing selection mechanism 15-2 is not selected (S6: No), the third selection mechanism processing unit 15-3 sets the timing selection mechanism 15 to the arbitrary timing selection mechanism 15-1 (S8).

[0074] Also, in step S4, if the third selection mechanism processing unit 15-3 does not determine that using the automatic timing selection mechanism 15-2 is optimal (S4: No), it presents a recommendation to use the arbitrary timing selection mechanism 15-1 (S9).

[0075] Next, the participant uses the options to select whether to use the automatic timing selection mechanism 15-2 or the arbitrary timing selection mechanism 15-1, and the third selection mechanism processing unit 15-3 determines whether the use of the arbitrary timing selection mechanism 15-1 has been selected (S10).

[0076] If the use of the arbitrary timing selection mechanism 15-1 is selected (S10: Yes), the third selection mechanism processing unit 15-3 sets the timing selection mechanism 15 to the arbitrary timing selection mechanism 15-1 (S11).

[0077] If the use of the arbitrary timing selection mechanism 15-1 is not selected (S10: No), the third selection mechanism processing unit 15-3 sets the timing selection mechanism 15 to the automatic timing selection mechanism 15-2 (S12). The third selection mechanism processing unit 15-3 transmits the selected timing selection mechanism 15 to the transmission processing unit 14.

[0078] In this way, by using not only the participant information input in advance but also the participant's intention as a choice from options to select the timing selection mechanism 15, the participant can determine an effective timing selection mechanism 15 even in a situation where the purpose of the conversation changes from time to time or where the purpose of the conversation is unknown.

[0079] 11 is a diagram showing an example of another embodiment of the information processing system 2. The information processing system 2 includes a sensing means 1010, an information processing device 1, and a server device 3.

[0080] The computer of the information processing device 1 executes a program to realize, as functional units, a participant attribute / subjective information acquisition unit 16, a third selection mechanism processing unit 15-3, and a transmission processing unit 14. The computer of the server device 3 executes a program to realize, as functional units, an attention amount analysis processing unit 12 and an output control processing unit 13.

[0081] First, in the computer of the information processing device 1, the participant attribute / subjective information acquisition unit 16 reads participant attribute information or subjective information from the storage unit 1020. Next, the third selection mechanism processing unit 15-3 determines the timing selection mechanism 15 to be used by the transmission processing unit 14 to transmit output information based on the processing flow shown in FIG. 10. Here, the timing selection mechanism 15 determined by the third selection mechanism processing unit 15-3 is, for example, the arbitrary timing selection mechanism 15-1 shown in FIG. 3 and the automatic timing selection mechanism 15-2 shown in FIG. 5. The third selection mechanism processing unit 15-3 transmits information indicating the determined timing selection mechanism 15 to the transmission processing unit 14. Thereafter, the information processing device 1 requests measurement from the sensing means 1010. The sensing means 1010 measures participant information and transmits the measurement information to the computer of the information processing device 1. The information processing device 1 transmits the measurement information to the server device 3.

[0082] Next, in the computer of the server device 3, the attention amount analysis processing unit 12 analyzes the attention amount based on the measurement information transmitted from the information processing device 1 and the participant attributes or subjective information that has been read out in advance. The calculated attention amount is stored in the storage unit 1030 of the server device 3. The output control processing unit 13 reads out the attention amount from the storage unit 1030 of the server device 3, calculates output information based on the attention amount, and stores the output information in the storage unit 1030 of the server device 3.

[0083] The output information stored in the storage unit 1030 of the server device 3 is transmitted to the computer of the information processing device 1, and the transmission processing unit 14 performs transmission processing to send the output information to the participants. The transmission processing unit 14 adjusts the timing of transmitting the output information to the participants using the timing selection mechanism 15 (e.g., the arbitrary timing selection mechanism 15-1 and the automatic timing selection mechanism 15-2) determined by the third selection mechanism processing unit 15-3. If it is expected in advance that the calculation time required for attention amount analysis will be long, the attention amount analysis and output control processing can be performed within the server device 3, so that the output information can be transmitted to the participants with as little delay as possible relative to the time moment selected by the timing selection mechanism 15 in the transmission processing.

[0084] <<Hardware Block Configuration of Computers of Information Processing Device 1 and Server Device 3>> Fig. 12 is a diagram showing an example of the configuration of hardware blocks of a computer. The computer 1100 shown in Fig. 12 includes a CPU (Central Processing Unit) 1101, a ROM (Read Only Memory) 1102, a RAM (Random Access Memory) 1103, a GPU 1104, a VRAM 1105, an input I / F 1106, a monitor I / F 1107, an output I / F 1108, a LAN I / F 1109, a WLAN I / F 1110, an audio I / F 1111, and a storage unit 1112, which are connected via a bus.

[0085] The CPU 1101 controls the overall operation of the computer 1100. The ROM 1102 stores programs used to drive the CPU 1101. The RAM 1103 is used as a work area for the CPU 1101. The storage unit 1112 is an HDD or SSD and stores various data. The GPU 1104 mainly performs graphics processing, and the VRAM 1105 is used for graphics processing by the GPU 1104.

[0086] The input unit I / F 1106 is an interface for connecting an input device, such as a keyboard, a mouse, a touch input panel, or a camera.

[0087] A monitor I / F 1107 is an interface for connecting a monitor, which is a display 1006 such as a liquid crystal display or an organic electroluminescence (EL) display.

[0088] The output I / F 1108 is an interface for outputting a motor, a heater, or light.

[0089] The LANI / F 1109 is an interface for connecting to a LAN, and the WLANI / F 1110 is an interface for connecting to a wide area LAN.

[0090] The audio I / F 1111 is an interface to which a microphone and a speaker are connected and which inputs and outputs audio signals.

[0091] <<Usage scenario 1>> 13 is a diagram showing a first example of a usage scene. As shown in Fig. 13, an information processing device 1, an IWB (Interactive White Board: an electronic white board with a blackboard function that allows mutual communication) 301, and a camera 302 are provided in a venue 300. The information processing device 1 in the venue 300 is connected to a network N.

[0092] Presenter (participant) A1 gives a presentation by displaying the contents of the presentation on IWB 301 while displaying the facial expressions of the participants on presenter A1's information processing device 1. Video of presenter A1's presentation is captured in real time by camera 302. Note that "capture" is used synonymously with "capture image" since the image is captured by a camera. The captured video and the display information on IWB 301 are transmitted to presenter A1's information processing device 1.

[0093] The information processing device 1 of the presenter A1 inputs the gaze and voice of the presenter A1 using input means such as a camera 1007 or a microphone 1008, and transmits the captured image and display information, etc., to the information processing device 1 of the participant A2 connected to the network N.

[0094] The information processing device 1 of participant A2 has sensing means such as a camera 1007 and a microphone 1008, and a display 1006. The display 1006 displays captured images of the venue, display information of the presentation content, etc. The information processing device 1 of participant A2 captures images of participant A2 with the camera 1007, measures participant A2's movements, etc. with the sensing means, and transmits this information to the information processing device 1 of presenter A1 via the network N.

[0095] <<Usage scenario 2>> Fig. 14 is a diagram showing a second example of a usage scene, in which the device is used for personal interviews or sales.

[0096] As shown in Fig. 14, a booth 400 used for interviews or the like is provided with a housing 401 that houses a computer. The housing 401 is provided with a camera 402, a half mirror 203, a speaker 404, and sensing means 405. The camera 402 is a camera that photographs the upper body of a subject. In addition, a camera 406 that photographs the entire body of the subject is also provided.

[0097] The computer in each booth 400 is connected to a network N and communicates with, for example, the computer of the person being interviewed. Note that a notebook PC may also be connected to the network N for communication.

[0098] <<Usage scenario 3>> Fig. 15 is a diagram showing a third example of a usage scenario. Fig. 15 shows a scenario in which the system is used in a meeting. As shown in Fig. 15, each participant in the meeting connects to the network N using a laptop PC or the like at their desk.

[0099] <<Usage scenario 4>> Fig. 16 is a diagram showing a fourth example of a usage scenario. Fig. 16 shows a form in which the system is used for a meeting. As shown in Fig. 16, booths 500 are provided for the meeting, and each booth 500 is connected to a network N by a computer 1100 in each booth 500.

[0100] 16 is equipped with a computer 1100, a camera 501, and sensing means 502. Participants hold meetings while wearing head-mounted displays 503. Head-mounted displays 503 may be equipped with a camera that captures the viewpoints of participants.

[0101] <<Example 1 of output information>> FIG. 17 is a diagram illustrating a first example of output information m1. The information processing device 1 of participant A1, who is participating in a conversation, receives output information m1 regarding the amount of attention transmitted from the information processing device 1 of participant A2, who is also participating in the same conversation, via the network N. The information processing device 1 of participant A1 changes the light state of the light source 2000 provided in the information processing device 1 of participant A1 based on the output information m1 regarding the amount of attention from the information processing device 1 of participant A2. By changing the color type, blinking interval, or light intensity of the light source 2000 of participant A1's information processing device 1, participant A1 can determine what and how much attention participant A2 is paying. In participant A1's information processing device 1, the type of attention and the color of light are associated in advance, such as red when participant A2 is paying attention to participant A1's speech, and green when participant A2 is paying attention to participant A1's visual information, such as body language and facial expressions. In addition, when the amount of attention is high, the blinking interval is shortened and the light intensity is increased. Furthermore, because participant A1 perceives simple output information m1, which is a change in light, the amount of attention allocated to the output information m1 itself can be reduced, allowing participant A1 to concentrate on the conversation with participant A2. Furthermore, because the output information m1 is separated from information such as videos and images, which require a relatively long time to process on the information processing device 1, it is possible to transmit the output information m1 with reduced delays in the transmission timing determined by participant A2, such as the color, flashing, and light intensity used to convey the output information m1.

[0102] <<Example 2 of output information>> Fig. 18 is a diagram showing a second example of output information m1. As shown in Fig. 18, a vibrator 3000 is attached directly to the arm A1-1 of participant A1 or attached to the chair 600 on which participant A1 sits, and vibrations generated by the vibrator 3000 are used as output information m1.

[0103] For example, the information processing device 1 of participant A1 changes the vibration magnitude, frequency, vibration interval, or vibration pattern of the vibrator 3000 of participant A1. Participant A1 can determine from the vibration of the vibrator 3000 what and how much attention participant A2 is paying to.

[0104] Specifically, when participant A2 is paying attention to what participant A1 is saying, vibrations of a magnitude corresponding to the change in sound pressure of participant A1's speech are given to participant A1. Also, when participant A2 is paying attention to the body language, which is visual information from participant A1, vibrations of a magnitude corresponding to the volume of the body language are given to participant A1. By giving output information m1 in the form of vibrations, attention to participant A2 and attention to participant A2's speech are not interrupted, allowing for smooth conversation.

[0105] <<Example 3 of output information>> FIG. 19 is a diagram showing a third example of output information m1. As shown in FIG. 19, output information m1 relating to the amount of attention of participant A2 is displayed using icons or the like on the display 1006 of participant A1's information processing device 1. For example, a first group of icons 4010 corresponds to the sensory organs to which participant A2 is paying attention, with icon 4011 corresponding to vision and icon 4012 corresponding to hearing. A second group of icons 4020 corresponds to the objects to which participant A2 is paying attention, with icon 4021 corresponding to participant A1's visual information and icon 4022 corresponding to the materials shared by participant A1. Materials include not only visually perceptible text information, image information, and video information, but also audio information. When participant A2 is paying attention to the materials visually, the sizes of icons 4011 and 4022 on the display 1006 become larger than those of icons 4012 and 4021 at the timing determined by the timing selection mechanism 15. This allows participant A1 to recognize that participant A2 is paying attention to the materials. By using an icon whose shape itself has a meaning as the output information m1, participants can instantly and easily grasp the output information m1.

[0106] <<Example 4 of Output Information>> 20 is a diagram showing a fourth example of output information m1. As shown in FIG. 20, in a situation such as a lecture in which participant A1 is unilaterally speaking to multiple participants A2, A3, A4, and A5, output information m1 from the multiple participants can be displayed collectively as icons. For example, when the sensory organs used by participants other than participant A1 to pay attention coincide with the objects to which they are paying attention at a certain moment, participant A1 can recognize representative output information m1 related to attention collectively by changing the size of the corresponding icon 4011, icon 4012, icon 4021, or icon 4022.

[0107] If an icon is provided for each participant, participant A1 cannot see all the output information m1 at once. On the other hand, by adopting the configuration of Example 4, participant A1 can selectively see only the information commonly output by multiple participants, allowing the conversation to proceed more efficiently.

[0108] <<Example 5 of Output Information>> 21 is a diagram showing a fifth example of output information m1. As shown in Fig. 21, output information m1 related to the amount of attention of participant A2 is implemented by temporal and spatial changes in the color or pattern of the background of participant A2 displayed on display 1006. As output information m1 related to the amount of attention, for example, the type of color, the shape of the pattern, or the spacing between patterns may be changed.

[0109] Participant A1 can determine what and how much attention participant A2 is paying by seeing changes in participant A2's background 5001. For example, the type of attention and the background color can be associated in advance, such as red 5002 when participant A2 is paying attention to participant A1's speech, and green 5003 when participant A2 is paying attention to participant A1's visual information, such as body language and facial expressions. Also, when participant A2 is paying attention to both speech and visual information, it is possible to use blue, which is different from red or green, or a mixture of red and green on display 1006.

[0110] In this way, in Example 5, unlike the light source, a two-dimensional color distribution can be provided. Also, in Example 5, the area where a visual change occurs is larger than the light source, so that participant A1 can more easily recognize the output information m1. Also, in Example 5, the amount of attention can be grasped using not only color but also pattern.

[0111] <<Example 6 of Output Information>> FIG. 22 is a diagram illustrating a sixth example of output information m1. As shown in FIG. 22, the display 1006 displays the visual information of participant A2 and the materials shared by participants A1 and A2. Areas for displaying each piece of information (participant display area 1006a and material display area 1006b) are simultaneously provided on the display 1006. Output information m1 regarding participant A2's attention level is implemented by temporally changing the size of each of the participant display area 1006a and material display area 1006b. For example, when participant A2 is paying attention to material B1, the area of ​​the material display area 1006b is changed from its normal state. The normal state refers to the state immediately after the conversation begins, meaning that the areas of each display area are in their initial state. For example, enlarging the material display area 1006b is associated with participant A2's attention to material B1. The greater the attention participant A2 pays to material B1, the larger the material display area 1006b is.

[0112] In Example 6, participant A1 is physically larger than the light source and can see only the minimum information necessary for conversation, so that he or she can easily associate the size of the display area with the amount and target of attention and grasp the output information m1. Each display area may be provided by dividing the display area on the display 1006 into multiple areas, or one area may be displayed superimposed on another area.

[0113] <<Example of selecting timing selection mechanism 15 by detecting body movement>> FIG. 23 illustrates an example in which the timing selection mechanism 15 is selected by detecting a physical movement (gesture). Assume that participant A2 is paying attention to participant A1 or document B1 displayed on the display 1006. For example, it is predetermined that the hand movement or shape corresponding to participant A1 is sign R1 when the target of attention is participant A1, and the hand movement or shape corresponding to document B1 is sign R2 when the target of attention is document B1. Participant A2 changes the shape of participant A2's hand to sign R1 or sign R2. The change in hand shape is detected by a detector 800, such as a camera, LiDAR, RADAR, or speckle imaging, and output information m1 is transmitted to participant A1. The information transmitted to participant A1 includes information about the target to which participant A2 is paying attention, and this information changes depending on the shape of participant A2's hand. If the hand shape is sign R1, the target to which participant A2 is paying attention is participant A1. If the hand shape is sign R2, the target to which participant A2 is paying attention is the displayed document B1. The types of hand changes are not limited to hand shape alone, and may also include changes in hand position over time.

[0114] <<Example of speech time threshold of arbitrary timing selection mechanism 15-1>> 24A and 24B are diagrams showing an example of an embodiment of the speech time threshold of the arbitrary timing selection mechanism 15-1. Fig. 24A shows the speech times of participants A1 and A2 for each time period and the cumulative sum of the speech times of participants A1 and A2 for each time period.

[0115] When the cumulative sum of speech times exceeds a certain threshold, it means that participants are conversing while paying a certain amount of attention to each other's speech information. When threshold t1 is exceeded, output information m1 is sent to the participants, indicating that they are paying attention to the speech. The participants recognize that the conversation is progressing smoothly and are conscious of maintaining the conversation. Next, by setting multiple thresholds, such as threshold t2, and sending output information m1 to the participants, participants can converse while always being conscious of paying attention to each other's speech. In a creative problem-solving situation where participants A1 and A2 exchange many opinions and aim to generate ideas, making them aware that they are paying attention to each other's speech accelerates communication and ultimately improves it.

[0116] The number of thresholds to be set is not limited to 2. Furthermore, the threshold values ​​may not be fixed, but may be changed depending on the purpose of the conversation and the number of participants.

[0117] On the other hand, Figure 24(b) shows an example of calculating the cumulative difference in speech time between participant A1 and participant A2 for each time interval. When the cumulative difference in speech time falls below a certain threshold, it means that participant A2, who has spoken less, has become a listener and is paying a lot of attention to the speech of the other participant A1. When the cumulative difference in speech time falls below a certain threshold, output information m1, which means that participant A2 is paying attention to the speech of participant A1, is sent to participant A2. In a sales situation where a seller and buyer of a product are negotiating, making the seller aware that the seller, corresponding to participant A2, is listening more to what the buyer, corresponding to participant A1, is saying, can facilitate smooth communication and increase the probability of successful negotiations and contracts.

[0118] The number of thresholds is not limited to one. Furthermore, the threshold value is not fixed, and may be changed depending on the purpose of the conversation, the number of participants, etc.

[0119] The functions of the embodiments and modifications may be provided by being implemented in an ASIC (Application Specific Integrated Circuit), or by having a computer execute a program. In the latter case, the program may be pre-installed in a ROM, HDD, or the like and provided as a functional unit. In this case, the CPU reads the program and executes it step by step to realize the various functional units. The program may also be recorded on a computer-readable recording medium and provided as a computer program product. For example, the program may be provided by being recorded on a recording medium such as a flexible disk, CD-R, DVD (Digital Versatile Disk), Blu-ray Disc (registered trademark), or semiconductor memory in the form of an installable or executable file.

[0120] The program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed in the embodiment and each modification may be provided or distributed via a network such as the Internet.

[0121] Although the embodiments and modifications of the present invention have been described above, the embodiments and examples are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and examples can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and examples are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0122] 1. Information processing equipment 12 Attention amount analysis means (attention amount analysis processing unit) 13 Output control means (output control processing unit) 14 Transmission means (transmission processing unit) 15 Timing selection mechanism 15-1 Arbitrary timing selection mechanism 15-2 Automatic timing selection mechanism 15-3 Third selection mechanism processing section 16 Participant attributes / subjective information acquisition section [Prior art documents] [Patent documents]

[0123] [Patent Document 1] International Publication No. 2018 / 174088

Claims

1. An information processing device that processes information related to communication between multiple participants, an information acquisition means for acquiring participant information relating to the participant acquired from the participant in the communication; an attention amount analysis means for acquiring attention amount information relating to the amount of attention of the participants to information used in the communication; presentation information determination means for determining presentation information to be presented to the plurality of participants based on the attention amount information; a transmitting means for transmitting the presentation information to a presenting means provided for each of the plurality of participants; the transmitting means adjusts a timing for transmitting the presentation information to other participants included in the plurality of participants. Information processing device.

2. The information processing device according to claim 1 , wherein the participant information includes at least one of information relating to the living body of the participant, audio information, and video information.

3. The information processing apparatus according to claim 1 , wherein the transmitting means determines a timing for transmitting the presentation information to the other participants based on an operation input by the participant.

4. The information processing device according to claim 1 , wherein the transmitting means determines a timing for transmitting the presentation information to the other participants based on participant information related to the participant or attention amount information related to the participant.

5. The information processing device according to claim 4 , wherein the transmission means determines the timing for transmitting the presentation information to the other participants based on the timing at which the attention amount information relating to the participant exceeds a predetermined threshold.

6. The information processing device according to claim 5 , wherein the attention amount information is an amount related to the time that the participant continuously gazes at image information of the other participant.

7. the information relating to the communication includes at least one of text information, image information, and video information; The information processing device according to claim 5 , wherein the attention amount information is an amount related to the time the participant continuously gazes at at least one of the text information, the image information, and the video information.

8. 3. The information processing device according to claim 1, wherein the transmitting means determines a timing for transmitting the presentation information to the other participant associated with the body movement in advance, based on information obtained by detecting the body movement of the participant.

9. the information relating to the communication includes voice information of the plurality of participants; the participant information includes information relating to speaking times of the plurality of participants, The attention amount information is acquired based on a speaking time of the participant and a speaking time of the other participant. The information processing device according to claim 5 .

10. the information acquisition means acquires the plurality of pieces of participant information from the participants; The attention amount analysis means acquiring a plurality of pieces of attention amount information for a predetermined time period based on a valid number, which is the number of pieces of participant information determined to be valid based on a predetermined threshold, and the plurality of pieces of participant information for the predetermined time period; The information processing device according to claim 1 .

11. The information processing device according to claim 2 , wherein the information relating to the living body of the participant is information relating to the line of sight of the participant.

12. The transmitting means 3. The information processing device according to claim 1, wherein the timing at which to transmit is determined based on a first timing determined based on the participant's operation input or a second timing determined based on participant information relating to the participant or attention amount information relating to the participant, based on the participant's timing selection input.

13. a sensing means for acquiring the participant information; The information processing device according to claim 1 .

14. A plurality of information processing devices according to any one of claims 1 to 13; a sensing means capable of communicating with the information processing device and acquiring the participant information; the presentation means capable of communicating with the information processing device; Equipped with The information processing system, wherein the plurality of information processing devices are capable of communicating with each other.

15. The information processing system according to claim 14, the plurality of information processing devices are capable of transmitting and receiving information related to the communication with each other, Communication support system.

16. An information processing method performed by an information processing device that processes information related to communication between multiple participants, In the communication, acquiring participant information related to the participant acquired from the participant; acquiring attention amount information relating to the amount of attention of the participants to information used in the communication; determining presentation information to be presented to the plurality of participants based on the attention amount information; adjusting a timing for transmitting the presentation information to other participants included in the plurality of participants; transmitting the presentation information to a presentation means provided for each of the plurality of participants; An information processing method including:

17. A computer of an information processing device that processes information related to communication between multiple participants, an information acquisition means for acquiring participant information relating to the participant acquired from the participant in the communication; an attention amount analysis means for acquiring attention amount information relating to the amount of attention of the participants to information used in the communication; presentation information determination means for determining presentation information to be presented to the plurality of participants based on the attention amount information; a transmitting means for transmitting the presentation information to a presenting means provided for each of the plurality of participants; It functions as the transmitting means adjusts a timing for transmitting the presentation information to other participants included in the plurality of participants. A program that can be executed on a computer.

Images