Information processing device, information processing method, and program product

Abstract

The present invention enables the state of a conversation to be grasped in more detail. The speech of each participant of a shared conversation is detected by processing a voice signal. The state of each participant of a shared conversation, for example, the gaze direction of each participant, is detected by processing an image signal. The state and existence of a conversation are determined based on the utterance and state of each participant. Therefore, the state and existence of a conversation can be determined more accurately than when the state and existence of a conversation are determined based only on the utterances of the participants.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more specifically, to an information processing device and the like for grasping the situation of a conversation (discussion, meeting, group work, active learning, etc.) constructed by multiple participants. Background Art

[0002] In a conversation, it is very important to grasp the situation of the conversation. For example, Patent Document 1 proposes a technology for recording and displaying the duration of speech of a conversation participant. In addition, for example, Patent Document 2 proposes a technology for presenting a correlation diagram by grasping the situation of the conversation from voice data obtained from name tag type sensors worn by multiple participants. These technologies can acquire voice and movements such as nodding through an acceleration sensor, but this is not enough.

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 4-323689

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2013-058221 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] The purpose of this technology is to allow a more detailed understanding of the context of a conversation.

[0009] Solution to the problem

[0010] The concept of this technology is an information processing device, including:

[0011] a speech processing unit that detects utterances of a plurality of participants of a conversation from the speech signal;

[0012] an image processing unit that detects states of the plurality of participants of the conversation from an image signal; and

[0013] A determination unit determines the state of the conversation based on the speeches of the plurality of participants and the states of the plurality of participants.

[0014] In the present technology, a speech processing unit detects the speech of multiple participants of a conversation from a speech signal. In addition, the status of multiple participants of the conversation is detected from an image signal. Then, a determination unit determines the status of the conversation based on the speech of multiple participants and the status of multiple participants.

[0015] For example, the state of a session may include a session relationship between the participants of the session. In addition, for example, the state of a session may include at least one of the content of a speech, the number of speeches, or the duration of a speech. In addition, for example, the state of a session may include the establishment of a session. In this case, for example, when one of the participants is looking at the other party of the session and is speaking, it can be determined that the session has been established.

[0016] For example, the image processing unit can detect the direction in which one of the participants is looking as one of the states of the participants. Then, in this case, for example, the image processing unit can detect the direction in which one of the participants is looking from the facial orientation or sight direction of one of the participants.

[0017] For example, when the second participant speaks after the first participant speaks and the direction the second participant is looking at is consistent with the direction of the first participant, the determination unit may determine that there is a conversation between the first participant and the second participant.

[0018] As described above, in the present technology, the state of a conversation is determined based on the speeches of multiple participants in the conversation detected from the voice signal and the states of multiple participants in the conversation detected from the image signal. Therefore, compared with the technology of determining the state of a conversation only by the speeches of the participants, the state of the conversation can be determined with higher accuracy.

[0019] Note that, for example, the present technology may further include an image generation unit that generates a floor plan map image of a space for a conversation, wherein the floor plan map image may include a participant display showing each participant participating in the conversation, and the floor plan map image may include a conversation line of two participant displays corresponding to the detected conversation connection. This makes it possible to easily identify who has had a conversation with whom, for example, visually.

[0020] For example, the image processing unit may process the image signal to further detect the position of each participant participating in the conversation, and the image generation unit may change the position of the participant display showing each participant on the planar map image according to the detected position of each participant. This makes it easy to visually identify the position of each participant participating in the conversation.

[0021] In addition, for example, the image generation unit may change the position of the participant display showing each participant on the plan map image according to the number of sessions of each participant. In this case, for example, as the number of sessions increases, the image generation unit may arrange the position of the participant display showing each participant closer to the center of the plan map image. This makes it easy to visually identify who has conducted a larger number of sessions. In addition, in this case, for example, as the number of sessions between the two participants increases, the image generation unit may make the interval between the participant displays of the two participants closer. This makes it easy to visually identify who has conducted a larger number of sessions with whom.

[0022] For example, the participant display may have a size corresponding to the number of speeches of the corresponding participant. In addition, for example, the participant display may be supplemented with characters showing the number of speeches of the corresponding participant. This makes it easy to visually identify the extent of the number of speeches of each participant.

[0023] For example, the conversation line may have a thickness corresponding to the number of conversations between the corresponding participants. In addition, for example, the conversation line may be attached with characters showing the number of conversations between the corresponding participants. This makes it easy to visually recognize the extent of the number of conversations between each two participants.

[0024] For example, a display unit may be further provided, which displays a plan map image generated by the image generation unit. This makes it possible to check the plan map image on the display unit. In addition, a server that holds, for example, a plan map image generated by the image generation unit may be further provided. This makes it possible to reproduce and check past plan map images from the server. In addition, a communication unit may be further provided, which, for example, sends the plan map image generated by the image generation unit to a mobile terminal. This allows the mobile terminal to check the plan map image.

[0025] Effects of the Invention

[0026] According to the present technology, the situation of the conversation can be grasped in more detail. Note that the effects described here are not necessarily restrictive, and any effects described in the present disclosure can be applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram showing a configuration example of an information processing apparatus serving as the first embodiment.

[0028] Figure 2 is a block diagram showing a configuration example of an information processing apparatus serving as the second embodiment.

[0029] Figure 3 is a diagram showing an example of a floor plan of a room in which a conversation is conducted.

[0030] Figure 4 is a diagram showing another example of a floor plan of a room in which a conversation is conducted.

[0031] Figure 5 is a diagram showing an example of a plan map image generated by the image generating unit.

[0032] Figure 6 is a diagram showing another example of a plan map image generated by the image generating unit.

[0033] Figure 7 is a diagram showing another example of a plan map image generated by the image generating unit.

[0034] Figure 8 is a diagram showing another example of a plan map image generated by the image generating unit.

[0035] Fig. 9 is a diagram showing another example of a plan map image generated by the image generating unit.

[0036] Fig.10 It is a diagram for explaining the processing flow of the information processing device.

[0037] Fig.11 It is a diagram showing an example of a plan map image in a case where a conversation is conducted in a plurality of divided groups in a large room (venue).

[0038] Fig.12 is a diagram showing an example of a plan map image in a case where a participant located in another room participates in one session.

[0039] Fig.13 : is a diagram showing an example of a plan map image in the case where information other than the participant display, the number of speeches, and the number of sessions is displayed on the plan map or an area outside the map.

[0040] Fig.14 : is a diagram for explaining an example of specifying the position of a participant display showing each participant according to the number of sessions of each participant.

[0041] Fig.15 This is a diagram for explaining a case where the size of the participant display of each participant is changed according to the duration of the speech.

[0042] Fig.16 This is a diagram for explaining that evaluation is performed based on the statements and conversation situations of the participants, and displays the evaluation values ​​of the evaluation.

[0043] Fig.17 is a graph for explaining the evaluation level of each participant.

[0044] Fig.18 The diagram shows an example of screen display on a monitor or a mobile terminal.

[0045] Fig.19 The diagram shows an example of screen display on a monitor or a mobile terminal.

[0046] Fig. 20 The diagram shows an example of screen display on a monitor or a mobile terminal.

[0047] Fig.21 The diagram shows an example of screen display on a monitor or a mobile terminal.

[0048] Fig. 22 The diagram shows an example of screen display on a monitor or a mobile terminal.

[0049] Fig.23 The diagram shows an example of screen display on a monitor or a mobile terminal. DETAILED DESCRIPTION

[0050] A mode for carrying out the present invention (hereinafter, referred to as an "embodiment") will be described below. Note that the description will be given in the following order.

[0051] 1. First Embodiment

[0052] 2. Second Embodiment

[0053] 3. Modifications

[0054] <1. First Embodiment>

[0055] [Information processing device]

[0056] Figure 1 There is shown a configuration example of an information processing apparatus 100 serving as a first embodiment. The information processing apparatus 100 includes a speech processing unit 101 , an image processing unit 102 , and a determination unit 103 .

[0057] A voice signal obtained by detecting the voice of each participant participating in a conversation (discussion, meeting, group work, active learning, etc.) using a microphone is provided to the voice processing unit 101. In this case, the voice can be detected using a microphone attached to each participant, or a microphone array composed of a plurality of microphones can be used to detect the voice.

[0058] The speech processing unit 101 detects the speech of a plurality of participants participating in the conversation from the speech signal. In this case, for example, the volume is measured from the speech signal for each participant, and when the state where the measured volume is equal to or higher than the threshold value continues for a certain period of time or longer, it is determined that there is a speech. When the state where the measured volume is equal to or higher than the threshold value is maintained for less than a certain period of time, the utterance is considered to be related to agreement, refusal, agreement, etc., and it is determined that there is no speech.

[0059] The image processing unit 102 is provided with an image signal obtained by capturing an image of a participant participating in the conversation using a camera. In this case, a camera capable of imaging the entire surrounding environment (such as a 360-degree camera) may be arranged at the center of all participants to image each participant, or a plurality of cameras may be arranged to cover the entire surrounding environment to image each participant.

[0060] The image processing unit 102 detects the states of the multiple participants participating in the conversation from the image signal. The state of the participant detected in this way is a state required for determining the state of the conversation and the existence according to the speech of the participant. For example, the state of the participant includes the direction the participant is looking at. In order to detect the direction the participant is looking at, the image processing unit 102 detects the facial orientation of the participant or detects the direction of the participant's sight line through image processing.

[0061] The determination unit 103 determines the state and existence of the conversation based on the speeches of the multiple participants detected by the voice processing unit 101 and the states of the multiple participants detected by the image processing unit 102. For example, the state of the conversation is the conversation relationship between the participants of the conversation. In addition, for example, the state of the conversation includes at least one of the content of the speech, the number of speeches, or the duration of the speech. In addition, for example, the state of the conversation is the establishment of the conversation. For example, when the participant is looking at the other party of the conversation and talking, the determination unit determines that the conversation has been established.

[0062] In addition, for example, when the second participant speaks after the first participant speaks or immediately before the end of the speech, and the direction the second participant is looking at is consistent with the direction of the first participant, the determination unit determines that there is a conversation between the first participant and the second participant. Note that even when the direction the second participant is looking at is inconsistent with the direction of the first participant, it can be determined that there is a conversation. For example, even when the second participant speaks after the first participant speaks, or when the face of the second participant moves up and down (echoes) after the first participant speaks, it can be determined that there is a conversation.

[0063] Will briefly describe Figure 1The operation of the information processing device 100 shown in . A voice signal obtained by detecting the voice of each participant participating in the conversation by using a microphone is provided to a voice processing unit 101. In the voice processing unit 101, the voice signal is processed, and the speech of each participant participating in the conversation is detected. In addition, an image signal obtained by capturing an image of each participant participating in the conversation by using a camera is provided to an image processing unit 102. In the image processing unit 102, the image signal is processed, and the state of each participant participating in the conversation, for example, the direction in which each participant is looking, is detected.

[0064] The speech of each participant participating in the conversation detected by the voice processing unit 101 and the state of each participant participating in the conversation detected by the image processing unit 102 are provided to the determination unit 103. In the determination unit 103, the state and existence of the conversation are determined based on the speech of each participant and the state of each participant, and determination information is obtained.

[0065] As mentioned above, in Figure 1 In the information processing device 100 shown, the state and existence of a conversation are determined based on the speech of each participant participating in the conversation and the state detected by image processing. Therefore, compared with the technology of determining the state and existence of a conversation only by the speech of the participants, the state and existence of a conversation can be determined with higher accuracy.

[0066] <2. Second Embodiment>

[0067] [Information processing device]

[0068] Figure 2 An example of a configuration of an information processing apparatus 200 serving as a second embodiment is shown. The information processing apparatus 200 includes N (N is an integer) cameras 201-1 to 201-N, M (M is an integer) microphones 202-1 to 202-M, an image and voice control system 203, a monitor 204, a server 205, and a mobile terminal 206.

[0069] Cameras 201-1 to 201-N are used to capture images of participants participating in the session. Cameras 201-1 to 201-N are installed so as to image the participants participating in the session, particularly the faces. In this case, a camera (such as a 360-degree camera) capable of imaging the entire surrounding environment may be installed at the center of the participant, or a plurality of cameras may be installed so that the entire surrounding environment can be imaged.

[0070] Figure 3An example of a floor plan map of a room in which a conversation is taking place is shown. In this example, the circular displays labeled "A" to "D" represent participant displays showing the participants participating in the conversation, and each participant is located near a table placed in the center of the room. In this example, a 360-degree camera capable of imaging the entire surrounding environment is mounted on the table. In this case, adjustments are made so that the orientation and position of the camera on the floor plan map are in a constant state. Note that although this example shows a state in which there is a table in the room and four participants are located around the table, the number and shape of the tables, the number of participants, etc. are arbitrary.

[0071] Figure 4 Another example of a floor plan map of a room in which a conversation is taking place is shown. In this example, the circular displays labeled "A" to "D" represent the participant displays in the conversation, and each participant is located near a table placed in the center of the room. In this example, camera 1, which is a 360-degree camera capable of imaging the entire surrounding environment, is installed on top of the table, and additionally, camera 2 is installed to capture a whiteboard that exists at the periphery of the room. Camera 2 can image the whiteboard, and in addition can also image the face of the participant who is writing on the whiteboard when he explains the writing in front of the whiteboard. In this case, when cameras 1 and 2 are installed, the position and orientation of the cameras on the floor plan map are adjusted.

[0072] Back to Figure 2 , microphones 202-1 to 202-M are used to detect the voice of each participant participating in the conversation. In this case, microphones 202-1 to 202-M may be microphones attached to each participant, or may be a plurality of microphones constituting a microphone array for detecting the voice of each participant.

[0073] The image and voice control system 203 processes the image signals obtained by the cameras 201-1 to 201-N and the voice signals obtained by the microphones 202-1 to 202-M, and generates a floor plan map image of the room in which the conversation takes place. The image and voice control system 203 provides the generated floor plan map image to the monitor 204, and displays the floor plan map image on the monitor 204.

[0074] Furthermore, the image and voice control system 203 provides the generated plan map image to the mobile terminal 206 in place of the monitor 204 or together with the monitor 204 according to the operation of the user, and displays the plan map image on the mobile terminal 206. In addition, the image and voice control system 203 provides the generated plan map image to the server 205 to record the provided plan map image therein, and further, reproduces the plan map image from the server 205 according to the operation of the user to provide the plan map image to the monitor 204 or the mobile terminal 206 and display the reproduced plan map image thereon.

[0075] The image and voice control system 203 includes a control unit 231, a user operation unit 232, a voice processing unit 233, an image processing unit 234, a determination unit 235, an image generation unit 236, and a communication unit 237. The control unit 231 controls the action of each unit of the image and voice control system 203. The user operation 232 is a key, a button, a touch panel, a remote controller, etc. for a user to perform various operations.

[0076] In the preparatory stage, the face of each participant may be registered in the image processing unit 234. Here, in the case where the voice of each participant is detected by attaching a microphone to each participant, the face of each registered participant may be associated with the number of microphones attached to each participant. With this configuration, voice detection associated with each participant can be achieved.

[0077] When registering the facial image of each participant, the face of each participant is imaged using the camera. At this time, the size of the participant's face and the distance from the camera to the participant are measured in association with each other. This makes it possible to estimate the distance from the camera to the participant based on the size of the participant's face imaged by the camera.

[0078] Note that, in the case where it is difficult to estimate the distance from the camera to the participant by the above method, or in the case where it is not necessary to estimate the distance, it is also conceivable to regard the distance from the camera to the participant as fixed. In addition, it is also conceivable to measure the distance from the camera to the participant by constructing a stereo camera using two or more cameras. This technology is described in "Japanese Patent Application Laid-Open No. 62-080768", for example, but it goes without saying that any method other than this technology can be adopted.

[0079] The image processing unit 234 performs face sensing processing on the image signal (captured image) to specify the position and state of the face of each participant, for example, the direction in which each participant is looking. The direction in which each participant is looking is specified by specifying the face orientation or the line of sight direction. For example, a technique for specifying the face position and the line of sight is described in "Japanese Patent Application Laid-Open No. 2012-226665", but it goes without saying that any method other than this technique can be adopted. In this case, the image processing unit 234 performs personal authentication by comparing the face sensed in the face sensing processing with the pre-registered face, and finds out which participant's face each sensed face coincides with.

[0080] In addition, the image processing unit 234 specifies the position (direction and distance) of the predetermined participant on the plan map based on the position and size of the participant's face within the imaging angle of view. In this case, the direction of the participant viewed from the camera on the plan map can be specified based on the position of the participant's face within the imaging angle of view. In addition, as described above, since the size of the participant's face and the distance from the camera to the participant are measured in association with each other at the time of registration, the distance from the camera to the participant can be specified based on the ratio between these sizes of the face at the time of registration and the current size of the face.

[0081] Note that in the above, facial sensing processing is performed on the image captured by the camera to specify the position of each participant's face, etc., but the participant does not always face the camera. For example, there is a case where the image captured by the camera does not contain the participant's face, for example, when the participant is writing on a whiteboard during a session. In the case where the participant's face cannot be sensed, the point at which the face was last sensed is fixed as the participant's position (facial position). Alternatively, in this case, for example, object tracking can be performed from the point where the face becomes invisible using object tracking technology, and the participant's position (facial position) according to the face on the plane map can be updated. This object tracking technology is described, for example, in "Japanese Patent Application Laid-Open No. 6-169458", but it goes without saying that any method other than this technology can be adopted.

[0082] The voice processing unit 233 processes the voice signal corresponding to each participant and detects the speech of each participant participating in the conversation. Here, in the case of using a microphone attached to each participant to detect the voice, the voice processing unit 233 can easily grasp which participant each detected voice belongs to. Therefore, matching between the voice and the participant is performed.

[0083] At the same time, in the case where a voice is detected by the microphone array, the voice processing unit 233 can grasp which participant each detected voice belongs to. That is, for example, the microphone array is installed at or around the same position as a 360-degree camera that can image the entire surrounding environment, and detects the voice while detecting the direction from which the detected voice is emitted. Therefore, the voice processing unit 233 refers to the position information about the face of each participant detected by the image processing unit 234 as described above, and grasps which participant the voice detected by the microphone array belongs to. Therefore, matching between voice and participant is performed. Note that the technology of detecting the direction of voice using a microphone array is described in, for example, "Japanese Patent Application Laid-Open No. 9-251299", but it goes without saying that any method other than this technology can be adopted.

[0084] The voice processing unit 233 processes the voice signal corresponding to each participant and detects the speech of each participant participating in the conversation. In this case, for example, the volume is measured from the voice signal for each participant, and when the state where the measured volume is equal to or higher than the threshold value continues for a certain period of time or longer, it is determined that there is a speech. When the state where the measured volume is equal to or higher than the threshold value is maintained for less than a certain period of time, the utterance is considered to be related to agreement, refusal, agreement, etc., and it is determined that no speech is made.

[0085] The determination unit 235 detects a conversation based on the speech of each participant obtained by the voice processing unit 233 and the state of each participant obtained by the image processing unit 234. In this case, for example, when the second participant speaks after the first participant speaks or immediately before the end of the speech, and the direction the second participant is looking at coincides with the direction of the first participant, it is determined that a conversation exists between the first participant and the second participant.

[0086] The image generation unit 236 generates a floor plan map image of a space such as a room for a conversation. In this case, the image generation unit 236 adds a participant display showing each participant participating in the conversation detected by the image processing unit 234 to the floor plan map image. Then, the image generation unit 236 dynamically changes the position of the participant display according to the position (face position) of the relevant participant specified by the image processing unit 234. In addition, the image generation unit 236 adds a conversation line connecting the corresponding two participant displays to the floor plan map image based on the conversation determination information from the determination unit 235.

[0087] Figure 5 An example of a plan map image generated by the image generation unit 236 is shown. In this example, the circular displays labeled "A" to "D" represent participant displays showing participants participating in the conversation. Note that in addition to Figure 5 In addition to the symbols shown, the participant display can be displayed in the format of name, registered face, color, or a combination of multiple of these items. In addition, the display format can be fixed, or can be set or switched by the user.

[0088] In addition, in this example, the participant display "A" to "D" indicates the current position of the participants "A" to "D". In addition, this example includes a conversation line linking the participant displays "B" and "C" and indicating that there is a conversation between the participants "B" and "C". Note that in the case where the position of the corresponding participant display moves, the display position of the conversation line moves accordingly. The conversation line displayed on the flat map image in this way can be deleted after being displayed for a certain period of time from the detection of the conversation.

[0089] Figure 6 Another example of a plan map image generated by the image generation unit 236 is shown. Also in this example, the circular displays labeled "A" to "D" show participant displays of participants participating in a conversation. This example contains corresponding conversation lines linking participant displays "B" and "C", "A" and "B", and "A" and "D", and indicates that a conversation exists between the corresponding participants.

[0090] In this case, the number of sessions is accumulated, and a character indicating the number of sessions is appended to each session line. Here, as described above, each time it is determined that a session exists between participants, the count of the number of sessions between the participants is increased. In this example, it is indicated that there are five sessions between participants "B" and "C", one session between participants "A" and "B", and two sessions between participants "A" and "D". The accumulation of the number of sessions may include the number of all sessions since the start of the session, or the number of sessions that occurred within a certain period of time in the past (e.g., within the past five minutes) may be accumulated.

[0091] Note that instead of appending a character indicating the number of sessions to the session line, you can also Figure 7 As shown, the thickness of the conversation line is set according to the number of conversations to indirectly indicate the number of conversations. In addition, both adding characters indicating the number of conversations to the conversation line and setting the thickness of the conversation line according to the number of conversations are also conceivable.

[0092] Figure 8 Another example of a plan map image generated by the image generation unit 236 is shown. Figure 6 In the example shown in , a character indicating the number of speeches is further attached to the participant display corresponding to each participant. In this example, it is indicated that participant "A" has three speeches, participant "B" has six speeches, participant "C" has five speeches, and participant "D" has two speeches.

[0093] Note that instead of displaying additional characters indicating the number of speeches to the participants, Fig. 9 As shown, the size of the participant display is set according to the number of speeches to indirectly indicate the number of speeches. In addition, it is also conceivable to use both the addition of characters indicating the number of speeches to the participant display and the setting of the size of the participant display according to the number of speeches. In addition, it is also conceivable to allow characters indicating the duration of the speech rather than the number of speeches to continue to the participant display.

[0094] return Figure 2 , the image generation unit 236 provides the generated plan map image to the monitor 204, and causes the monitor 204 to display the provided plan map image. In addition, the image generation unit 236 provides the generated plan map image to the server 205, and causes the server 205 to maintain the provided plan map image. In addition, the image generation unit 236 provides the generated plan map image to the mobile terminal 206 instead of the monitor 204 or together with the monitor 204 through the communication unit 237 according to the operation of the user, and causes the mobile terminal 206 to display the provided plan map image. In addition, the image generation unit 236 reproduces the plan map image from the server 205 according to the operation of the user to provide the plan map image to the monitor 204 or the mobile terminal 206, and causes the monitor 204 or the mobile terminal 206 to display the reproduced plan map image. Note that it is also conceivable that an instruction to reproduce the plan map image from the server 205 is given from the mobile terminal 206 via the communication unit 237.

[0095] Will describe Figure 2 The process flow of the information processing device 200 shown in FIG. Fig.10 As shown in (a), as a preliminary preparation, cameras 201-1 to 201-N are installed so that each person participating in the session is imaged, especially the face is imaged. For example, a camera capable of imaging the entire surrounding environment (such as a 360-degree camera) is installed at the center of the participant, or a plurality of cameras are installed so that the entire surrounding environment can be imaged. In this case, adjustments are made so that the orientation and position of the camera on the plane map are in a constant state.

[0096] In addition, microphones 202-1 to 202-M are installed as a preliminary preparation. For example, a microphone is attached to each person participating in the conversation, or a microphone array is installed at the center of the participant. Here, in the case where a microphone is attached to each person participating in the conversation, the microphone is associated with the participant. In addition, the face of each participant is captured using a camera and registered as a preliminary preparation. In this case, imaging is performed while the distance between the participant and the camera is kept constant, and the size of the face is measured and maintained together with the face. Note that in this case, information about the distance can be maintained together. In addition, in this case, when a participant registers for the first time, a new registration is performed; however, when there are already registered participants, it is sufficient to additionally register the remaining participants.

[0097] After completing the pre-preparation, repeat as the session progresses Fig.10 (b) shows the main process. In step ST1, a captured image including each participant participating in the conversation is obtained using cameras 201-1 to 201-N. Then, in step ST2, the image processing unit 234 processes the captured image signal to specify the position and state of the face of each participant, for example, the direction in which each participant is looking (the orientation or sight line of the face).

[0098] Furthermore, in step ST3, the voice of each participant participating in the conversation is detected using the microphones 202-1 to 202-M. Next, in step ST4, the voice processing unit 233 processes the voice signal and detects the utterance of each participant participating in the conversation.

[0099] Then, in step ST5, the determination unit 235 detects a conversation based on the speeches of all or some of the participants obtained by the voice processing unit 233 and the states of all or some of the participants obtained by the image processing unit 234. In this case, for example, when the second participant speaks after the first participant speaks or immediately before the end of the speech, and the direction the second participant is looking at is consistent with the direction of the first participant, it is determined that a conversation exists between the first participant and the second participant. In addition, for example, when the second participant speaks after the first participant speaks, or when the face of the second participant moves up and down (echoes) after the first participant speaks, it is determined that a conversation exists between the first participant and the second participant.

[0100] Next, in step ST6, the image generation unit 236 generates a floor plan map image of a space such as a room for conversation. In this case, a participant display showing each participant participating in the conversation detected by the image processing unit 234 is added to the floor plan map image, and the position of the participant display is dynamically changed according to the position (face position) of the relevant participant specified by the image processing unit 234. In addition, in this case, based on the conversation determination information from the determination unit 235, a conversation line connecting the corresponding two participant displays is added to the floor plan map image, and in addition, a display of the number of speeches of each participant, the number of conversations between the two participants, and the like is added.

[0101] Next, in step ST7, the image generation unit 236 outputs the created plan map image. In this case, the plan map image is provided to the monitor 204, and the monitor 204 is caused to display the plan image. In addition, in this case, the plan map image is provided to the server 205, and the server 205 is caused to accumulate and hold the sequentially generated plan map images. In addition, in this case, the plan map image is sent to the mobile terminal 206 through the communication unit 237, and the mobile terminal 206 is caused to display the plan map image.

[0102] As mentioned above, in Figure 2 In the information processing device 200 shown, the determination unit 235 determines the state and existence of a conversation based on the speech of each participant participating in the conversation and the state detected by image processing. Therefore, compared with the technology of determining the state and existence of a conversation only by the speech of the participant, it is possible to determine the state and existence of a conversation with higher accuracy.

[0103] In addition, Figure 2 In the information processing device 200 shown, the plane map image generated by the image generation unit 236 includes a participant display showing each participant participating in the conversation, and the plane map image includes a conversation line displayed based on two corresponding participants connected by the detected conversation. Therefore, it is easy to visually recognize who has had a conversation with whom.

[0104] In addition, Figure 2 In the information processing device 200 shown, the image processing unit 234 processes the image signal to further detect the position of each participant participating in the conversation, and the image generation unit 236 changes the position of the participant display showing each participant on the planar map image according to the detected position of each participant. Therefore, the position of each participant participating in the conversation can be easily visually recognized.

[0105] In addition, Figure 2In the information processing device 200 shown, the participant display included in the plan map image generated by the image generation unit 236 has a size corresponding to the number of speeches of the corresponding participant, or the participant display is attached with characters showing the number of speeches of the corresponding participant. Therefore, the extent of the number of speeches of each participant can be easily visually recognized.

[0106] In addition, Figure 2 In the information processing device 200 shown, the conversation line included in the plane map image generated by the image generation unit 236 has a thickness corresponding to the number of conversations between the corresponding participants, or the conversation line is attached with characters showing the number of conversations between the corresponding participants. Therefore, the degree of the number of conversations between each two participants can be easily visually recognized.

[0107] In addition, Figure 2 In the information processing apparatus 200 shown, the floor plan map image generated by the image generating unit 236 is provided to the monitor 204 as a display unit, and the floor plan map image is displayed on the monitor 204. Therefore, for example, a facilitator of a conversation can check the floor plan map image having various types of information about the speech and the conversation in the monitor 204.

[0108] In addition, Figure 2 In the information processing apparatus 200 shown, the plan map image generated by the image generating unit 236 is provided to the server 205 and is held in the server 205. Therefore, the past plan map image can be reproduced from the server 205 and provided to, for example, the monitor 204 to be displayed thereon, so that the past plan map image can be easily checked.

[0109] In addition, Figure 2 In the information processing device 200 shown, as needed, a floor plan map image or a past floor plan map image read from the server 205 is provided to the mobile terminal 206 by the image generation unit 236 through the communication unit 237, and the current or past floor plan map image is displayed on the mobile terminal 206. Therefore, for example, the leader of the conversation can check the floor plan map image having various types of information about the speech and the conversation in the mobile terminal 206.

[0110] <3. Modification>

[0111] Note that in the above-described embodiment, personal authentication is performed by pre-registering the faces of the participants and comparing the faces detected from the images with the registered faces. However, in this process, appropriate display cannot be performed in the case where an unregistered person participates. Therefore, it is also conceivable that, in the case where personal authentication fails, the face that failed in personal authentication is newly registered as a participant.

[0112] In this case, since there is no relationship between face size and distance registered for the new participant, the exact distance cannot be known unless a device such as a stereo camera capable of measuring distance is used. Therefore, the distance from the camera to the new participant is fixed, or the distance to the camera is simply calculated using the average size of the face.

[0113] Furthermore, in the above-described embodiment, an example has been shown where a conversation occurs in one group; however, there is also a case where a conversation occurs in a plurality of divided groups in a large room (venue). In this case, participants are sometimes exchanged between groups. In this case, as Fig.11 As shown, cameras are arranged for each group, and faces appearing on the camera of a particular group are designated as participants of that particular group. Fig.11 In this example, Group 1 consists of four participants "A" to "D", Group 2 consists of four participants "E" to "H", and Group 3 consists of three participants "I" to "K".

[0114] Then, the location of the participants, the number of speeches, and the number of conversations are displayed for each group. As for determining whether the members of the group have been exchanged, the exchange can be determined at the moment when the face of a specific participant appears on the camera used to image the other group, or the exchange can be determined after a preset time period has passed after the exchange. In this case, the time when the face first appears on the camera is also recorded, so that the exchange is confirmed after the preset time period has passed, and then the number of conversations, etc. are displayed.

[0115] Furthermore, in the above-described embodiments, a conversation that takes place in one room (space) has been described. However, there are also cases where a participant located in another room participates in a conversation through a video conference, a telephone conference, a web conference, etc. In this case, Fig.12 As shown in FIG, a floor plan of multiple spaces where participants are located is prepared, and the locations of all participants are displayed. Fig.12 In the example shown in , four participants "A" through "D" appear in room 1, while two participants "F" and "G" appear in room 2.

[0116] Conversation lines are displayed similarly to what is described above with respect to conversations between participants in the same space, but are displayed across the planar map with respect to conversations between participants located in different spaces. Note that although Fig.12 The example in shows an example of two rooms, but this is similarly applicable to the case of three or more rooms. In addition, the establishment of a session between participants can be determined based on whether the participants located in different rooms are looking at each other's monitors.

[0117] Furthermore, in the above-described embodiment, the positions of the participants of the conversation, the number of speeches, and the number of conversations are displayed on a flat map; however, Fig.13 As shown, information other than the participant display, number of speeches, and number of sessions can be displayed on the flat map and areas outside the map. Fig.13 As shown, the content of the speech can be translated into a sentence and displayed from the acquired speech by speech recognition. In this case, as the content of the speech, for example, it is conceivable to extract and display only keywords, or to display the entire sentence of the speech content.

[0118] In addition, in the above-described embodiment, the participants of the conversation are displayed at the positions on the plane map observed on the camera images of the participants. Fig.14 As shown in (a) and 14 (b), the position of the participant display showing each participant can be specified according to the number of conversations of each participant. For example, a person with a larger total number of conversations is arranged closer to the center of the screen, while a person with a smaller total number of conversations is arranged on the outside of the screen. In addition, in the case where the number of conversations between two participants is large, the two participants are arranged closer to each other, and in the case where the number of conversations between the two participants is small, the two participants are arranged farther from each other.

[0119] As an arrangement method, for example, Coulomb's law equation in magnetic force is used. Coulomb force F is represented by the following formula (1). Here, k represents a proportional constant, m1 and m2 represent the magnetic quantities of objects 1 and 2, respectively, and r represents the distance between the two objects.

[0120] F = km1m2 / r 2 ...... (1)

[0121] First, a virtual object 0 is arranged in the center of the plan view as a magnetic quantity m0, and in the presence of four participants, objects 1 to 4 are given magnetic quantities m1 to m4 proportional to the number of sessions, respectively. As the number of sessions increases, the magnetic quantity becomes larger. The magnetic quantities m0 and m1 to m4 attract each other through the Coulomb force calculated by formula (1). On the other hand, although m1 to m4 repel each other, the repulsive force decreases as the number of sessions between each other increases, and therefore, the magnetic quantity corresponding to the number of sessions is transformed into an inverse, such as m1'=1 / m1. By replacing m1 in formula (1) with m1', the calculated Coulomb force repels each other with a force inversely proportional to the number of sessions.

[0122] Using the attraction force relative to m0, the positions of all participants try to approach the center. On the other hand, the positions of all participants repel each other with a force inversely proportional to the number of sessions, and thus try to move away from each other. In a state where the balance between the attraction and repulsion forces is maintained, the positions of the participants stop and are specified. When the number of sessions changes, the displayed positions of the participants on the planar map also change.

[0123] Therefore, as the number of conversations increases, the location of the participant display is arranged closer to the center of the flat map picture, so that it is easy to visually identify who has a larger number of conversations. In addition, since the participant displays of the two participants are closer to each other as the number of conversations between the two participants increases, it is easy to visually identify who has had a larger number of conversations with whom. For example, Fig.14 (a) shows a state where the number of sessions of participant "A" is small and each of the remaining three persons has a larger number of sessions, and it can be easily visually recognized from the figure that such a state has occurred. Fig.14 (b) shows a state in which the number of sessions of participant "A" is larger and participant "A" serves as a kind of session center, and it can be easily visually recognized from the figure that such a state has occurred.

[0124] Note that even in the case where the position of the participant display showing each participant is specified according to the number of conversations of each participant in this way, characters showing the number of conversations may be attached to the conversation line, or the thickness of the conversation line may be set according to the number of conversations, and further, characters showing the number of speeches of each participant may be attached to each participant display, or the size of each participant display may be formed to a size corresponding to the number of speeches of each participant or the duration of the speech (refer to Figures 6 to 9 ).

[0125] For example, Fig.15 (a) and 15(b) show the Fig.14 (b) The case where the size of the participant display of each participant is changed according to the duration of the speech. Fig.15 In the case of (a), it can be seen that participant "A" has a long speech duration and only keeps making a scene like a solo performance. Fig.15 In case (b), we can see a scenario where participant "A" has many conversations but not so many long speaking durations, that is, a scenario where participant "A" acts as a kind of moderator or facilitator of the conversation group.

[0126] In this way, by configuring the participant display showing each participant at a position corresponding to the number of sessions, the status of the session can be objectively grasped. A viewer such as a guide can be allowed to switch between a plurality of display methods for a floor plan map (such as a floor plan map based on a position observed by a camera and a floor plan map based on a position corresponding to the number of sessions).

[0127] In addition, although not described above, a device for performing evaluation and displaying the evaluation value of the evaluation will be described according to the situation of the speech or conversation of the participant. Fig.16 As shown, for each conversation group, the evaluation level of the conversations in each conversation group is displayed. Fig.16 In the table, the evaluation levels are displayed in three stages, A, B, and C, starting from the top level. For example, the group displayed on the upper right is evaluated as "comprehensive evaluation A".

[0128] In this way, when the evaluation level is displayed for each group, the facilitator can easily know which group's conversation should be supported. Fig.17 The evaluation level of each participant is shown, and it can be known which participant has hesitated to participate in the conversation.

[0129] Regarding the evaluation of each group, the evaluation is performed based on, for example, the number of speeches and the number of conversations. For example, the evaluation value Et of each group is calculated using the following formula (2). Here, Ngs represents the number of speeches, Ngc represents the number of conversations, and αg and βg represent coefficients. The evaluation level is calculated by processing the evaluation value calculated by formula (2) using a predefined threshold value. Note that the evaluation value can be displayed as it is without converting it into an evaluation level.

[0130] Et=αg·Ngs +βg·Ngc......(2)

[0131] In addition, the evaluation value Em of each participant is similarly calculated from the number of speeches and the number of conversations using the following formula (3). Here, Nms represents the number of speeches, Nmc represents the number of conversations, and αm and βm represent coefficients. The evaluation level is calculated by processing the evaluation value calculated by formula (3) using a predefined threshold value. Note that the evaluation value can be displayed as it is without converting it into an evaluation level.

[0132] Em =αm·Nms +βm·Nmc...... (3)

[0133] In addition, for the evaluation value of the group, since a lower evaluation is given if only some people are motivated, the evaluation value of each participant can be added to the evaluation. For example, the evaluation value Et' of each group is calculated based on the following formula (4). Here, min(Em) represents the lowest evaluation value among all participants participating in the relevant conversation group, and γg represents a coefficient. With this configuration, when there is a participant with a low evaluation value, the evaluation of the entire group is lowered.

[0134] Et' =αg·Ngs +βg·Ngc +γg·min (Em) ... (4)

[0135] Note that although the number of utterances is used in formulas (2) to (4), the duration of utterances may be used instead of the number of utterances, or both the number of utterances and the duration of utterances may be used.

[0136] In addition, in the above, an example has been shown in which the participants and the conversation group participating in the conversation are evaluated; however, it is also conceivable to evaluate the content that has affected the end of the conversation together with or separately from the above evaluation.

[0137] Furthermore, in the above, an example has been shown in which monitor 204 and mobile terminal 206 display evaluation values ​​of participants participating in a conversation and conversation groups; however, in the case where a teacher monitors multiple conversation groups of students, it is conceivable to present the following information to the teacher or students.

[0138] As information to be presented to the teacher, information for suggesting intervention to the teacher can be conceived. In this case, the state in which each group is located is visualized. In addition, in this case, priorities are assigned so that the group that needs visualization the most among multiple groups is known. For example, a group including students who do not make any speeches at all is given a higher priority. The teacher can place priority on visualizing the state of the group with a higher priority to understand the state in which the group is located.

[0139] In addition, as information to be presented to the teacher, information on the score or grade automatically assigned to each student or each group can be conceived. The score or grade is appropriately displayed as, for example, a cumulative or instant score or grade according to the teacher's operation.

[0140] Meanwhile, as information to be presented to students, information for assisting students can be conceived. For example, a summary of the conversation up to the current time is displayed to prompt students to participate in the next conversation. In addition, as information to be presented to students, information for warning students can be conceived. For example, the number of speeches and the amount of speeches that contribute to the entire process are displayed.

[0141] In addition, in the above-described embodiment, an example has been shown in which information on speeches, conversations, etc. is used as information for checking the situation of conversations; however, it is also conceivable to also utilize information on utterances related to agreement, refusal, agreement, etc. as information for checking the situation of conversations. For example, the number of utterances related to agreement, refusal, agreement, etc. may also be counted so that the counted number of utterances is displayed on the plan map image together with the number of conversations and the number of speeches.

[0142] In addition, in the above-mentioned embodiments, an example in which the microphone and the camera are configured as different bodies has been shown; however, it is also conceivable that these microphones and cameras are integrated. For example, conceivable examples include an example in which the microphone is integrated with a 360-degree camera, an example in which the microphone is integrated with each of a plurality of cameras covering the entire surrounding environment, and the like.

[0143] Furthermore, in the above-described embodiment, the direction in which the participant is looking has been indicated as the state of the participant detected by processing the image signal; however, the state of the participant to be detected is not limited to this case. For example, as the state of the participant, the participant's facial expression (happy, angry, sad, pleasant, surprised, etc.), posture (finger, hand and arm movement), facial movement (nodding in the up and down directions (agreement, agreement), shaking head in the left and right directions (rejection)), biometric information (changes in body temperature and changes in pulse), etc. can be detected. In addition, for example, a person who nods a lot is likely to actively participate in the conversation even if the person does not make a speech during the conversation; therefore, such information can be counted and displayed together with the number of conversations and the number of speeches.

[0144] Furthermore, in the above-described embodiments, an example has been shown of detecting the speech of a participant of a conversation from a voice signal and detecting the status of a participant of a conversation from an image signal; however, it is also conceivable to detect both the speech of a participant and the status of a participant from an image signal.

[0145] also, Figures 18 to 23 An example of screen display on the monitor 204 or the mobile terminal 206 is shown. Fig.18 The screen display example in shows the status of conversations in PODs (conversation groups) A to D. POD identifiers of PODs A to D are displayed corresponding to the displays of the respective PODs. In addition, in each POD display, as indicated by arrow P1, a participant is represented by an image including a face.

[0146] Furthermore, in each POD display, the timeline of the discussion is displayed in the area indicated by the arrow P2. In this case, as indicated by the arrow P3, a thumbnail image is arranged at the head of each speech, and further, as indicated by the arrow P4, name characters are arranged so that it is possible to know who made the speech. Note that either the thumbnail image or the name characters may be adopted. Whenever a speech is made, the display in the area displaying the timeline of the discussion is automatically updated so that the new speech is displayed at the latest position; however, the user can check the past speeches by performing a scroll operation on the screen.

[0147] Fig.19 The screen display example is Fig.18 The display example of similarly shows the conversation status of POD (conversation group) A to D. The POD identifiers of POD A to POD D are displayed corresponding to the display of each POD. In addition, in each POD display, as shown by arrow P1, the participants are represented by images including faces. In addition, in each POD display, the timeline of the discussion is displayed in the area indicated by arrow P2.

[0148] In this example, in addition, in the participant display represented by the arrow P1, the color of the image of each participant changes dynamically according to the number of speeches. For example, a participant with a large number of speeches is colored yellow, a participant with a medium number of speeches is colored red, a participant with a small number of speeches is colored green, and so on. Note that in the example shown, the color change is represented by a change in the pattern superimposed on the image. Of course, the pattern superimposed on the image of each participant can be dynamically changed according to the number of speeches. In addition, not only the number of speeches, but also the number of agreed or echoed voices and the number of conversations can be similarly represented by color and pattern. According to this configuration, for example, a teacher as a user can easily visually check the number of speeches of each participant in each POD, etc., and can easily intervene in the POD where the teacher feels a deadlock in the discussion.

[0149] Fig. 20 The screen display example is Fig.18 The display example of similarly shows the conversation status of POD (conversation group) A to D. The POD identifiers of POD A to POD D are displayed corresponding to the display of each POD. In addition, in each POD display, the timeline of the discussion is displayed in the area indicated by the arrow P2. Moreover, in this example, in addition to the status displayed for the participant with a facial image, etc., the area indicated by the arrow P1 can be switched to a shared screen in the POD. For example, a teacher as a user can perform the operation of switching the screen as described above, and understand what material or description is used as the basis for performing a discussion in each POD.

[0150] Note that in each POD display, the speeches of the session participants displayed in the area indicated by the arrow P2 are accumulated in the server 205, and a timeline is sent to each student's mobile terminal 206 in the form of minutes at the end of the course, for example. When a student clicks a speech in the timeline, the saved video (camera image) can also be reviewed from the time of the clicked speech. In this case, the time point of each speech is maintained in association with the time axis of the camera image.

[0151] Fig.21 The screen display example in shows the state of a conversation between four participants. In the area indicated by arrow Q1, the participants are represented by images including faces. In addition, the timeline of the discussion is displayed in the area indicated by arrow Q2. In this case, as shown by arrow Q4, name characters are arranged at the head of each speech so that it is possible to know who made the speech. Thumbnails can be used instead of name characters. In addition, in the area indicated by arrow Q3, slides and text on the board used in the discussion are displayed. Note that in the example shown, among the four participants, the participant corresponding to the image displayed in the upper right part of the area shown by arrow Q1 faces backward because the participant is writing on the board at this point in time; therefore, the facial image cannot be captured, and character tracking is performed so that the image is cropped. Note that it is also conceivable to correspond the participant to the speech by adopting the same color between the color of the frame surrounding the image of each participant and the color of the character part of the name at the head of each speech. In the example shown, the line type is changed instead of the frame color to indicate the correspondence.

[0152] Fig. 22 The image display example in shows the status of a conversation between four participants. The facial images of the four participants are displayed at the four corners of a rectangular frame. In this case, circles corresponding to the respective participants are displayed, and the circles move on a diagonal line. Note that in order to make it easy to know the correspondence between the participants and the circles, the same color is adopted between the color of the circle and the color of the frame of the image of the participant corresponding to the circle. In the example shown, the line type is changed instead of the frame color to indicate the correspondence. The circle of each participant expands as the number of speeches increases, and in addition, the distance from the center of the rectangular frame to the center of the circle shortens as the number of conversations increases. In the example shown, it can be known that the participant displayed in the upper right image has a larger number of speeches and a larger number of conversations, while the participant displayed in the upper left image has a smaller number of speeches and a smaller number of conversations.

[0153] Fig.23The image display example in shows the status of a conversation between four participants. Facial images representing the four participants are displayed. In this case, the facial image of the participant having a larger number of speeches is displayed larger. In addition, conversation lines linking the participants are displayed, and are displayed thicker as the number of conversations increases. In the example shown, it can be known that the participant shown in the upper left image has a larger number of speeches, and in particular has a larger number of conversations with the participant shown in the lower right image. In addition, it can be known that the participant shown in the upper right image has a medium number of speeches, but in particular has a larger number of conversations with the participant shown in the lower right image, and has no conversations with the participant shown in the lower left image.

[0154] Reference numerals list

[0155] 100 Information processing device

[0156] 101 Speech Processing Unit

[0157] 102 Image Processing Unit

[0158] 103 judgment unit

[0159] 200 Information processing device

[0160] Cameras 201-1 to 201-N

[0161] 202-1 to 202-M Microphones

[0162] 203 Image and Voice Control System

[0163] 204 monitors

[0164] 205 Server

[0165] 206 Mobile Terminal

[0166] 231 Control Unit

[0167] 232 User operation unit

[0168] 233 Voice Processing Unit

[0169] 234 Image Processing Unit

[0170] 235 judgment unit

[0171] 236 Image Generation Unit

[0172] 237 Communication Unit

Claims

1. An information processing device, comprising: a speech processing unit that detects utterances of a plurality of participants of a conversation from the speech signal; an image processing unit that detects the states of the plurality of participants of the conversation from an image signal; a determination unit, configured to determine the state of the conversation based on the speeches of the plurality of participants and the states of the plurality of participants; as well as an image generation unit that generates a plan map image of a space for the session, wherein the plan map image includes a participant display representing each participant participating in the session, wherein, when generating the plan map image, a virtual object having a specific magnetic quantity is arranged in the center of the plan map image, a first magnetic quantity proportional to the number of sessions of each participant is given to each participant display, so that an attractive force corresponding to Coulomb's law is generated between each participant display and the virtual object, and a second magnetic quantity proportional to the inverse of the number of sessions of each participant is given to each participant display, so that a repulsive force corresponding to Coulomb's law is generated between each participant display, and In which, using the attraction relative to the virtual object, the position displayed by each participant tries to approach the center, and the respective participant displays repel each other with the repulsive force. While maintaining the balance between the attraction and the repulsive force, the position displayed by each participant stops and is specified.

2. The information processing device according to claim 1, wherein The state of the session includes the session relationship between the participants of the session.

3. The information processing device according to claim 1, wherein The state of the conversation includes at least one of the content of the speech, the number of speeches, or the duration of the speech.

4. The information processing device according to claim 1, wherein The state of the session includes the establishment of the session.

5. The information processing device according to claim 4, wherein When one of the participants is talking while looking at the other party of the conversation, the determination unit determines that the conversation has been established.

6. The information processing device according to claim 1, wherein The image processing unit detects a direction in which one of the participants is looking as one of the states of the participant.

7. The information processing device according to claim 6, wherein The image processing unit detects a direction in which one of the participants is looking based on a facial orientation or a sight line direction of one of the participants.

8. The information processing device according to claim 6, wherein The determination unit determines that a conversation exists between the first participant and the second participant when the second participant speaks after the first participant speaks and the direction the second participant is looking at coincides with the direction of the first participant.

9. The information processing device according to claim 1, wherein The voice processing unit measures a volume from a voice signal, and determines that there is a speech if a state in which the measured volume is equal to or higher than a threshold value continues for a certain period of time or longer.

10. The information processing device according to claim 9, wherein The voice processing unit determines that an utterance related to approval, rejection, or agreement is made in a case where a state in which the measured volume is equal to or higher than a threshold value is maintained for less than the certain period of time. 11 . The information processing apparatus according to claim 1 , wherein the plan map image includes a conversation line displayed to connect two corresponding participants based on the conversation that has been detected.

12. The information processing device according to claim 11, wherein The image processing unit processes the image signal to further detect the position of each participant participating in the conversation, and The image generation unit changes the position of a participant displayed on the plan map image representing each participant according to the detected position of each participant.

13. The information processing device according to claim 11, wherein The image generation unit changes a position at which a participant representing each participant is displayed on the plan map image according to the number of sessions of each participant.

14. The information processing device according to claim 13, wherein The image generation unit arranges a position of a participant display representing each participant closer to a center of the plan map image as the number of sessions increases.

15. The information processing device according to claim 13, wherein The image generation unit causes an interval between participant displays of two participants to become shorter as the number of conversations between the two participants increases.

16. The information processing apparatus according to claim 11, wherein The participant display has a size corresponding to the number of speeches made by the corresponding participant.

17. The information processing apparatus according to claim 11, wherein The participant display is appended with characters indicating the number of speeches made by the corresponding participant.

18. The information processing device according to claim 11, wherein The conversation line has a thickness corresponding to the number of conversations between corresponding participants.

19. The information processing apparatus according to claim 11, wherein The conversation line is appended with characters indicating the number of conversations between corresponding participants.

20. The information processing device according to claim 11, further comprising A display unit displays the plan map image generated by the image generation unit.

21. The information processing device according to claim 11, further comprising A server that holds the plan map image generated by the image generating unit.

22. The information processing device according to claim 11, further comprising A communication unit transmits the plan map image generated by the image generation unit to a mobile terminal.

23. An information processing method, comprising: A speech processing step of detecting, by a speech processing unit, the speech of a plurality of participants of a conversation from a speech signal; An image processing step of detecting the states of a plurality of participants of a conversation from an image signal by an image processing unit; a detection step of determining the state of a conversation based on the speeches of the plurality of participants and the states of the plurality of participants by a conversation detection unit; as well as a step of generating a plan map image of a space for the session by an image generating unit, wherein the plan map image includes a participant display representing each participant participating in the session, wherein, when generating the plan map image, a virtual object having a specific magnetic quantity is arranged in the center of the plan map image, a first magnetic quantity proportional to the number of sessions of each participant is given to each participant display, so that an attractive force corresponding to Coulomb's law is generated between each participant display and the virtual object, and a second magnetic quantity proportional to the inverse of the number of sessions of each participant is given to each participant display, so that a repulsive force corresponding to Coulomb's law is generated between each participant display, and In which, using the attraction relative to the virtual object, the position displayed by each participant tries to approach the center, and the respective participant displays repel each other with the repulsive force. While maintaining the balance between the attraction and the repulsive force, the position displayed by each participant stops and is specified.

24. A program product, It enables the computer to be used as: a speech processing device that detects utterances of a plurality of participants of a conversation from a speech signal; an image processing device that detects the states of the plurality of participants of the conversation from the image signal; A determination device for determining the speeches of the plurality of participants and the states of the plurality of participants; as well as an image generating device for generating a plan map image of a space for the session, wherein the plan map image includes a participant display representing each participant participating in the session, wherein, when generating the plan map image, a virtual object having a specific magnetic quantity is arranged in the center of the plan map image, a first magnetic quantity proportional to the number of sessions of each participant is given to each participant display, so that an attractive force corresponding to Coulomb's law is generated between each participant display and the virtual object, and a second magnetic quantity proportional to the inverse of the number of sessions of each participant is given to each participant display, so that a repulsive force corresponding to Coulomb's law is generated between each participant display, and In which, using the attraction relative to the virtual object, the position displayed by each participant tries to approach the center, and the respective participant displays repel each other with the repulsive force. While maintaining the balance between the attraction and the repulsive force, the position displayed by each participant stops and is specified.

Images