Conference support control device and conference support method

Abstract

To help keep meetings moving smoothly when the meetings get stalled.SOLUTION: A meeting help control device 80 includes a situation detection unit 82 that extracts the status of participants from video footage of a conference venue attended by multiple participants and from the captured audio of the venue, and detects whether the venue status is good or bad on the basis of the proportion of participants in a specified status among all participants, and a necessity determination unit 83 that determines whether it is necessary to accept mid-venue participation in the venue on the basis of the venue status detected by the situation detection unit 82.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a conference support control device and a conference support method. [Background technology]

[0002] Conventionally, there have been conference support systems that assist conferences. A technology is known that allows conference participants to share their understanding and smoothly advance the conference (see, for example, Patent Document 1). The technology described in Patent Document 1 provides conference participants with conference support information generated based on the speech of the conference participants. Frequently occurring keywords are extracted from the speech, and the degree of coherence of the discussion is determined as a convergence degree based on the trend of changes in the keywords. By providing participants with conference support information including the convergence degree, the participants are able to appropriately recognize the current state of the conference. By participants sharing their understanding, the discussion advances smoothly. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-112545 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 uses information obtained from the voices of the participants. Therefore, when a participant is stuck for words, it is difficult to present appropriate information to the participant. As such, there is room for improvement in supporting the smooth progress of a meeting when the meeting is stalled.

[0005] The present disclosure aims to provide support for smoothly progressing a meeting when the meeting is at a standstill. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objectives, the conference support control device of the present disclosure comprises a situation detection unit that extracts the status of participants from video footage of a conference venue attended by multiple participants and from audio captured from the venue, and detects whether the status of the venue is good or bad based on the proportion of participants in a specified status among all participants, and a necessity determination unit that determines whether or not it is necessary to accept mid-venue participation in the venue based on the status of the venue detected by the situation detection unit.

[0007] The conference support method of the present disclosure includes a situation detection step of extracting the status of participants from video footage of a conference venue attended by multiple participants and from audio captured from the venue, and detecting whether the status of the venue is good or bad based on the proportion of participants in a specified status among all participants, and a necessity determination step of determining whether or not it is necessary to accept mid-venue participation in the venue based on the status of the venue detected by the situation detection step. Effect of the Invention

[0008] According to the present disclosure, when a meeting is at a standstill, assistance can be provided to help the meeting proceed smoothly. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a conference support system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the information terminal device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of the configuration of a server device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of the conference supporting device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a display of visual information. [Figure 6] FIG. 6 is a diagram showing another example of display of visual information. [Figure 7] FIG. 7 is a flowchart showing the conference supporting method according to the first embodiment. [Figure 8] FIG. 8 is a block diagram illustrating an example of the configuration of a conference supporting device according to the second embodiment. [Figure 9] FIG. 9 is a flowchart showing a conference supporting method according to the second embodiment. [Figure 10] FIG. 10 is a block diagram illustrating an example of the configuration of a conference supporting device according to the third embodiment. [Figure 11] FIG. 11 is a flowchart showing a conference supporting method according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a conference support control device and a conference support method according to the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiment.

[0011] [First embodiment] (Meeting support system) 1 is a schematic diagram showing an example of the configuration of a conference support system according to a first embodiment. The conference support system 1 includes one or more information terminal devices 10 and a server device 30. The conference support system 1 is used, for example, in a conference held face-to-face with a plurality of participants (hereinafter referred to as a "face-to-face conference"), a conference held online via a conference application (hereinafter referred to as a "WEB conference"), or a hybrid conference that combines these. A WEB conference includes a virtual space conference that is linked to a face-to-face conference and set up in a virtual space.

[0012] (Information terminal device) 2 is a block diagram showing a configuration example of an information terminal device according to the first embodiment. Each information terminal device 10 starts a conference application to hold a face-to-face conference or a WEB conference with another information terminal device 10 via a server device 30. The information terminal device 10 is, for example, an information terminal device such as a desktop or notebook personal computer, a smartphone, or a tablet terminal used when a participant participates in a face-to-face conference or a WEB conference.

[0013] In this embodiment, all participants use the information terminal device 10 in any of the face-to-face conference, the Web conference, and the hybrid conference.

[0014] The information terminal device 10 includes a camera 11, a microphone 12, an audio output unit 13, an operation unit 14, a biosensor 15, a communication unit 18, a display unit 19, and a terminal control device 20. The biosensor 15 is not an essential component.

[0015] The camera 11 photographs the participants facing the information terminal device 10. The camera 11 photographs the participants using the information terminal device 10. The camera 11 is, for example, an in-camera provided on the display unit 19 side. The camera 11 may be an external camera connected to the information terminal device 10. The camera 11 outputs the photographed image to the photographing control unit 21 of the terminal control device 20.

[0016] The microphone 12 picks up the voices of the participants using the information terminal device 10. The microphone 12 is, for example, a built-in microphone arranged in the information terminal device 10. The microphone 12 may be an external microphone connected to the information terminal device 10. The microphone 12 outputs audio information related to the picked-up voice to the voice acquisition unit 22 of the terminal control device 20.

[0017] The audio output unit 13 is an audio output device that outputs the audio of the participant or other participants. The audio output unit 13 is, for example, a built-in speaker arranged in the information terminal device 10. The audio output unit 13 may be an external speaker or headphones connected to the information terminal device 10.

[0018] The operation unit 14 is an input device capable of inputting various operations to the information terminal device 10. The operation unit 14 outputs an operation signal corresponding to the received input operation to the terminal control device 20. The operation unit 14 includes, for example, a keyboard, a mouse, a touch panel arranged over the display unit 19, buttons, switches, etc. The operation unit 14 can accept, for example, a key input operation, a click operation, or a touch operation.

[0019] The biosensor 15 is a sensor capable of detecting, for example, the heart rate or the amount of sweat of a participant using the information terminal device 10. The biosensor 15 outputs the detected sensor data to the bioinformation acquisition unit 29 of the terminal control device 20.

[0020] The communication unit 18 is a communication unit for performing wired communication or wireless communication. The communication unit 18 performs communication by a communication method using, for example, Wi-Fi (registered trademark) or a mobile phone line, and is connected to an intranet or the Internet. The communication of the communication unit 18 is controlled by a communication control unit 26 of the terminal control device 20. The communication unit 18 performs data communication with the server device 30. More specifically, the communication unit 18 transmits to the server device 30 the voice of the participant picked up by the microphone 12, the video of the participant captured by the camera 11, and the biometric information detected by the biometric sensor 15. The communication unit 18 receives the video of the shared screen material, and the video and audio of the other participants from the server device 30.

[0021] The display unit 19 displays various types of video including characters and images. The display unit 19 is, for example, a display including a liquid crystal display, an organic EL (Electro-Luminescence), etc. The display unit 19 displays a conference screen including the video of the participant and other participants, and shared materials.

[0022] (Terminal Control Device) The terminal control device 20 controls each part of the information terminal device 10. The terminal control device 20 has, for example, a control device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The terminal control device 20 executes a program that controls the operation of the information terminal device 10 according to the present disclosure. The terminal control device 20 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The terminal control device 20 loads a stored program into a memory and executes instructions included in the program. The terminal control device 20 includes an internal memory such as the above-mentioned RAM, and the internal memory is used for temporary storage of data in the terminal control device 20. The terminal control device 20 may be realized by a combination of hardware and software.

[0023] The terminal control device 20 has, as functional blocks realized by circuits or programs executed, an imaging control unit 21, an audio acquisition unit 22, an audio output control unit 23, an operation control unit 24, a storage unit 25, a communication control unit 26, a display control unit 27, an application control unit 28, and a biometric information acquisition unit 29. The biometric information acquisition unit 29 is not an essential component.

[0024] The photographing control unit 21 controls the camera 11 to shoot an image. The photographing control unit 21 acquires image data of the image shot by the camera 11. More specifically, the photographing control unit 21 acquires image data of the participants using the information terminal device 10, which is shot by the camera 11. The image data acquired by shooting with the camera 11 is video data of a moving image.

[0025] The voice acquiring unit 22 acquires voice information related to the voice detected by the microphone 12. The voice acquiring unit 22 acquires voice information related to the speech of the participant using the information terminal device .

[0026] The audio output control unit 23 controls the output of audio in the audio output unit 13. For example, the audio output control unit 23 controls the audio output unit 13 to output the audio of the participant and other participants.

[0027] The operation control unit 24 acquires an operation signal corresponding to an operation on the operation unit 14 from the operation unit 14. The operation control unit 24 outputs a control signal corresponding to the acquired operation signal to each unit.

[0028] The storage unit 25 stores information such as the contents of calculations and programs of the terminal control device 20. The storage unit 25 is composed of, for example, a RAM, a main storage device such as a ROM, and a storage device such as an SSD (Solid State Drive).

[0029] The communication control unit 26 controls the communication unit 18 to control data communication between the information terminal device 10 and an external device. More specifically, the communication control unit 26 controls, for example, the communication unit 18 to control communication between the information terminal device 10 and the server device 30. The communication control unit 26 controls, for example, the communication unit 18 to transmit and receive information about a conference with another information terminal device 10 to and from the server device 30. More specifically, the communication control unit 26 controls the communication unit 18 to transmit to the server device 30 the voice of the participant picked up by the microphone 12, the video of the participant captured by the camera 11, and the biometric information detected by the biometric sensor 15. The communication control unit 26 controls the communication unit 18 to receive from the server device 30 the video of the shared screen material and the video and audio of the other participants.

[0030] The display control unit 27 controls the display of the image on the display unit 19. For example, the display control unit 27 controls the display unit 19 to display information resulting from the execution of a conference application (hereinafter referred to as a "conference screen").

[0031] The display control unit 27 may generate the conference screen and cause the display control unit 27 to display it on the display unit 19. The display control unit 27 may receive the conference screen generated by the server device 30 as video data and cause the display control unit 27 to display it on the display unit 19.

[0032] The application control unit 28 controls the operation of various applications that run on the information terminal device 10. The application control unit 28 controls the operation of an application used for a conference, such as a conference tool. The application control unit 28 may run an application such as a conference tool in the server device 30. The application control unit 28 may cause the information terminal device 10 and the server device 30 to cooperate with each other to operate an application such as a conference tool in a distributed manner. In other words, the application control unit 28 may be a functional block realized only in the information terminal device 10, a functional block realized only in the server device 30, or a functional block realized by the cooperation of the information terminal device 10 and the server device 30.

[0033] The biometric information acquiring unit 29 acquires biometric information from the biometric sensor 15 .

[0034] (Server device) 3 is a block diagram showing a configuration example of a server device according to the first embodiment. The server device 30 is, for example, a server device installed by a company that provides a conference tool. The server device 30 includes a communication unit 31 and a server control device 40.

[0035] The communication unit 31 is a communication unit for performing wired communication or wireless communication. The communication unit 31 performs communication by a communication method using, for example, Wi-Fi or a mobile phone line, and is connected to an intranet or the Internet. The communication of the communication unit 31 is controlled by the communication control unit 41 of the server control device 40. The communication unit 31 performs data communication with the information terminal device 10. More specifically, the communication unit 31 receives the voice of the participant, the video of the participant captured by the camera 11, and the biometric information detected by the biometric sensor 15 from the information terminal device 10. The communication unit 31 transmits the video of the shared screen material, and the video and audio of the other participants to the information terminal device 10.

[0036] (Server control device) The server control device 40 controls each part of the server device 30. The server control device 40 has, for example, a conference support control device such as a CPU or an MPU, and a storage device such as a RAM or a ROM. The server control device 40 may be realized by, for example, an integrated circuit such as an ASIC or an FPGA. The server control device 40 may be realized by a combination of hardware and software. The server control device 40 executes a program that controls the operation of the server device 30 according to the present disclosure.

[0037] The server control device 40 includes a communication control unit 41 and a storage unit 42 as functional blocks realized by circuits or programs to be executed.

[0038] The communication control unit 41 controls the communication unit 31 to control data communication between the server device 30 and the information terminal device 10. More specifically, the communication control unit 41 controls, for example, the communication unit 31 to control communication between the server device 30 and the information terminal device 10. The communication control unit 41 controls, for example, the communication unit 31 to receive, from the information terminal device 10, the voices of the participants, the images of the participants captured by the camera 11, and the biometric information detected by the biometric sensor 15. The communication control unit 41 controls, for example, the communication unit 31 to transmit, to the information terminal device 10, the images of the materials shared on the screen, and the images and voices of the other participants.

[0039] The storage unit 42 stores information such as the contents of calculations and programs of the communication control unit 41. The storage unit 42 is composed of, for example, a RAM, a main storage device such as a ROM, and a storage device such as an SSD or a hard disk drive (HDD).

[0040] (Meeting support equipment) 4 is a block diagram showing an example of the configuration of a conference support device according to the first embodiment. The conference support device 70 determines whether or not it is necessary to join a conference midway through based on the quality of the situation (atmosphere) of the conference detected from the status of the participants. The conference support device 70 displays the situation of the conference in real time to non-participants in the conference, thereby sharing the situation of the conference.

[0041] Non-participants may be, for example, all employees of a company.

[0042] The conference support device 70 is a conceptual device realized by either one of the information terminal device 10 and the server device 30, or by the cooperation of the two devices. Either the information terminal device 10 or the server device 30 may be called the conference support device 70, and the conference support system 1 configured with the information terminal device 10 and the server device 30 may be called the conference support device 70. Each unit of the conference support device 70 is realized by either the information terminal device 10 or the server device 30, or by both devices.

[0043] The conference support device 70 includes a display unit 71 and a conference support control device (control unit) 80.

[0044] The display unit 71 displays various images including text and images indicating at least one of the status of the conference venue and whether or not late participation in the conference is required.

[0045] The display unit 71 is, for example, a display including a liquid crystal display, an organic EL (Electro-Luminescence) display, etc., that is arranged on the outer wall of the conference room where the face-to-face conference is held. The display unit 71 may be a projection device that projects an image on the outer wall of the conference room where the face-to-face conference is held.

[0046] (Conference support control device) The conference support control device 80 judges whether or not a participant needs to join the conference midway based on the quality of the situation of the conference detected from the status of the participant. The conference support control device 80 displays the situation of the conference in real time to non-participants to share the situation of the conference. The conference support control device 80 is a conceptual device realized by either one of the terminal control device 20 and the server control device 40 or by the cooperation of the two devices. Either the terminal control device 20 or the server control device 40 may be referred to as the conference support control device 80, and the conference support system 1 configured by the terminal control device 20 and the server control device 40 may be referred to as the conference support control device 80. Each unit of the conference support control device 80 is realized by either the terminal control device 20 or the server control device 40, or by both devices.

[0047] The conference support control device 80 includes a situation detection unit 82 , a necessity determination unit 83 , a display control unit 88 , and a storage unit 89 .

[0048] The situation detection unit 82 detects the situation of a conference venue in which multiple participants participate. More specifically, the situation detection unit 82 extracts the states of the participants from a video of the conference venue in which multiple participants participate and from audio of the venue, and detects whether the situation of the venue is good or bad based on the ratio of participants in a predetermined state to all participants. The situation detection unit 82 includes a state extraction unit 821 and a ratio calculation unit 822.

[0049] The status extraction unit 821 extracts the status of each participant by a known method from the voice data, video data, or biometric information of the multiple participants acquired from the multiple information terminal devices 10. The status extraction unit 821 stores the good or bad status of each participant in the storage unit 89 as status data of the conference participant.

[0050] A participant's condition is deemed to be "good" when he or she is actively speaking in a conference. For example, when the voice data shows that the number of vocabulary words per given time is on the rise and that the number of positive statements is on the rise, the participant's condition is "good." For example, when the video of the participant shows that there is no change in the line of sight and the inclination of the posture from the start of the conference (steady state), the participant's condition is "good." For example, when biometric information shows that the heart rate, amount of sweat, etc. tend to increase and decrease, the participant's condition is "good."

[0051] A participant's status is deemed "bad" when speech in a conference is decreasing. For example, when the voice data indicates that the number of words per given time is decreasing and the number of negative phrases is increasing, the participant's status is "bad." For example, when the video of the participant indicates that the inclination of the line of sight and posture from the start of the conference (steady state) is increasing, the participant's status is "bad." For example, when biometric information indicates that there is little change in heart rate, amount of sweat, etc., the participant's status is "bad."

[0052] The extraction criteria for the condition extraction unit 821 as to whether the condition of a participant is good or bad are stored in the storage unit 89 .

[0053] The ratio calculation unit 822 calculates the ratio of participants whose status is "no" based on the total number of participants in the conference and the number of participants for each status calculated from the status data stored in the memory unit 89, which is the extraction result of the status extraction unit 821.

[0054] The situation detection unit 82 detects the situation of the field as bad (bad) when, for example, the ratio of participants whose status is bad among all participants calculated by the ratio calculation unit 822 is 50% or more. The situation detection unit 82 detects the situation of the field as good (good) when, for example, the ratio of participants whose status is bad among all participants calculated by the ratio calculation unit 822 is less than 50%.

[0055] The necessity determination unit 83 determines whether or not mid-way participation is required, in other words, whether mid-way participation is required, based on the situation of the event detected by the situation detection unit 82. The necessity determination unit 83 determines that mid-way participation is required, for example, when the situation of the event is not good, in other words, when the meeting is stagnating and problem solving is at an impasse. The necessity determination unit 83 determines that mid-way participation is not required, for example, when the situation of the event is good, in other words, when the meeting is not stagnating. The necessity determination unit 83 determines that mid-way participation is not required when the proportion of all event participants at which the following trends are detected exceeds a predetermined value.

[0056] The criteria for determining whether or not mid-game participation is required by the necessity determining unit 83 are stored in the memory unit 89 .

[0057] The display control unit 88 controls the display unit 71 to display, as visual information, information indicating at least one of the situation of the place detected by the situation detection unit 82 and the determination result of the necessity determination unit 83. The display control unit 88 may further control the display unit 71 to display, as visual information, keyword information indicating the contents of the conference.

[0058] The situation of the venue is displayed, for example, by distinguishing between good and bad situations (atmosphere) of the conference venue using characters, colors, icon images, etc. For example, the background color, background image, message image, icon image, etc. of the display unit 71 may be displayed in red when the situation of the venue is "bad" and in blue when the situation of the venue is "good."

[0059] The result of the determination as to whether or not mid-game participation is accepted is displayed, for example, with text, a color, an icon image, or the like indicating that mid-game participation is accepted.

[0060] The keyword information is text information generated from conversation contents extracted in real time from the voice data of multiple participants acquired from multiple information terminal devices 10. Using a known method, text information indicating the gist of the topic, minutes, or keywords is generated from the voice data.

[0061] When the necessity determining unit 83 determines that late participation acceptance is necessary, the display control unit 88 displays visual information including participation request information encouraging participation in the conference.

[0062] FIG. 5 is a diagram showing an example of display of visual information. In FIG. 5, visual information 100 is displayed outside a conference room R in a real space where a face-to-face conference is being held. In the example of FIG. 5, the display unit 71 is a projection device arranged in the conference room R. The projection device projects the visual information 100 onto a wall surface Ra facing the passage in front of the conference room R, which is formed of a transparent or translucent material that transmits light. The visual information 100 includes characters indicating mid-way participation acceptance and keyword information indicating the contents of the conference. The background color of the visual information 100 may change depending on the situation of the venue. For example, a person U1 who has confirmed the visual information 100 including information indicating mid-way participation acceptance projected onto the wall surface Ra considers mid-way participation in the conference because it is his specialty based on the keyword information. For example, a person U2 who has confirmed the visual information 100 projected onto the wall surface Ra decides not to join the conference mid-way because it is not his specialty based on the keyword information.

[0063] FIG. 6 is a diagram showing another example of display of visual information. In FIG. 6, the display unit 71 is the display unit 19 of the information terminal device 10 used by participants and non-participants using the conference application. In FIG. 6, visual information 100 of the conference is displayed on the display unit 71 via a conference application or the like executed in the information terminal device 10. For example, the face icon image in the upper right corner indicates that the situation is "no". The display shown in FIG. 6 may be displayed, for example, when a non-participant is executing the conference application. Or, for example, when the situation is "no", an icon image may be displayed on the task bar, or a pop-up screen may be displayed.

[0064] The storage unit 89 stores information such as the calculation contents of the conference support control device 80 and programs.

[0065] (Meeting support method) Fig. 7 is a flowchart showing a conference supporting method according to the first embodiment. The process shown in Fig. 7 is started when the conference is started. When the conference is started, display of the conference screen in each information terminal device 10 is started.

[0066] In the following description of this embodiment, an example in which the present invention is used for a face-to-face conference will be described.

[0067] The conference support control device 80 detects the situation of the venue (step S101). More specifically, the conference support control device 80 extracts the status of the participants from the video of the conference venue in which multiple participants participate and the audio of the venue, using the situation detection unit 82, and detects whether the situation of the venue is good or bad based on the ratio of participants in a predetermined state to all participants. The conference support control device 80 detects the situation of the venue as bad (bad) using the situation detection unit 82, for example, when the ratio of participants in a bad state to all participants is 50% or more. The conference support control device 80 proceeds to step S102.

[0068] The conference support control device 80 determines whether mid-way participation in the conference is necessary (step S102). More specifically, the conference support control device 80 determines whether mid-way participation is necessary or not based on the situation of the venue detected by the situation detection unit 82, by the necessity determination unit 83. The conference support control device 80 determines that mid-way participation is necessary by the necessity determination unit 83, for example, when the situation of the venue is not good. The conference support control device 80 determines that mid-way participation is not possible by the necessity determination unit 83, for example, when the situation of the venue is good. If the conference support control device 80 determines that mid-way participation is necessary (Yes in step S102), the conference support control device 80 proceeds to step S103. If the conference support control device 80 does not determine that mid-way participation in the conference is necessary (No in step S102), the conference support control device 80 proceeds to step S104.

[0069] If it is determined that mid-way participation in the conference is necessary (Yes in step S102), the conference support control device 80 displays the situation of the conference room and participation request information on the display unit of the conference room (step S103). More specifically, the conference support control device 80 controls the display control unit 88 to project, as visual information, information indicating at least one of the situation of the conference room detected by the situation detection unit 82 and the determination result of the necessity determination unit 83, via the display unit 71 of the conference room where the face-to-face conference is being held. The conference support control device 80 proceeds to step S104.

[0070] The conference support control device 80 determines whether the conference has ended (step S104). More specifically, the conference support control device 80 determines that the conference has ended, for example, when the conference end time has passed or when the conference has ended due to an operation by a participant. If it is determined that the conference has ended (Yes in step S104), the conference support control device 80 ends the processing of this flowchart. If it is not determined that the conference has ended (No in step S104), the conference support control device 80 executes the processing of step S101 again.

[0071] In this way, in a face-to-face conference, the necessity of mid-way participation in a face-to-face conference is determined based on the quality of the situation detected from the status of the participants. Then, visual information including the situation of the conference is displayed in real time on the wall Ra facing the passage in front of the conference room R for non-participants to share the situation of the conference.

[0072] (effect) As described above, in this embodiment, the condition of the conference venue is detected from the video and audio captured at the conference venue. In this embodiment, the necessity of accepting mid-session participation is determined based on the condition of the conference venue. In this embodiment, it is possible to detect that the conference is stalled and objectively determine that mid-session participation is necessary. In this way, according to this embodiment, it is possible to support the conference to proceed smoothly when the conference is stalled.

[0073] According to this embodiment, when a meeting is stalled due to a deadlock in problem-solving, people around the meeting room can join the meeting midway. According to this embodiment, advice from people who join midway can be obtained, which can lead to problem-solving in the meeting. According to this embodiment, conversations with people with whom one does not usually communicate can lead to the creation of new ideas.

[0074] In this embodiment, information indicating at least one of the status of the conference venue and the determination result of the necessity determination unit is displayed as visual information. In this embodiment, when it is determined that mid-session acceptance is necessary, participation request information that encourages participation in the conference is displayed in addition to the visual information. According to this embodiment, the status of the conference venue can be shared in real time with people who are not participating in the conference.

[0075] In this embodiment, keyword information indicating the contents of the conference is displayed as part of the visual information. According to this embodiment, people who are interested in the contents of the conference can join the conference midway. According to this embodiment, by receiving appropriate advice from people who join midway, it is possible to solve problems in the conference.

[0076] [Second embodiment] A conference supporting device 70A according to a second embodiment will be described with reference to Figs. 8 and 9. Fig. 8 is a block diagram showing a configuration example of the conference supporting device according to the second embodiment. Fig. 9 is a flowchart showing a conference supporting method according to the second embodiment. In the following description, components similar to those in the conference supporting device 70 are given the same or corresponding reference symbols, and detailed description thereof will be omitted. This also applies to the following embodiments. The conference supporting device 70A differs from the first embodiment in that it includes an information acquisition unit 84A and a participation request unit 85A, and in the processing in a display control unit 88A.

[0077] In this embodiment, the case where the system is used for a WEB conference will be described as an example. In the WEB conference of this embodiment, there are participants who gather in a conference room in real space, and participants and non-participants who participate online via a conference application. There may be multiple conference rooms in the virtual space, including the conference room.

[0078] In this embodiment, online participants and non-participants who are at least in the vicinity of the conference are displayed as avatars in the virtual space.

[0079] In a conference application executed by a non-participant on an information terminal, a conference is held in a conference room in a virtual space. In the virtual space displayed on the screen, avatars of non-participants are displayed around the conference room in the virtual space. Attribute information indicating attributes such as the specialized field in which the participant has knowledge is linked to the avatars of non-participants. In this embodiment, a non-participant may be a person who is not participating in the conference in the conference room but is participating in another conference room in the virtual space. Also, a non-participant may be a person who is strolling around the conference room in the virtual space.

[0080] The display control unit 88A displays visual information in a scene in real space and a scene in virtual space.

[0081] The location in the real space is the location of the conference room in the real space (the actual conference room). The location in the virtual space is the location of the conference room in the virtual space.

[0082] When the necessity determining unit 83 determines that mid-game participation acceptance is necessary, the information acquiring unit 84A acquires attribute information of avatars present around the location in the virtual space.

[0083] The participation request unit 85A requests participation from avatars that have a high degree of match between the text information based on the wording acquired from the voice of the venue and the attribute information acquired by the information acquisition unit 84A. For example, the participation request unit 85A displays a message requesting participation from avatars that have a high degree of match. For example, when there is no avatar that has a high degree of match, the participation request unit 85A displays a message requesting participation in the virtual space without specifying a target person to be requested.

[0084] Next, a conference support method will be described with reference to Fig. 9. The processes in steps S111 to S113 and S118 in the flowchart shown in Fig. 9 are similar to those in steps S101 to S103 and S104 in the flowchart shown in Fig. 7.

[0085] The conference support controller 80A acquires attribute information of avatars in the virtual space (step S114). More specifically, the conference support controller 80A acquires attribute information of avatars present around the place in the virtual space by the information acquisition unit 84A. The conference support controller 80A proceeds to step S115.

[0086] The conference support control device 80A determines whether or not an avatar that highly matches the conference content exists (step S115). More specifically, if the conference support control device 80A determines that an avatar that highly matches the conference content exists (Yes in step S115), the conference support control device 80A proceeds to step S116. If the conference support control device 80A does not determine that an avatar that highly matches the conference content exists (No in step S115), the conference support control device 80A proceeds to step S117.

[0087] If it is determined that there is an avatar that highly matches the conference content (Yes in step S115), the conference support controller 80A causes the participation request unit 85A to display a participation request to the avatar that highly matches the conference content (step S116). The conference support controller 80A proceeds to step S118.

[0088] If it is not determined that there is an avatar that highly matches the conference content (No in step S115), the conference support controller 80A requests participation by displaying participation request information in the conference room in the virtual space through the participation request unit 85A (step S117). The conference support controller 80A proceeds to step S118.

[0089] (effect) As described above, in this embodiment, participation is requested to avatars whose character information based on the words acquired from the voices in the conference match the avatar information to a high degree. According to this embodiment, it is possible to request participation of people who are suitable for the content of the conference via the avatars.

[0090] In this embodiment, if there is no avatar whose avatar information matches closely with text information based on the words acquired from the voice of the conference, participation can be requested by displaying participation request information in the conference room in the virtual space. According to this embodiment, people who are interested in the contents of the conference can join the conference midway.

[0091] In this way, according to this embodiment, even in a Web conference, when the conference is at a standstill, it is possible to provide support for the conference to proceed smoothly.

[0092] [Third embodiment] A conference support control device 80 according to a third embodiment will be described with reference to Figs. 10 and 11. Fig. 10 is a block diagram showing a configuration example of a conference support device according to a third embodiment. Fig. 11 is a flowchart showing a conference support method according to the third embodiment. A conference support device 70B differs from the first embodiment in that it includes an information acquisition unit 84B and a participation request unit 85B, and in that it includes a memory unit 89B.

[0093] In this embodiment, a case where the device is used for a face-to-face conference will be described as an example.

[0094] The storage unit 89B stores attribute information of participants and non-participants. The storage unit 89B stores attribute information indicating attributes such as a specialized field of knowledge for each participant and non-participant in association with each other.

[0095] When the necessity determining unit 83 determines that late participation acceptance is necessary, the information acquiring unit 84B acquires attribute information of non-participants stored in the storage unit 89B.

[0096] The participation request unit 85B requests non-participants who have a high degree of match between the character information based on the wording acquired from the voice of the meeting and the attribute information acquired by the information acquisition unit 84B as participation candidates, and transmits participation request information to a predetermined communication unit. More specifically, when a participation candidate is located in the vicinity of the meeting room, the participation request unit 85B requests participation by the display unit of the meeting room or by voice. When a participation candidate is present but is not located in the vicinity of the meeting room, the participation request unit 85B requests the participation candidate to be the request target and transmits participation request information to the information terminal device 10 of the participation candidate via a predetermined communication unit (not shown). The participation request unit 85B may transmit the request to the participation candidate via a meeting application.

[0097] Next, a conference support method will be described with reference to Fig. 11. The processes in steps S121 to S123 and step S129 in the flowchart shown in Fig. 11 are similar to those in steps S101 to S103 and step S104 in the flowchart shown in Fig. 7.

[0098] The conference support control device 80B acquires attribute information of non-participants around the conference room (step S124). More specifically, the conference support control device 80B acquires attribute information of non-participants who are currently at the office in the real space from the memory unit 89B by the information acquisition unit 84B. The conference support control device 80B proceeds to step S125.

[0099] The conference support control device 80B determines whether or not there is a non-participant (participant candidate) whose content highly matches the conference content (step S125). More specifically, if the conference support control device 80B determines that there is a non-participant (participant candidate) whose content highly matches the conference content (Yes in step S125), the conference support control device 80B proceeds to step S126. If the conference support control device 80B does not determine that there is a non-participant (participant candidate) whose content highly matches the conference content (No in step S125), the conference support control device 80B proceeds to step S129.

[0100] If it is determined that there is a non-participant (participant candidate) whose content matches the conference content to a high degree (Yes in step S125), the conference support control device 80B determines whether the participant candidate is present in the vicinity of the conference room (step S126). The conference support control device 80B acquires location information of the participant candidate who is a non-participant whose content matches the conference content to a high degree. If the conference support control device 80B determines that the participant candidate is present in the vicinity of the conference room (Yes in step S126), it proceeds to step S127. If the conference support control device 80B does not determine that the participant candidate is present in the vicinity of the conference room (No in step S126), it proceeds to step S128.

[0101] If it is determined that the potential participant is present in the vicinity of the conference room (Yes in step S126), the conference support control device 80B makes a request to the vicinity of the conference room by the participation request unit 85B (step S127). More specifically, the conference support control device 80B makes a participation request by the participation request unit 85B on the display unit of the venue or by voice. The conference support control device 80B proceeds to step S129.

[0102] If it is not determined that the participant candidate is present in the vicinity of the conference room (No in step S126), the conference support controller 80B causes the participation request unit 85B to display a request on the information terminal device 10 used by the participant candidate (step S128). The conference support controller 80B proceeds to step S129.

[0103] (effect) As described above, in this embodiment, the request targets non-participant candidate participants who have a high degree of match between the character information based on the wording acquired from the voice of the venue and the attribute information, and if the participant candidate is not in the vicinity of the conference room, the participation request information is sent to a predetermined communication unit. In this embodiment, the request targets non-participant candidate participants who have a high degree of match between the character information based on the wording acquired from the voice of the venue and the attribute information, and if the participant candidate is in the vicinity of the conference room, the participation request information is requested on the display unit of the venue or by voice. According to this embodiment, it is possible to appropriately request participation from non-participants (participant candidates) who have a high degree of match. According to this embodiment, it is possible to request participation from people who are suitable for the content of the conference. In this way, according to this embodiment, it is possible to support the smooth progress of the conference when the conference is stagnating.

[0104] Each component of the illustrated conference support system is a functional concept, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of the devices may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0105] The configuration of each device in the conference support system is realized, for example, as software, by a program loaded into a memory. In the above embodiment, these functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms by only hardware, only software, or a combination of both.

[0106] The components described above include those that a person skilled in the art can easily imagine and those that are substantially the same. Furthermore, the configurations described above can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the scope of the present invention.

[0107] The above third embodiment has been described taking an example where the system is used for a face-to-face conference, but in the case of a hybrid conference, a participation request may be sent to potential participants as follows: When an avatar of a potential participant is located near a conference room in the virtual space, the participation request unit 85B may request the participant to participate by displaying participation request information in the conference room in the virtual space, or may display a message requesting the participant to participate to the avatar of the potential participant. [Explanation of symbols]

[0108] 1. Meeting support system 10 Information terminal equipment 11 Camera 12 Microphones 13 Audio output section 14 Control section 15 Biometric sensors 18 Communications Department 19 Display section 20 Terminal control device 21 Shooting control section 22 Voice Acquisition Unit 23 Audio output control section 24 Operation control section 25 Memory section 26 Communication control section 27 Display control section 28 Application control section 29 Biometric information acquisition unit 30 Server device 31 Communications Department 40 Server control device 41 Communication control section 42 Storage section 70 Meeting Support Equipment 71 Display section 80 Conference support control device 82 Situation detection section 821 State Extraction Unit 822 Percentage Calculation Section 83 Necessity determination section 88 Display control section 89 Memory section

Claims

1. a situation detection unit that extracts the states of the participants from a video of a conference venue attended by multiple participants and from audio of the venue, and detects whether the situation of the venue is good or bad based on the ratio of participants in a predetermined state to the total number of participants; a necessity determination unit that determines whether or not mid-game participation in the event is required based on the event situation detected by the situation detection unit; A conference support control device comprising:

2. a display control unit that controls to display, as visual information, information indicating at least one of the situation at the site detected by the situation detection unit and the determination result of the necessity determination unit; Further equipped with When the necessity determination unit determines that late participation acceptance is necessary, the display control unit causes the visual information to include participation request information that prompts the user to participate in the conference. The conference support control device according to claim 1 .

3. The display control unit displays the visual information in the scene in a real space and in the scene in a virtual space; an information acquisition unit that acquires attribute information of avatars present around the location in the virtual space when the necessity determination unit determines that mid-game participation is necessary; a participation request unit that requests participation from an avatar having a high degree of match between character information based on a phrase acquired from the voice of the event and the attribute information acquired by the information acquisition unit; The conference support control device of claim 2 further comprising:

4. A storage unit in which attribute information of participants and non-participants is stored; Equipped with the information acquisition unit, when the necessity determination unit determines that late participation acceptance is necessary, acquires attribute information of non-participants stored in the storage unit; The participation request unit requests the non-participants who have a high degree of match between character information based on a phrase acquired from the voice of the event and the attribute information acquired by the information acquisition unit, and transmits participation request information to a predetermined communication unit. The conference support control device according to claim 3 .

5. a situation detection step of extracting the states of the participants from a video of a conference venue attended by a plurality of participants and from audio of the venue, and detecting whether the situation of the venue is good or bad based on the ratio of participants in a predetermined state to the total number of participants; a necessity determination step of determining whether or not mid-game participation in the event is required based on the event status detected by the status detection step; A meeting support method comprising:

Images