Conference support device and method

Abstract

To provide a conference support device and a method that can present useful information to each participant in a real time.SOLUTION: A conference support method executed by a conference support device that supports a conference determines each participant's behavior / state on the basis of the output of each sensor that acquires information regarding each participant's behavior / state, determines information to be presented to the participant on the basis of the determined behavior / state of the participant and the preset position of each participant, and presents the determined information to the participant via a terminal provided on the corresponding participant's side.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present invention relates to a conference support device and method, and is suitable for application to a conference support system that supports, for example, online conferences and hybrid conferences.

Background Art

[0002] With the recent trend of DX (Digital Transformation), the visualization of conferences (including workshops) has been demanded. In addition, while face-to-face meetings have been the mainstream in the past, in recent years, online meetings have also been promoted due to the spread of infectious diseases around the world. Furthermore, the hybrid conference format, which combines face-to-face and online meetings, is also widely used.

[0003] Regarding the visualization of conferences, Patent Document 1 below discloses operating an avatar corresponding to a first user displayed on the terminal of a second user on the other side in an operation pattern corresponding to the non-verbal behavior of the first user.

[0004] In addition, Patent Document 2 discloses that in an online video conference, each conference attendee is sensed using sensors such as cameras and microphones, and based on the movement information of each attendee obtained by the sensing, the speaker of each speech in the audio data is identified, and the audio data of each attendee is output as a timeline in the time series of the speech.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, as disclosed in Patent Documents 1 and 2, when participating in an online meeting, it is difficult to see the facial expressions and appearances of other participants, so the speech (voice), images, and various sensor information of other participants are visualized.

[0007] However, a problem arose in that the amount of information displayed at this time was large, making it difficult for the individual to identify the information they needed. Furthermore, the information displayed only concerned the meeting as a whole and the other party, without considering the participant's own actions or state at the time, meaning it was often not very useful to the participant.

[0008] This invention was made in consideration of the above points, and aims to propose a conference support device and method that can present useful information to each participant in real time. [Means for solving the problem]

[0009] To solve these problems, the present invention provides a conference support device for supporting a conference, comprising: an action / state determination unit that determines the action / state of each participant in real time based on the output of each sensor that acquires information on the action / state of each participant; a presentation information determination unit that determines information to be presented to the participant based on the action / state of the participant determined by the action / state determination unit and the pre-set position of each participant; and a presentation processing unit that presents the information determined by the presentation information determination unit to the participant via a terminal provided on the corresponding participant's side. For each combination of the aforementioned position and the aforementioned actions / states, items to be presented to the participant are predetermined, and the information presentation unit determines, for each participant, the information of the aforementioned items corresponding to the participant's aforementioned position and the aforementioned actions / states as the information to be presented to that participant. I did that.

[0010] Furthermore, the present invention provides a conference support method performed by a conference support device, comprising: a first step of determining the behavior and state of each participant in real time based on the output of each sensor that acquires information on the behavior and state of each participant; a second step of determining information to be presented to the participant based on the determined behavior and state of the participant and the pre-set position of each participant; and a third step of presenting the determined information to the participant via a terminal provided on the corresponding participant's side. For each combination of the aforementioned position and the aforementioned behavior / state, items to be presented to the participant are predetermined, and in the second step, the conference support device determines, for each participant, the information of the aforementioned items corresponding to the participant's aforementioned position and the aforementioned behavior / state as information to be presented to that participant. I did that.

[0011] According to the conference support device and method of the present invention, information corresponding to the participant's position and their actions and state at that time is presented to the participant in real time via their terminal. [Effects of the Invention]

[0012] According to the present invention, a conference support device and method can be realized that can present useful information to each participant in real time. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram showing the overall configuration of the conference support system according to this embodiment. [Figure 2] This is a block diagram showing the logical configuration of the conference support system. [Figure 3] This diagram shows an example of the configuration of a participant attribute management table. [Figure 4] This diagram shows an example configuration of a sensing information management table. [Figure 5] This diagram shows an example of the configuration of a decision-making behavior / state management table. [Figure 6] (A) is a diagram showing an example of the configuration of a display rule management table for facilitation, and (B) is a diagram explaining the information used when determining whether each display item meets the display requirements. [Figure 7](A) is a chart showing a configuration example of a display rule management table for supervisors, and (B) is a chart for explaining information used when determining whether each display item meets the display requirements. [Figure 8] (A) is a chart showing a configuration example of a display rule management table for subordinates, and (B) is a chart for explaining information used when determining whether each display item meets the display requirements. [Figure 9] (A) is a chart showing a configuration example of a display rule management table for guests, and (B) is a chart for explaining information used when determining whether each display item meets the display requirements. [Figure 10] A chart showing a configuration example of a display requirement management table. [Figure 11] A chart showing a configuration example of a display information management table. [Figure 12] A flowchart showing the processing procedure of the action / status determination process. [Figure 13] A flowchart showing the processing procedure of the display information determination process.

Embodiments for Carrying Out the Invention

[0014] The following describes in detail an embodiment of the present invention with reference to the drawings.

[0015] (1) Configuration of the Conference Support System According to the Present Embodiment In FIG. 1, reference numeral 1 indicates the conference support system according to the present embodiment as a whole. This conference support system 1 includes a face-to-face participant side system 2 installed at the venue of a conference (including workshops, etc.), online participant side systems 3 respectively provided for each participant who participates in the conference online, and a conference support device 4.

[0016] The face-to-face participant system 2 consists of a sensor group 10A that acquires various information about the actions and status of each participant participating in the face-to-face meeting through sensing, and a plurality of display devices 14A provided in accordance with each of these participants. The sensor group 10A consists of an imaging device 11A, a sound collection device 12A, and a vibration sensor 13A provided in accordance with each participant participating in the face-to-face meeting.

[0017] The imaging device 11A consists of a general-purpose camera and is used to capture the facial expressions of the corresponding participants. However, instead of providing an imaging device 11A for each participant attending the meeting face-to-face, a 360° camera may be used, for example. Each imaging device 11A transmits the video signal obtained from its capture to the meeting support device 4.

[0018] The sound collection device 12A consists of general-purpose microphones and is used to collect sound emitted from the corresponding participants. However, instead of providing a separate sound collection device 12A for each participant attending the meeting face-to-face, a multi-channel microphone array capable of identifying the direction of the sound source may be used, for example. Each sound collection device 12A transmits an audio signal corresponding to the collected sound to the conference support device 4.

[0019] The vibration sensor 13A is composed of, for example, an acceleration sensor and is installed on the seat of each chair where each participant sitting in a face-to-face meeting is seated. The vibration sensor 13A detects vibrations caused by the nodding of the participant seated in the chair on which it is installed and outputs a vibration signal corresponding to the detected vibration to the meeting support device 4.

[0020] The display device 14A consists of, for example, a small liquid crystal monitor and is used to display necessary information. Each display device 14A is installed near the seating position of the corresponding participant so that the participant can visually view the displayed information.

[0021] The online participant system 3 consists of a sensor group 10B that acquires various information about the actions and status of the corresponding participant participating in the online meeting through sensing, and a display device 14B that displays the meeting screen and other various information to the participant, and is connected to the meeting support device 4 via the network 5. The sensor group 10B consists of an imaging device 11B, a sound collection device 12B, and a vibration sensor 13B.

[0022] The configuration and installation of the imaging device 11B, sound collection device 12B, vibration sensor 13B, and display device 14B are the same as those of the imaging device 11A, sound collection device 12A, vibration sensor 13A, and display device 14A of the face-to-face participant system 2, so a detailed explanation is omitted here. The video signal output from the imaging device 11B, the audio signal output from the sound collection device 12B, and the vibration signal output from the vibration sensor 13B are each transmitted to the conference support device 4 via the network 5.

[0023] The conference support device 4 is a computer device that determines the current behavior and state of each participant in real time based on various information about the behavior and state of each participant collected by the sensor group 10A of the face-to-face participant system 2 and the sensor group 10B of each online participant system 3, and has the function of displaying information on each participant's display device 14A, 14B as appropriate, along with the participant's attributes and information corresponding to the participant's behavior and state at that time.

[0024] This conference support device 4 is comprised of information processing resources such as a CPU (Central Processing Unit) 20, memory 21, storage device 22, and input device 23. The CPU 20 is a processor that controls the operation of the entire conference support device 4. The memory 21 is composed of, for example, volatile semiconductor memory and is used as the working memory of the CPU 20.

[0025] The storage device 22 consists of a non-volatile, high-capacity storage device such as a hard disk drive or an SSD (Solid State Drive), and is used to hold various programs and data that needs to be stored for a long period of time. When the conference support device 4 is started up or when necessary, the required programs are loaded from the storage device 22 into the memory 21, and the CPU 20 executes the programs loaded into the memory 21, thereby executing various processes for the conference support device 4 as a whole, as described later.

[0026] In this embodiment, the memory 21 stores the image processing program 30, voice processing program 31, nod detection program 32, behavior / state determination program 33, display information determination program 34, and display processing program 35, which are loaded from the storage device 22, and the storage device 22 stores an information management database 36, which consists of a plurality of management tables as described below.

[0027] The input device 23 consists of, for example, a mouse and keyboard, and is used by the facilitator (hereinafter referred to as the facilitator or facilit) to input various data and perform various operations on the conference support device 4.

[0028] Figure 2 shows the logical configuration of the conference support device 4 according to this embodiment. As shown in Figure 2, the conference support device 4 comprises an image processing unit 40, an audio processing unit 41, a nodding detection unit 42, an action / state determination unit 43, a display information determination unit 44, and a display processing unit 45 as functional units, and is configured with a participant attribute management table 50, a sensing information management table 51, a determined action / state management table 52, multiple display rule management tables 53, a display requirements management table 54, and a display information management table 55 stored in the information management database 36 as management tables.

[0029] In the following, it is assumed that information indicating which imaging device 11A, 11B, sound collection device 12A, 12B, vibration sensors 13A, 13B, and display devices 14A, 14B are associated with each participant in the meeting has been provided in advance to the image processing unit 40, the sound processing unit 41, the nodding detection unit 42, and the display processing unit 45, respectively. Furthermore, in the following, it is assumed that each participant is assigned a unique identifier (hereinafter referred to as the participant ID).

[0030] The image processing unit 40 is a functional unit that is realized when the CPU 20 (Figure 1) executes the image processing program 30 shown in Figure 1, which is loaded into the memory 21 (Figure 1). Based on the video signals provided by each imaging device 11A and 11B, the image processing unit 40 determines the facial expression of each participant at a predetermined determination period (e.g., a 1-second period), and when it detects a smile or an expression that suggests the participant wants to speak (e.g., an expression such as silently opening and closing the mouth as if to express an opinion), it notifies the behavior / state determination unit 43 of that fact and the participant ID of that participant.

[0031] Furthermore, the image processing unit 40 determines the orientation of each participant's face at the meeting based on the video signals provided by each imaging device 11A of the face-to-face participant system 2, according to the aforementioned determination cycle. If it determines that any participant is looking at another participant, it notifies the behavior / state determination unit 43 of that participant's participant ID and the participant ID of the other participant that that participant was looking at.

[0032] The voice processing unit 41 is a functional unit that is realized when the CPU 20 executes the voice processing program 31 shown in Figure 1, which is loaded into the memory 21. Based on the voice signals provided by each of the sound collection devices 12A and 12B, the voice processing unit 41 measures the volume of any participant's speech at the aforementioned determination cycle and notifies the behavior / state determination unit 43 of the participant ID of the participant who made the speech and the measured volume of that speech.

[0033] The nodding detection unit 42 is a functional unit that is realized when the CPU 20 executes the nodding detection program 32 shown in Figure 1, which is loaded into the memory 21. The nodding detection unit 42 analyzes the vibration signals provided by each vibration sensor 13A and 13B at the aforementioned determination period, and when it detects that any participant is nodding, it notifies the behavior / state determination unit 43 of that fact and the participant ID of that participant.

[0034] The behavior / state determination unit 43 is a functional unit that is realized when the CPU 20 executes the behavior / state determination program 33 shown in Figure 1, which is loaded into the memory 21. The behavior / state determination unit 43 sequentially stores and manages the information notified by the image processing unit 40, the voice processing unit 41, and the nodding detection unit 42, respectively, in the sensing information management table 51 described later, according to the determination cycle described above. The behavior / state determination unit 43 also determines the behavior / state of each participant at that time based on the information stored in the sensing information management table 51 according to the determination cycle described above, and registers and manages the determination result in the determined behavior / state management table 52.

[0035] In this embodiment, the behavior / state determination unit 43 determines whether the participant is "speaking," "not speaking and listening attentively," or "not speaking and not interested / doing something else."

[0036] "Speaking" refers to the state in which the corresponding participant is speaking. "Non-speaking and listening attentively" refers to the state in which the corresponding participant is not speaking (non-speaking) and is attentively listening to what the participant who is speaking at that time (hereinafter referred to as the speaker) is saying. Furthermore, "Non-speaking and not interested / working on something else" refers to the state in which the corresponding participant is not speaking (non-speaking) and is not interested in what the speaker is saying at that time or is working on something unrelated to the meeting.

[0037] The display information determination unit 44 is a functional unit that is realized when the CPU 20 executes the display information determination program 34 shown in Figure 1, which is loaded into the memory 21. Based on the information stored in the sensing information management table 51 and the judgment action / state management table 52, respectively, and the information stored in the participant attribute management table 50, each display rule management table 53, and the display requirement management table 54, respectively, the display information determination unit 44 determines, at the aforementioned determination cycle, whether the current situation is such that any information should be displayed on the participant's display devices 14A and 14B.

[0038] The display information determination unit 44 then determines that there is a situation in which some information should be displayed for any participant, and refers to the display information management table 55 (described later) to determine the information to be displayed on the participant's display devices 14A and 14B at that time, and notifies the display processing unit 45 of the determined information along with the participant's participant ID.

[0039] The display processing unit 45 is a functional unit that is realized when the CPU 20 executes the display processing program 35 shown in Figure 1, which is loaded into the memory 21. Based on the information notified by the display information determination unit 44 and the participant ID, the display processing unit 45 transmits the information to the display devices 14A and 14B of the participant to which the participant ID is assigned. As a result, the information is displayed on the participant's display device 14A and 14B.

[0040] On the other hand, the participant attribute management table 50 is a table used to manage the pre-defined attributes of each participant. Here, "participant attributes" refers to the participant's position in the meeting. In this embodiment, for example, if the meeting is an internal company meeting, four attributes are defined for such "participant attributes (positions)": "facilitator" who is the moderator, "superior" who is in a higher position among the participants, "subordinate" who is in a lower position than the "superior" among the participants, and "guest" who is a guest such as a client.

[0041] As shown in Figure 3, the participant attribute management table 50 is configured to include a participant ID column 50A and an attribute column 50B. In the participant attribute management table 50, each row corresponds to one participant. The participant ID column 50A stores the participant ID of the corresponding participant, and the attribute column 50B stores the attribute (position) of that participant in that meeting, which has been set in advance.

[0042] Therefore, in the example in Figure 3, the participant with the participant ID "0001" has the attribute (position) of "Facilitator," and the participant with the participant ID "0002" has the attribute (position) of "Supervisor."

[0043] The sensing information management table 51 is a table used to store information based on each notification provided by the image processing unit 40, the voice processing unit 41, and the nodding detection unit 42, respectively, at the judgment cycle, and is generated separately for each participant.

[0044] As shown in Figure 4, the sensing information management table 51 is configured to include an elapsed time column 51A, an expression column 51B, a direction column 51C, a speech column 51D, a volume column 51E, a speech time column 51F, and a nod column 51G. In the sensing information management table 51, one record (row) corresponds to one analysis result notified by the image processing unit 40, the sound processing unit 41, and the nod detection unit 42, respectively, in the judgment cycle described above. In the following, the timing at which the image processing unit 40, the sound processing unit 41, and the nod detection unit 42 analyze the video signal, sound signal, or nod detection signal in such a judgment cycle will be referred to as the data analysis timing.

[0045] The elapsed time column 51A stores the elapsed time from the start of the meeting for the corresponding data analysis timing. The facial expression column 51B stores a value corresponding to the facial expression of the corresponding participant detected by the image processing unit 40 during the period from the previous data analysis timing to the corresponding data analysis timing (hereinafter referred to as the previous data analysis period). For example, if the image processing unit 40 detected a smile as the facial expression of the corresponding participant during the previous data analysis period, "1" is stored in the facial expression column 51B; if it detected an expression that suggested it wanted to speak, "2" is stored; and otherwise, "0" is stored.

[0046] Furthermore, the orientation column 51C stores the participant ID of the other participant that the image processing unit 40 notified the behavior / state determination unit 43 when the image processing unit 40 determined that the face of the corresponding participant who was participating in the meeting face-to-face was facing the other participant (i.e., the corresponding participant was looking at the other participant) during the immediately preceding data analysis period.

[0047] The utterance field 51D stores a flag (hereinafter referred to as the utterance flag) indicating whether or not the speech processing unit 41 detected the utterance of the corresponding participant during the immediately preceding data analysis period. In the example in Figure 4, if the speech processing unit 41 detected the utterance of the corresponding participant during the data analysis period, the utterance flag is set to "1", and if it did not detect the utterance, the utterance flag is set to "0".

[0048] Furthermore, the volume column 51E stores the volume of the utterance (speech volume) measured by the speech processing unit 41 when the speech processing unit 41 detected the utterance of the corresponding participant during the immediately preceding data analysis period, and the speech time column 51F stores the duration of the utterance from the start.

[0049] Furthermore, the nodding column 51G stores a flag (hereinafter referred to as the nodding flag) indicating whether or not the nodding detection unit 42 detected that the corresponding participant was nodding during the immediately preceding data analysis period. In the example in Figure 4, if the nodding detection unit 42 detected that the corresponding participant was nodding during the data analysis period, the nodding flag is set to "1", and if it did not detect nodding, the nodding flag is set to "0".

[0050] Therefore, in the example of Figure 4, it is shown that the participant corresponding to Figure 4, for example, during the period from "00:00:01" to "00:00:02" after the start of the meeting, did not smile or show any signs of wanting to speak, nor did they speak at all, but instead turned their face towards the participant with participant ID "1", and nodded at least once during this time.

[0051] The judgment action / state management table 52 is a table used to manage the actions or states that each participant was performing at a given time, as determined by the action / state determination unit 43 at the determination cycle based on the information stored in the sensing information management table 51. As shown in Figure 5, it is configured to include a participant ID column 52A and a judgment action / state column 52B. In the judgment action / state management table 52, one record (row) corresponds to one participant.

[0052] The participant ID field 52A stores the participant ID of the corresponding participant, and the determined action / state field 52B stores the action the participant was performing at that time or the participant's state at that time, as determined by the action / state determination unit 43.

[0053] In this embodiment, as described above, the behavior / state determination unit 43 determines whether the participant is "speaking," "not speaking and listening attentively," or "not speaking and not interested / doing other work." Therefore, the determination behavior / state column 52B of each record will, in principle, store information such as "speaking," "not speaking and listening attentively," or "not speaking and not interested / doing other work," or "-" which means that none of these apply.

[0054] However, if the participant is a "facilitator," and the behavior / state determined by the behavior / state determination unit 43 is "not speaking and listening attentively" or "not speaking and not interested / doing something else," then the information "not speaking" is stored in the determined behavior / state column 52B (see the determined behavior column for participant ID "0001" in Figure 5).

[0055] Therefore, for "facilitator," if the corresponding judgment behavior / state column 52B contains the information "not speaking," it means that the behavior at that time was either "not speaking and listening attentively" or "not speaking and not interested / doing something else."

[0056] On the other hand, the display rule management table 53 is a table used to manage predetermined information display rules, such as what kind of information items should be displayed on the participant's display device 14A, 14B, depending on the participant's attributes (position), what actions they take, or what state they are in, and what requirements they must meet.

[0057] The display rule management table 53 has four types, corresponding to each attribute of the participant: one for "facilitator," one for "supervisor," one for "subordinate," and one for "guest." These four types of display rule management tables 53 are created in advance and stored in the information management database 37.

[0058] Figure 6(A) shows the display rule management table 53A for "facilitation". As shown in Figure 6(A), the display rule management table 53A for "facilitation" consists of a first region 53AA, which corresponds to the case when the facilitator's action / state is speaking, and a second region 53AB, which corresponds to the case when the facilitator's action / state is not speaking.

[0059] The first area 53AA consists of one or more columns (hereinafter referred to as display item columns) 53AAA, each corresponding to an item (hereinafter referred to as display item) that should be displayed on the facilitator's display devices 14A and 14B depending on the situation when the facilitator is speaking. In this embodiment, when the facilitator is speaking, the only display item to be displayed on the facilitator's display devices 14A and 14B is the display item "Many Agreeers 1". Therefore, the first area 53AA consists only of the display item column 53AAA "Number of Agreeers 1". The "Number of Agreeers 1" is a display item that displays on the facilitator's display devices 14A and 14B the number of participants who agreed with the content of the facilitator's utterance while the facilitator is speaking.

[0060] Similarly, the second area 53AB consists of one or more display item sequences 53ABA, 53ABB, and 53ABC, which correspond to each display item that displays information on the facilitator's display devices 14A and 14B depending on the situation when the facilitator is not speaking. In this embodiment, when the facilitator is not speaking, three display items are defined as display items that should display information corresponding to the facilitator's display devices 14A and 14B depending on the situation: "level of excitement," "number of agreeers (2)," and "people who wish to speak." Therefore, the second area 53AB consists of three display item sequences 53ABA to 53ABC: "level of excitement," "number of agreeers (2)," and "people who wish to speak."

[0061] "Engagement Level" is a display item that shows the degree of engagement in the meeting based on the contributions of participants other than the facilitator on the facilitator's display devices 14A and 14B. Details of "Engagement Level" will be described later. "Number of Agreeers 2" is a display item that shows the number of other participants who agreed with the contributions of participants other than the facilitator on the facilitator's display devices 14A and 14B. "Participants Wanting to Speak" is a display item that shows the participant ID of a participant who appears to want to speak on the facilitator's display devices 14A and 14B.

[0062] Furthermore, the display item columns 53AAA, 53ABA to 53ABC in the first and second regions 53AA and 53AB are divided into display item column 53AC, display priority column 53AD, and display information column 53AE, respectively.

[0063] The display item column 53AC stores the names of the display items corresponding to the display item columns 53AAA, 53ABA, and 53ABC, and the display priority column 53AD stores the display priority of those display items. The display priority is the priority given to displaying the information of the corresponding display item, and is determined independently for each case where the participant to whom the display rule management table 53 applies is speaking or not speaking.

[0064] When a participant to whom the display rule management table 53 applies is speaking or not speaking, and a situation arises where information from multiple display items should be displayed, only the information from the display item with the highest display priority will be displayed on the participant's display device, and information from the other display items will not be displayed. Furthermore, if there are multiple display items with the highest display priority, information from all of these display items will be displayed on the participant's display devices 14A and 14B.

[0065] Furthermore, the display information number column 53AE stores an identification number (hereinafter referred to as the display information number) unique to the information that should be displayed when the display requirements for displaying the information of the corresponding display item are met. When the display requirements for any display item are met, the display information number stored in the display information number column 53AE of the display item column 53AAA, 53ABA~53ABC corresponding to that display item is used as a search key to search for the information that should be displayed at that time on the display information management table 55 described later, and the detected information is displayed on the display devices 14A, 14B of the participants to which the display rule management table 53 applies.

[0066] On the other hand, Figures 7(A), 8(A), and 9(A) show the display rule management tables 53B, 53C, and 53D for "supervisors," "subordinates," and "guests," respectively. As shown in Figures 7(A) to 9(A), the basic structure of the display rule management tables 53B to 53D for "supervisors," "subordinates," and "guests" is the same as the display rule management table 53A for "facilitators."

[0067] However, if a participant's attribute is "superior," "subordinate," or "guest," three display items are specified as display items that, depending on the situation, display information corresponding to the participant's display devices 14A and 14B when that participant is speaking: "number of consenters," "support," and "speaking time."

[0068] Therefore, in the display rule management tables 53B to 53D for "supervisors," "subordinates," or "guests," the first area 53BA, 53CA, 53DA consists of display item columns 53BAA, 53CAA, 53DAA corresponding to the display item "Many Agreeers," display item columns 53BAB, 53CAB, 53DAB corresponding to the display item "Support," and display item columns 53BAC, 53CAC, 53DAC corresponding to the display item "Speech Time."

[0069] "Many Agreeers 1" is a display item that, when it is determined that a large number of other participants agree with the content of a participant's utterance during the utterance of a participant ("Supervisor," "Subordinate," or "Guest") to whom the display rule management table 53 (53B~53D) applies, will display that number on the participant's display device 14A, 14B.

[0070] Furthermore, "Encouragement" is a display item that displays a message encouraging the utterance on the participant's display device 14A, 14B when the participant to whom the display rule management table 53 (53B~53D) applies is speaking. Additionally, "Speech duration" is a display item that displays the duration of the utterance on the participant's display device 14A, 14B while the participant to whom the display rule management table 53 (53B~53D) applies is speaking.

[0071] Furthermore, when a participant's attribute is "supervisor," "subordinate," or "guest," the behavior / state of "not speaking" is subdivided into the behavior / states of "listening attentively" and "not interested / working on another task." "Listening attentively" corresponds to the participant's behavior / state of "not speaking and listening attentively," while "not interested / working on another task" corresponds to the participant's behavior / state of "not speaking and not interested / working on another task."

[0072] Furthermore, when a participant to whom the display rule management table 53 (53B~53D) applies is in the state of "not speaking and listening attentively," two display items, "interjection" and "influence / effect," are specified as display items to be displayed according to the participant's display devices 14A and 14B, depending on the situation.

[0073] Furthermore, if the participant is "not speaking and not interested / doing other work," a display item called "non-listening time" is specified as a display item that displays information corresponding to the participant's display device, depending on the situation.

[0074] Therefore, in the display rule management tables 53 (53B~53D) for "supervisors," "subordinates," or "guests," the second area 53BB, 53CB, 53DB is further divided into areas 53BBX, 53CBX, 53DBX for "listening attentively" and areas 53BBY, 53CBY, 53DBY for "not interested / doing something else."

[0075] The domains 53BBX, 53CBX, and 53DBX, which represent "listening attentively," consist of display item columns 53BAB, 53CBA, and 53DBA, which correspond to the display item "interfering," and display item columns 53BBB, 53CBB, and 53DBB, which correspond to the display item "influence / effect." The domains 53BBY, 53CBY, and 53DBY, which represent "not interested / working on a side job," consist of display item columns 53BBC, 53CBC, and 53DBC, which correspond to the display item "not interested / working on a side job."

[0076] Furthermore, "Interjection" is a display item that, when a participant is not speaking and appears to want to speak, displays a message prompting that participant to speak on their display device 14A, 14B at the same time as other participants have finished speaking. Additionally, "Influence / Effect" is a display item that, while a discussion on a topic initiated by the target participant is continuing, displays the number of participants who have spoken on that topic and the current duration of that topic on the target participant's display device 14A, 14B. Finally, "Non-listening Time" is a display item that, when a target participant is acting or in a state that suggests they are not interested in the meeting or are working on a task unrelated to the meeting, displays the duration of that action or state (hereinafter referred to as "non-listening time").

[0077] Furthermore, in the display rule management tables 53 (53B, 53C, 53D) for "supervisors," "subordinates," and "guests," for display items where "Hidden" is indicated in the display information fields 53BE, 53CE, 53DE of the display item columns (53BAA~53BAC, 53BBA~53BBC), (53CAA~53CAC, 53CBA~53CBC), (53DAA~53DAC, 53DBA~53DBC), no determination is made as to whether the current situation is such that the information for that display item should be displayed on the target participant's display device 14A, 14B. As a result, the information for these display items will not be displayed on the target participant's display device 14A, 14B.

[0078] On the other hand, the display requirements management table 54 is a table used to manage the requirements (hereinafter referred to as "display requirements") for displaying information for each display item defined in the display rule management tables 53 (53A-53D) for "facilitators," "supervisors," "subordinates," and "guests" on the display devices 14A and 14B of the target participants. The display requirements for each display item are predetermined and stored and managed in the display requirements management table 54 in advance.

[0079] As shown in Figure 10, the display requirements management table 54 is configured to include a display item column 54A and a display requirements column 54B. In the display requirements management table 54, one record (row) corresponds to one display item.

[0080] The display item field 54A stores the name of the corresponding display item, and the display requirements field 54B stores the requirements (display requirements) for displaying the information of that display item on the participant's display device 14A, 14B.

[0081] Therefore, in the example shown in Figure 10, for example, the display item "Number of Agreeers: 1" is a display item that shows the number of other participants who agreed with the content of the utterance made by the target participant (in this case, only "facilitator") while that participant was speaking. This indicates that the requirement for displaying this item is that the target participant is speaking.

[0082] The determination of whether or not the display requirements for each display item are met (i.e., whether or not the information for that display item is in a state to be displayed) is made using the information marked with "○" in Figures 6(B), 7(B), 8(B), and 9(B).

[0083] In Figures 6(B) to 9(B), the "Facial Expression," "Orientation," "Speech," "Speech Time," and "Nodding" columns 53AG, 53BG, 53CG, and 53DG in the "Self" column 53AF, 53BF, 53CF, and 53DF correspond to the facial expression column 51B, orientation column 51C, speech column 51D, volume column 51E, speech time column 51F, and nodding column 51G of the sensing information management table 51 (Figure 4) of the target participant (in this case, "Facilitator"). Corresponding to each of these, the "Facial Expression," "Orientation," "Speech," "Volume," "Speech Time," and "Nodding" columns 53AI, 53BI, 53CI, and 53DI of the "Others" column 53AH, 53BH, ​​53CH, and 53DH correspond to the facial expression column 51B, orientation column 51C, speech column 51D, volume column 51E, speech time column 51F, and nodding column 51G respectively in the sensing information management table 51 for participants other than the target participant.

[0084] Therefore, for example, when determining whether the display requirement for displaying information for a display item such as "Number of consenters: 1" has been met, it is assumed that the target participant ("facilitator") is speaking by referring to the value of the utterance flag stored in the utterance column 51D of the sensing information management table 51 of the target participant. Then, the number of people who have agreed with the content of the target participant's utterance is counted using the information stored in the facial expression column 51B, orientation column 51C, and nod 51G of the sensing information management table 51 of each participant other than the target participant.

[0085] The display information management table 55 is used to manage the correspondence between the display information numbers stored in the display information number columns 53AE, 53BE, 53CE, and 53DE of each display rule management table 53 (53A to 53D), and the specific information that should actually be displayed on the corresponding participant's display device 14A and 14B. This table is created in advance and stored in the information management database 37.

[0086] As shown in Figure 11, the display information management table 55 is configured to include a display information number field 55A and a display information field 55B. The display information field 55B stores the specific content of the information to be displayed on the corresponding participant's display device 14A, 14B, and the display information number field 55A stores the display information number unique to that information that has been assigned to it.

[0087] In Figure 11, information enclosed in parentheses indicates that the specific numerical value of that information should be displayed on the corresponding participant's display device 14A, 14B, while information enclosed in square brackets indicates that the message within the square brackets should be displayed on the corresponding participant's display device 14A, 14B.

[0088] Therefore, in the example in Figure 11, for example, the information assigned the display information number "1" indicates the "number of agreeers" (i.e., the number of participants who agree with what the speaker said), and the information assigned the display information number "4" indicates the message "Like!".

[0089] (2) Specific processing details of each functional unit Next, we will explain the specific processing details of each functional unit of the conference support device 4 described above. However, the processing details of the image processing unit 40, audio processing unit 41, nod detection unit 42, and display processing unit 45 are general-purpose processing details that utilize existing technologies, so we will omit their explanation. Here, we will particularly explain the method for determining the actions of each participant by the action / state determination unit 43 and the method for determining the display information by the display information determination unit 44.

[0090] (2-1) Specific processing details of the action / state determination unit The behavior / state determination unit 43 uses the sensing information management table 51 (Figure 4) for each participant to determine the current behavior of each participant at each of the data analysis timings described above, as follows.

[0091] The behavior / state determination unit 43 first refers to the speech column 51D of the latest record in the sensing information management table 51 of the participant to be processed (target participant). Then, if the speech flag stored in the speech column 51D is "1", the behavior / state determination unit 43 determines that the target participant's current behavior is speaking.

[0092] Furthermore, if the speech flag is "0", the behavior / state determination unit 43 determines that the target participant is currently not speaking. If the behavior / state determination unit 43 determines that the target participant is not speaking, it refers to the speech column 51D of the latest record in the sensing information management table 51 for each participant other than the target participant, and determines whether each participant other than the target participant is speaking or not in the same manner as described above. If the behavior / state determination unit 43 determines that any participant other than the target participant is speaking, it determines whether the target participant is listening attentively to what that participant (speaker) is saying or not.

[0093] In this embodiment, a participant is considered to be listening attentively to a speaker if they nod at least once after the speaker begins speaking, or if they have been facing the speaker for a certain percentage of time (for example, 90%, which will be referred to as the first percentage below) since the speaker began speaking (limited to participants in face-to-face meetings).

[0094] The action / state determination unit 43 first refers to the speaker's sensing information management table 51 to determine the time when the speaker started speaking (hereinafter referred to as the speaking start time). This speaking start time is the earliest time when the value of the speaking flag stored in the speaking column 51D of the sensing information management table 51 began to be "1" continuously or nearly continuously up to the present time.

[0095] Next, the behavior / state determination unit 43 refers to the sensing information management table 51 of the target participant and determines whether the nodding flag stored in the nodding column 51G of at least one record from the record of the speaker's utterance start time to the latest record is set to "1", and whether the proportion of records in which the speaker's speaker ID is stored in the orientation column 51C of such records from the record of the utterance start time to the latest record is equal to or greater than the first proportion.

[0096] The behavior / state determination unit 43 then determines that the current behavior of the participant is "not speaking and listening attentively" if it obtains a positive result from at least one of the two determinations. Note that this determination of the participant's behavior / state based on the direction of their face is limited to participants in face-to-face meetings. The same applies below.

[0097] Furthermore, if the behavior / state determination unit 43 determines that any participant other than the target participant is speaking, it determines whether the target participant is not interested in what that speaker is saying or is performing work unrelated to the meeting (side work).

[0098] In this embodiment, if a participant has not nodded even once since the speaker began speaking, or if the time the participant has been facing the speaker from the time the speaker began speaking until now is less than a certain percentage (for example, 20%, which will be referred to as the second percentage below), it will be determined that the participant is not interested in what the speaker is saying or is performing work unrelated to the meeting (side work).

[0099] The behavior / state determination unit 43 then refers to the sensing information management table 51 of the target participant and determines whether the nodding flags stored in the nodding column 51G of each record from the record of the speaker's utterance start time to the latest record are all set to "0", and whether the proportion of records among those records from the record of the utterance start time to the latest record in which the speaker's speaker ID is stored in the orientation column 51C is less than or equal to the second proportion.

[0100] The behavior / state determination unit 43 then determines that the current behavior of the participant is "not speaking and showing no interest or working on something else" if it obtains a positive result in at least one of the two determinations.

[0101] The behavior / state determination unit 43 performs the above behavior determination process for all participants to determine whether each participant's behavior at that time falls under one of the following categories: "speaking," "not speaking and listening attentively," or "not speaking and not interested or doing other work." The determination results are then stored in the aforementioned behavior / state management table 52 (Figure 5).

[0102] (2-2) Specific processing details of the display information determination unit The display information determination unit 44 obtains the attributes of the target participant from the participant attribute management table 50 (Figure 3). The display information determination unit 44 also obtains the target participant's current behavior ("speaking", "not speaking and listening attentively", "not speaking and not interested / doing something else", or "-"), as determined by the behavior / state determination unit 43, from the determined behavior / state management table 52 (Figure 5).

[0103] The display information determination unit 44 then, based on the acquired attributes of the target participant and the current behavior and status of that participant, refers to the display rule management table 53 corresponding to the attributes of that participant and determines whether there are any display items that should be displayed according to the display information of that participant's display devices 14A and 14B, depending on the situation.

[0104] Furthermore, if such a display item exists, the display information determination unit 44 retrieves the display requirements for that display item from the display requirements management table 54 and determines whether the current situation meets those display requirements. If the display information determination unit 44 determines that the current situation meets those display requirements, it determines the information to be displayed on the target participant's display devices 14A and 14B as follows and outputs the determined information to the display processing unit 45.

[0105] (2-2-1) When the target participant is "facilitating" and "speaking" If the attribute of the target participant is "facilitator" and the current action of that participant is "speaking," then referring to the display rule management table 53A for "facilitator" (Figure 6), the only corresponding display item is "Number of consenters: 1."

[0106] This "Number of Agreeers 1" is a display item that, as described above, displays on the facilitator's display devices 14A and 14B the number of other participants who agreed with the content of the facilitator's utterance while the facilitator was speaking. Referring to the display requirements management table 54 (Figure 10), it is stipulated that the display requirement for "Number of Agreeers 1" is that the target participant ("facilitator") is speaking. In this case, since the facilitator is speaking, the display information determination unit 44 determines that the current situation satisfies the display requirement for "Number of Agreeers 1".

[0107] Therefore, the display information determination unit 44 reads the display information number stored in the display information number column 53AE of the display information column 53AA (Figure 6) for "Number of Agreeers 1" in the display information management table 53A for "Facilitator," and obtains the display information to which that display information number is assigned from the display information management table 55 (Figure 11). In the example in Figure 11, the display information is the "Number of Agreeers," that is, the number of participants who agree with the content of "Facilitator's" utterance, so the display information determination unit 44 counts the number of other participants who agree with the content of "Facilitator's" utterance.

[0108] In this embodiment, if any other participant has nodded at least once since the speaker began speaking, or if any other participant is facing the speaker and their expression is smiling, it is determined that the other participant agrees with what the speaker has said.

[0109] The display information determination unit 44 then refers to the sensing information management tables 51 (Figure 4) of each participant other than "Facilitator" and counts the total number of other participants who agree with the content of "Facilitator's" speech as the number of other participants who agree with the content of "Facilitator's" speech, which is the sum of the number of other participants whose value stored in the facial expression column 51B is "1" and whose participant ID of "Facilitator" is stored in the direction column 51C, and the nodding flag stored in the nodding column 51G of at least one record among the records from the time "Facilitator" started speaking to the present in the corresponding sensing information management table 51.

[0110] The display information determination unit 44 then determines the count result of this count as display information to be displayed on the target participant's display device 14A, 14B, and outputs this count result along with the target participant's participant ID to the display processing unit 45.

[0111] (2-2-2) When the target participant is a "facilitator" and "not speaking" If the attribute of the target participant is "facilitator" and the current behavior / state of that participant is "non-speaking" ("non-speaking and listening attentively" or "non-speaking and not interested / doing something else"), referring to the display rule management table 53A for "facilitators", in the example in Figure 6, three display items are defined for "facilitator" when they are non-speaking: "level of engagement", "number of agreeers: 2", and "number of people wishing to speak".

[0112] Furthermore, since each of these display items, "level of excitement," "number of agreeers (2)," and "number of people wishing to speak," has a display priority of "1," it is necessary to display the corresponding display information for the target participants' display devices 14A and 14B for all display items that meet the display requirements. Therefore, the display information determination unit 44 determines whether the current situation meets the display requirements for each of these three display items, and determines the display information for the display items that meet the display requirements.

[0113] For example, the "level of engagement" display item, as mentioned above, displays the level of engagement in the meeting based on the contributions of participants other than the target participant (in this case, the "facilitator"). Referring to the display requirements management table 54, it is stipulated that the display requirement for "level of engagement" is that participants other than the target participant are speaking. The display information determination unit 44 then determines whether the current situation satisfies this display requirement.

[0114] Specifically, the display information determination unit 44 refers to the sensing information management table 51 for each participant other than the target participant to check whether there is a participant whose speech flag stored in the speech field 51D of the latest record is set to "1". If such a participant exists, the display information determination unit 44 determines that the current situation satisfies the display requirements for the display item "level of excitement".

[0115] Then, if the display information determination unit 44 determines that the current situation meets the display requirements for the display item "level of excitement," it reads the display information number stored in the display information number column 53AE of the "level of excitement" display information column 53ABA in the display rule management table 53A for "facilitation," and obtains the information to which that display information number is assigned from the display information management table 55. In Figure 11, since "level of excitement" is the display information, the display information determination unit 44 calculates the level of excitement of the current meeting in the immediately preceding data analysis period.

[0116] In this embodiment, the level of excitement during the data analysis period is calculated by normalizing and summing the total volume of speech from all participants during the previous data analysis period, the number of participants who had a smiling expression during that period, and the number of participants who nodded during that period.

[0117] The display information determination unit 44 then refers to the sensing information management table 51 for each participant other than "Facilitator," reads all the volume values ​​stored in the volume column 51E of the latest record in each sensing information management table 51, sums them up, and normalizes them. The display information determination unit 44 also normalizes the number of participants other than "Facilitator" whose facial expression column 51B of the latest record in the corresponding sensing information management table 51 contains "1." Furthermore, the display information determination unit 44 normalizes the number of participants other than "Facilitator" whose nod flag in the nod column 51G is set to "1." Finally, the display information determination unit 44 calculates the sum of these normalized values ​​as the level of excitement during the data analysis period.

[0118] The display information determination unit 44 then determines that the excitement level calculated in this way should be displayed on the display devices 14A and 14B of the target participants ("facilitators"), and outputs this excitement level along with the participant ID of the target participant to the display processing unit 45.

[0119] Furthermore, the "Number of Agreeers: 2" display item, as mentioned above, displays the number of other participants who agreed with the content of the statements made by participants other than the "facilitator." Referring to the display requirements management table 54, it is stipulated that the display requirement for "Number of Agreeers: 2" is that a participant other than the "facilitator" is speaking. Therefore, the display information determination unit 44 determines whether the current situation satisfies this display requirement.

[0120] Specifically, the display information determination unit 44 refers to the sensing information management table 51 for each participant other than "facilitator" and checks whether there is a participant whose utterance flag stored in the utterance field 51D of the latest record is set to "1". If such a participant exists, the display information determination unit 44 determines that the current situation satisfies the display requirement for the display item "Number of consenters: 2".

[0121] Then, if the display information determination unit 44 determines that the current situation satisfies the display requirement for the display item "Number of consenters: 2", it reads the display information number stored in the display information number column 53AE of the "Number of consenters: 2" display information column 53ABB in the display information management table 53A for "facilitation", and obtains the display information to which that display information number is assigned from the display information management table 55. In the example in Figure 11, the display information is "Number of consenters", that is, the number of other participants who agree with the content of the speaker's utterance at that time. Therefore, the display information determination unit 44 counts the number of participants who agreed with the content of the speaker's utterance at that time during the immediately preceding data analysis period.

[0122] Specifically, the display information determination unit 44 refers to the sensing information management table 51 of each participant other than the speaker and counts the total number of participants whose numerical value stored in the facial expression column 51B is "1" and whose speaker's participant ID is stored in the direction column 51C, and participants whose nod flag stored in the nod column 51G of at least one record among the records from the time the speaker started speaking to the present in the corresponding sensing information management table 51 is "1", as the number of other participants who agree with the content of the speaker's speech at that time.

[0123] The display information determination unit 44 then determines the count result of this count as display information to be displayed on the target participant's ("facilitator") display device 14A, 14B, and outputs this count result along with the target participant's participant ID to the display processing unit 45.

[0124] Furthermore, the "Participant Wants to Speak" display item displays the participant ID of any participant who appears to want to speak, as described above. Referring to the display requirements management table 54, it is stipulated that the display requirements for "Participant Wants to Speak" are that a participant other than the "facilitator" is speaking, and that there is a participant who appears to want to speak. The display information determination unit 44 then determines whether the current situation satisfies these display requirements.

[0125] Specifically, the display information determination unit 44 determines whether a participant other than "Facilitator" is speaking, in the same manner as for "Number of Agreeers: 2". If the display information determination unit 44 obtains a positive result in this determination, it refers to the sensing information management table 51 for each participant other than "Facilitator" and the current speaker, and determines whether there is a participant whose value stored in the facial expression column 51B of the latest record in the corresponding sensing information management table 51 is "2". If such a participant exists (if there is a participant who seems to want to speak), the display information determination unit 44 determines that the current situation satisfies the display requirements for the display item "Participants Want to Speak".

[0126] Then, if the display information determination unit 44 determines that the display requirements for the display item "Participant Wants to Speak" are met, it reads the display information number stored in the display information number column 53AE of the "Participant Wants to Speak" display information column 53ABC in the display information management table 53A for "facilitator," and obtains the display information to which that display information number is assigned from the display information management table 55. In Figure 11, the participant ID of the participant who looks like they want to speak is the display information, so the display information determination unit 44 obtains the participant ID of that participant.

[0127] The display information determination unit 44 then determines that the acquired participant ID should be displayed on the target participant's display device 14A, 14B, and outputs this participant ID together with the target participant's participant ID to the display processing unit 45.

[0128] (2-2-3) When the participant in question is a "supervisor" and is "currently speaking" On the other hand, if the attribute of the target participant is "supervisor" and the current action of that participant is "speaking", referring to the display rule management table 53B for "supervisors", the corresponding display items are "Number of agreeers: 1" and "Speaking time".

[0129] Furthermore, since the display priority for each of these display items, "Number of Consent 1" and "Speech Time," is "1," it is necessary to display the corresponding display information for the target participants' display devices 14A and 14B for all display items that meet the display requirements. Therefore, the display information determination unit 44 determines whether the current situation meets the display requirements for each of these two display items, and determines the display information for the display items that meet the display requirements.

[0130] For example, the "Many Agreeers 1" display item, as mentioned above, displays the number of other participants who agreed with the content of the target participant's (in this case, "supervisor") statement while the participant was speaking. Referring to the display requirements management table 54, it is stipulated that a large number of other participants agreeing with the content of the target participant's statement is a requirement for displaying "Many Agreeers 1".

[0131] In this embodiment, "Many Agreeers 1" is defined as a situation where the sum of the normalized number of participants whose facial expressions were smiling during the data analysis period, the number of participants who were looking at (facing) the target participant during the data analysis period, and the number of participants who nodded during the data analysis period is equal to or greater than a predetermined threshold (hereinafter referred to as the first agreement threshold), in which case a large number of other participants agreed with the content of the target participant's speech.

[0132] The display information determination unit 44 then normalizes the number of participants other than the target participant whose facial expression column 51B of the latest record in the corresponding sensing information management table 51 contains "1". The display information determination unit 44 also normalizes the number of participants other than the target participant whose participant ID of the target participant is stored in the orientation column 51C of the latest record in the corresponding sensing information management table 51. Furthermore, the display information determination unit 44 normalizes the number of participants other than the target participant whose nod flag in the nod column 51G of the latest record in the corresponding sensing information management table 51 is set to "1".

[0133] The display information determination unit 44 then determines whether the sum of the values ​​obtained through these normalizations is equal to or greater than the first threshold for the number of consenters. If the sum is equal to or greater than the first threshold for the number of consenters, the display information determination unit 44 determines that the current situation satisfies the display requirement for the display item "Number of consenters: 1".

[0134] Then, if the display information determination unit 44 determines that the display requirements for the display item "Many Agreeers 1" are met, it reads the display information number stored in the display information number column 53BE of the display information column 53BAA for "Many Agreeers 1" in the display information management table 53B for "Supervisor," and retrieves the display information to which that display information number is assigned from the display information management table 55. In Figure 11, since "Number of Agreeers" is the display information, the display information determination unit 44 counts the number of participants who agreed with the content of the speaker's utterance during the immediately preceding data analysis period.

[0135] Specifically, the display information determination unit 44 first calculates the sum of the following: the number of participants other than the target participant whose facial expression column 51B of the latest record in the corresponding sensing information management table 51 contains "1"; the number of participants other than the target participant whose participant ID of the target participant is stored in the orientation column 51C of the latest record in the corresponding sensing information management table 51; and the number of participants other than the target participant whose nodding flag in the nodding column 51G of the latest record in the corresponding sensing information management table 51 is set to "1".

[0136] Next, the display information determination unit 44 subtracts the number of participants who are counted multiple times from this total value. Participants who are counted multiple times include those in whom "1" is stored in the facial expression column 51B of the latest record in the corresponding sensing information management table 51, the participant ID of the target participant is stored in the orientation column 51C of that record, and the nodding flag in the nodding column 51G of that record is set to "0". Participants in whom "1" is stored in the facial expression column 51B of the latest record in the corresponding sensing information management table 51, the participant ID of the target participant is not stored in the orientation column 51C of that record, and the nodding flag in the nodding column 51G of that record is set to "1". Furthermore, participants in whom "0" or "1" is stored in the facial expression column 51B of the latest record in the corresponding sensing information management table 51, the participant ID of the target participant is stored in the orientation column 51C of that record, and the nodding flag in the nodding column 51G of that record is set to "1". Furthermore, participants whose latest record in the corresponding sensing information management table 51 has "1" stored in the facial expression column 51B, whose participant ID is stored in the orientation column 51C of that record, and whose nodding flag in the nodding column 51G of that record is set to "1" are also included.

[0137] The display information determination unit 44 then determines the number of other participants who agreed with the content of the target participant's speech, as determined in this way, to be displayed on the target participant's ("supervisor's") display devices 14A and 14B, and outputs this number along with the target participant's participant ID to the display processing unit 45.

[0138] Furthermore, the "Speech Time" display item, as mentioned above, displays the speech time of the target participant on the target participant's display device 14A, 14B. Referring to the display requirements management table 54, it is stipulated that the participant must be speaking as a display requirement for "Speech Time". Therefore, if the target participant is speaking, it is immediately determined that the display requirement for the "Speech Time" display item is met.

[0139] Then, if the display information determination unit 44 determines that the display requirements for the display item "speech time" are met, it reads the display information number stored in the display information number column 53BE of the "speech time" display information column 53BAC in the display information management table 53B for "supervisor," and obtains the display information to which that display information number is assigned from the display information management table 55.

[0140] In the example shown in Figure 11, the displayed information is the elapsed time from when the participant started speaking until the present. Therefore, the display information determination unit 44 retrieves the speech time stored in the speech time column 51F of the latest record in the participant's sensing information management table 51.

[0141] The display information determination unit 44 then determines the acquired utterance time as display information to be displayed on the target participant's display device 14A, 14B, and outputs this utterance time along with the target participant's participant ID to the display processing unit.

[0142] (2-2-4) When the target participant is a "supervisor" and "is not speaking and is listening attentively" If the attribute of the target participant is "supervisor" and the current behavior of that participant is "not speaking and listening attentively," then referring to the display rule management table 53B for "supervisors," the only corresponding display item is "interfering."

[0143] The "Interjection" display item, as described above, is a display item that prompts the target participant to speak when they look like they want to speak and when other participants have finished speaking. Referring to the display requirements management table 54, the display requirements for "Interjection" are defined as the target participant looking like they want to speak and the other participants having finished speaking. The display information determination unit 44 then determines whether the current situation satisfies these display requirements.

[0144] Specifically, the display information determination unit 44 refers to the sensing information management table 51 of the target participant and determines whether "2" is stored in the facial expression field 51B of the latest record. Furthermore, the display information determination unit 44 refers to the sensing information management table 51 of each participant other than the target participant and determines whether there is a participant whose utterance flag stored in the utterance field 51D of the record immediately preceding the latest record is "1", and whose utterance flag stored in the utterance field 51D of the latest record is set to "0". If the display information determination unit 44 obtains a positive result from both of these determinations, it determines that the current situation satisfies the display requirements for the display item "interjection".

[0145] Then, if the display information determination unit 44 determines that the current situation satisfies the display requirements for the display item "interfering", it reads the display information number stored in the display information number column 53BE of the "interfering" display information column 53BBA in the display information management table 53B for "supervisor", and obtains the display information to which that display information number is assigned from the display information management table 55.

[0146] In the example in Figure 11, the message "Now is your chance to speak!" is the display information. Therefore, the display information determination unit 44 determines that this message should be displayed on the display devices 14A and 14B of the target participant, and outputs the text data of this message along with the participant ID of the target participant to the display processing unit 45.

[0147] (2-2-5) When the target participant is a "supervisor" and is "not speaking and not interested / doing other work" If the target participant's attribute is "supervisor" and their behavior is "not speaking and not interested / working on another task," then referring to the display rule management table 53B for "supervisors" reveals no display items. Therefore, in the case of "not speaking and not interested / working on another task," regardless of the target participant's ("supervisor") behavior, no information is displayed on the target participant's display devices 14A and 14B. This takes into account the target participant's attribute (position) as "supervisor."

[0148] (2-2-6) When the participant in question is a "subordinate" and is "currently speaking" On the other hand, if the attribute of the target participant is "subordinate" and the current action of that participant is "speaking", referring to the display rule management table 53C for "subordinates", there are two corresponding display items: "Many agreeers 2" and "Supporter".

[0149] Furthermore, since both the "Many Agreeers 2" and "Support" display items have a display priority of "1," all display items that meet the display requirements must display the corresponding display information for the target participants' display devices 14A and 14B. Therefore, the display information determination unit 44 determines whether the current situation meets the display requirements for each of these two display items, and determines the display information for the display items that meet the display requirements.

[0150] For example, the "Many Agreeers 2" display item, as mentioned above, displays the number of other participants who agreed with the content of the target participant's (in this case, "subordinate") statement while the target participant was speaking. Referring to the display requirements management table 54, it is stipulated that a large number of other participants, other than the target participant, agreed with the content of the target participant's statement as a display requirement for "Many Agreeers 2".

[0151] In this embodiment, "Many Agreeers 2" is defined as a large number of other participants agreeing with the content of the target participant's speech if the sum of the normalized numbers of participants who were smiling during the data analysis period, the number of participants who were looking at (turning their faces toward) the target participant during the data analysis period, and the number of participants who nodded during the data analysis period is equal to or greater than a predetermined threshold (a value smaller than the first agreeer threshold, hereinafter referred to as the second agreeer threshold). The second agreeer threshold is set smaller than the first agreeer threshold in order to build confidence in participants with the attribute of "subordinate."

[0152] The display information determination unit 44 then determines whether the current situation satisfies the display requirements for the display item "Many Agreeers 1," in the same manner as in the case of "Many Agreeers 1." If the display information determination unit 44 determines that the display requirements for the display item "Many Agreeers 2" are met, it counts the number of other participants who agreed with the content of the target participant's utterance, in the same manner as in the case of "Many Agreeers 1."

[0153] The display information determination unit 44 then determines the number of other participants who agreed with the content of the target participant's speech, as determined in this way, to be displayed on the target participant's ("subordinate") display devices 14A and 14B, and outputs this number along with the target participant's participant ID to the display processing unit 45.

[0154] Furthermore, the "Encourage" display item, as mentioned above, is a display item that displays a message encouraging the participant to speak on the participant's display devices 14A and 14B when the participant is speaking. Referring to the display requirements management table 54, it is stipulated that the participant must be speaking as a display requirement for "Encourage". Therefore, when the participant is speaking, it is immediately determined that the display requirements for the "Encourage" display item are met.

[0155] Then, if the display information determination unit 44 determines that the display requirements for the display item "support" are met, it reads the display information number stored in the display information number column 53CE of the "support" display information column 53CAB in the display information management table 53C for "subordinates," and obtains the display information to which that display information number is assigned from the display information management table 55.

[0156] In the example in Figure 11, the message "Like!" is the display information, so the display information determination unit 44 determines that this message should be displayed on the display devices 14A and 14B of the target participant, and outputs the text data along with the participant ID of the target participant to the display processing unit 45.

[0157] (2-2-7) When the target participant is a "subordinate" and "not speaking and listening attentively" If the attribute of the participant in question is "subordinate" and the participant's current behavior is "not speaking and listening attentively," then referring to the display rule management table 53C for "subordinates," the corresponding display items are "interjection" and "influence / effect."

[0158] Furthermore, in the display category "Non-speaking and listening attentively," the display priority for the "Interrupting" category is set to "1," while the display priority for the "Not interested / Working on something else" category is set to "2."

[0159] Therefore, the display information determination unit 44 first determines whether the current situation meets the display requirements for the display item "interference," and if the display requirements are met, it determines the display information for that display item.

[0160] Specifically, the display information determination unit 44, in the same manner as in the case of "supervisor" "interference," determines whether the current situation of the target participant meets the display requirements for "interference." If it determines that the current situation meets the display requirements for the display item "interference," it determines that the display information to be displayed on the target participant's display devices 14A and 14B is the message "Now is your chance to speak up!" and outputs the text data along with the target participant's participant ID to the display processing unit 45.

[0161] On the other hand, if the display information determination unit 44 determines that the current situation does not meet the display requirements for the "interference" display item, it determines whether the current situation meets the display requirements for the "impact / effect" display item. If the display information determination unit 44 determines that the display requirements are met, it determines the display information for that display item.

[0162] For example, the "Impact / Effect" display item, as mentioned above, displays the number of participants who have spoken on a topic and the current duration of that topic on the participant's display device, while the discussion on that topic continues. Referring to the display requirements management table 54, it is stipulated that the topic initiated by the participant is still ongoing as a display requirement for "Impact / Effect".

[0163] In a typical meeting, a "discussion" takes place for a certain period of time in which one or more participants speak continuously on a given topic. This is followed by a period of silence (a period during which no participants speak), after which a "discussion" on the next topic begins. Therefore, by referring to each participant's sensing information management table 51, the most recent period of silence (a period during which no participants speak) exceeding a certain duration can be detected. If the target participant spoke at the beginning of that period of silence, it can be determined that the topic initiated by that participant is still being discussed.

[0164] The display information determination unit 44 then refers to the sensing information management table 51 for each participant and, working backward from the most recent record, detects a period of silence in which the utterance flag stored in the utterance column 51D of the sensing information management table 51 for all participants is "0" for a certain period of time. The display information determination unit 44 then determines whether or not the first person to utter after the end of that period of silence was the target participant.

[0165] Specifically, the display information determination unit 44 determines whether the utterance flag stored in the utterance column 51D of the first record after the end of the silence period in the sensing information management table 51 of the target participant is "1". If the utterance flag is "1", the display information determination unit 44 determines that the target participant initiated the current topic and that the display requirements for the display item "Influence / Effect" are met.

[0166] Then, if the display information determination unit 44 determines that the display requirements for the display item "impact / effect" are met, it reads the display information number stored in the display information number column 53CE of the "impact / effect" display information column 53CBB in the display information management table 53C for "subordinates," and retrieves the display information to which that display information number is assigned from the display information management table 55.

[0167] In the example shown in Figure 11, the number of participants in the topic and the duration of the topic are the displayed information, so the display information determination unit 44 calculates these participant numbers and durations, respectively.

[0168] Specifically, the display information determination unit 44 first refers to the sensing information management table 51 of the target participant and obtains the elapsed time stored in the elapsed time column 51A of the record at the time the current topic started and the elapsed time stored in the elapsed time column 51A of the latest record, respectively, and calculates the duration of the current topic by subtracting the former elapsed time from the latter elapsed time.

[0169] The display information determination unit 44 also refers to the speech flag in the speech column 51D of each participant's sensing information management table 51 and counts the number of participants whose speech flag stored in the speech column 51D of at least one record from the record at the time the current topic started to the latest record is "1". This number is the number of participants who have joined the current topic.

[0170] The display information determination unit 44 then determines the number of participants in the current topic and the duration of the current topic, which have been determined in this manner, as display information to be displayed on the target participants' display devices 14A and 14B, and outputs these participant numbers and durations, along with the participant IDs of the target participants, to the display processing unit 45.

[0171] (2-2-8) When the target participant is a "subordinate" and "not speaking and not interested / doing other work" If the attribute of the target participant is "subordinate" and the current behavior of that participant is "not speaking" and "not interested / working on another task", then referring to the display rule management table 53C for "subordinates" will show that the only corresponding display item is "non-listening time".

[0172] The "Non-listening time" display item, as mentioned above, is used to show the time spent (non-listening time) when a participant is presumed to be uninterested in the meeting or engaged in work unrelated to the meeting.

[0173] Referring to the display requirements management table 54, if the participant is an online meeting participant, the display requirement for "non-listening time" is stipulated as the participant not nodding even once in the recent period of time. If the participant is an in-person meeting participant, the display requirement for "non-listening time" is stipulated as the participant not looking at the speaker (not facing the speaker) even once in the recent period of time.

[0174] Therefore, if the target participant is an online meeting participant, the display information determination unit 44 refers to the target participant's sensing information management table 51 and determines whether all nod flags stored in the nod column 51G of each record within the most recent period of time are "0". If the display information determination unit 44 obtains a negative result in this determination, it determines that the current situation satisfies the display requirements for the display item "non-listening time".

[0175] Furthermore, if the target participant is participating in the meeting in person, the display information determination unit 44 refers to the sensing information management table 51 of each participant other than the target participant to identify all participants who have spoken within the most recent fixed time period, for each time period. The display information determination unit 44 then refers to the target participant's sensing information management table 51 to determine whether the participant ID of the speaker for that time period has never been stored in the orientation column 51C of the record for that time period within the most recent fixed time period. If the target participant receives a positive result in this determination, the display information determination unit 44 determines that the current situation satisfies the display requirements for the display item "non-listening time".

[0176] Then, if the display information determination unit 44 determines that the current situation satisfies the display requirements for the display item "non-listening time," it reads the display information number stored in the display information number column 53CE of the "non-listening time" display information column 53CBC in the display information management table 53C for "subordinates," and retrieves the display information to which that display information number is assigned from the display information management table 55. In the example in Figure 11, since "non-listening time" is the display information, the display information determination unit 44 retrieves the most recent non-listening time of the target participant up to that point.

[0177] Specifically, if the target participant is an online meeting participant, the display information determination unit 44 refers to the target participant's sensing information management table 51 and retrieves the elapsed time stored in the elapsed time column 51A of the oldest record in which the nodding flag value stored in the nodding column 51G is consecutively "0", working backward from the latest record. The display information determination unit 44 then calculates the time when the participant is not nodding as the non-listening time by subtracting the elapsed time obtained as described above from the elapsed time stored in the elapsed time column 51A of the latest record.

[0178] The display information determination unit 44 then determines that the non-listening time calculated in this manner should be displayed on the display devices 14A and 14B of the target participant, and outputs this non-listening time, along with the participant ID of the target participant, to the display processing unit 45.

[0179] Furthermore, if the target participant is a face-to-face meeting participant, the display information determination unit 44 refers to the target participant's sensing information management table 51 and retrieves the elapsed time stored in the elapsed time column 51A of the oldest record in which the participant ID of the participant who spoke during that time period is not consecutively stored in the direction column 51C, working backward from the latest record. The display information determination unit 44 then calculates the time during which the target participant is not facing the speaker as non-listening time by subtracting the elapsed time obtained as described above from the elapsed time stored in the elapsed time column 51C of the latest record.

[0180] The display information determination unit 44 then determines that the non-listening time calculated in this manner should be displayed on the display devices 14A and 14B of the target participant, and outputs this non-listening time, along with the participant ID of the target participant, to the display processing unit 45.

[0181] (2-2-9) When the target participant is a "guest" and "currently speaking" When the attribute of the target participant is "Guest" and the target participant's current action is "Speaking," the processing content of the display information determination unit is the same as when the "Subordinate" is "Speaking." Therefore, the explanation here is omitted.

[0182] (2-2-10) When the target participant is a "guest" and "not speaking and listening attentively" If the attribute of the target participant is "Guest" and the current behavior of that participant is "Non-speaking and listening attentively," then referring to the display rule management table 53D for "Guests," the only corresponding display item is the display item "Interfering."

[0183] Therefore, the display information determination unit 44 determines, with respect to the display item "interfering," whether the current situation meets the display requirements for that item, in the same way as for "supervisor's interference." If the display requirements are met, it determines the display information for that item to be "Now is your chance to speak up," and outputs the determined display information along with the participant ID of the participant in question to the display processing unit 45.

[0184] (2-2-11) When the target participant is a "guest" and "not speaking and not interested / doing other work" If a participant's attribute is "Guest" and their behavior is "Non-speaking and uninterested / working on another task," then referring to the display rule management table 53D for "Guests" reveals no display items. Therefore, in the case of "Non-speaking and uninterested / working on another task," no information is displayed on the participant's display devices 14A and 14B, regardless of the participant's behavior. This takes into account the participant's attribute (position) as a "Guest."

[0185] (2-3) Processing flow of the action / state determination unit and the display information determination unit (2-3-1) Action / State Determination Process Next, the flow of a series of processes executed by the action / state determination unit 43 (hereinafter referred to as the action / state determination process) will be explained with reference to Figure 12. Note that the specific processing details of the action / state determination unit 43 at each step in Figure 12 have already been described in detail, so any redundant explanations will be omitted.

[0186] Figure 12 shows the flow of the action / state determination process executed by the action / state determination unit 43 in the determination cycle described above. The action / state determination unit 43 determines the action of each participant at that time according to the processing procedure shown in Figure 12.

[0187] In practice, when the behavior / state determination unit 43 starts this behavior / state determination process, it first stores the various information of each participant provided by the image processing unit 40, the voice processing unit 41, and the nodding detection unit 42 into the corresponding participant's sensing information management table 51 (Figure 4) (S1).

[0188] Specifically, the behavior / state determination unit 43 stores numerical values ​​corresponding to the facial expressions of each participant, as notified by the image processing unit 40, in the facial expression column 51B of the corresponding sensing information management table 51. The behavior / state determination unit 43 also stores the participant IDs of other participants that each participant is facing, as notified by the image processing unit 40, in the direction column 51C of that participant's sensing information management table 51.

[0189] Furthermore, based on the information notified by the voice processing unit 41 about whether or not each participant is speaking, the behavior / state determination unit 43 sets the speech flag stored in the speech column 51D of the participant's sensing information management table 51 to "1" if the participant is speaking, and sets the speech flag to "0" if the participant is not speaking. The behavior / state determination unit 43 also stores the speech volume and speech time of each participant, notified by the voice processing unit 41, in the volume column 51E and speech time column 51F of the participant's sensing information management table 51, respectively.

[0190] Furthermore, based on the information received from the nodding detection unit 42 regarding whether each participant is nodding, the behavior / state determination unit 43 sets the nodding flag stored in the nodding column 51G of the participant's sensing information management table 51 to "1" if the participant is nodding, and sets the nodding flag to "0" if the participant is not nodding.

[0191] Next, the behavior / state determination unit 43 selects one participant whose steps from step S3 onwards have not yet been processed, and refers to the speech column 51D of the sensing information management table 51 of the selected participant (hereinafter referred to as the selected participant in the explanation of Figure 12) to determine whether or not the selected participant is currently speaking (S2).

[0192] Then, if the behavior / state determination unit 43 obtains a positive result in this determination, it determines that the behavior / state of the selected participant at that time is "speaking" (S4), and stores the determination result in the determined behavior / state management table 52 (Figure 5) in association with the participant ID of the selected participant (S10).

[0193] Furthermore, if the behavior / state determination unit 43 obtains a negative result in step S3, it refers to the sensing information management table 51 of the selected participant to determine whether or not the selected participant is seriously listening to what the speaker said at that time (S5).

[0194] Then, if the behavior / state determination unit 43 obtains a positive result in this determination, it determines that the behavior / state of the selected participant at that time is "not speaking and listening attentively" (S6), and stores the determination result in the determined behavior / state management table 52 in association with the participant ID of the selected participant (S10).

[0195] Furthermore, if the behavior / state determination unit 43 obtains a negative result in step S5, it refers to the sensing information management table 51 of the selected participant to determine whether the selected participant is not interested in the content of the speaker's utterance at that time, or whether they are performing an action unrelated to the meeting (S7).

[0196] Then, if the behavior / state determination unit 43 obtains a positive result in this determination, it determines that the behavior / state of the selected participant at that time is "not speaking and not interested / doing something else" (S8), and stores the determination result in the determined behavior / state management table 52 in association with the participant ID of the selected participant (S10).

[0197] In response, if the behavior / state determination unit 43 obtains a negative result in the determination in step S7, it determines that the behavior / state of the selected participant at that time does not fall under any of the following categories: "speaking," "not speaking and listening attentively," or "not speaking and not interested / doing something else" (S9), and stores the determination result in the determination behavior / state management table 52 in association with the participant ID of the selected participant (S10). In this case, "-" is stored in the determination behavior / information field 52B of the record corresponding to the selected participant in the determination behavior / state management table 52 (see the record with participant ID "0004" in Figure 5).

[0198] Next, the behavior / state determination unit 43 determines whether the processing in steps S3 to S10 has been completed for all participants (S11). If the behavior / state determination unit 43 obtains a negative result in this determination, it returns to step S2, and then repeats the processing in steps S2 to S11, sequentially switching the participant selected in step S2 to other participants whose processing in steps S3 and beyond has not yet been completed.

[0199] Through this iterative process, the actions and states of each participant at that time are sequentially determined, and the determined actions and states are sequentially stored in the determined actions and states management table 52.

[0200] The behavior / state determination unit 43 then finishes determining the behavior / state of all participants at that time and stores the determination results in the determined behavior / state management table 52. If a positive result is obtained in step S11, the behavior / state determination process is terminated.

[0201] (2-3-2) Display Information Determination Process Next, the flow of a series of processes executed by the display information determination unit 44 (hereinafter referred to as the display information determination process) will be explained with reference to Figure 13. Note that the specific processing details of the display information determination unit at each step in Figure 13 have already been described in detail, so any redundant explanations will be omitted.

[0202] Figure 13 shows the flow of the display information determination process performed by the display information determination unit 44 in the determination cycle described above. The display information determination unit 44 determines the information to be displayed at that time on the necessary participant's display devices 14A and 14B according to the processing procedure shown in Figure 13.

[0203] In practice, when the display information determination unit 44 starts this display information determination process, it first selects one participant whose steps from step S21 onwards have not yet been processed (S20). The display information determination unit 44 also obtains the attributes of the selected participant (hereinafter referred to as the selected participant in the explanation of Figure 13) from the participant attribute management table 50 (Figure 3), and obtains the selected participant's current behavior and state from the determined behavior and state management table 51 (Figure 4) (S21).

[0204] Next, the display information determination unit 44, based on the attributes of the selected participant and their current behavior / status acquired in step S21, refers to the display rule management table 53 corresponding to the attributes of the selected participant and identifies the display items that should be displayed on the selected participant's display devices 14A and 14B according to the situation (S22).

[0205] Next, in step S22, the display information determination unit 44 determines whether it was able to identify the display items that should be displayed on the selected participant's display devices 14A and 14B depending on the situation (S23). If the display information determination unit 44 was unable to identify the display items in this determination (for example, in the case of "supervisor" who is "not speaking and not interested / doing other work"), the unit proceeds to step S29.

[0206] In response, if the display information determination unit 44 obtains a positive result in the determination in step S23, it identifies all the display items with the highest display priority among the display items identified in step S22 (S24). The display information determination unit 44 also refers to the display requirements management table 54 (Figure 10) for each display item identified in step S24 and determines whether the current situation satisfies the display requirements for displaying the information of that display item (S25). If the display information determination unit 44 obtains a negative result in this determination, it proceeds to step S29.

[0207] In response, if the display information determination unit 44 obtains a positive result in the determination in step S25, it refers to the display information management table 55 (Figure 11) for all display items for which a positive result was obtained and identifies the information to be displayed (S26). The display information determination unit 44 also obtains the identified information from the display information management table 55 by reading or calculation (S27).

[0208] The display information determination unit 44 then outputs the information acquired in step S27 along with the participant ID of the selected participant to the display processing unit 45 (S28), and then determines whether or not the processing from step S21 onwards has been completed for all participants (S29).

[0209] If the display information determination unit 44 obtains a negative result in this determination, it returns to step S20, and thereafter repeats the processing from step S20 to step S29, sequentially switching the participant selected in step S20 to other participants that have not yet been processed in step S21 and beyond.

[0210] Through this repetitive process, the information to be displayed and the participant ID of each participant are sequentially transmitted to the display processing unit 45 for all participants whose information should be displayed on the corresponding display devices 14A and 14B. As a result, the information is displayed on the display devices 14A and 14B of these participants.

[0211] Then, the display information determination unit 44 finishes executing the processes in steps S21 to S28 for all participants, and when it obtains a positive result in step S29, it terminates the display information determination process.

[0212] (3) Effects of this embodiment As described above, the conference support device 4 of this embodiment determines the current behavior and state of each participant in real time based on various information regarding the behavior and state of each participant collected by the sensor group 10A of the face-to-face participant system 2 and the sensor group 10B of each online participant system 3, and appropriately displays information corresponding to the participant's attributes and the participant's behavior and state at that time on the participant's display devices 14A and 14B, respectively.

[0213] Therefore, with the conference support device 4, information corresponding to each participant's position and their actions / state at that time is presented to the participant in real time via their terminal, thus enabling the presentation of useful information to each participant in real time.

[0214] (4) Other embodiments In the above-described embodiment, a display device 14A and 14B are placed on each participant's side as terminals for presenting information (hereinafter referred to as information presentation devices), and the necessary information is displayed on the display devices 14A and 14B to be presented to the necessary participants. However, the present invention is not limited to this, and for example, robots may be placed on some or all participants' sides as such information presentation terminals instead of display devices 14A and 14B, and information may be presented to participants by the robot's movements rather than by display. Alternatively, a display device capable of displaying information may be provided on the robot so that information is presented not only by the robot's movements but also by display. Furthermore, a speaker may be provided on the robot so that it can speak, and messages or other information may be presented to participants by voice or sound.

[0215] Furthermore, while the above-described embodiment mentions a case where the attributes (positions) of meeting participants are of four types: "facilitator," "supervisor," "subordinate," and "guest," the present invention is not limited to this, and other attributes may be defined. For example, if the meeting is a workshop, the attributes of the participants may be limited to two types: "instructor" and "student." [Industrial applicability]

[0216] The present invention relates to a conference support device and method, and can be applied to a conference support device that supports a conference including online participants. [Explanation of symbols]

[0217] 1...Conference support system, 2...In-person participant system, 3...Online participant system, 4...Conference support device, 11A,11B...Imaging device, 12A,12B...Sound collection device, 13A,13B...Vibration sensor, 14A,14B...Display device, 20...CPU, 40...Image processing unit, 41...Audio processing unit, 42...Nodding detection unit, 43...Action / state determination unit, 44...Display information determination unit, 45...Display processing unit, 50...Participant attribute management table, 51...Sensing information management table, 52...Judgment action / state management table, 53,53A~53D...Display rule management table, 54...Display requirements management table, 55...Display information management table.

Claims

1. A conference support device that assists in meetings, An action / state determination unit that determines the action / state of each participant in real time based on the output of each sensor that acquires information about the action / state of each participant, A presentation information determination unit determines the information to be presented to the participant based on the participant's behavior and state determined by the behavior / state determination unit and the predetermined position of each participant. A presentation processing unit presents the information determined by the presentation information determination unit to the participant via a terminal provided on the participant's side. Equipped with, For each combination of the aforementioned position and the aforementioned actions / states, the items for which information should be presented to the participant are predetermined. The aforementioned information presentation determination unit, For each participant, the information to be presented to that participant shall be determined to be the information corresponding to the participant's position and the aforementioned actions / states. A conference support device characterized by the following features.

2. Each of the aforementioned items has pre-defined requirements for presenting the information of that item to the participant. The aforementioned information presentation determination unit, The information of the aforementioned items that meets the aforementioned requirements is determined to be the information to be presented to the aforementioned participants. The conference support device according to feature 1.

3. If multiple items are set for a single combination of the aforementioned position and the aforementioned actions / states, a priority order is set for each of the aforementioned items. The aforementioned information presentation determination unit, If there are multiple items that satisfy the aforementioned requirements, the information of the item with the highest priority will be determined as the information to be presented to the participant. The conference support device according to feature 2.

4. At least a portion of the information to be presented to the participant in accordance with the predetermined actions and states differs depending on the position. The conference support device according to feature 1.

5. A conference support method performed by a conference support device that assists in conferences, The first step involves determining the behavior and state of each participant in real time based on the output of each sensor that acquires information about the behavior and state of each participant, A second step involves determining the information to be presented to the participant based on the judged participant's behavior and status, and the pre-set position of each participant. A third step is to present the determined information to the participant via a terminal provided on the participant's side. Equipped with, For each combination of the aforementioned position and the aforementioned actions / states, the items for which information should be presented to the participant are predetermined. In the second step described above, the conference support device is For each participant, the information to be presented to that participant shall be determined to be the information corresponding to the participant's position and the aforementioned actions / states. A method for supporting meetings characterized by the following features.

6. Each of the aforementioned items has pre-defined requirements for presenting the information of that item to the participant. In the second step described above, the conference support device is The information of the aforementioned items that meets the aforementioned requirements is determined to be the information to be presented to the aforementioned participants. The conference support method according to feature 5.

7. If multiple items are set for a single combination of the aforementioned position and the aforementioned actions / states, a priority order is set for each of the aforementioned items. In the second step described above, the conference support device is If there are multiple items that satisfy the aforementioned requirements, the information of the item with the highest priority will be determined as the information to be presented to the participant. The conference support method described in feature 6.

8. At least a portion of the information to be presented to the participant in accordance with the predetermined actions and states differs depending on the position. The conference support method according to feature 5.

Images