Meeting assistance system, meeting assistance method, and meeting assistance program
The meeting support system enhances meeting quality by using image analysis to provide real-time feedback to speakers, addressing issues of low engagement and poor speaking skills.
Patent Information
- Application Number
- JP2023077661
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-21
AI Technical Summary
Speakers with little experience or poor speaking skills often lead to meetings with low audience engagement and poor quality due to interruptions and lack of interest.
A meeting support system that includes an imaging unit to capture speaker and audience images, determines the meeting status based on facial expressions and gestures, and provides support information to enhance the presentation.
Improves meeting quality by providing speakers with actionable feedback to increase audience engagement and maintain interest.
Smart Images

Figure 2024162292000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a technology for supporting speakers who speak at meetings. [Background technology]
[0002] Conventionally, there are known techniques for evaluating the liveliness of a meeting such as a conference. For example, there is known a device that calculates motion information representing the motions or phenomena of meeting participants individually, calculates correlation data between the meeting participants, and evaluates the liveliness of a meeting by taking into consideration the relationship between the meeting participants, such as the degree of conversation elicitation between the meeting participants (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2009-163431 A Summary of the Invention [Problem to be solved by the invention]
[0004] For example, when a speaker (such as a presentation presenter, conference chairperson, or participant) speaks at a meeting, if he or she has little experience or poor speaking skills, the audience will lose focus and interest, resulting in a decline in the quality of the meeting.
[0005] An object of the present disclosure is to provide a meeting support system, a meeting support method, and a meeting support program that are capable of improving the quality of a meeting by supporting speakers in the meeting. [Means for solving the problem]
[0006] A meeting support system according to one aspect of the present disclosure includes an acquisition processing unit, a determination processing unit, and a presentation processing unit. The acquisition processing unit acquires captured images captured by an imaging unit that captures a speaker who will speak at a meeting and listeners who listen to the speaker's remarks. The determination processing unit determines a status of the meeting based on the captured images acquired by the acquisition processing unit. The presentation processing unit presents support information to the speaker according to the status of the meeting determined by the determination processing unit.
[0007] A meeting support method according to another aspect of the present disclosure is a method in which one or more processors execute the following steps: acquiring an image captured by an imaging unit that captures an image of a speaker speaking at a meeting and an audience listening to the speaker's remarks; determining a status of the meeting based on the captured image; and presenting support information to the speaker according to the status of the meeting.
[0008] A meeting support program according to another aspect of the present disclosure is a program for causing one or more processors to execute the following operations: acquiring images captured by an imaging unit that captures images of a speaker speaking at a meeting and attendees listening to the speaker's remarks; determining the status of the meeting based on the captured images; and presenting support information to the speaker according to the status of the meeting. Effect of the Invention
[0009] According to the present disclosure, it is possible to provide a meeting support system, a meeting support method, and a meeting support program that are capable of improving the quality of a meeting by supporting speakers in the meeting. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a schematic diagram showing an application example of a meeting support system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a functional block diagram showing a configuration of the meeting support system according to an embodiment of the present disclosure. [Diagram 3]FIG. 3 is a diagram illustrating an example of score information used in the meeting support system according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of a support information list used in the meeting support system according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a diagram illustrating an example of a meeting support screen displayed on the operation terminal according to the embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating an example of an image display field of the meeting support screen displayed on the operation terminal according to the embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example of an image display field of the meeting support screen displayed on the operation terminal according to the embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating another example of the image display field of the meeting support screen displayed on the operation terminal according to the embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating an example of a meeting support screen displayed on the operation terminal according to the embodiment of the present disclosure. [Figure 10] FIG. 10 is a flowchart illustrating an example of a procedure of a meeting support process executed by the meeting support system according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating another example of the meeting support screen displayed on the operation terminal according to the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments are examples of the present disclosure and are not intended to limit the technical scope of the present disclosure.
[0012] The meeting support system 10 according to this embodiment includes a meeting support device 1, an operation terminal 2, and a camera 3. The meeting support system 10 is applied to various types of meetings, such as presentations and seminars where a presenter makes a presentation or gives a lecture to listeners (audience, students, participants, etc.), and conferences (e.g. Web conferences) attended by multiple participants. FIG. 1 shows a specific example of the meeting support system 10 applied to a presentation meeting. For example, in a conference room R1, presenter A (presenter) makes a presentation to multiple listeners. Presenter A also makes the presentation using the operation terminal 2.
[0013] The camera 3 is a digital camera that captures an image of a subject and outputs it as digital image data. The camera 3 has a communication function that transmits the digital image data to the meeting supporting device 1 through the network N1. The camera 3 may be one or more. Specifically, the camera 3 is installed in a place where it can capture an image of a predetermined area to be captured (image capture area), captures the image capture area at a predetermined frame rate, and sequentially transmits image data (frame images) to the meeting supporting device 1. For example, as shown in FIG. 1, the camera 3 is installed in a meeting room R1 at a position where it can capture images of the bodies (faces, hands, etc.) of the participants (presenter A, audience). The camera 3 may be a camera that is installed in advance in the meeting room R1, or a camera that is mounted on the meeting supporting device 1 or the operation terminal 2.
[0014] 2 is a functional block diagram showing the configuration of the meeting support system 10. The meeting support device 1 is connected to an operation terminal 2 and a camera 3 via a network N1 so as to be able to perform data communication. The meeting support system 10 performs a meeting support process in which it performs image analysis on the captured image acquired from the camera 3, estimates the emotions and interest levels of each participant (presenter A, audience), determines the status of the meeting (presentation), and presents presenter A with support information according to the status of the meeting. The specific configuration of the meeting support process will be described below.
[0015] In this embodiment, the meeting support device 1 alone corresponds to the meeting support system according to the present disclosure, but the meeting support system according to the present disclosure may include one or more of the components of the meeting support device 1, the operation terminal 2, and the camera 3. For example, when the components of the meeting support device 1, the operation terminal 2, and the camera 3 cooperate to share and execute a meeting support process (see FIG. 10) described below, a system including the components that execute the process can be regarded as the meeting support system according to the present disclosure. Also, for example, the meeting support device 1 and the operation terminal 2 may constitute the meeting support system according to the present disclosure.
[0016] [Meeting support device 1] 2, the meeting supporting device 1 includes a control unit 11, a storage unit 12, an operation display unit 13, and a communication unit 14. The meeting supporting device 1 may be an information processing device (server device) such as a personal computer. The meeting supporting device 1 may also be configured as a cloud server.
[0017] The communication unit 14 is a communication interface for connecting the meeting support device 1 to a network N1 by wire or wirelessly and for executing data communication in accordance with a predetermined communication protocol between the meeting support device 1 and external devices such as the operation terminal 2 and the camera 3 via the network N1. The network is formed of, for example, a LAN, the Internet, etc.
[0018] The operation display unit 13 is a user interface that includes a display unit such as a liquid crystal display or an organic EL display that displays various information, and an operation unit such as a touch panel, a mouse, or a keyboard that accepts operations.
[0019] The storage unit 12 is a non-volatile storage unit such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory that stores various information, or a volatile memory such as an SRAM or a DRAM. The storage unit 12 stores a control program such as a meeting support program for causing the control unit 11 to execute a meeting support process (see FIG. 10) described below. For example, the meeting support program is non-temporarily recorded on a computer-readable recording medium such as a CD or a DVD, and is read by a reading device (not shown) such as a CD drive or a DVD drive provided in the meeting support device 1 and stored in the storage unit 12. The meeting support program may be distributed from a cloud server and stored in the storage unit 12.
[0020] The storage unit 12 also stores image data acquired by the meeting support device 1 from the camera 3. For example, the storage unit 12 accumulates image data (frame images) captured by the camera 3 at a predetermined frame rate.
[0021] The memory unit 12 also stores score information D1 relating to scores (evaluation values) corresponding to each participant's feelings, level of interest, etc. regarding the meeting (presentation) estimated from the captured images, and a support information list D2 relating to support information corresponding to the situation of the meeting.
[0022] FIG. 3 is an example of the score information D1. In the score information D1, a person ID and a score are registered in association with each other. The person ID is identification information for identifying the participants of the meeting (presenter A, audience), and is given any identification information by the control unit 11. The score is an evaluation value indicating each participant's feelings and degree of interest in the meeting. The control unit 11 calculates the score at a predetermined time interval (for example, for each frame) and registers it in the score information D1. In addition, the control unit 11 updates the minimum value (minimum score), maximum value (maximum score), and average value (average score) each time the score is calculated. The control unit 11 registers and updates the score while identifying and tracking each participant from the captured image. In addition, the control unit 11 calculates the average score of all participants.
[0023] FIG. 4 is an example of the support information list D2. In the support information list D2, a support ID, a situation, and support information are registered in association with each other. The support ID is identification information for identifying the support information. The situation is information indicating the situation of the meeting. For example, the situation includes the situation of the change in the score, the facial expressions, gestures, and actions of the audience and the presenter, and the interest level of the audience. The support information is information corresponding to the situation, for example, information for supporting the presentation of presenter A. The situation and the support information are registered in advance in the support information list D2 by a registration operation by a user. Although details will be described later, in the meeting support process, when the control unit 11 determines the situation of the meeting based on the calculated score, the control unit 11 refers to the support information list D2 to acquire support information corresponding to the situation, and notifies the operation terminal 2 of the presenter A of the support information. Since the presenter A can take appropriate action by acquiring the support information, the quality of the presentation can be improved.
[0024] In another embodiment, the image data (captured image), the score information D1, and some or all of the support information list D2 may be stored in another server accessible via the network N1 from the meeting support device 1. In this case, the control unit 11 of the meeting support device 1 may acquire the information from the other server and execute various processes such as the meeting support process (see FIG. 10) described below.
[0025] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processing. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processing are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 11 controls the meeting support device 1 by executing the various control programs stored in advance in the ROM or the storage unit 12 by the CPU.
[0026] Specifically, as shown in Fig. 2, control unit 11 includes various processing units such as acquisition processing unit 111, determination processing unit 112, display processing unit 113, and presentation processing unit 114. Control unit 11 functions as each processing unit by executing various processes in accordance with the meeting support program using the CPU. Some or all of the processing units included in control unit 11 may be configured with electronic circuits. The meeting support program may be a program for causing a plurality of processors to function as the various processing units.
[0027] The acquisition processing unit 111 acquires image data of an image captured by the camera 3 from the camera 3. Specifically, the acquisition processing unit 111 sequentially acquires captured images of a predetermined imaging area from the camera 3 frame by frame. The acquisition processing unit 111 registers the acquired captured images in the storage unit 12. In the example shown in FIG. 1, the acquisition processing unit 111 acquires captured images of a presenter A and audience members from the camera 3 at a meeting (presentation) held in a conference room R1. The acquisition processing unit 111 is an example of an acquisition processing unit of the present disclosure.
[0028] The determination processing unit 112 determines the status of the meeting based on the captured image acquired by the acquisition processing unit 111. Specifically, the determination processing unit 112 recognizes at least one of the facial expressions and gestures of the attendees based on the facial images of the attendees included in the captured image, and determines the status of the meeting based on the recognition result. The determination processing unit 112 also determines the degree of interest of the attendees in the meeting based on the recognition result, and determines the status of the meeting based on the degree of interest.
[0029] For example, the determination processing unit 112 extracts feature points of each part of the face in a face image of a person included in the captured image, recognizes the facial expression and gestures of the person based on the movement of the feature points, and estimates the emotion of the person (anger, disgust, sadness, contempt, surprise, fear, etc.) based on the recognized facial expression and gestures. Note that the technology of estimating the emotion of a person based on the face image of the person is well known, and the determination processing unit 112 may estimate the emotion of the meeting participants (presenter A, audience) using the well known technology.
[0030] The determination processing unit 112 calculates a score (evaluation value) corresponding to the emotion, interest level, etc. of the participant regarding the meeting based on the facial expression, gesture, and emotion determined for the participant. For example, the determination processing unit 112 calculates the possibility (probability) that the gesture of the audience matches each of a plurality of preset gestures, and multiplies the probability of each gesture by a predetermined weight (coefficient) to calculate the score of the audience. For example, the determination processing unit 112 uses the trained model to calculate the probability of each gesture matching the gesture of the audience for each of the preset gestures 1 to 5, as "gesture 1: 5%", "gesture 2: 10%", "gesture 3: 70%", "gesture 4: 15%", and "gesture 5: 0%". Then, the judgment processing unit 112 multiplies each probability by a predetermined weight (for example, "gesture 1: -0.7", "gesture 2: 0.5", "gesture 3: -0.2", "gesture 4: 1", "gesture 5: 0.3") and calculates the sum of each multiplication result (5% x (-0.7) + 10% x 0.5 + 70% x (-0.2) + 15% x 1) as the score for that listener.
[0031] For example, for a given listener, if the gestures corresponding to "happiness" are 100% and the gestures corresponding to other emotions are 0%, the determination processing unit 112 calculates the score as "100".
[0032] The judgment processing unit 112 calculates the score for each participant and registers it in the score information D1 (see FIG. 3). The judgment processing unit 112 also calculates the score every time a captured image is acquired and registers it in the score information D1. The judgment processing unit 112 also calculates the average value of the scores of all participants and registers it in the score information D1.
[0033] In another embodiment, the determination processing unit 112 may set priorities according to participant information about the participants, and calculate the average score of all the participants using weights according to the priorities. The participant information includes, for example, the distance from the presenter A to the audience, the seating positions in the conference room R1 (front row, middle row, back row, etc.), the frequency of speech of the audience, and the status of the audience (job title, qualifications, etc.). For example, an audience member who speaks a lot in a meeting will have a higher score than an audience member who speaks a little, and an audience member seated in the front row will have a higher score than an audience member seated in the back row. The determination processing unit 112 calculates the average score of all the participants by adding weights according to the priorities to the scores of each participant.
[0034] The determination processing unit 112 determines the situation of the meeting based on the scores calculated as described above (the score of each participant, the overall score). For example, the determination processing unit 112 identifies a matching "situation" from among the "situations" registered in the support information list D2 (see FIG. 4) based on the scores. The determination processing unit 112 is an example of a determination processing unit of the present disclosure.
[0035] The display processing unit 113 displays various information on the operation terminal 2. Specifically, the display processing unit 113 displays the captured image acquired by the acquisition processing unit 111 on the operation terminal 2 of the presenter A. FIG. 5 shows an example of a display screen of the operation terminal 2. For example, when a meeting starts, the display processing unit 113 displays a meeting support screen P1 on the operation terminal 2 of the presenter A. The meeting support screen P1 includes a material display field K1 that displays materials for presentation, an image display field K2 that displays the state (live video) of the conference room R1, and a support display field K3 that displays support information. The display processing unit 113 displays materials in the material display field K1 and switches pages of the materials according to the operation of the presenter A. In addition, when a display for audience members is installed in the conference room R1, the display processing unit 113 may share the screen between the operation terminal 2 and the display, and display the display contents of the material display field K1 on the display.
[0036] Furthermore, the display processing unit 113 displays the image captured by the camera 3 in the image display field K2. The display processing unit 113 switches the display image at a predetermined frame rate. Furthermore, the display processing unit 113 displays an index indicating the interest level of the listener in the image display field K2 in association with the face image. For example, the display processing unit 113 displays a face icon image f1 representing the facial expression, gesture, and emotion of the listener determined by the determination processing unit 112 in association with the face image of the live video. The face icon image f1 is an example of an index of the present disclosure.
[0037] For example, the display processing unit 113 displays face icon image f1 so as to overlap the face image of each audience member in the live video shown in Fig. 6 (see Fig. 7). The display processing unit 113 may display face icon image f1 next to the face image. The display processing unit 113 may also switch the display mode of the live video screen between a display mode of displaying face icon image f1 (see Fig. 7) and a display mode of not displaying face icon image f1 (see Fig. 6) in response to an operation by presenter A.
[0038] By displaying the face icon image f1, presenter A can check the emotions of each audience member and therefore grasp the status of the meeting.
[0039] As another embodiment, the display processing unit 113 may display a score image f2 indicating the score calculated by the judgment processing unit 112 instead of the face icon image f1, as shown in FIG. 8. With this configuration, the presenter A can grasp the interest level of each audience member in numerical value, and therefore can grasp the situation of the meeting more accurately. The display processing unit 113 may display text information of the score instead of the score image f2. The display processing unit 113 may also switch between a display mode in which the face icon image f1 is displayed (see FIG. 7) and a display mode in which the score image f2 is displayed (see FIG. 8) on the live video screen according to an operation by the presenter A. The score image f2 and the text information of the score are examples of indicators of the present disclosure.
[0040] In another embodiment, the display processing unit 113 may display the face icon image f1 and the score image f2 in a color according to the score. For example, for each audience member, when the score is equal to or greater than a predetermined value, the display processing unit 113 displays the background of the face icon image f1 and the numerical value of the score image f2 in blue, and when the score is less than the predetermined value, the display processing unit 113 displays the background of the face icon image f1 and the numerical value of the score image f2 in red. When the score of the entire participants is equal to or greater than a predetermined value, the display processing unit 113 displays the background of the face icon image f1 and the numerical value of the score image f2 of all audience members in blue, and when the score of the entire participants is less than the predetermined value, the display processing unit 113 displays the background of the face icon image f1 and the numerical value of the score image f2 of all audience members in red. The display processing unit 113 is an example of a display processing unit in the present disclosure.
[0041] The presentation processing unit 114 presents to presenter A support information according to the situation of the meeting determined by the determination processing unit. Specifically, the presentation processing unit 114 presents to presenter A support information that increases the level of interest of the audience, that is, support information that supports the presentation by presenter A. In addition, the presentation processing unit 114 presents to presenter A support information that suggests changing the topic of the meeting or changing the progress of the meeting.
[0042] For example, the presentation processing unit 114 refers to the support information list D2 (see FIG. 4) to acquire support information corresponding to the "situation" identified by the determination processing unit 112, and displays the support information in the support display field K3 of the meeting support screen P1 displayed on the operation terminal 2 of the presenter A (see FIG. 9). For example, when the determination processing unit 112 determines that the situation of the meeting is "the overall score is declining" based on the scores of each participant, the presentation processing unit 114 displays the support information corresponding to the situation (support ID "1001") in the support display field K3 of the operation terminal 2 of the presenter A (see FIG. 9). In addition, the presentation processing unit 114 may identify a presentation theme (topic) corresponding to the situation of the meeting from among a plurality of presentation themes registered in advance, and present information on the identified presentation theme to the presenter A. In this way, when the presenter A checks the support information displayed on the operation terminal 2, he or she can improve the current situation by taking an action to change the topic, for example, by changing the presentation theme.
[0043] The presentation processing unit 114 may calculate an average score for the entire meeting, for example, after the meeting is over, and present evaluation information corresponding to the average score (overall evaluation, points, etc.) to presenter A. The presentation processing unit 114 may also calculate an average score for each unit time (e.g., 20 minutes) of the meeting, and present evaluation information corresponding to the average score to presenter A, after the meeting is over. The presentation processing unit 114 may also calculate an average score for each presented topic, and present evaluation information corresponding to the average score to presenter A, after the meeting is over. The presentation processing unit 114 may also calculate a trend in the change in score for each audience member, and present evaluation information corresponding to the trend to presenter A, after the meeting is over.
[0044] The presentation processing unit 114 may also present to presenter A evaluation information comparing the current meeting with a past meeting (e.g., the previous meeting). Furthermore, the presentation processing unit 114 may present to presenter A support information corresponding to the next meeting based on past evaluation information (score history, evaluation history, etc.). The presentation processing unit 114 is an example of a presentation processing unit of the present disclosure.
[0045] [Operation terminal 2] 2, the operation terminal 2 includes a control unit 21, a storage unit 22, an operation display unit 23, and a communication unit 24. The operation terminal 2 is an information processing device such as a personal computer, a tablet terminal, a smartphone, or a mobile phone.
[0046] The communication unit 24 is a communication interface for connecting the operation terminal 2 to the network N1 via a wired or wireless connection, and for performing data communication between the meeting support device 1 and external devices such as the camera 3 via the network N1 in accordance with a predetermined communication protocol.
[0047] The operation display unit 23 is a user interface that includes a display unit such as a liquid crystal display or an organic EL display that displays information such as various web pages, and an operation unit such as a mouse, keyboard, or touch panel that accepts operations.
[0048] The storage unit 22 is a non-volatile storage unit such as an HDD, SSD, or flash memory that stores various information. For example, a control program such as a browser program is stored in the storage unit 22. Specifically, the browser program is a control program that causes the control unit 21 to execute communication processing with an external device such as the meeting supporting device 1 according to a communication protocol such as HTTP (Hypertext Transfer Protocol). The browser program may also be a dedicated application for executing communication processing with the meeting supporting device 1 according to a predetermined communication protocol.
[0049] The control unit 21 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processing. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 21 controls the operation terminal 2 by executing, by the CPU, various control programs stored in advance in the ROM or the storage unit 22.
[0050] Specifically, the control unit 21 functions as a browser processing unit by executing various processes according to the browser program stored in the storage unit 22. The browser processing unit is capable of executing browser processing for displaying a web page provided from the meeting supporting device 1 via the network N1 on the operation display unit 23 and inputting operations for the operation display unit 23 to the meeting supporting device 1. That is, the operation terminal 2 can function as an operation terminal for the meeting supporting device 1 by the control unit 21 executing the browser program. Note that some or all of the processing units included in the control unit 21 may be configured with electronic circuits.
[0051] For example, control unit 21 acquires web page data of a meeting support service site from meeting support device 1, and displays the web page of the meeting support service site on operation display unit 23. Furthermore, if a dedicated application (meeting support application) compatible with meeting support device 1 is installed in operation terminal 2, the user of operation terminal 2 (e.g., presenter A) performs an operation to start up the dedicated application, whereby the web page is displayed on operation display unit 23.
[0052] For example, when presenter A accesses the meeting support service site and performs an operation to start a meeting, the control unit 21 causes the operation display unit 23 to display a meeting support screen P1 (see FIG. 5) based on data acquired from the meeting support device 1. The control unit 21 also causes the meeting support screen P1 to display live video (captured images), support information, and the like based on various information acquired from the meeting support device 1 (see FIGS. 5 to 9).
[0053] [Meeting Support Processing] Hereinafter, the meeting support process executed in the meeting support system 10 will be described with reference to Fig. 10. Specifically, in this embodiment, the meeting support process is executed by the control unit 11 of the meeting support device 1. The present disclosure can be regarded as a meeting support method (one example of the meeting support method of the present disclosure) that executes one or more steps included in the meeting support process, and one or more steps included in the meeting support process described here may be omitted as appropriate. The steps in the meeting support process may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the control unit 11 executes each step in the meeting support process, the method may also be a meeting support method in which multiple processors execute each step in the meeting support process in a distributed manner.
[0054] First, in step S1, the control unit 11 determines whether or not to start the meeting. For example, when presenter A or the facilitator inputs an instruction to start the meeting on the operation terminal 2, the control unit 11 starts the meeting. The control unit 11 executes the meeting support process described below when the meeting starts. When the control unit 11 starts the meeting (S1: Yes), it shifts the process to step S2. The control unit 11 waits until the meeting starts (S1: No).
[0055] In step S2, the control unit 11 acquires a captured image (frame image) from the camera 3. The control unit 11 acquires a frame image at a predetermined time after the start of the meeting.
[0056] Next, in step S3, the control unit 11 calculates scores (evaluation values) for the participants of the meeting (presenter A, listeners) corresponding to their feelings about the meeting, their degree of interest, and the like.
[0057] For example, the control unit 11 calculates the possibility (probability) that the participant's gesture matches each of a plurality of preset gestures, and multiplies the probability of each gesture by a predetermined weight to calculate the score of the participant. For example, when five gestures 1 to 5 are preset, the control unit 11 calculates the score E1 of each participant by the following formula. E1 = [(probability of matching gesture 1 (%) × weight corresponding to gesture 1) + (probability of matching gesture 2 (%) × weight corresponding to gesture 2) + (probability of matching gesture 3 (%) × weight corresponding to gesture 3) + (probability of matching gesture 4 (%) × weight corresponding to gesture 4) + (probability of matching gesture 5 (%) × weight corresponding to gesture 5)]
[0058] Furthermore, when the control unit 11 has already calculated a score for each participant based on a previously acquired captured image, it calculates the average value of the score calculated based on the currently acquired captured image. Furthermore, the control unit 11 calculates the minimum and maximum values of the multiple scores for each participant. Furthermore, the control unit 11 calculates the average score of all participants. The control unit 11 registers each calculated score in score information D1 (see FIG. 3).
[0059] In another embodiment, the control unit 11 may calculate the average score of all the participants by using weights according to the priority according to the participant information of the participants. For example, when there are three participants, the control unit 11 calculates the average score E2 by the following formula. E2 = [(Participant 1's score × Participant 1's priority) + (Participant 2's score × Participant 2's priority) + (Participant 3's score × Participant 3's priority)]
[0060] The control unit 11 registers the calculated score of each participant and the average score of all participants in the score information D1 (see FIG. 3).
[0061] Next, in step S4, the control unit 11 determines the status of the meeting. Specifically, the control unit 11 determines the status of the meeting based on the calculated scores (the score of each participant, and the average score of all participants). For example, the control unit 11 identifies a matching "situation" from among the "situations" registered in the support information list D2 (see FIG. 4) based on the score.
[0062] Next, in step S5, the control unit 11 displays an index indicating the degree of interest of the participants in the meeting. Specifically, the control unit 11 displays the meeting support screen P1 on the operation terminal 2 of the presenter A, and displays the presentation materials in the material display field K1 and the state of the conference room R1 (live video) in the image display field K2 in the meeting support screen P1 (see FIG. 5). The control unit 11 also displays an index indicating the degree of interest of the audience in the image display field K2 in association with the face image. For example, as shown in FIG. 7, the control unit 11 displays a face icon image f1 representing the determined facial expression, gesture, and emotion of the audience in association with the face image of the live video. As another embodiment, as shown in FIG. 8, the control unit 11 may display a score image f2 indicating the calculated score in association with the face image of the live video.
[0063] Next, in step S6, the control unit 11 determines whether or not there is support information corresponding to the situation of the meeting. Specifically, the control unit 11 refers to the support information list D2 (see FIG. 4) and determines whether or not support information corresponding to the "situation" identified in step S4 is registered. If the support information is registered (S4: Yes), the control unit 11 shifts the process to step S5. On the other hand, if the support information is not registered (S4: No), the control unit 11 shifts the process to step S8.
[0064] In step S7, control unit 11 presents the support information to presenter A. Specifically, control unit 11 causes the support information to be displayed in support display field K3 of meeting support screen P1 displayed on operation terminal 2 of presenter A (see FIG. 9).
[0065] In step S8, the control unit 11 determines whether or not to end the meeting. For example, when presenter A or the facilitator inputs an instruction to end the meeting on the operation terminal 2, the control unit 11 ends the meeting. When the meeting ends, the control unit 11 ends the meeting support process (S8: Yes). The control unit 11 repeatedly executes the above-mentioned process until the meeting ends (S8: No). During the meeting, each time a captured image is acquired, the control unit 11 calculates a score to determine the status of the meeting, and causes the operation terminal 2 of presenter A to display an index and support information according to the status of the meeting. The meeting support device 1 executes the meeting support process in the above manner.
[0066] [Other embodiments] In the above embodiment, an example has been shown in which the meeting support system 10 is applied to a presentation meeting, but the meeting support system 10 may also be applied to, for example, a Web conference (online conference) in which a conversation takes place by connecting a plurality of different locations to a network N1. In a Web conference, each of a plurality of conference participants who participate in the conference uses an operation terminal 2, and the meeting support device 1 executes a Web conference application to display a meeting support screen P2 (see FIG. 11) on each operation terminal 2. In addition, the meeting support device 1 acquires from each operation terminal 2 a captured image captured by a camera 3 mounted on each operation terminal 2.
[0067] 11 on the operation terminal 2 of the conference participant, and displays, in the image display field K2 of the meeting support screen P2, a facial image of each conference participant and a score (evaluation value) corresponding to the conference participant's emotion, degree of interest, etc., regarding the conference. Note that the control unit 11 may display a facial icon image f1 (see FIG. 7) in place of the score in association with the facial image of each conference participant.
[0068] The control unit 11 also displays support information according to the state of the meeting (conference) determined based on the scores of each conference participant in the support display field K3 of the conference support screen P2. In the example shown in FIG. 11, since the score of the conference participant B4 is "0", the control unit 11 determines that the conference participant B4 is dozing off and includes a question about whether to wake up the conference participant B4 in the support information. In this case, if the conference participant who has acquired the support information selects "Yes", the control unit 11 communicates with the communication terminal held by the conference participant B4 and notifies the conference participant B4 via the communication terminal held by the conference participant B4 to wake up the conference participant B4. As one example, the control unit 11 outputs a predetermined sound from the speaker of the communication terminal held by the conference participant B4. As another example, the control unit 11 lights up the screen of the communication terminal held by the conference participant B4. As yet another example, the control unit 11 vibrates the communication terminal held by the conference participant B4 using a vibration mechanism provided in the communication terminal held by the conference participant B4 so as not to affect the progress of the conference and other conference participants.
[0069] The control unit 11 may display the image display field K2 and the support display field K3 on the meeting support screen P2 only on the operation terminal 2 of a predetermined user. For example, the control unit 11 displays the image display field K2 and the support display field K3 only on the operation terminal 2 of the conference chairperson, facilitator, manager, etc., and displays only the material display field K1 on the meeting support screen P2 of other conference participants without displaying the image display field K2 and the support display field K3.
[0070] As another embodiment, the control unit 11 may calculate the score taking into consideration the voices of the conference participants. For example, the control unit 11 may calculate the score taking into consideration the remarks of the conference participants (replies, questions, etc.). As another embodiment, the control unit 11 may calculate the score taking into consideration the environment of the conference. For example, the control unit 11 may calculate the score taking into consideration environmental information such as the room temperature and humidity of the conference room.
[0071] As described above, in the meeting support system 10 of the present disclosure, the meeting support device 1 acquires an image captured by a camera 3 that captures an image of a speaker (e.g., presenter A) who will be speaking at a meeting and the audience listening to the speaker's remarks, determines the status of the meeting based on the acquired image, and presents support information to the speaker according to the determined status of the meeting.
[0072] According to the above configuration, a speaker at a meeting can take appropriate action by acquiring the support information. Therefore, by supporting a speaker at a meeting, it is possible to improve the quality of the meeting.
[0073] Furthermore, the meeting support device 1 recognizes at least one of the facial expressions and gestures of the listeners based on the facial images of the listeners included in the captured image, and determines the status of the meeting based on the recognition result. The meeting support device 1 also determines the listeners' level of interest in the meeting based on the recognition result, and determines the status of the meeting based on the level of interest. With the above configuration, the listeners' level of interest in the meeting can be determined from their facial expressions and gestures, and therefore the status of the meeting can be accurately grasped.
[0074] The meeting support device 1 also displays the captured image on the operation terminal 2 of the speaker, and also displays an index indicating the degree of interest in the face image, allowing the speaker to grasp at a glance the degree of interest of the audience in his / her speech.
[0075] Furthermore, the meeting support device 1 presents the speaker with support information that increases the interest level of the listeners. For example, the meeting support device 1 presents the speaker with support information that suggests changing the topic of the meeting or changing the progress of the meeting. This allows the speaker to take action in response to the support information, thereby increasing the interest level of the listeners.
[0076] Furthermore, the meeting support device 1 displays the acquired captured image, the determined meeting status, and the support information on the operation terminal 2 of the speaker. For example, the meeting support device 1 displays the captured image, the meeting status, and the support information on the same screen (meeting support screen P1) (see FIG. 9).
[0077] [Disclosure Note] The following is an outline of the disclosure extracted from the embodiments. Note that the configurations and processing functions described in the following supplementary notes can be selected and combined as desired.
[0078] <Appendix 1> an acquisition processing unit that acquires an image captured by an imaging unit that captures an image of a speaker who is speaking at a meeting and a listener who is listening to the speaker's remarks; a determination processing unit that determines a situation of the meeting based on the captured image acquired by the acquisition processing unit; a presentation processing unit that presents to the speaker support information according to the situation of the meeting determined by the determination processing unit; A meeting support system equipped with:
[0079] <Appendix 2> the determination processing unit recognizes at least one of facial expressions and gestures of the listeners based on face images of the listeners included in the captured images, and determines the situation of the meeting based on a recognition result. 2. A meeting support system as described in appendix 1.
[0080] <Appendix 3> the determination processing unit determines a degree of interest of the listeners in the meeting based on the recognition result, and determines a status of the meeting based on the degree of interest. 2. A meeting support system as described in appendix 2.
[0081] <Appendix 4> a display processing unit that displays the captured image acquired by the acquisition processing unit on an operation terminal of the speaker and displays an index indicating the degree of interest in the speaker in association with the face image, 4. A meeting support system as described in appendix 3.
[0082] <Appendix 5> The presentation processing unit presents the support information that increases the interest level of the listener to the speaker. 5. The meeting support system according to claim 3 or 4.
[0083] <Appendix 6> The presentation processing unit presents the support information proposing a change in the topic of the meeting or a change in the progress of the meeting to the speaker. 6. A meeting support system as described in appendix 5.
[0084] <Appendix 7> a display processing unit that displays the captured image acquired by the acquisition processing unit and the situation of the meeting determined by the determination processing unit on an operation terminal of the speaker, The presentation processing unit causes the support information to be displayed on the operation terminal. A meeting support system according to any one of appendix 1 to 6. [Explanation of symbols]
[0085] 1:Meeting support device 2: Operation terminal 3: Camera 10: Meeting support system 11: Control section 12: Storage section 13: Operation display section 14: Communications Department 21: Control section 22: Storage section 23: Operation display section 24: Communications Department 111: Acquisition processing unit 112: Judgment processing unit 113: Display processing unit 114: Presentation processing unit D1: Score information D2: Support information list P1: Meeting support screen P2: Meeting support screen K1: Material display column K2: Image display column K3: Support display field f1: Face icon image f2 :Score image
Claims
1. an acquisition processing unit that acquires an image captured by an imaging unit that captures an image of a speaker who is speaking at a meeting and a listener who is listening to the speaker's remarks; a determination processing unit that determines a situation of the meeting based on the captured image acquired by the acquisition processing unit; a presentation processing unit that presents to the speaker support information according to the situation of the meeting determined by the determination processing unit; A meeting support system equipped with:
2. the determination processing unit recognizes at least one of facial expressions and gestures of the listeners based on face images of the listeners included in the captured images, and determines the situation of the meeting based on a recognition result. The meeting support system according to claim 1 .
3. the determination processing unit determines a degree of interest of the listeners in the meeting based on the recognition result, and determines a status of the meeting based on the degree of interest. The meeting support system according to claim 2.
4. a display processing unit that displays the captured image acquired by the acquisition processing unit on an operation terminal of the speaker and displays an index indicating the degree of interest in the speaker in association with the face image, The meeting support system according to claim 3.
5. The presentation processing unit presents the support information that increases the interest level of the listener to the speaker. The meeting support system according to claim 3.
6. The presentation processing unit presents the support information proposing a change in the topic of the meeting or a change in the progress of the meeting to the speaker. The meeting support system according to claim 5.
7. a display processing unit that displays the captured image acquired by the acquisition processing unit and the situation of the meeting determined by the determination processing unit on an operation terminal of the speaker, The presentation processing unit causes the support information to be displayed on the operation terminal. The meeting support system according to any one of claims 1 to 6.
8. acquiring an image captured by an imaging unit that captures an image of a speaker making a speech at a meeting and a listener listening to the speech of the speaker; determining a situation of the meeting based on the captured images; and presenting support information according to a situation of the meeting to the speaker; The meeting support method is executed by one or more processors.
9. acquiring an image captured by an imaging unit that captures an image of a speaker making a speech at a meeting and a listener listening to the speech of the speaker; determining a situation of the meeting based on the captured images; and presenting support information according to a situation of the meeting to the speaker; A meeting support program for executing the above on one or more processors.
Citation Information
Patent Citations
Communication calculation device, function calculation device for meeting, and function calculation method and program for meeting
JP2009163431A
Cited By
Information processing device, estimating method and estimating program
WO2026126307A1
