Conference activation level improvement support system and operational method thereof

Abstract

To provide a system that supports the improvement of a conference activation level by evaluating the conference activation level based on the number of movements of visual line and the number of eye contacts made by conference participants and providing stimulating sounds at required timing.SOLUTION: A conference support device according to the present invention includes: a visual line analysis unit that continuously receives participant information which is used to obtain positions and visual lines of conference participants from participant information acquisition means, and detects the number of movements of visual line which is the number of times that a participant looks at the faces of other participants, and the number of eye contacts which is the number of times that the participants turn their faces to each other; a conference activation level evaluation unit that estimates the excitement of the conference from the number of movements of visual line per specific time; a conference participant activation level evaluation unit that estimates participants who find it difficult to speak from the number of movements of visual line and the number of eye contacts; and a conference promotion support unit that controls stimulus generating means to emit stimuli to encourage conference participants to speak if it is determined that the number of movements of visual line per specific time is less than a threshold value or if a participant who finds it difficult to speak is identified.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a meeting activity improvement support system and an operation method thereof, and in particular to a meeting activity improvement support system and an operation method thereof that evaluates the meeting activity level based on the number of eye movements and eye contacts of meeting participants and supports the improvement of the meeting activity level by providing stimulation at the timing when the meeting activity level decreases and support is needed. [Background technology]

[0002] In activities involving multiple people, meetings are often held to make decisions or generate new ideas. In recent years, due to the threat of infectious diseases caused by new viruses, telecommuting has become commonplace, and with the development of online conferencing systems, opportunities for online meetings have increased.

[0003] However, when it comes to generating ideas for new product development or free discussions on specific themes, face-to-face meetings are more efficient and tend to generate new ideas, so there is a high demand for face-to-face meetings. Meetings have various purposes, but especially in meetings where such discussion is required, it is desirable for participants to exchange various opinions and for the meeting to proceed with high activity. However, depending on the composition of meeting participants, it is possible that only a few participants have the opportunity to speak, or the meeting as a whole is not lively, resulting in a meeting that does not produce the desired results. Therefore, various methods have been proposed to ensure that meetings proceed smoothly.

[0004] Patent Document 1 discloses a meeting support method that acquires biometric information such as voice and images of meeting participants, calculates the activity level of the meeting from the acquired biometric information, and encourages participants to speak by pointing out on a device participants who are expected to speak based on changes in facial expressions, etc.

[0005] Patent document 2 also discloses a method of calculating the activity level of each conference participant from their voice, facial expression, etc., determining their short-term activity level up to the present time and their long-term activity level, and selecting a conversation partner based on these activity levels and providing audio to encourage the conversation partner to speak.

[0006] In the method of Patent Document 1, participants who are expected to speak are directly nominated, but if the nominated participants are not given a topic to talk about, they will not speak, and if there is no person like a facilitator, it may not function as expected. In addition, since it does not voluntarily encourage meeting participants to speak, it is difficult to improve essential intellectual productivity, and there is a problem that the mental burden of being directly nominated is large for participants.

[0007] Even in the method of Patent Document 2, speakers are selected and encouraged to speak, so the speech is not spontaneous and the speaker places a large mental burden. In addition, both Patent Documents 1 and 2 evaluate meetings mainly using voice data, but since there is little voice information available from participants who find it difficult to speak and who should be supported, a new method of evaluating meetings using something other than voice is required. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2022-068904 [Patent Document 2] JP 2020-187605 A Summary of the Invention [Problem to be solved by the invention]

[0009] The present invention has been made in consideration of the problems with the conventional meeting activity improvement support systems described above, and an object of the present invention is to provide a meeting activity improvement support system and an operation method thereof which evaluates meeting activity level based on the number of eye movements and eye contacts made by meeting participants, and supports the improvement of meeting activity level by providing stimulation at times when the meeting activity level is declining and support is needed. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the conference activity improvement support system according to the present invention is a conference activity improvement support system having a conference support device that evaluates conference activity and supports improvement of conference activity based on the evaluation result of the conference activity level, the conference support device being characterized by comprising: a gaze analysis unit that continuously receives participant information from a participant information acquisition means that acquires participant information from which the positions and gaze of the conference participants can be determined, and detects the number of gaze movements, which is the number of times a participant looks at the faces of other participants, and of these, the number of eye contacts, which is the number of times the participants turned their faces towards each other, based on the participant's position and gaze data; a conference activity evaluation unit that estimates the excitement of the conference from the number of gaze movements per specific time; a conference participant activity evaluation unit that estimates participants who find it difficult to speak from the number of gaze movements and the number of eye contacts; and a conference promotion support unit that controls a stimulus generation means to emit a specific stimulus to encourage conference participants to speak when it is determined that the number of gaze movements per specific time is less than a threshold value or when a participant who finds it difficult to speak is identified.

[0011] It is preferable that the meeting promotion support unit has a speaker as a stimulus generating means, and if it determines that the number of eye movements per specific time is less than a threshold value, or if it is recognized that a participant is having difficulty speaking, it controls the speaker to emit a specific sound to encourage the participants in the meeting to speak.

[0012] In order to achieve the above-mentioned object, the conference activity evaluation method of the present invention is a conference activity evaluation method using a conference activity improvement support system having a conference support device that evaluates conference activity and supports improvement of conference activity based on the conference activity evaluation results, and is characterized by having a step of continuously receiving participant information from a participant information acquisition means that acquires participant information from which the positions and gazes of conference participants can be determined, and detecting the number of gaze movements, which is the number of times that participants looked at other participants' faces, based on the participants' gaze data, and the number of eye contacts, which is the number of times that participants turned their faces towards each other, a step of estimating the excitement of the conference from the number of gaze movements per specific period of time, and a step of estimating participants who find it difficult to speak up from the number of gaze movements and the number of eye contacts.

[0013] In order to achieve the above-mentioned object, the method for improving meeting activity according to the present invention is a method for improving meeting activity using a meeting activity improvement support system having a meeting support device which evaluates meeting activity and supports improvement of meeting activity based on the evaluation results of the meeting activity, and is characterized in that it includes a step of issuing a specific stimulus to encourage meeting participants to speak when it is determined based on the evaluation results of the meeting activity evaluation method described above that the number of eye movements per specific time period is less than a threshold value or when it is determined that a participant finds it difficult to speak.

[0014] It is preferable to have a step of emitting a specific sound from a speaker to encourage conference participants to speak if it is determined that the number of eye movements per specified time period is less than a threshold value, or if a participant is found to be having difficulty speaking. Effect of the Invention

[0015] According to the meeting activity improvement support system and its operation method of the present invention, the evaluation of the activity level of the meeting is based on the gaze of the meeting participants, so a microphone for collecting voice is not required. In addition, the participant information for determining the position and gaze of the meeting participants may be image information from a camera that can always obtain images of the faces of the meeting participants so that the gaze can be obtained from the movement of the pupils, but the meeting support device according to the embodiment of the present invention can be equipped with a participant movement estimation unit that obtains skeletal information from point cloud data of a depth sensor and estimates head movement from the transition of the skeletal information, so point cloud data from a depth sensor that can grasp the movement of the meeting participants may also be used. In this case, there is no need to install a camera according to the direction of the participants' faces.

[0016] Furthermore, according to the meeting activity improvement support system and its operation method of the present invention, when the activity level of a meeting decreases and support is required, stimuli that people can feel, such as sound, light, wind, etc., or a combination of these are provided in an appropriate amount and strength as environmental stimuli for the meeting, rather than being targeted at specific participants, to encourage them to speak, so that meeting participants can easily seize the opportunity to speak without feeling the burden of being called on. When providing sounds as stimuli, natural sounds are effective, but this is not limited to these, and basically any sound that cannot be recognized as language can be an appropriate stimulus without disrupting the progress of the meeting. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an outline of a configuration of a meeting activity improvement support system according to an embodiment of the present invention; [Diagram 2] 1 is a diagram for explaining a schematic operation method of a meeting activity improvement support system according to an embodiment of the present invention; [Diagram 3] 13 is a diagram illustrating an example of a threshold management table for each type of meeting according to an embodiment of the present invention. [Figure 4] 10 is a diagram illustrating an example of a correspondence relationship between the activity level of a meeting and the number of eye movements of participants during the meeting. [Diagram 5]13 is a diagram illustrating an example of the relationship between the number of eye movements and the number of eye contacts depending on the number of comments made during a meeting. FIG. [Figure 6] FIG. 13 is a diagram showing an example of the results of a subjective evaluation of ease of conversation when various sounds are provided as stimuli during a meeting. [Figure 7] FIG. 13 shows examples of the results of subjective evaluation of how easily participants became nervous when various sounds were provided as stimuli during a meeting. [Figure 8] 2 is a flowchart illustrating a method of operating a meeting activity improvement support system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Next, a specific example of an embodiment for carrying out the meeting activity improvement support system and its operation method according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a meeting activity improvement support system according to an embodiment of the present invention.

[0019] Referring to FIG. 1, a meeting activity improvement support system 1 according to an embodiment of the present invention includes a participant information acquisition means 20 that acquires the status of meeting participants during the meeting, a meeting support device 10 that acquires information from the participant information acquisition means 20, evaluates the status of the meeting, and attempts to liven up the meeting based on the result of the evaluation, and a stimulus generation means 30 that receives output from the meeting support device 10 and provides stimuli to the meeting participants.

[0020] A conference liveliness improvement support system 1 according to an embodiment of the present invention is a system in which a conference support device 10 continuously receives participant information from a participant information acquisition means 20 that acquires participant information from which the positions and gaze of conference participants can be determined, detects the number of gaze movements, which is the number of times a participant looks at the faces of other participants, and the number of eye contacts, which is the number of times participants turned their faces towards each other, based on the participant's position and gaze data, estimates the excitement of the conference from the number of gaze movements per specific period of time, and evaluates the liveliness of the conference by estimating participants who find it difficult to speak up from the number of gaze movements and the number of eye contacts.

[0021] In addition, the meeting activity improvement support system 1 is also a system that supports improving the meeting activity level by providing a specific stimulus via the stimulus generating means 30 to encourage meeting participants to speak up when the meeting support device 10 evaluates the meeting activity level and determines that the meeting activity level is insufficient, specifically when it determines that the number of eye movements per specific time period is less than a threshold value, or when it recognizes that a participant is having difficulty speaking up. Here, the specific stimulus is a stimulus that affects the human senses, such as sound, light from lighting, wind strength from an air conditioner, wind temperature, smell, etc., and is any one of these or a combination of several of these.

[0022] Since a meeting is basically a place for discussion, it is usually judged whether the meeting is lively or not based on the situation of what is being said. Various systems have been proposed to evaluate the lively state of meetings and to support making them more lively, but most of them use microphones to capture what the meeting participants are saying, and many of them are configured to combine this with a camera to capture the faces of the participants as necessary. However, even in a meeting where many participants say things, if the speech is biased towards a few participants, the microphones will not be able to pick up the situation of the many participants who do not say much, and the outcome of the meeting may differ from the conclusion based on the consensus of the participants.

[0023] Therefore, in the meeting activity improvement support system 1 according to the embodiment of the present invention, a microphone is not required to evaluate the meeting activity level, but instead, a participant information acquisition means 20 is used to acquire participant information that can determine the positions and gazes of meeting participants by focusing on the gaze movements of the meeting participants. The fact that the gaze behavior of the speaker and listener affects whether or not a person will be the next speaker in a meeting is described, for example, in the reference "Will the speaker's gaze be directed at the next speaker? (Enomoto et al.: Social Language Sciences, Vol. 14, No. 1, pp. 97-109, September 2011)" and the like.

[0024] The participant information acquisition means 20 only needs to capture the gaze movements of the conference participants, and can be realized by a camera that is installed at a fixed point and can capture the faces of the participants. Recently, even if the faces of the participants cannot be captured, a method has been put to practical use in which, for example, a depth sensor that can capture a target person in three dimensions is used to detect skeletal information of the target person from point cloud data of the target person acquired by the depth sensor, and head movement is determined. A depth sensor to which such a method can be applied may be used as the participant information acquisition means 20.

[0025] One participant information acquisition means 20 is sufficient if all the participants in the conference can be photographed or three-dimensionally sensed, but if one is not enough to photograph or sense all the participants, multiple participant information acquisition means 20 are provided. In the conference activity improvement support system 1, it is necessary to detect which of the other participants the conference participant has directed his / her gaze at, so when multiple participant information acquisition means 20 are provided, it is desirable to fix the position and orientation of each participant information acquisition means 20 in the conference room in advance. If the layout of the conference room, such as the arrangement of desks, is frequently changed depending on the situation of the conference participants, the arrangement of the participant information acquisition means 20 may also be changed each time. In such a case, for example, fixed indicators for identifying the position may be installed in multiple places in the conference room, and when the participant information acquisition means 20 photographs or senses the participants in the conference room, the fixed indicators may also be photographed or sensed together, so that the orientation and position of the participant information acquisition means 20 can be identified based on the indicators.

[0026] In the embodiment, the participant information acquiring means 20 includes a control unit 21 , an input / output unit 22 , a storage unit 23 , an image data acquiring unit 24 , and a depth data acquiring unit 25 . The input / output unit 22 includes a wired or wireless communication means for outputting participant information, acquired by the participant information acquisition means 20, from which the positions and gaze directions of the participants of the conference can be obtained, to the conference supporting device 10. If the position and orientation of the participant information acquisition means 20 are fixed, the position and orientation of the participant information acquisition means 20 may be input in advance to the conference supporting device 10. This makes it possible to obtain the positional relationship and gaze direction of the participants using only the captured images or 3D depth data of the participants. If there is a possibility that the position and orientation of the participant information acquisition means 20 may be changed for each conference, the positional relationship and gaze direction of the participants can be obtained by also capturing or sensing the above-mentioned fixed indicators and including them in the participant information. The storage unit 23 stores a control program that controls the operation of the participant information acquiring means 20 and temporarily stores the participant information acquired by the participant information acquiring means 20 .

[0027] The image data acquisition unit 24 photographs the faces of the conference participants to acquire image data of the faces. The movement of the gaze can be obtained by various methods, such as detecting the position of the pupil relative to a reference point such as the inner corner or outer corner of the eye, or detecting the positional relationship between the corneal reflex and the pupil when infrared light is irradiated, and a camera such as an infrared camera can be used in addition to a normal camera depending on each detection method. Therefore, the image data acquisition unit 24 may be equipped with an imaging means for acquiring a visible light image, or may be equipped with an imaging means for acquiring an infrared image.

[0028] The depth data acquisition unit 25 acquires point cloud data using a depth sensor that captures participants in three dimensions so that the head movements of the participants in the conference can be detected. As the depth sensor, Microsoft's Azure Kinect (registered trademark) DK or the like can be applied. The control unit 21 executes a control program stored in the storage unit 23, and controls each element of the participant information acquisition means 20 to perform the above-mentioned functions. In the above embodiment, both the image data acquisition unit 24 and the depth data acquisition unit 25 are provided, but only one of them may be sufficient as long as the position and line of sight of the participant can be obtained.

[0029] The conference support device 10 has a control unit 11, an input / output unit 12, a memory unit 13, a display unit 14, a gaze analysis unit 15, a conference activity level evaluation unit 16, a conference participant activity level evaluation unit 17, a conference promotion support unit 18, and a participant movement estimation unit 19. The gaze analysis unit 15 continuously receives participant information from the participant information acquisition means 20, and detects the number of gaze movements, which is the number of times a participant looked at the faces of other participants, and the number of eye contacts, which is the number of times participants turned their faces towards each other, based on the participant's positions and gaze data.

[0030] The line of sight analysis unit 15 first determines the positional relationship between the participants in the conference in order to detect whether the gaze of a participant is directed at the face of another participant. If multiple participant information acquisition means 20 are required to acquire the participant information of all participants, the positional relationship between the participants is determined by combining the participant information acquired from each participant information acquisition means 20 based on the position and orientation of each participant information acquisition means 20. If the position and orientation of the participant information acquisition means 20 are fixed, they may be input to the line of sight analysis unit 15 in advance, or, as described above, multiple fixed indices may be photographed together with the participants, and the position and orientation may be determined from the multiple fixed indices in the photographed image data.

[0031] Next, the gaze analysis unit 15 detects the gaze direction of the participant from the eye movement in the image data of the participant of the conference, and judges whether the detected gaze direction is directed at another participant based on the positional relationship between the participants. If it is detected that the detected gaze direction is directed at the face of another participant, the number of gaze movements of the participant is counted as one. If the gaze of the other participant is also detected as being directed at the face of the participant who was gazed at at this time, eye contact between the two participants is established, and the number of eye contacts between the two participants is counted as one for each participant. Eye contact may be established only for a moment, or may continue for a certain period of time as the two continue to discuss with each other. In order to distinguish between these situations, if eye contact continues for a certain period of time, the number of eye contacts may be counted as one for each predetermined period of time.

[0032] In this way, the gaze analysis unit 15 detects the gaze movements of all participants while the conference is continuing, and counts the number of gaze movements and eye contacts for each participant. The detected number of gaze movements or eye contacts is associated with the participant and stored in the memory unit 13 together with the time of detection.

[0033] In the above, the participant information includes image data including the participant's face. However, if the participant information is acquired as point cloud data from a depth sensor, it is difficult to directly detect eye movements. In this case, the gaze analysis unit 15 acquires data that is converted by the participant movement estimation unit 19 to detect the head movements of the participants, as described below, and estimates the gaze movements of the participants from the head movements. The gaze analysis unit 15 counts the number of gaze movements and eye contacts based on the estimated gaze movements.

[0034] The meeting activity evaluation unit 16 estimates the excitement of the meeting from the number of eye movements per specific time. In a meeting where many participants speak, the discussion also becomes lively, and the meeting is likely to become lively. In such a case, the number of eye movements is likely to increase because the gaze of each participant is likely to be directed to the new speaker as the speaker is replaced. On the other hand, if there are few speakers or only some participants speak, the meeting is not lively and the number of eye movements is unlikely to increase. In addition, the excitement of a meeting may differ between the first half and the second half, or the excitement may change due to a change in the topic during the meeting, and the excitement situation may change even within a single meeting. Therefore, the meeting activity evaluation unit 16 estimates the excitement of the meeting from the number of eye movements per specific time. The meeting activity evaluation unit 16 may always calculate the number of eye movements per specific time most recently, and store the calculated result in the memory unit 13 at regular time intervals so that the passage of time can be known.

[0035] The conference participant activity evaluation unit 17 estimates participants who find it difficult to speak from the number of eye movement and eye contact. In a conference, the atmosphere of the conference room changes depending on the agenda and the member composition of the participants. Therefore, some participants may find it difficult to get an opportunity to speak even if they have an opinion. In such a case, eye contact can provide an opportunity to speak. Conversely, even if a participant is ready to speak and looks at the faces of other participants in search of an opportunity to speak, if the other participants do not return their gaze, the situation in which it is difficult to speak will continue. From this perspective, the conference participant activity evaluation unit 17 estimates participants who do not increase their eye contact count in proportion to the increase in the number of eye movement as participants who find it difficult to speak based on the number of eye movement and the number of eye contact for each participant. For this reason, a specific threshold may be set for the ratio of the number of eye contacts to the number of eye movement, and whether or not the participant finds it difficult to speak may be determined based on whether or not the ratio exceeds this threshold. In another embodiment, a threshold may be set for the number of eye contacts themselves, and whether or not the participant finds it difficult to speak may be determined based on whether or not the ratio exceeds this threshold.

[0036] When the meeting promotion support unit 18 determines that the number of eye movements per specific time period is less than a threshold value, or when it recognizes that a participant is having difficulty speaking, it controls the stimulus generating means 30 to emit a specific stimulus to encourage the meeting participants to speak.

[0037] There are various types of meetings. For example, in the case of a free discussion where no materials are at hand, the conversation is the main focus, so the participants of the meeting tend to move their eyes to follow the speakers who change one after another, and the number of eye movements is likely to be large. In contrast, in a meeting where the discussion is based on materials at hand, there are more opportunities for participants to speak while looking at the materials or to look at the materials when other participants are speaking, so the number of eye movements is less likely to increase. If the number of eye movements is fixed at a specific threshold, the excitement of the meeting may not be properly evaluated depending on the type of meeting. Therefore, in the embodiment, the threshold number of eye movements is set to be different depending on the type of meeting. Similarly, when determining whether or not a participant feels uncomfortable speaking based on a specific threshold in the ratio of the number of eye contacts to the number of eye movements, the threshold may be changed depending on the type of meeting. In the embodiment, a threshold management table for each type of meeting is created in advance and stored in the storage unit 13, and a corresponding threshold is used for judgment depending on the conference mode selected by the operator.

[0038] When the conference promotion support unit 18 determines that the number of eye movement per specific time period is less than a threshold, which varies depending on the type of conference, or when it recognizes a participant who feels uncomfortable speaking, it provides a stimulus to the conference participants via the stimulus generation means 30. When the stimulus is light, the stimulus generation means 30 is, for example, a light with a dimming function in a conference room, and the conference promotion support unit 18 outputs a control signal to the stimulus generation means 30 to change the color tone or brightness of the illumination light.

[0039] Furthermore, when the stimulus is sound, the stimulus generating means 30 is, for example, a speaker in the conference room, and the conference promotion support unit 18 outputs a control signal to the stimulus generating means 30 to emit a sound. The sound provided at this time is preferably, but not limited to, natural sounds such as birds singing or the sound of waves. However, it is preferable that the sound is not recognizable as language. In addition, since sounds that are too loud can actually disrupt the progress of the conference, the sound provided is controlled to be within a specific volume range.

[0040] When the stimulus is wind, the stimulus generating means 30 is, for example, an air conditioning device in a conference room, and the conference promotion support unit 18 outputs a control signal to change the strength of the wind provided by the stimulus generating means 30 or to change the temperature of the wind. In addition, the stimulus provided via the stimulus generating means 30 may be an odor or a scent. The stimuli exemplified here may be provided individually or in combination.

[0041] The stimuli provided by the stimulus generating means 30 are not aimed at specific participants, but rather bring about a change in the atmosphere of the conference room that is noticeable to the participants, so that the participants do not feel the mental stress that they would feel if they were specifically called upon to speak. However, the provision of the stimuli changes the atmosphere of the conference room, which tends to encourage participants to speak. Since the stimuli provided by the stimulus generating means 30 are intended to increase the liveliness of the conference, if it is determined that the liveliness of the conference has increased, the state before the provision of the stimuli may be restored. Furthermore, the stimuli do not need to be constant until the liveliness of the conference is increased, and may be provided with fluctuations, such as 1 / f fluctuations.

[0042] If the participant information acquired from the participant information acquisition means 20 includes point cloud data from a depth sensor, the participant movement estimation unit 19 identifies each participant from the point cloud data, detects skeletal information for each participant, and creates data for detecting head movement from the detected skeletal information. Since eye movement cannot be detected using only the point cloud data from the depth sensor, it is possible to detect gaze movement from data for detecting head movement by taking the direction of the head as the direction of the gaze. The participant movement estimation unit 19 provides the created data for detecting head movement to the gaze analysis unit 15, which uses the data to detect the number of gaze movements and the number of eye contacts. If the participant information acquired from the participant information acquisition means 20 does not include point cloud data from a depth sensor, the participant movement estimation unit 19 may not be provided.

[0043] The input / output unit 12 includes a wired or wireless communication means for acquiring participant information from the participant information acquiring means 20 and outputting a control signal for generating a stimulus to the stimulus generating means 30. It also includes a keyboard, a mouse, and the like for an operator to input data such as the conference mode. The storage unit 13 stores data such as the acquired participant information and the number of gaze movements or eye contacts detected by the gaze analysis unit 15, and also stores a control program that controls the operation of the conference support device 10 itself. The storage unit 13 is realized by a memory IC, a memory module, or the like.

[0044] The display unit 14 displays data such as a threshold management table for each type of conference, participant information, the number of eye movement or eye contact, etc. Also, if necessary, displays programs such as a control program for the conference supporting device 10. The display unit 14 is realized by a liquid crystal display or the like. The control unit 11 controls each element of the conference supporting device 10 to perform the above-mentioned functions in accordance with a control program stored in the storage unit 13.

[0045] FIG. 2 is a diagram for explaining a typical method of operating the meeting activity improvement support system according to the embodiment of the present invention. Referring to FIG. 2, four participants are in a meeting in a conference room 40. The gazes (41, 42) indicated by solid or dashed arrows indicate the direction of gaze of each participant detected by gaze analysis unit 15. The gazes 41 of the two participants on the right are directed separately at the faces of the two participants on the left. In this state, the number of gaze movements of the two participants on the right is counted once each. However, the gazes 42 of the two participants on the left are not directed at the faces of the two participants on the right, so the number of eye contacts of the two participants on the right is not counted. On the other hand, the gazes 42 of the two participants on the left are directed toward each other's faces, and in this state, the number of gaze movements and eye contacts of the two participants on the left is counted as one for each participant.

[0046] Based on the number of gaze movements and eye contacts counted in this manner, if the meeting promotion support unit 18 determines that the number of gaze movements per specific time period is less than a threshold value, or if it recognizes that a participant is having difficulty speaking up, a specific stimulus is provided via the stimulus generating means 30 to encourage the meeting participants to speak up.

[0047] In Fig. 2, a speaker 31, lighting equipment 32, and air conditioner 33 are provided as stimulus generating means 30 on the ceiling of a conference room 40. As a result, if the stimulus is sound, the stimulus is provided through the speaker 31, and if the stimulus is light, the stimulus is provided through the lighting equipment 32. If the stimulus is wind, the stimulus is provided through the air conditioner 33. Stimuli may be provided in combination, such as light and sound. It is expected that these stimuli will change the environment of conference room 40, encouraging participants to speak up and making the meeting more lively.

[0048] FIG. 3 is a diagram illustrating an example of a meeting type threshold management table according to an embodiment of the present invention. Referring to FIG. 3, a threshold management table 50 for each type of meeting according to an embodiment of the present invention has columns for a meeting mode 51, a meeting type 52, and a threshold 53, and the threshold 53 includes columns for a threshold 54 for the number of eye movements and a threshold 55 for the number of eye contacts.

[0049] The conference modes 51 are numbers individually assigned to correspond to the conference forms 52 so that the operator of the conference support device 10 can easily select them. Mode "1" is a case where the conference form 52 is a free discussion. In this case, the thresholds 53 are set to Ta1 as the threshold 54 for the number of eye movements and Tb1 as the threshold 55 for the number of eye contacts. Similarly, mode "2" is a conference where the conference form 52 is based on handouts, mode "3" is a conference where the conference form 52 is based on materials projected by a projector, mode "4" is a conference where the conference form 52 involves writing on a whiteboard, and mode "5" is a conference where the conference form 52 is a discussion on idea creation using sticky notes. In addition, the threshold 54 for the number of eye movements and the threshold 55 for the number of eye contacts are described for each conference mode 51.

[0050] Mode "1" free discussion is a type of meeting where there are generally many opportunities to talk while looking at each other's faces, and the number of eye movements and eye contacts is likely to be high52, whereas in mode "2" meetings based on handouts, for example, there are more opportunities to talk while looking at the materials at hand or to listen to other participants, and the number of eye movements and eye contacts is likely to be less than in free discussions. In modes "3" and "4," participants have more opportunities to focus on shared projected materials or whiteboards, so the number of eye movements and eye contacts is also less likely to increase than in free discussions. Therefore, in meetings such as free discussions where the number of eye movements and eye contacts is likely to be high, the threshold 53 is set high, and for other modes, the threshold 53 is set for each mode of the meeting51 based on a prior evaluation, etc. In this manner, in this embodiment, the set value of the threshold 53 is changed for each type of meeting 52, the liveliness of the meeting is evaluated for each type of meeting 52, and assistance is provided to make the meeting more lively in accordance with the evaluation results.

[0051] FIG. 4 is a diagram showing an example of the correspondence between the activity level of a meeting and the number of eye movements of participants during the meeting. Figure 4 shows the average number of eye movements per person over time when a free discussion meeting was held among four participants, when there was continuous conversation between participants and when there was continuous silence. The circles in the figure show the number of eye movements when there was continuous conversation, and the triangular markers show the number of eye movements when there was continuous silence. The regression lines for each case are also shown in the figure.

[0052] Referring to Figure 4, when a conversation continues, the number of eye movements tends to increase as the duration increases, but when silence continues, the number of eye movements only increases slightly even if the duration increases, and a clear difference can be seen between the two. Therefore, for example, a state in which the number of eye movements is less than two in 20 seconds can be judged to be a state of silence. In this way, by evaluating the number of eye movement for each meeting format 52 in advance, it is possible to determine the threshold value 54 for the number of eye movement. In addition, it is also possible to accumulate results for each meeting format 52 and appropriately revise the threshold value 54 for the number of eye movement based on the average value or the like.

[0053] FIG. 5 is a diagram showing an example of the relationship between the number of eye movements and the number of eye contacts depending on the number of comments made during a meeting. Figure 5 shows the total number of eye movements and eye contacts made by each participant during a 10-minute period during a meeting. The circles in the figure indicate the number of eye contacts relative to the number of eye movements made by participants who spoke 20 or more times during the 10-minute period, and the triangles indicate the number of eye contacts relative to the number of eye movements made by participants who spoke less than 20 times during the 10-minute period. The number of eye contacts made by participants who spoke 20 or more times tends to increase as the number of eye movements increases, whereas the number of eye contacts made by participants who spoke less than 20 times does not increase even if the number of eye movements increases. From this situation, for example, it can be assumed that participants who spoke less than 20 times have difficulty finding an opportunity to speak if they cannot make eye contact even when they look at the faces of other participants. Therefore, it can be assumed that such participants are participants who find it difficult to speak up.

[0054] In one embodiment, to determine whether a participant feels uncomfortable speaking up, the ratio of the number of eye contacts to the number of gaze movements is set to a specific value as the eye contact number threshold 55. In another embodiment, the absolute value of the number of eye contacts is set to a specific value as the eye contact number threshold 55. The threshold 55 for the number of eye contacts for participants who find it difficult to speak may be determined and set based on an initial evaluation, or subsequent performance values ​​may be accumulated and the threshold 55 for the number of eye contacts may be revised as appropriate based on the average value, etc.

[0055] FIG. 6 shows examples of the results of subjective evaluation of ease of conversation when various sounds are provided as stimuli during a meeting. Figure 6 shows a conference held with four participants in each group, consisting of workers and students, in which various sounds were provided as stimuli when there was a lull in the conversation and the sounds were stopped when the conversation resumed. After the conference ended, each participant subjectively evaluated the ease of conversation for each sound, and the results are summarized by subject attributes.

[0056] The sounds provided were brown noise, human speech-like noise, which is a conversation noise in which the content cannot be heard, and natural sounds including birds singing and the sound of waves, and were provided at two sound intensities of 30dB and 40dB. These were compared with a situation where no stimuli were provided, i.e., a situation where only indoor background noise was provided.

[0057] When the subject attributes were workers, the ease of conversation was slightly higher on average when all sounds were 30 dB louder than when they were indoor background noise, with natural sounds being rated slightly higher as a type of sound. On the other hand, when the subject attributes were students, the average was higher when all sounds were 30 dB louder than when they were indoor background noise, and natural sounds were also rated higher. Also, for students, sounds at 40 dB louder were rated lower than sounds at 30 dB louder, showing that sounds that are actually louder than those that make it easier to talk. It was confirmed that by providing a stimulating sound of an appropriate volume at a stage when the conversation pauses in this way, it is possible to improve the ease of conversation compared to when there is no stimulating sound.

[0058] FIG. 7 shows examples of results of subjective evaluation of ease of tension when various sounds are provided as stimuli during a meeting. Figure 7 also shows the results of a meeting with four participants per group, consisting of workers and students. Various sounds were provided as stimuli when there was a lull in the conversation, and the sounds were stopped when the conversation resumed. After the meeting ended, each participant subjectively evaluated how easy it was for each sound to make them feel nervous, and the results are summarized by subject attributes. The types and conditions of sounds provided were the same as in Figure 6.

[0059] When the subject attributes were workers, no clear improvement in ease of tension was observed even when stimulating sounds were provided. However, when the subject attributes were students, there was a tendency for ease of tension to increase when stimulating sounds were provided, with human speech-like noise and natural sounds being given higher ratings in particular. Also, contrary to the ease of conversation in Figure 6, sounds at 40 dB were given a higher rating than sounds at 30 dB. It was confirmed that by providing a stimulating sound of an appropriate volume at a point when the conversation paused in this way, it was possible to improve the tendency for people to become tense compared to when there was no stimulating sound.

[0060] FIG. 8 is a flowchart illustrating a method of operating the meeting activity improvement support system according to the embodiment of the present invention. FIG. 8 shows a method in which the conference support device 10 acquires participant information from the participant information acquisition means 20, evaluates the liveliness of the conference, and supports improvement of the liveliness of the conference at a timing when support is required.

[0061] 8, in step S800, the participant information acquisition means 20 acquires participant information of the participants of the conference. The participant information is information that can determine the positions and gazes of the conference participants, and is, for example, image data captured by a camera that can confirm the positions and eye movements of the participants. In another example, the participant information is point cloud data captured by a depth sensor that can grasp the positions and head orientations of the participants. For this reason, the participant information acquisition means 20 may be a camera or a depth sensor.

[0062] In step S810, the participant information acquiring means 20 outputs the acquired participant information to the conference supporting device 10. The participant information acquiring means 20 then determines in step S820 whether the process is to end, and if the conference is to continue, returns to step S800 to repeat the acquisition and output of participant information.

[0063] 8 shows that the participant information acquisition means 20 includes a means for determining whether to end processing, but it is sufficient for the participant information acquisition means 20 to continuously acquire and output participant information during the duration of the conference, and in one embodiment, the acquisition and output of participant information is terminated upon receiving an end instruction from the conference support device 10. Furthermore, without being limited to this, the acquisition and output of participant information may be terminated by an operator's operation at the timing of the end of the conference.

[0064] When the conference supporting device 10 receives the participation information from the participant information acquisition means 20, in step S830, it detects the number of gaze movements and eye contacts from images of the conference participants, which are the participant information. If the participant information is not image data from a camera but point cloud data from a depth sensor, the conference supporting device 10 identifies each participant based on the point cloud data from the depth sensor within the conference supporting device 10, detects skeletal information for each participant, and creates data that detects head movement from the detected skeletal information. It further obtains gaze movement based on the data that detects head movement, and detects the number of gaze movements and eye contacts of the participant.

[0065] Next, in step S840, the conference supporting device 10 estimates the excitement of the conference, which corresponds to the liveliness of the conference, from the number of eye movement movements per specific time. As described above, the number of eye movement movements may vary depending on the type of conference 52. Therefore, in the embodiment, the conference supporting device 10 stores in advance a threshold management table 50 by type of conference as shown in FIG. 3, and evaluates the liveliness of the conference based on the number of eye movement movements detected in step S830 according to the type of conference to be held. In this case, the conference supporting device 10 needs to identify the type of the conference to be evaluated. Therefore, when a conference is held, the conference supporting device 10 accepts input of the conference mode 51 from the operator, and evaluates the liveliness of the conference using the threshold 54 of the number of eye movement movements corresponding to the input mode 51.

[0066] Furthermore, in step S850, the conference support device 10 estimates participants who find it difficult to speak from the number of eye contacts and the number of gaze movements. In this case, as in step S840, the conference support device 10 estimates participants who find it difficult to speak by using the corresponding eye contact number threshold value 55 from the conference type threshold management table 50 based on the conference mode 51 accepted from the operator when the conference is held.

[0067] In this way, the conference support device 10 can evaluate whether the conference is active or not based on whether the number of gaze movements per specific time exceeds the threshold 54 for the number of gaze movements and whether the number of eye contacts exceeds the threshold 55 for the number of eye contacts. As a result, if it is determined that there is a participant who feels uncomfortable speaking because the number of gaze movements per specific time falls below the threshold 54 for the number of gaze movements or the number of eye contacts falls below the threshold 55 for the number of eye contacts, it is determined that the conference is not active, that is, it is a time when support is needed, and an instruction is given to provide a specific stimulus at step S860. The instruction to provide the stimulus is output to the stimulus generating means 30 corresponding to the content of the stimulus. If the content of the stimulus is predetermined, the stimulus generating means 30 may be a fixed one.

[0068] Furthermore, for example, if the content of the stimulus is determined for each type of meeting 52, the type of stimulus generating means 30 may be associated with a threshold management table 50 for each type of meeting, the stimulus generating means 30 may be confirmed based on the meeting mode 51 received from the operator, and an instruction to provide a stimulus may be output to the confirmed stimulus generating means 30. In addition, the conference support device 10 may receive an input of a type of stimulus from an operator when a conference is held, and output an instruction to provide a stimulus to the corresponding stimulus generating means 30 based on the received type of stimulus.

[0069] Finally, in step S870, the conference support device 10 judges whether the conference has ended, and if it is judged that the conference has not ended, the process returns to before step S830, receives participant information again, and repeats the process from step S830. If it is judged in step S860 that the timing for support is necessary, and in the process from step S830 after instructing the provision of stimuli, the conference support device 10 instructs the stimulus generating means 30 to stop providing stimuli when it is judged that there is no participant who feels uncomfortable speaking, by determining that the activity level of the conference has increased, the number of gaze movements per specific time exceeds the threshold 54 for the number of gaze movements, and the number of eye contacts exceeds the threshold 55 for the number of eye contacts.

[0070] If it is determined in step S870 that the conference has ended, the processing of the conference supporting device 10 is terminated. The conference may be automatically ended at a preset scheduled end time, or the end process may be performed at the discretion of the operator when the conference ends.

[0071] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical scope of the present invention. [Explanation of symbols]

[0072] 1. Meeting Liveliness Improvement Support System 10. Conference Support Equipment 11, 21 Control section 12, 22 Input / output section 13, 23 Storage section 14 Display section 15 Line of sight analysis section 16 Meeting Activity Evaluation Department 17 Conference Participant Activity Evaluation Section 18 Conference Promotion Support Department 19 Participant motion estimation section 20 Participant information acquisition means 24 Image data acquisition section 25 Depth data acquisition section 30 Stimulus generation means 31 Speaker 32 Lighting equipment 33 Air conditioner 40 Conference Rooms 41, 42 line of sight 50 Meeting type threshold management table 51 Meeting Mode 52 Meeting format 53 Threshold 54 Eye Movement Threshold 55 Eye Contact Threshold

Claims

1. A meeting activity improvement support system having a meeting support device that evaluates a meeting activity level and supports improvement of the meeting activity level based on an evaluation result of the meeting activity level, The conference support device includes: a gaze analysis unit that continuously receives participant information from a participant information acquisition means that acquires participant information that can determine the positions and gazes of conference participants, and detects the number of gaze movements, which is the number of times a participant looks at the faces of other participants, and the number of eye contacts, which is the number of times the participants turn their faces to each other, based on the participant position and gaze data; a meeting activity evaluation unit that estimates the liveliness of the meeting from the number of eye movements per specific time period; a conference participant activity evaluation unit that estimates participants who feel uncomfortable speaking up based on the number of eye movement and the number of eye contacts; A meeting activity improvement support system characterized by comprising a meeting promotion support unit that controls a stimulus generating means to emit a specific stimulus to encourage meeting participants to speak when it is determined that the number of eye movements per specific time period is less than a threshold value, or when it is recognized that a participant is having difficulty speaking.

2. The meeting activity support system described in claim 1, characterized in that the meeting promotion support unit has a speaker as a stimulus generating means, and when it determines that the number of eye movements per specific period of time is less than a threshold value, or when it is recognized that a participant is having difficulty speaking, it controls the speaker to emit a specific sound to encourage the participants in the meeting to speak.

3. A meeting activity evaluation method using a meeting activity improvement support system having a meeting support device that evaluates a meeting activity level and supports improvement of the meeting activity level based on a result of the meeting activity evaluation, A step of continuously receiving participant information from a participant information acquisition means for acquiring participant information that can determine the positions and gazes of the participants of the conference, and detecting the number of gaze movements, which is the number of times the participants looked at the faces of other participants, and the number of eye contacts, which is the number of times the participants turned their faces towards each other, based on the gaze data of the participants; A step of estimating the excitement of the meeting from the number of eye movements per specific time period; A method for evaluating the liveliness of a meeting, comprising a step of estimating participants who find it difficult to speak from the number of eye movement and the number of eye contacts.

4. A method for improving a meeting activity level by a meeting activity level improvement support system having a meeting support device for evaluating a meeting activity level and supporting improvement of the meeting activity level based on an evaluation result of the meeting activity level, Based on the evaluation result of the meeting activity evaluation method according to claim 3, A method for improving meeting liveliness, comprising the step of issuing a specific stimulus to encourage meeting participants to speak when it is determined that the number of eye movements per specific time period is less than a threshold value, or when a participant is found to be having difficulty speaking.

5. The method for improving the liveliness of a meeting as described in claim 4, characterized in that if it is determined that the number of eye movements per specific time period is less than a threshold value, or if it is recognized that a participant is having difficulty speaking, a step of emitting a specific sound from a speaker to encourage the participants in the meeting to speak.

Images