Communication assistance system, communication assistance method, and program

Abstract

A communication assistance system (100) comprises: a voice acquisition unit (203) that acquires voice information uttered in communication performed by a plurality of persons; a condition setting unit (threshold value condition setting unit (205a)) that, on the basis of voice information for a duration from the start of communication until a prescribed period elapses from within the acquired voice information, sets an estimation condition for estimating the state of communication; an estimation unit (205) that estimates the state of communication using the set estimation condition; and an intervention control unit (part of the function of a control unit (208)) that, on the basis of the estimated communication state, controls equipment (151) for imparting an intervention stimulus to the plurality of persons.

Description

Communication support system, communication support method, and program

[0001] The present disclosure relates to a communication support system, a communication support method, and a program.

[0002] When communication between people is lively (in other words, activated), many benefits arise, such as smooth discussions, the ability to exchange various opinions, and the ability to acquire new knowledge by combining multiple opinions. Therefore, the degree of activation is important when communicating. Conventionally, a communication support system that supports the activation of communication has been known (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2017-201479

[0004] However, conventional communication support systems have not always been able to provide appropriate support. Therefore, the present disclosure provides a communication support system and the like that more appropriately supports communication.

[0005] A communication support system according to one aspect of the present disclosure includes a voice acquisition unit that acquires voice information uttered in communication between multiple people, a condition setting unit that sets estimation conditions for estimating a state of the communication based on voice information from the acquired voice information during a predetermined period from the start of the communication, an estimation unit that estimates the state of the communication using the set estimation conditions, and an intervention control unit that controls equipment to provide intervention stimuli to the multiple people based on the estimated state of the communication.

[0006] In addition, a communication support method according to one aspect of the present disclosure is a communication support method executed on a server for a communication support system, and includes: a voice acquisition step of acquiring voice information spoken in communication between multiple people; a condition setting step of setting estimation conditions for estimating a state of the communication based on voice information from the acquired voice information during a predetermined period from the start of the communication; an estimation step of estimating the state of the communication using the set estimation conditions; and an intervention control step of controlling equipment for providing intervention stimuli to the multiple people based on the estimated state of the communication.

[0007] Furthermore, a program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described communication support method.

[0008] According to a communication support system according to an aspect of the present disclosure, communication can be supported more appropriately.

[0009] FIG. 1 is a block diagram showing a functional configuration of a communication support system according to an embodiment. FIG. 2 is a diagram showing an example of a first database in the communication support system according to an embodiment. FIG. 3 is a diagram showing an example of a second database in the communication support system according to an embodiment. FIG. 4 is a diagram showing an example of a third database in the communication support system according to an embodiment. FIG. 5 is a flowchart showing an example of operation of the communication support system according to an embodiment. FIG. 6A is a flowchart showing a detailed example of operation of an estimation unit according to an example of an embodiment. FIG. 6B is a flowchart showing a detailed example of operation of a control unit according to an example of an embodiment. FIG. 7A is a flowchart showing a detailed example of operation of an estimation unit according to another example of an embodiment. FIG. 7B is a flowchart showing a detailed example of operation of a control unit according to another example of an embodiment. FIG. 8A is a flowchart showing a detailed example of operation of an estimation unit according to yet another example of an embodiment. FIG. 8B is a flowchart showing a detailed example of operation of a control unit according to yet another example of an embodiment.

[0010] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0011] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0012] (Embodiment) The overall configuration of a communication support system according to an embodiment will be described below. Fig. 1 is a block diagram showing the functional configuration of a communication support system according to an embodiment.

[0013] In the communication support system of the present disclosure, as an example of communication between people, communication between participants in a meeting where the date, time, location, and participants are predetermined will be described. However, communication to which the communication support system can be applied is not limited to meetings, and can also be any situation involving conversation between people, such as spontaneous chats, presentations, seminars, etc.

[0014] As shown in FIG. 1, the communication support system 100 is realized using a device installed in a space 150 (corresponding to a conference room in this case) where the meeting takes place, and a server 200 realized by an information processing server such as a cloud server or an edge server.

[0015] The space 150 is equipped with a device 151 , a selection device 152 , an input device 153 , a camera 154 , and a microphone 155 that can input and output information to and from the server 200 .

[0016] Each of the devices 151 acts on a conference participant and stimulates one of the participants' five senses. In other words, the devices 151 are devices that provide an intervention stimulus to the conference participants. The devices 151 have a configuration according to the five-sensory stimulus to be provided. A plurality of devices 151 are provided in the space 150. In this example, the devices 151 are devices that serve both as environmental devices that control the environment of the space 150 and as devices that provide intervention stimuli to the conference participants. In other words, the devices 151 provide intervention stimuli while controlling the environment. Note that the environmental devices that control the environment and the devices that provide intervention stimuli to the conference participants may be provided separately.

[0017] The devices 151 include a sound device, a lighting device, an airflow generating device, and a fragrance generating device.

[0018] An acoustic device is an electrical device that provides interventional stimuli through auditory stimulation. Therefore, the acoustic device generates sounds that stimulate human hearing. The acoustic device outputs so-called environmental sounds into the space 150, such as the rustling of trees, the chirping of birds, the chirping of insects, the sound of a bonfire, the sound of water such as the babbling of a stream, or the sound of wind such as a gentle breeze. The acoustic device may also play music. In other words, the acoustic device may function as a music playback device. The acoustic device may be a combination of a sound output device such as a speaker, a signal converter, a gain amplifier, and other devices.

[0019] A lighting device is an electrical device that provides interventional stimuli through visual stimulation. Therefore, the lighting device generates and irradiates light that stimulates human vision. The lighting device may also generate visually stimulating images using the irradiated light. For example, the lighting device irradiates various parts of the space 150 with light, such as green light that evokes a forest, and reflects the light to illuminate the space 150 with a forest image color. The lighting device may also irradiate the floor with light that simulates sunlight filtering through the trees, or may display images depicting forest scenes, including plants such as trees, animals, or rivers. The lighting device may be an indirect lighting device, such as a wall light, or a direct lighting device, such as a spotlight, rather than an ambient lighting device, such as a base light. Examples of lighting devices include light source devices that emit light from a light source such as an LED, and display devices such as a projector (i.e., a video projection device). When displaying images using a lighting device, both still images and moving images may be displayed. Thus, lighting devices include devices that only provide illumination based on brightness and chromaticity, and devices that irradiate light that contains information in the light itself, such as images. In the following description, the term "image projection device" may be used when describing only image projection devices among lighting devices.

[0020] The airflow generating device is an electrical device that provides interventional stimuli through tactile stimulation. The airflow generating device generates wind that stimulates a person's sense of touch by generating an airflow. The airflow generating device generates, for example, wind that blows in a forest. A blower or the like can be used as the airflow generating device. Alternatively, an air conditioner (air conditioner) can be used as the airflow generating device. This can adjust the temperature and humidity in the space 150, thereby generating wind with varying temperature and humidity as a stimulating element for tactile stimulation. The airflow generating device may also generate an airflow for diffusing fragrance components supplied from a fragrance generating device (described below) into the space 150. For this purpose, the airflow generating device may be installed near the fragrance generating device. Alternatively, the fragrance generating device may be installed near the airflow generating device. As a preferred example of such an arrangement, the airflow generating device and the fragrance generating device may be configured as an integrated device. In this way, the fragrance components supplied by the fragrance generating device are carried by the airflow and quickly diffused throughout the space 150, allowing efficient olfactory stimulation by the fragrance components.

[0021] The scent emitting device is an electrical device that provides interventional stimuli through olfactory stimulation. The scent emitting device emits a scent that stimulates a person's sense of smell. The scent emitting device supplies a scent that makes the person feel like they are in a forest (e.g., a plant scent) or a scent component that makes the person perceive a scent into the space 150. An aroma diffuser or the like can be used as the scent emitting device.

[0022] It has been explained that these devices 151 may be used to control the environment or to provide an intervention stimulus to one or more specific conference participants. When used to control the environment, the devices 151 need to act on the entire environment of the space 150. On the other hand, when used to provide an intervention stimulus to a conference participant, the devices 151 need to act locally on the position of the participant in the space 150. Therefore, the devices 151 are configured to be able to act on each unit position within the space 150. Furthermore, the devices 151 can also act on the entire group of unit positions.

[0023] The selection device 152 is a device that can be used by the meeting participants to cause the communication support system 100 to accept operations. The selection device 152 may be a device installed in the space 150, or may be an information terminal owned by at least one of the participants. The selection device 152 accepts a selection regarding the type of meeting and transmits the selection to the server 200. Examples of types of meetings include brainstorming meetings, decision-making meetings, and casual conversation meetings. Such types of meetings may be appropriately set depending on the purpose that the meeting organizer wants to achieve through the meeting. The above examples of types of meetings are merely examples, and other types may also be set.

[0024] For example, a method of providing intervention stimuli that is appropriate for one type of meeting may not be appropriate for another type of meeting. This is because the direction of activation, i.e., the participants whose speech is suppressed or encouraged by intervention stimuli, changes depending on the purpose of the meeting. Therefore, in this embodiment, the type of meeting corresponding to the purpose is selected in advance, and then an appropriate intervention stimuli is provided for that type of meeting, thereby appropriately activating the meeting.

[0025] Furthermore, depending on the type of meeting, appropriate environmental control can be effectively applied according to the direction of activation, depending on the purpose to be achieved through the meeting. In other words, if environmental control can suppress and encourage speech in accordance with the overall progress of the meeting, more efficient interaction can be achieved. Therefore, in this embodiment, the type of meeting is selected in advance according to the purpose of the meeting, and then appropriate environmental stimuli are provided for that type of meeting, allowing the meeting to proceed efficiently. In particular, since one or more different scenes exist depending on the type of meeting, changing the environmental control over time according to the scene makes it possible to proceed with the meeting more efficiently. For example, from the perspective of proceeding with the meeting more efficiently, it is possible to perform only environmental control according to the type of meeting according to the scene, without providing intervention stimuli according to the type of meeting.

[0026] FIG. 2 is a diagram illustrating an example of a first database in a communication support system according to an embodiment. FIG. 2 illustrates a portion of information stored in the database 209. As illustrated in FIG. 2, how environmental control changes over time (i.e., scenarios) is shown for each type of meeting. For example, for a brainstorming meeting, icebreaker environmental control is performed as the first environment. In the icebreaker environmental control, the environment is controlled to provide green lighting evocative of nature, a scent such as marjoram pine, and sounds such as birdsong in the space 150 to help participants relax and open up. Next, a divergent environment environmental control is performed as the second environment. In the divergent environment environmental control, the environment is controlled to provide bright warm lighting, fluctuating lighting whose illuminance and color change periodically, a relaxing scent such as lavender, and relaxing music such as jazz in the space 150 to create a pleasant atmosphere that encourages participants to talk more while guiding them to topics related to the meeting agenda. Next, a convergent environment environmental control is performed as the third environment. In the environmental control of the convergence environment, the environment is controlled to provide the space 150 with cool, somewhat dim lighting that is easy to concentrate on in order to unify the contents.

[0027] Next, as the fourth environment, environmental control of a divergent environment is performed. Next, as the fifth environment, environmental control of a convergent environment is performed. Next, as the sixth environment, environmental control of a divergent environment is performed. Next, as the seventh environment, environmental control of a convergent environment is performed. In this way, for the type of brainstorming meeting, a scenario is set in which environmental control of a divergent environment and environmental control of a convergent environment are repeatedly performed after environmental control of an icebreaker. Note that when one scenario includes multiple environmental controls, for example, each environmental control may be performed for a time period calculated by dividing the scheduled time from the start to the end of the meeting by the number of each environmental control. In other words, all environmental controls may be performed evenly within the scheduled time. Alternatively, a duration may be set for each environmental control. In other words, environmental control may be performed for the respective durations regardless of the scheduled time of the meeting.

[0028] In a conference to be decided, environmental control of a reporting environment is performed as the first environment. In the environmental control of the reporting environment, the environment is controlled so that the presentation of the participants is centered around the presenter, and so that the monitor screen is easy to see, and cool, somewhat bright lighting is provided in the space 150. Next, environmental control of a diverging environment is performed as the second environment. Next, environmental control of a converging environment is performed as the third environment.

[0029] In casual conversation meetings, the first environment is an ice-breaking environment control.

[0030] Returning to FIG. 1, the selection device 152, the details of which will be described later, has a function of accepting the type of team when the participants of the conference are considered as a team, in addition to the selection of the type of conference.

[0031] The input device 153 is a device that can be used by the conference participants to cause the communication support system 100 to accept operations. The input device 153 may be a device installed in the space 150 or may be an information terminal owned by at least one of the participants. The input device 153 functions as a trigger for causing the conference participant to provide an intervention stimulus at their own timing. Therefore, it is preferable that an input device 153 is provided for each conference participant, and that each participant can operate the input device 153 at their own timing.

[0032] The camera 154 is a photographing device for photographing moving images within the space 150. The camera 154 transmits the photographed moving images to the server 200.

[0033] Microphone 155 is a sound collection device for collecting sound within space 150. Microphone 155 transmits the collected sound to server 200 so that the time position can be associated with the moving image captured by camera 154. Alternatively, as long as the time position of the moving image can be associated with the sound collected by microphone 155, it is not necessary that the time position of the sound can be associated.

[0034] The server 200 processes information received from each device in the space 150 and performs processing to cause the equipment 151 to perform environmental control and interventional stimulation. The server 200 is realized by executing a predetermined program using a processor and memory. The server 200 includes a selection unit 201, an acquisition unit 202, an audio acquisition unit 203, a video acquisition unit 204, an estimation unit 205, an input unit 206, a scenario determination unit 207, and a control unit 208. The estimation unit 205 includes a threshold condition setting unit 205a, and the control unit 208 includes a database 209.

[0035] The selection unit 201 accepts the selection of the type of conference from the selection device 152. The selection unit 201 passes information related to the accepted selection of the type of conference to the control unit 208.

[0036] The acquisition unit 202 accepts the selection of the team type from the selection device 152. Alternatively, the acquisition unit 202 may estimate and determine the type of team of participants currently holding the conference based on at least one of video images and audio captured or collected by the camera 154 and microphone 155, and acquire the team type based on the determination result. The acquisition unit 202 passes the acquired information on the team type to the control unit 208.

[0037] The speech acquisition unit 203 acquires speech collected by the microphone 155. The speech acquisition unit 203 passes speech information of the acquired speech to the estimation unit 205.

[0038] The video acquisition unit 204 acquires a video captured by the camera 154. The video acquisition unit 204 passes video information of the acquired video to the estimation unit 205.

[0039] The estimation unit 205 estimates the state of the conference based on the audio information and the video image information. Here, the estimation unit 205 performs processing to detect a speech duration, which is the duration of speech of a person currently speaking among the participants in the conference, a silent period, which is a period during which all participants are silent (no speech), the number of times the speaker changes, and speech after a silent period of a certain length or longer.

[0040] These processes may all be processes that apply existing technology as long as the above parameters can be calculated from speech. Furthermore, to calculate the above parameters from speech, only audio information is required. However, since sounds and the like caused by environmental control are supplied into space 150, it may be difficult to identify the direction of the speaker relative to microphone 155. Therefore, in this example, the above parameters are accurately calculated by identifying the direction from which the sound is coming using video information. If technological advances make it possible to accurately calculate the above parameters from audio information alone, camera 154 and video acquisition unit 204 may not be provided.

[0041] Furthermore, the estimation unit 205 estimates the state of the conference at that time based on whether each of the parameters detected as described above satisfies a threshold condition (determining whether the parameter is greater or smaller than a threshold). In other words, the state of the conference is a concept indicated by whether each of one or more parameters is greater or smaller than a threshold. While the threshold may be experimentally or empirically set in advance, the threshold condition setting unit 205a dynamically sets the threshold based on audio information collected during a predetermined period from the start of the conference. The threshold condition setting unit 205a is an example of a condition setting unit. The setting of the threshold condition by the threshold condition setting unit 205a will be described later. The estimation unit 205 estimates the state of the conference using the set threshold condition and passes the estimation result to the control unit 208. Therefore, the threshold condition can also be interpreted as an estimation condition for estimating the state of the conference.

[0042] By dynamically setting threshold conditions using the threshold condition setting unit 205a, an appropriate conference state can be estimated based on how the conference is actually progressing (such as the activity level), and intervention stimuli based on the estimation results can be provided. This allows more appropriate intervention stimuli to be given to participants for each conference, thereby providing more appropriate communication support from the perspective of supporting the conference.

[0043] As described above, in this embodiment, the method of providing an intervention stimulus is changed depending on the type of conference. Fig. 3 is a diagram showing an example of a second database in the communication support system according to the embodiment. As shown in Fig. 3, a database is used which is referenced when providing an intervention stimulus, and which associates the type of communication, the state of communication, and information regarding device control for providing the intervention stimulus.

[0044] As shown in the figure, if the type of meeting is a brainstorming meeting, the aim is to create a pleasant atmosphere and encourage speaking. Since it is necessary to encourage speaking among all participants, for example, if the calculated parameters indicate that the speaking time is longer than a threshold (i.e., the presence of a participant who speaks for a long time is detected), an intervention stimulus is applied to that participant using spotlighting. Alternatively, if a silence period longer than the threshold is detected, an intervention stimulus is applied to all participants using airflow and fragrance. Alternatively, if a speaker change is detected (here, whether or not there is a change is determined by comparing with a threshold of once), an intervention stimulus is applied to all participants using lighting effects. Alternatively, if a participant breaking the silence is detected (similarly, by comparing with a threshold of once), an intervention stimulus is applied to all participants, including that participant, using video effects. In this embodiment, such intervention stimuli are used to support the progress of the meeting so that participants can communicate with each other according to the type of meeting.

[0045] Similarly, for other types of meetings, a database is compiled that lists what intervention stimuli will be given depending on the state of the meeting, and by referring to this database, it is possible to give appropriate intervention stimuli to support the meeting.

[0046] Furthermore, FIG. 4 is a diagram illustrating an example of a third database in the communication support system according to the embodiment. The database illustrated in FIG. 4 describes what type of intervention stimulus should be applied for each team type (i.e., team composition). An example of applying an intervention stimulus will be described below with reference to FIGS. 3 and 4 . For example, if the type of meeting is a brainstorming meeting and the type of team is a "quiet" team, and the calculated parameters indicate that the speaking time is longer than a threshold value, no intervention stimulus is applied. On the other hand, if a silence period longer than the threshold value is detected, a subtle intervention stimulus is applied to all participants using airflow and fragrance. The intervention stimulus here is applied using stimulus parameters such as relatively weak airflow and fragrance compared to standard stimulus parameters. Furthermore, if one or more speaker changes, which is the threshold value, are detected, an intervention stimulus is applied to all participants using wall lighting to change the atmosphere. Furthermore, if a participant breaking the silence is detected one or more times, which is the threshold value, an intervention stimulus is applied to all participants, including that participant, using visual effects. 3 and 4, if it is set that an intervention stimulus is to be given to both, an intervention stimulus is given in that conference state. On the other hand, if it is not set that an intervention stimulus is to be given to either of the two, an intervention stimulus is not given in that conference state.

[0047] Furthermore, the conditions (database) for intervention stimuli according to the type of meeting and the type of team can include conditions for whether or not to provide an intervention stimulus, as well as various other conditions, such as a parameter threshold for determining whether to provide an intervention stimulus (i.e., a threshold for estimating the state of the meeting in the estimation unit), the device 151 used to provide the intervention stimulus, and stimulus parameters such as the intensity and length of the stimulus when providing the intervention stimulus. Because many factors contribute to the setting of such conditions, for example, intervention stimuli provided in advance during a meeting can be evaluated by participants' subjective evaluations or creativity tests, and a large number of evaluation results can be collected and used to set the conditions using an intervention control model constructed by machine learning along with a large number of intervention stimulus conditions. The conditions for intervention stimuli set by such an intervention control model can sometimes be further optimized by re-learning, so it is preferable to configure the model and the conditions (database) for intervention stimuli to be updatable.

[0048] The input unit 206 accepts input from a participant via the input device 153 and executes an intervention stimulus as an interrupt. To this end, the input unit 206 notifies the control unit 208 that the input has been made. The conditions, etc., when the input is accepted by the input unit 206 are also used to update the conditions of the intervention stimulus.

[0049] The scenario determination unit 207 determines a scenario for controlling the environment of the space 150 according to the type of conference. The scenario determination unit 207 determines the scenario by selecting a predetermined scenario according to the type of conference. The scenario determination unit 207 passes the determined scenario to the control unit 208.

[0050] The control unit 208 is an example of an environmental control unit and an intervention control unit, and controls environmental equipment to control the environment of the space 150 according to the type of meeting, and also controls equipment to provide intervention stimuli to meeting participants based on the type of meeting, the state of the meeting, and the type of team.

[0051] The operation of the control unit 208 and the operation of the communication support system will be described below with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the operation of the communication support system according to the embodiment.

[0052] As shown in Fig. 5, first, the control unit 208 accepts and acquires the selection of the type of meeting (selection step S11). Next, the control unit 208 accepts and acquires the selection of the type of team (acquisition step S12). The control unit 208 refers to the database 209 (referring to the information exemplified in Figs. 2 to 4) and reads out the conditions for providing intervention stimulation (step S13). Meanwhile, the scenario determination unit 207 determines a scenario (step S14) and passes it to the control unit 208.

[0053] Thereafter, as soon as the conference starts, the control unit 208 starts environmental control in accordance with the scenario (step S15).

[0054] If an input is made to the input unit 206 during the conference (Yes in step S16), the control unit 208 controls the device 151 to provide an intervention stimulus specified in the input or set in advance (step S17). At this time, the control unit 208 updates the conditions for providing the intervention stimulus based on the conditions at the time of the input (step S18). The control unit 208 also rereads the updated conditions (step S19), returns to step S16, and continues to determine whether to provide the intervention stimulus using the latest conditions.

[0055] On the other hand, if there is no input to the input unit 206 (No in step S16), the server 200 acquires audio information and video information (step S20, which includes an audio acquisition step and a video acquisition step). The estimation unit 205 estimates the state of the conference by calculating various parameters (estimation step S21). If the estimated conference state satisfies the conditions (Yes in step S22), the control unit 208 controls the device 151 to provide an intervention stimulus based on the read conditions (intervention control step S23). If the conference state does not satisfy the conditions (No in step S22), the control unit 208 skips the intervention control step S23 and determines whether the conference has ended (step S24). If the conference has not ended (No in step S24), the process is repeated from step S16. On the other hand, if it is determined that the conference has ended (Yes in step S24), for example, if the audio in the space 150 has been silent for a certain period of time, the process is terminated.

[0056] Here, we will explain, with reference to Figures 6A to 8B, the threshold conditions set by the threshold condition setting unit 205a, the state of the conference estimated by those threshold conditions, and some examples of intervention stimuli resulting from the control of the equipment 151 based on such estimation results.

[0057] 6A and 6B are flowcharts showing a more detailed example of operation in an embodiment. Fig. 6A shows an example of the operation of estimation step S21, in which the estimation unit 205 estimates the state of the conference. Fig. 6B shows an example of the operation of intervention control step S23, in which the control unit 208 controls the device 151.

[0058] 6A , in this example, a fourth threshold is set for the length of time during which at least one of the participants continues speaking. That is, the threshold condition includes a fourth threshold for the length of time during which the participant continues speaking. The fourth threshold is set to prevent one participant from continuing speaking for a long period of time.

[0059] The threshold condition setting unit 205a sets the fourth threshold from a portion of the acquired audio information obtained from the start of the conference until a predetermined period of time has elapsed (step S21a). For example, if the portion of the audio information shows a relatively uneven distribution of speakers and a tendency for one participant to speak for a long time, the threshold condition setting unit 205a reduces the fourth threshold. Conversely, if the portion of the audio information shows a relatively uneven distribution of speakers and a tendency for one participant to speak for a short time, the threshold condition setting unit 205a increases the fourth threshold. Note that the predetermined period can be extended by continuing to acquire the portion of the audio information until the number of samples required for such a determination is included. Alternatively, if a necessary and sufficient period has been determined experimentally or empirically as the predetermined period, the threshold condition can be set using only a portion of the audio information for a fixed predetermined period.

[0060] The estimation unit 205 determines whether the speech time in the audio information is shorter than the set fourth threshold or longer than the fourth threshold (step S21b), estimates the state of the conference including the determination result, and generates an estimation result (step S21c).

[0061] 6B , when the state of the conference indicates that the speaking time is equal to or longer than the fourth threshold, the control unit 208 controls the lighting device to shine a spotlight on the person (speaker) who has been speaking for a period equal to or longer than the fourth threshold (step S23a). The spotlight is illuminated for a period of time, such as 5 seconds, sufficient to provide awareness. Furthermore, to encourage other participants to speak, the control unit 208 controls the airflow generator and the fragrance generator to generate airflow and spray fragrance toward the person who has spoken the least in the last 60 seconds (the person who has spoken the least) (step S23b).

[0062] 7A and 7B are flowcharts showing a more detailed example of operation in another example of the embodiment. Fig. 7A shows an example of operation of estimation step S21, in which the estimation unit 205 estimates the state of the conference. Fig. 7B shows an example of operation of intervention control step S23, in which the control unit 208 controls the device 151.

[0063] 7A, in this example, a first threshold is set for the length of a silent period during which multiple people are silent. That is, the threshold condition includes a first threshold for the length of a silent period. The first threshold is set to avoid a long period of silence.

[0064] The threshold condition setting unit 205a sets the first threshold based on a portion of the acquired audio information obtained from the start of the conference until a predetermined period of time has elapsed (step S21d). For example, if the portion of the audio information indicates a tendency for relatively few speakers, the threshold condition setting unit 205a reduces the first threshold, and conversely, if the portion of the audio information indicates a tendency for relatively many speakers, the threshold condition setting unit 205a increases the first threshold. Note that the portion of the audio information may be continuously acquired and the predetermined period extended until the number of samples required for such a determination is included. Alternatively, if a necessary and sufficient period has been determined experimentally or empirically as the predetermined period, the threshold condition may be set based on only a portion of the audio information for a fixed predetermined period.

[0065] The estimation unit 205 determines whether the silence period in the audio information is shorter than the set first threshold or longer than the first threshold (step S21e), estimates the state of the conference including the determination result, and generates an estimation result (step S21f).

[0066] 7B , if the conference status indicates a silence period equal to or longer than the first threshold, the control unit 208 controls the airflow generator and the fragrance generator to generate an airflow and spray a fragrance toward the person who has spoken the least in the last 60 seconds (the person with the least amount of speech) to encourage participants to speak (step S23c). Furthermore, if the silence is broken by providing this intervention stimulus, i.e., if any participant speaks, the control unit 208 provides an intervention stimulus to praise the participant, thereby controlling the control unit 208 to encourage further speech. Specifically, after step S23c, the control unit 208 determines whether any participant has spoken (step S23d). If a participant has spoken (Yes in step S23d), the control unit 208 controls, for example, a video projection device to project a video of praise onto a table or the like in the space 150 where the participants are seated (step S23e). On the other hand, if no participant has spoken (No in step S23d), the control unit 208 skips step S23e and proceeds to the next process (step S24).

[0067] 8A and 8B are flowcharts showing a more detailed example of operation in yet another example of the embodiment. Fig. 8A shows an example of operation of estimation step S21, in which the estimation unit 205 estimates the state of the conference. Fig. 8B shows an example of operation of intervention control step S23, in which the control unit 208 controls the device 151.

[0068] 8A , in this example, a first threshold for the length of a silent period during which multiple people are silent and a second threshold for the number of turns, which is the number of times a speaker takes over from one speaker, are set. That is, the threshold condition includes a first threshold for the length of a silent period and a second threshold for the number of turns. The first threshold is set to avoid a long period of silence, and the second threshold is set to encourage participants to take turns speaking.

[0069] The threshold condition setting unit 205a sets the first threshold and the second threshold from a portion of the acquired audio information obtained from the start of the conference until a predetermined period of time has elapsed (step S21g). For example, if the portion of the audio information indicates a tendency for relatively few speakers, the threshold condition setting unit 205a increases the first threshold. Conversely, if the portion of the audio information indicates a tendency for relatively many speakers, the threshold condition setting unit 205a decreases the first threshold. Furthermore, if the portion of the audio information indicates a tendency for one participant to speak continuously and for few turns, the threshold condition setting unit 205a sets the second threshold to 5 times. Conversely, if the portion of the audio information indicates a tendency for speakers to be relatively dispersed and for many turns, the threshold condition setting unit 205a sets the second threshold to 10 times. Note that the predetermined period can be extended by continuing to acquire the portion of the audio information until the number of samples required for such a determination is included. Alternatively, if a necessary and sufficient period of time has been determined experimentally or empirically, the threshold condition can be set using only a portion of the audio information for a fixed predetermined period. The number of times set as the second threshold value is an example, and the second threshold value may be set to a number other than 5 times and 10 times, and is not particularly limited to the above example.

[0070] The estimation unit 205 determines whether the silence period in the audio information is shorter than a first threshold or longer than or equal to the first threshold (step S21h). The estimation unit 205 also determines whether the number of turns in the audio information is shorter than a second threshold or longer than or equal to the second threshold (step S21i). The estimation unit 205 then estimates the state of the conference including these determination results and generates an estimation result (step S21j).

[0071] 8B, the control unit 208 first resets the number of consecutive changes to 0 (step S23f). Then, the control unit 208 determines whether a change in speaker has occurred (step S23g). If a change in speaker has occurred (Yes in step S23g), the control unit 208 increments the number of consecutive changes by 1 (step S23h). On the other hand, if a change in speaker has not occurred (No in step S23g), the control unit 208 skips step S23h.

[0072] If the state of the conference indicates a silent period longer than the first threshold (Yes in step S23j), the control unit 208 changes the control mode of the light emitted from the lighting device. For example, the control unit 208 reduces the increased illuminated area of ​​the light (step S23m) and resets the number of consecutive alternations (return to step S23f). Alternatively, the control unit 208 may stop the illumination of the light by changing the control mode. If the state of the conference does not indicate a silent period longer than the first threshold (No in step S23j), the control unit 208 further determines whether the number of consecutive alternations is greater than or equal to the second threshold (step S23k).

[0073] If the number of consecutive changes does not exceed the second threshold (No in step S23k), the process returns to step S23g to continue determining whether a change has occurred. If the number of consecutive changes is equal to or greater than the second threshold (Yes in step S23k), the control unit 208 controls the lighting device to continue lighting (step S23l), while changing the lighting mode of the lighting device to provide a visual effect that praises the high number of consecutive changes.

[0074] Alternatively, the number of times multiple people nod can be used as an alternative concept to the number of turns. Because the number of times participants nod corresponds to the number of utterances that give a positive impression, such as being easily empathetic or agreeable, it is advisable to set the threshold condition so that the number of times multiple people nod is as large as possible. For example, the threshold condition setting unit 205a sets a third threshold for the number of times multiple people nod from a portion of the acquired audio information from the start of the conference until a predetermined period of time has elapsed. For example, the "second threshold" in the description of FIGS. 8A and 8B can be replaced with the "third threshold," the "number of turns" with the "number of nods," and the "number of consecutive turns" with the "number of consecutive agreements," which indicates the number of consecutive nods that are equal to or greater than the third threshold.

[0075] [Effects, etc.] As described above, the communication support system 100 according to the first aspect of the present disclosure includes the voice acquisition unit 203 that acquires voice information uttered in communication between multiple people, a condition setting unit (threshold condition setting unit 205 a) that sets estimation conditions for estimating a communication state based on voice information from the start of communication until a predetermined period of time has elapsed among the acquired voice information, the estimation unit 205 that estimates the communication state using the set estimation conditions, and an intervention control unit (part of the function of the control unit 208) that controls the device 151 for providing intervention stimuli to multiple people based on the estimated communication state.

[0076] This allows estimation conditions for estimating the state of communication to be set using voice information obtained after the actual start of communication. Therefore, estimation conditions can be set according to the actual communication, allowing for more appropriate estimation. Therefore, more appropriate communication support can be provided based on the more appropriately estimated state of communication.

[0077] Furthermore, the communication support system 100 according to a second aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the estimation conditions include a first threshold value for the length of a silent period during which multiple people are silent, and the estimation unit 205 estimates the state of communication, including whether the silent period in the communication is shorter than the first threshold value or longer than the first threshold value.

[0078] This allows a first threshold value for the length of a silence period during which multiple people are silent to be set as an estimation condition, and allows estimation of the state of communication, including whether the silence period in communication is shorter than the first threshold value or longer than the first threshold value.

[0079] Furthermore, a communication support system 100 according to a third aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the device 151 includes at least one of a lighting device, an airflow generating device, and a fragrance generating device, and the intervention control unit controls at least one of irradiating light from the lighting device, generating airflow from the airflow generating device, and spraying fragrance from the fragrance generating device when at least one of the plurality of people starts speaking after the communication state indicates that the silence period is longer than a first threshold.

[0080] According to this, when at least one of the plurality of people starts speaking after the communication state indicates that the silence period is longer than a first threshold, at least one of the following controls can be performed: irradiating light from a lighting device, generating an airflow from an airflow generating device, and spraying fragrance from a fragrance generating device toward at least one of the plurality of people who has started speaking.

[0081] Furthermore, the communication support system 100 according to a fourth aspect of the present disclosure is the communication support system 100 described in the first aspect, in which the intervention control unit controls the projection of an image from a lighting device into a space 150 where multiple people are present.

[0082] This allows control to be performed so that an image is projected from the lighting device into a space 150 where multiple people are present.

[0083] Furthermore, the communication support system 100 according to a fifth aspect of the present disclosure is the communication support system 100 described in the first aspect, wherein the estimation conditions include a second threshold value for the number of speaker turns in the communication, and the estimation unit 205 estimates the state of the communication, including whether the number of turns in the communication is less than the second threshold value or more than the second threshold value.

[0084] This allows a second threshold value for the number of speaker turns in communication to be set as an estimation condition, and allows estimation of the state of communication, including whether the number of turns in communication is less than the second threshold value or more than the second threshold value.

[0085] Furthermore, a communication support system 100 according to a sixth aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the device 151 includes a lighting device, and the intervention control unit changes the control mode of the light emitted from the lighting device when the state of communication indicates that the number of alternations is greater than or equal to a second threshold.

[0086] According to this, when the state of communication indicates that the number of alternations is equal to or greater than the second threshold, the control mode of the light emitted from the lighting device can be changed.

[0087] Furthermore, a communication support system 100 according to a seventh aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the estimation conditions further include a first threshold value for the length of a silent period during which multiple people are silent, the estimation unit 205 estimates the state of communication including whether the silent period in the communication is shorter than the first threshold value or longer than the first threshold value, and the intervention control unit changes the control mode of light emitted from the lighting device when the state of communication indicates that the number of turns is greater than or equal to a second threshold value, and stops the emission of light from the lighting device in the changed control mode when the state of communication indicates that the number of turns is greater than or equal to the second threshold value and then indicates that the silent period is longer than or equal to the first threshold value.

[0088] According to this, it is possible to set an estimation condition further including a first threshold for the length of a silence period during which multiple people are silent. The state of communication is estimated, including whether the silence period in communication is shorter than the first threshold or longer than the first threshold, and the intervention control unit can change the control mode of light emitted from the lighting device when the state of communication indicates that the number of alternations is greater than or equal to a second threshold, and can stop the emission of light from the lighting device in the changed control mode when the state of communication indicates that the number of alternations is greater than or equal to the second threshold and then indicates that the silence period is longer than or equal to the first threshold.

[0089] Furthermore, the communication support system 100 according to an eighth aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the estimation conditions include a third threshold value for the number of times that multiple people nod, and the estimation unit 205 estimates the state of communication, including whether the number of nods in the communication is less than the third threshold value or more than the third threshold value.

[0090] This allows a third threshold value for the number of nods made by multiple people to be set as an estimation condition, and allows estimation of the state of communication, including whether the number of nods made by multiple people in a communication is less than the third threshold value or more than the third threshold value.

[0091] Furthermore, a communication support system 100 according to a ninth aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the device 151 includes a lighting device, and the intervention control unit changes the control mode of the light emitted from the lighting device when the state of communication indicates that the number of nods is equal to or greater than a third threshold.

[0092] According to this, when the state of communication indicates that the number of nods is equal to or greater than the third threshold, the control mode of the light emitted from the lighting device can be changed.

[0093] Furthermore, the communication support system 100 according to a tenth aspect of the present disclosure is the communication support system 100 described in the first aspect, wherein the estimation conditions include a fourth threshold value for the length of speech time during which at least one of the plurality of people continues speaking, and the estimation unit 205 estimates the state of communication, including whether the speech time in the communication is shorter than the fourth threshold value or longer than the fourth threshold value.

[0094] This allows a fourth threshold value for the duration of speech during which at least one of the plurality of people continues speaking to be set as an estimation condition, and allows estimation of the state of communication, including whether the duration of speech in the communication is shorter than the fourth threshold value or longer than the fourth threshold value.

[0095] Furthermore, a communication support system 100 according to an eleventh aspect of the present disclosure is the communication support system 100 according to the first aspect, wherein the device 151 includes at least one of a lighting device, an airflow generating device, and a fragrance generating device, and when the state of communication indicates that the speaking time is longer than a fourth threshold, the intervention control unit controls at least one of irradiating light from the lighting device, generating airflow from the airflow generating device, and spraying fragrance from the fragrance generating device to a person among the plurality of people who is continuing to speak.

[0096] According to this, when the state of communication indicates that the speaking time is longer than a fourth threshold, at least one of the following controls can be performed on the person among the multiple people who is continuing to speak: irradiating light from a lighting device, generating airflow from an airflow generating device, and spraying fragrance from a fragrance generating device.

[0097] Furthermore, the communication support system 100 according to the twelfth aspect of the present disclosure is the communication support system 100 according to the first aspect, and further includes a video acquisition unit 204 that acquires video information of a plurality of people, and an estimation unit 205 that estimates the state of communication based on the video information and audio information.

[0098] According to this, in estimating the state of communication, video image information can be used in addition to audio information. Inferences that cannot be made using audio information alone can be made using video image information. For example, even if it is unclear from audio information alone who made a sound, it is possible to determine who made the sound from video image information and estimate the state of communication based on that.

[0099] In addition, a communication support method according to a thirteenth aspect of the present disclosure is a communication support method executed on a server for a communication support system, and includes: a voice acquisition step (part of step S20) of acquiring voice information spoken in communication between multiple people; a condition setting step (steps S21a, S21d, S21g, etc.) of setting estimation conditions for estimating the state of communication based on voice information from the acquired voice information during a predetermined period from the start of communication; an estimation step S21 of estimating the state of communication using the set estimation conditions; and an intervention control step S23 of controlling equipment for providing intervention stimuli to multiple people based on the estimated state of communication.

[0100] This provides the same effects as the communication support system 100 described above.

[0101] A program according to a fourteenth aspect of the present disclosure is a program for causing a computer to execute the communication support method according to the thirteenth aspect.

[0102] This provides the same effects as the above-described communication support method using a computer.

[0103] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0104] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0105] Furthermore, the communication support system of the present disclosure may be realized by multiple devices each having some of the multiple components, or may be realized by a single device having all of the multiple components. Also, some of the functions of a component may be realized as the functions of another component, and the functions may be distributed in any manner among the components. Any form having a configuration that substantially provides all of the functions that can realize the communication support system of the present disclosure is included in the present disclosure.

[0106] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0107] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0108] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0109] The present disclosure may also be realized as a communication support method executed by a communication support system, as a program for causing a computer to execute such a communication support method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0110] In addition, this disclosure also includes forms obtained by applying various modifications to the embodiments that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0111] REFERENCE SIGNS LIST 100 Communication support system 150 Space 151 Equipment 152 Selection device 153 Input device 154 Camera 155 Microphone 200 Server 201 Selection unit 202 Acquisition unit 203 Audio acquisition unit 204 Video image acquisition unit 205 Estimation unit 205a Threshold condition setting unit 206 Input unit 207 Scenario determination unit 208 Control unit 209 Database

Claims

1. A communication support system comprising: a voice acquisition unit that acquires voice information spoken in communication between multiple people; a condition setting unit that sets estimation conditions for estimating a state of the communication based on voice information from the start of the communication until a predetermined period of time has elapsed among the acquired voice information; an estimation unit that estimates the state of the communication using the set estimation conditions; and an intervention control unit that controls equipment to provide intervention stimuli to the multiple people based on the estimated state of communication.

2. The communication support system of claim 1, wherein the estimation conditions include a first threshold for the length of a silent period during which the plurality of people are silent, and the estimation unit estimates the state of the communication including whether the silent period in the communication is shorter than the first threshold or longer than the first threshold.

3. The communication support system of claim 2, wherein the equipment includes at least one of a lighting device, an airflow generating device, and a fragrance generating device, and the intervention control unit, when at least one of the plurality of people starts speaking after the communication state indicates that the silent period is longer than or equal to the first threshold, controls at least one of the following: irradiating light from the lighting device to at least one of the plurality of people who has started speaking, generating an airflow from the airflow generating device, and spraying fragrance from the fragrance generating device.

4. The communication support system according to claim 3, wherein the intervention control unit controls the lighting device to project an image into a space in which the plurality of people are present.

5. The communication support system of claim 1, wherein the estimation conditions include a second threshold for the number of speaker turns in the communication, and the estimation unit estimates the state of the communication including whether the number of turns in the communication is less than the second threshold or more than the second threshold.

6. The communication support system of claim 5, wherein the equipment includes a lighting device, and the intervention control unit changes a control mode of the light emitted from the lighting device when the state of communication indicates that the number of turns is greater than or equal to the second threshold value.

7. The communication support system of claim 6, wherein the estimation conditions further include a first threshold for the length of a silent period during which the plurality of people are silent, and the estimation unit estimates the state of the communication including whether the silent period in the communication is shorter than the first threshold or longer than the first threshold, and the intervention control unit changes a control mode of light emitted from the lighting device when the state of communication indicates that the number of turns is greater than or equal to the second threshold, and stops emitting light from the lighting device in the changed control mode when the state of communication indicates that the number of turns is greater than or equal to the second threshold and then indicates that the silent period is longer than or equal to the first threshold.

8. The communication support system of claim 1, wherein the estimation condition includes a third threshold for the number of times the plurality of people nod, and the estimation unit estimates the state of the communication including whether the number of times the plurality of people nod in the communication is less than the third threshold or is greater than or equal to the third threshold.

9. The communication support system of claim 8, wherein the device includes a lighting device, and the intervention control unit changes a control mode of the light emitted from the lighting device when the state of communication indicates that the number of nods is greater than or equal to the third threshold.

10. The communication support system of claim 1, wherein the estimation conditions include a fourth threshold for the length of time during which at least one of the plurality of people continues speaking, and the estimation unit estimates the state of the communication including whether the speech time in the communication is shorter than the fourth threshold or longer than the fourth threshold.

11. The communication support system of claim 10, wherein the equipment includes at least one of a lighting device, an airflow generating device, and a fragrance generating device, and the intervention control unit, when the state of communication indicates that the speaking time is longer than the fourth threshold, controls at least one of the following: irradiating light from the lighting device to a person of the plurality of people who is continuing to speak, generating an airflow from the airflow generating device, and spraying fragrance from the fragrance generating device.

12. A communication support system as described in any one of claims 1 to 11, further comprising a video acquisition unit that acquires video information of the plurality of people, and the estimation unit estimates the state of communication based on the video information and the audio information.

13. A communication support method executed in a server for a communication support system, comprising: a voice acquisition step of acquiring voice information spoken in communication between multiple people; a condition setting step of setting estimation conditions for estimating a state of the communication based on voice information from the start of the communication until a predetermined period of time has elapsed among the acquired voice information; an estimation step of estimating the state of the communication using the set estimation conditions; and an intervention control step of controlling equipment for providing intervention stimuli to the multiple people based on the estimated state of communication.

14. A program for causing a computer to execute the communication support method according to claim 13.

Images