Performance control system, performance control method, and program

Abstract

To enable assisting an information provider in grasping emotions of an audience.SOLUTION: A performance control system produces an environment of a place where an information provider provides information to an audience. The performance control system includes a performance control unit for controlling the production of the environment of the place based on a result of emotion analysis that estimates emotions of the information provider or the audience.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a performance control system, a performance control method, and a program. [Background technology]

[0002] When a speaker (information provider) is concentrating on his / her own remarks in a meeting or presentation, it can be difficult to identify what the audience is feeling. This situation makes it difficult for the speaker to accurately grasp the situation of the audience and adjust the progress. This tendency becomes more pronounced the larger the audience. In recent years, remote conferences or presentations connected by communication have been frequently carried out. In these conferences or presentations, it is more difficult for a speaker to accurately grasp what an audience is feeling than in a face-to-face conference or presentation, because the audience cannot meet face to face. As described above, it has been desirable for a speaker to be able to accurately grasp what the audience is feeling, regardless of whether the meeting is face-to-face or over the Internet. Meanwhile, a technique for calculating a sensory evaluation of an individual from an image of the individual's face or the like is known (see Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-037575 Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if the technology of Patent Document 1 or the like makes it possible to carry out a sensible evaluation of an individual from the facial expression of the individual, it is not easy for an information provider to grasp the state of the audience.

[0005] An object of the present invention is to provide a production control system, a production control method, and a program that assist an information provider in understanding the emotions of an audience. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a production control system for producing an environment in which an information provider provides information to an audience. The production control system includes a control unit that controls the production of the environment in which the information provider provides information to an audience based on a result of emotion analysis that estimates an emotion of the information provider or the audience.

[0007] The above-mentioned performance control system includes an output unit that outputs the result of performance control, either sound control, fragrance control, or lighting control, and the control unit causes the output unit to output the result of the performance control based on the result of the emotion analysis.

[0008] In the above-mentioned production control system, the control unit changes the environment of the venue by the acoustic control within a range that does not affect the transmission of the voice of the speaker who is the information provider.

[0009] In the above-mentioned performance control system, the control unit changes the environment of the location by the scent control so that the change in scent reaches the information provider.

[0010] In the above-mentioned performance control system, the control unit changes the environment of the venue by controlling the lighting so that the information provider can identify changes in the light of the venue.

[0011] In the above-mentioned performance control system, the control unit extends the usage time of the space when the result of the emotion analysis indicates a relatively positive tendency, and shortens the usage time of the space when the result of the emotion analysis indicates a relatively negative tendency.

[0012] In the above-mentioned performance control system, when the result of the emotion analysis indicates a relatively negative tendency, the control unit controls the environment of the venue to promote relaxation in the venue.

[0013] In the above-mentioned performance control system, when the control unit detects a high tendency towards "anger" or "fear" from the results of the emotion analysis, it controls the environment of the venue to create an environment that encourages relaxation.

[0014] In the above-mentioned performance control system, when the control unit detects a relatively high tendency for a "state of low arousal" from the results of the emotion analysis, it controls the environment of the venue to promote awakening.

[0015] In the above-mentioned performance control system, the output unit is provided in a room which is the location where the information provider is present.

[0016] A production control method according to one aspect of the present invention is a production control device that produces an environment in which an information provider provides information to an audience. The production control method includes a step of controlling the production of the environment in which the information provider provides information to an audience, based on a result of emotion analysis that estimates an emotion of the information provider or the audience.

[0017] One aspect of the program of the present invention is a program for causing a computer of a performance control device, which performs the performance of the environment of a place where an information provider provides information to an audience, to execute a step of controlling the performance of the environment of the place based on the results of an emotion analysis that estimates the emotions of the information provider or the audience. Effect of the Invention

[0018] According to the present invention, it is possible to provide a production control system, a production control method, and a program that assist an information provider in understanding the emotions of an audience. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a configuration diagram of a performance control system according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of a usage form of the performance control system according to the embodiment. [Diagram 3]FIG. 2 is a configuration diagram of an estimation unit according to the embodiment. [Figure 4] 10A to 10C are diagrams for explaining various types of information related to emotion analysis processing according to an embodiment. [Diagram 5] 11 is a flowchart of an emotion analysis process according to the embodiment. [Figure 6] 13 is a flowchart of a sentiment analysis process according to the second embodiment. [Figure 7] FIG. 2 is a diagram for explaining processing related to performance control in an embodiment. [Figure 8] 11 is a flowchart of an automatic extension process of a usage time according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The embodiment of the performance control system, the performance control method, and the program will be described with reference to the drawings.

[0021] First, comparative examples of the performance control system of the embodiment will be described. The systems exemplified as the comparative examples are as follows.

[0022] (Sound production within the facility) Conference rooms and presentation environments (hallways, etc.) are often silent, with no special sounds played for sound effects. Other examples of sound effects include playing sounds that make the environmental noise of the conference room less noticeable. Music may be played to entertain participants as part of the sound effects in conference rooms and presentation environments (hallways, etc.).

[0023] (Scenting within the facility) Aromas and other substances may be released to create a fragrant atmosphere in a conference room or presentation environment. For example, the aroma may be released continuously or at specific times using a schedule (timer). The times when the aroma is released may be selected to be times when it is expected to be effective, such as working hours.

[0024] (Lighting effects within the facility) Facilities such as conference rooms, presentation venues, and the like are used in a variety of ways, and may be configured to enable lighting effects to be created in a variety of ways. For example, in a conference room, lighting may be turned on / off and its illuminance and color temperature may be controlled by hand operation. For example, the on / off control of the lighting is performed by manually operating a switch or the like. In addition to the on / off control of the lighting, the illuminance of the room, the output of the luminaire, and the color temperature are adjusted by manually operating an operating device. In addition, lighting control aimed at saving energy when using a conference room may be linked to a conference room reservation system, etc., so that the lights are turned on only when the conference is to begin and during the time periods before and after the conference, or turned off when the conference is to end.

[0025] As a lighting design for a presentation venue, there are cases where lighting is turned on / off and illuminance and color temperature are controlled based on hand movements, similar to the aforementioned conference room. In addition, when controlling lighting for the purpose of saving energy when using a venue for a presentation, it may be linked to a conference room reservation system, etc., so that the lights are turned on only when the presentation is to be held and during the period before and after the presentation, or turned off when the presentation is to end.

[0026] As mentioned above, none of the comparison examples included a conference room or presentation environment with mechanisms based on emotion recognition results. In contrast, the emotion analysis system of the embodiment estimates the emotion of an information provider or an audience member, and outputs the result of emotion analysis based on this. Emotions that the emotion analysis system analyzes and identifies may include, for example, emotions such as "fun," "joy," "cheerful," "refreshing," "pleasure," "sad," "pain," "anger," "surprise," "discomfort," and "dislike." These will be explained in order below.

[0027] (First embodiment) Fig. 1 is a configuration diagram of a performance control system 1 according to an embodiment. Fig. 2 is a diagram for explaining an example of a usage form of the performance control system 1 according to the embodiment.

[0028] The production control system 1 uses the production control device 3 of the facility to produce the environment in which the information provider provides information to the audience. The performance control system 1 includes, for example, a performance control device 3. The performance control device 3 is an example of a device including a computer system. The performance control device 3 includes, for example, a control unit 31, a storage unit 32, a remote conference monitor unit 33, and an output unit .

[0029] The control unit 31 includes a processor such as a CPU. A part or the whole of the inside of the performance control device 3 may be a hardware function unit or a software function unit. The storage unit 32 includes, for example, a semiconductor memory for storing data, a magnetic storage device, etc. The storage unit 32 stores various data in addition to the programs required for the control of the control unit 31. The remote conference monitor unit 33 becomes a client of the remote conference system 7 and communicates with the remote conference system 7 to acquire and monitor video and audio of the remote conference provided by the remote conference system 7. The remote conference monitor unit 33 makes the acquired video and audio available for processing by the control unit 31. The output unit 34 outputs the result of the performance control, which is any one of the sound control, the scent control, and the lighting control, by the control unit 31 under the control of the control unit 31, and makes the result detectable by the user. For example, the output unit 34 is provided in a room where the information provider is present. The output unit 34 will be described in detail later. For example, as shown in FIG. 2, the performance control device 3 may be placed in a conference room, an auditorium where a presentation is given, or the like, in a position where it can be seen by the speaker.

[0030] FIG. 1 shows the following parts: The lighting control equipment 2 is an example of a lighting control equipment that controls the lighting status of the above-mentioned place. The terminal device 4 is, for example, a portable wireless communication device used as an operation unit of the performance control system 1. The terminal device 4 may be composed of one or more terminal devices 4. The type of the multiple terminal devices 4 may be the same or different. The terminal device 4 may be formed by executing processing by application software. The camera 5 is a camera including an imaging device, an optical system, etc., and transmits image data via the network NW. The microphone 6 is an example of a sound collecting device.

[0031] The performance control system 1 may be configured to include, for example, a lighting control facility 2, a terminal device 4, a camera 5, and a microphone 6. The lighting control facility 2, the terminal device 4, the camera 5, and the microphone 6 may each be installed in a conference room or the like. The performance control device 3, the lighting control equipment 2, the terminal device 4, and the camera 5 are connected to each other via a network NW so as to be able to communicate with each other. Furthermore, the performance control device 3 may be connected to, for example, a remote conference system 7 and a remote terminal device 8 via a network NW so that they can communicate with each other.

[0032] The details of each part of the performance control system 1 will be explained below in order.

[0033] (Lighting control equipment) The lighting control equipment 2 includes, for example, a first lighting control equipment 21 and a second lighting control equipment 22 having different uses. The first lighting control device 21 of the lighting control device 2 includes a luminaire 211 and a controller 212 thereof.

[0034] The luminaire 211 of the first lighting control equipment 21 is arranged in a specific area such as a conference room. The type of the luminaire 211 may be one of common types such as a ceiling type, a downlight type, a pendant type, a spotlight type, a bracket type, etc., or may be a combination of multiple types.

[0035] For example, the controller 212 of the first lighting control facility 21 and the luminaire 211 are communicatively connected. The controller 212 of the first lighting control facility 21 controls the lighting state of the luminaire 211 associated therewith.

[0036] The controller 212 of the first lighting control facility 21 can adjust the lighting state to dim the light, in addition to controlling the on / off of the luminaire 211. The "dimming" here may include adjustment of the amount of light emitted by the luminaire 211, that is, adjustment of the output of the luminaire 211, adjustment of the color temperature of the light emitted by the luminaire 211, and a combination of the above.

[0037] The second lighting control device 22 includes a luminaire 221 and its controller 222. The luminaire 221 of the second lighting control device 22 is arranged in the specific area or an area other than the specific area. The type of the luminaire 221 may be one of general types such as a ceiling type, a downlight type, a pendant type, a spotlight type, a bracket type, or the like, or may be a combination of multiple types.

[0038] The controller 222 of the second lighting control device 22 controls the on / off of the above-mentioned luminaires 221, but does not necessarily have to be configured to adjust the lighting state to adjust the dimming.

[0039] (Terminal Device) The terminal device 4 is an example of a device including a so-called computer system, such as a smartphone. For example, the terminal device 4 enables the setting of each part of the performance control system 1 and the control of the first lighting control facility 21 by executing an application program activated on an OS.

[0040] (Remote conference system) The remote conference system 7 is a server device that provides a remote conference service and the like to a user who uses the remote terminal device 8 by communicating with the remote terminal device 8. This remote conference service can be used from the remote terminal device 8. The remote conference system 7 selects and relays images, audio, and the like sent from a specific remote terminal device 8 to another remote terminal device 8.

[0041] (Remote Terminal Equipment) The remote terminal device 8 is an example of a device including a so-called computer system, such as a smartphone or a personal computer. For example, the remote terminal device 8 is configured to be able to use a remote conference service by communicating with the remote conference system 7 through execution of an application program activated on an OS. The remote terminal device 8 may be equipped with a microphone, a camera, etc. for collecting sound, and may be configured to be able to acquire information from an external microphone or camera. The remote terminal device 8 is positioned as a client of the remote conference system 7 . As will be described later, the performance control device 3 is also positioned as a client of the remote conference system 7.

[0042] (Performance control device) As described above, the performance control device 3 of the embodiment includes the control unit 31, the storage unit 32, the remote conference monitor unit 33, and the output unit .

[0043] The control unit 31 includes, for example, a facility use reservation information management unit 311, a performance control unit 312, and an estimation unit 313. The facility use reservation information management unit 311, the performance control unit 312, and the estimation unit 313 are examples of functional units formed by execution of a program. The control unit 31 has a clock unit (not shown) and can control the lighting control apparatus 2 using time information from the clock unit.

[0044] The facility reservation information management unit 311 and the performance control unit 312 will be described first.

[0045] The facility reservation information management unit 311 manages the reservation information for each facility in the building. The facility reservation information management unit 311 acquires the reservation information for each facility in the building from the terminal device 4, adds it to the storage unit 32, manages it as the reservation information for each facility, and uses it.

[0046] For example, when facility reservation information management unit 311 newly acquires reservation information for each facility in the building, it adds the reservation information for each facility in the building acquired from terminal device 4 to storage unit 32 . In addition, when the facility reservation information management unit 311 detects from the acquired information a change or a request for a change to the reservation information for each facility in the building stored in the memory unit 32, it updates the reservation information for each facility in the building stored in the memory unit 32 to the changed information.

[0047] As described above, the facility reservation information management unit 311 makes the reservation information for each facility in the building, which has been newly added to the storage unit 32, available to the performance control unit 312. For example, the facility reservation information management unit 311 may notify the performance control unit 312 that there has been a change in the reservation information for each facility in the building. For example, when making this notification, the facility reservation information management unit 311 may notify the performance control unit 312 of reservation information that has been newly added to the storage unit 32. This notification may be sent in the form of a command or mail, or may be sent via a shared memory in the storage unit 32.

[0048] The performance control unit 312 cooperates with the facility usage reservation information management unit 311 and the estimation unit 313 to carry out dimming control of the lighting control device 2, control of the output control unit 314, and the like.

[0049] The process of estimating the situation within a facility that is applicable to the first scenario will be described with reference to FIG.

[0050] First, the control unit 31 of the performance control system 1 uses the estimation unit 313 to detect the situation of the place within the facility. The estimation unit 313 of the embodiment estimates the emotions of the audience from the venue conditions of the facility, etc., based on either the collected audio or images, using either a predetermined discrimination criterion or a trained model.

[0051] FIG. 3 is a configuration diagram of the estimation unit 313 according to the embodiment. The estimation unit 313 includes, for example, a sensor information acquisition unit 3131 , a voice and sound detection unit 3132 , a natural language information understanding unit 3133 , an image and video information understanding unit 3134 , an inter-modal common understanding unit 3135 , and a state estimation unit 3136 .

[0052] The sensor information acquisition unit 3131 acquires information from each camera 5 installed in the facility. The camera 5 may be fixedly installed in each facility.

[0053] Alternatively, the camera 5 may be provided in the terminal device 4. For example, a user may bring the terminal device 4 into the facility, and the terminal device 4 and the performance control device 3 may be placed in a state in which they can communicate with each other. The sensor information acquisition unit 3131 may regard information from the terminal device 4 as information from the camera 5.

[0054] The voice and sound detection unit 3132 extracts features indicated by the voice and sound information based on the voice and sound information within the facility detected by the microphone 6. For example, the features indicated by the voice and sound information include the volume of the voice, the volume of the environmental sound within the facility, fluctuation, identification of the speaker's voice quality (male / female, etc.), the speaker's speech frequency, etc.

[0055] The natural language information understanding unit 3133 extracts natural language information from the voice and audio information in the facility detected by the microphone 6. For example, the feature amount indicated by the voice and audio information may include the volume of the voice and audio at the time when the word or sentence is identified, the volume of the environmental sound, fluctuation, etc., in addition to the meaning of the word or sentence.

[0056] The image video information understanding unit 3134 extracts information related to the usage status of the facility from the image information of the inside of the facility detected by the camera 5. For example, the information related to the usage status of the facility includes the number of users, the identification of the users, the identification of the user's behavior, the identification of the speaker, and the like.

[0057] By associating information regarding the situation within the facility during use, extracted by each of the voice and audio detection unit 3132, the natural language information understanding unit 3133, and the image and video information understanding unit 3134, with the time detected by the camera 5, it is possible to combine information written using the time information as a key. For example, the inter-modal common understanding unit 3135 uses time information as a key to combine information related to the situation within the facility being used, and extracts information that is significant for the intended analysis. For example, changes in "emotions" are detected as a later stage of processing, and the inter-modal common understanding unit 3135 extracts information suitable for this.

[0058] The state estimation unit 3136 detects a change in "emotion" by using the information extracted by the inter-modal common understanding unit 3135. The "emotion" to be detected may include voice quality, volume, meaning of words, speaker, intonation of speech, and the like. The estimation unit 313 improves the accuracy of identifying such changes in "emotion" by comprehensively determining the "emotion."

[0059] For example, the estimation unit 313 estimates "emotion" by using either the sound collected by the microphone 6 or the image collected by the camera 5, and a predetermined discrimination criterion. The estimation unit 313 estimates the actual usage status of the facility from the estimated "emotion."

[0060] More specifically, the "emotions" can be estimated by analyzing the voice information acquired through the microphone 6 and the image information acquired through the camera 5. There are times when it is judged from the "emotions" that it is not appropriate to continue the meeting or presentation as it is. In such cases, it is better to judge that it is more appropriate to end the situation before the permitted usage time expires, rather than continuing the situation until the expiration of the usage time.

[0061] The performance control system 1 uses the estimation unit 313 to detect the ongoing situation within the facility, and controls the performance of the conference room environment based on the result.

[0062] First scenario: First, the first scenario will be explained. In this first scenario, an example of a production control system 1 applied to a system that enables an information provider of a conference or presentation to casually know the reactions of the participants (hereinafter referred to as a "system for knowing reactions"). Note that the above "information provider" includes the speaker, organizer, presenter, facilitator, etc. Many of the "participants" will also be listeners. The "participants" may include the speaker (information provider). Participants in a conference, etc. will be listeners except when they are speaking.

[0063] A camera 5 installed in the conference room or a camera installed in a terminal device 4 such as a personal computer captures the facial expressions of each participant. For participants participating online, a camera installed in a remote terminal device 8 captures the facial expressions of each participant.

[0064] The control unit 31 acquires video showing the facial expressions of each individual participant and analyzes it in real time to classify the emotions of each individual participant into "boredom," "anger," "sadness," "joy," "surprise," etc. A known technology may be applied to identify such individual emotions.

[0065] The performance control system 1 of the present embodiment is an example of forming a "system for knowing reactions" by identifying "group emotions" and controlling the performance of the venue.

[0066] The "system for knowing reactions" (control unit 31) identifies "group emotions" by, for example, utilizing technology for identifying individual emotions. For example, the "system for knowing reactions" (control unit 31) counts the number of people expressing a specific emotion among the subjects of emotion analysis for each type of emotion, determines the "group emotion" as the atmosphere of the meeting or presentation, and casually conveys the "group emotion" to the organizer of the meeting or presentation by using lighting effects that involve changes in illuminance and color. In the "system for knowing reactions" (control unit 31), it is advisable to set in advance specific illuminance and color that characterize the environment for each type of "group emotion."

[0067] In addition, the lighting control equipment 2 in the embodiment may be configured to include not only general ceiling lighting, but also lighting installed on the walls and floors, as well as indirect lighting and projectors that illuminate the ceiling, walls, and floors.

[0068] As a more specific example, we will explain a case where one speaker was giving a lecture to an audience of 19 people using a combination of conference rooms and remote conferences. The speaker and some of the audience gather in a conference room, while the rest of the audience participate in a remote conference online using a remote conference service. During the lecture, the "reaction recognition system" (control unit 31) collects the "individual emotions" of each audience member.

[0069] <Step 1> In the "system for knowing reactions" (control unit 31) of the embodiment, initial values ​​of various information related to emotion analysis processing are set. With reference to FIG. 4, various information related to emotion analysis processing of the embodiment will be described. Fig. 4 is a diagram for explaining various information related to emotion analysis processing of the embodiment. Fig. 4 shows an example of a table used by the "system for knowing reactions" of the embodiment. This table includes, for example, "boredom," "anger," "sadness," "fun," "surprise," and "fear" as types of "emotions of a group" identified by the "system for knowing reactions." Some or all of these emotions are classified in advance as negative or positive emotions. For example, "boredom," "anger," "sadness," and "fear" are classified as negative emotions, and "fun" and "surprise" are classified as positive emotions. For example, a "▼" is marked in the classification column of Figure 4 for negative emotions, and a "△" is marked in the classification column of Figure 4 for positive emotions. A positive emotion is an example of an emotion that shows a relatively positive tendency.

[0070] The table shown in Figure 4 includes items of various data such as "volume judgment," "illuminance," "color," "scent," "sound effects," "number of people," and "judgment." Data indicating whether or not judgment based on volume is required is added to the "volume judgment" item. Furthermore, data for control according to "group emotion" determined based on the aggregated results of emotion analysis of each audience member is added to this table. The control items include "illuminance," "color," "scent," "sound effects," and so on. In addition to the above, this table also has the fields "Number of people" and "Judgment." Data on the results of sentiment analysis is added to these "Number of people" and "Judgment" fields. For example, the "Number of people" field is added with data on the total number of attendees and the aggregated results of sentiment analysis for each attendee. The "Judgment" field is added with data showing that the majority of attendees expressed the same emotion out of the total number of attendees.

[0071] For example, this will be applied to a lecture with an audience of 19 people. The majority of the audience will be 10 or more people. In the table shown in Figure 4, the effect of "boredom" expressed by the "reaction detection system" is preset as "illuminance 600 lux, color blue." Similarly, desired data can be preset for other types of emotions and the controlled "scents" and "sound effects." The "system for knowing reactions" may use the data in this table for sentiment analysis processing and control based on the results of that processing.

[0072] In the following explanation, using the table in Fig. 4, we will explain an example in which some or all of the types of "group emotions" are classified into two categories, negative emotions and positive emotions. There are no limitations on the types of emotions, their number, or the number of categories. For example, instead of classifying into two categories, emotions may be classified into three or more categories, such as negative emotions, positive emotions, and neutral emotions (leaving the category column in Fig. 4 unmarked).

[0073] <Step 2> FIG. 5 is a flowchart of the emotion analysis process according to the embodiment. As described above, after the data initialization is completed, the "reaction detection system" (estimation unit 313) detects and counts people expressing various emotions from images captured by camera 5 or the like (SA11). The "system for knowing the reaction" (estimation unit 313) identifies whether or not there is any type of "group emotion" that exceeds the majority (SA12) among the types of "group emotion". For example, the estimation unit 313 notifies the performance control unit 312 of the result. If there is a type of "group emotion" that exceeds the majority (type of emotion), the "system for knowing the reaction" (performance control unit 312) identifies that there is a specific tendency in the "emotions common to the group" of the audience (SA13), and carries out processing according to the type of emotion identified as a result (SA14), thereby completing the series of processes. On the other hand, if there is no type of emotion that exceeds the majority among the types of "group emotions," the "system for knowing the reaction" (performance control unit 312) ends the series of processes.

[0074] A more specific example will be given for explanation. It is assumed that, as a result of the counting in step SA11 above, 10 or more of the 19 people expressed the emotion of "boredom" as shown in FIG. In such a situation, the "system for knowing the reaction" (estimation unit 313) detects that 10 or more of the 19 people express the emotion of "boredom." In this case, the "system for knowing the reaction" (estimation unit 313) determines that the "emotion common to the group" is classified as "boredom."

[0075] If the "group emotion" is determined to be "boredom," the "system for knowing the reaction" (performance control unit 312) sets the lighting in the conference room to 600 lux and controls the lighting control equipment 2 to increase the ratio of blue components compared to white light, gradually changing the environment (light) in the conference room.

[0076] In addition, even if less than 10 people out of 19 people express the emotion of "boredom," the "system to know the reaction" (the production control unit 312) does not determine that the "emotion of the group" is "boredom." In this case, the "system to know the reaction" (the production control unit 312) does not change the environment (lighting) of the conference room from its normal state.

[0077] According to the above embodiment, the production control system 1 produces an environment in which a speaker, who is an information provider, provides information to an audience. The control unit 31 of the production control device 3 controls the production of the environment in the venue based on at least the result of emotion analysis that estimates the emotions of the audience. In this way, the production control system 1 can support the speaker, who is an information provider, in understanding the situation of the audience.

[0078] As described above, the control unit 31 may change the environment of the venue by controlling the lighting with the lighting control equipment 2 so that information providers such as speakers, organizers, speakers, and facilitators can identify changes in the lighting of the venue. This enables each information provider to understand the situation of the audience.

[0079] By controlling the presentation in this way, the speaker can discern changes in the conference room environment (lighting) and casually learn in real time that the audience is feeling bored. The speaker who detects this can then adjust the content of the presentation and the pace of the announcement.

[0080] In addition, the above process has shown that the conference room environment (lighting) has a higher blue component ratio than white light, and light with a high blue component ratio has the effect of waking people up. Therefore, by using light with a high blue component ratio, it is expected that not only can the speaker be informed of the participants' situation (emotions), but the audience in such a lighting environment can also be woken up and their sense of boredom reduced.

[0081] In this way, when the control unit 31 detects a high tendency for the subject to be in a "boredom" state, which is identified as a relatively "low level of arousal" from the results of the emotion analysis, it is preferable for the control unit 31 to control the environment of the location to promote arousal.

[0082] (A variation of the "first scenario" that applies sound processing) Furthermore, in addition to or instead of the image analysis by the control unit 31 described above, the volume of voice and sound (sound pressure level) is detected by a microphone 6 installed in the conference room or a microphone installed in the remote terminal device 8, and the "emotion of the group" is judged based on the detected volume of voice and sound (sound pressure level). For example, when the "system for knowing the reaction" (performance control unit 312) detects a voice and sound with a noise level exceeding a predetermined value (about 90 dB), it may presume that there are people in the conference room expressing the emotion of "anger" or "fear," and judge the "emotion of the group" to be "anger" or "fear" without counting the number of people. In this way, when using a microphone 6 or the like as a sensor, just as when using a camera 5 or the like as a sensor, it is advisable to adjust the illuminance and color rendering of the conference room according to the results of identifying the "emotions of the group" using the "reaction detection system" (performance control unit 312). When the result of the emotion analysis indicates a relatively negative tendency (negative emotion), the performance control unit 312 of the control unit 31 controls the environment of the venue to promote relaxation. For example, when the result of the emotion analysis indicates a high tendency of "anger" or "fear," the performance control unit 312 may control the environment of the venue to promote relaxation. The tendency of "anger" or "fear" may be identified based on the volume of sound. For example, the control unit 31 may control the performance of the environment of the venue based on the result of the emotion analysis that estimates the emotion of the person who raised his voice or yelled (information provider). This enables each information provider to grasp the situation of the audience.

[0083] Second embodiment The second scenario will now be described with reference to the above figures and FIG. The performance control system 1 of the second embodiment detects the ongoing situation in the facility using the estimation unit 313, and adjusts the scheduled use time of the conference room based on the result. An example of this is shown in the second scenario.

[0084] Second Scenario: In the second scenario illustrated here, as in the first scenario described above, we will explain the process according to the result of the determination of whether the "group emotion" is a "negative emotion" or a "positive emotion." After completing the "first scenario <phase 1>" described above, the following procedure is executed.

[0085] In this second scenario, the production control system 1 provides a function to change (extend or shorten) the duration of a conference or presentation by taking the emotions of the participants into account as a condition for adjusting the time (hereinafter referred to as the "duration change function"). The usage time reduction process by the performance control system 1 explained in this embodiment includes the following two processing aspects. The first mode is implemented in the middle of the usage time and will be described in <Phase 2-A>. The second mode is implemented towards the end of the usage time and will be described in <Phase 2-B>. After the initialization process shown in "<Step 1>" as described above, the performance control system 1 may implement either or both of the first and second modes. As a method of providing information used here, an example in which the following lighting control is applied will be described.

[0086] <Stage 2-A> (Emotion analysis processing in the middle of usage time and processing to reduce usage time) FIG. 6 is a flowchart of the emotion analysis process according to the embodiment.

[0087] When a predetermined time has elapsed since the start of the conference (SB11), the control unit 31 identifies the "emotion of the group" (SB12A) and, based on the result of the determination, determines whether or not the duration of the conference, in other words, the usage time of the facility to be used for the conference, needs to be adjusted (SB12B). If the result of this determination indicates that the facility usage time needs to be adjusted, the "reaction detection system" (performance control unit 312) identifies that there is a specific tendency in the "emotions common to the group" of audience members (SB13), and adjusts the facility usage time as processing according to the type of emotion identified (SB14), thereby completing the series of processes. On the other hand, if there is no need to adjust the facility usage time, the "system for knowing the reaction" (performance control unit 312) ends the series of processes.

[0088] The above processing can be arranged as follows. For example, a predetermined time from the start of a meeting may be associated with a configurable minimum meeting time, which may be preset, such as 20 minutes from the start of the meeting.

[0089] Also, in the above step SB12B, at any timing after the set shortest meeting time, the "system to know the reaction" (control unit 31) judges whether the "group emotion" is a negative emotion and detects it. If this condition is met, the process may proceed to step SB13 immediately as described above, or alternatively, if a preset fixed time (e.g., 2 minutes) continues thereafter, the process may proceed to step SB13. After the fixed time has elapsed, the control unit 31 may repeat step SB12A and step AS12B, and depending on the result, may notify the participants that the meeting or presentation should be ended as the process of step SB14. Details of this production control will be described later.

[0090] This will be described in relation to specific times with reference to FIG. FIG. 7 is a diagram for explaining the process related to the performance control of the embodiment. The graph in Figure 7 shows a model of the change in the environment caused by the performance control. For example, the horizontal axis shows the passage of time, and the vertical axis shows the quantified state of the environment. When controlling the performance using lighting equipment, the information may include the illuminance of the room, the output of the lighting fixtures, color temperature, etc.

[0091] Let us take an example of a conference that is scheduled to be held from time t10 (e.g., 9:00) to time t30 (e.g., 10:00). In this example, we focus on the "emotion of the group" at time t50 (not shown: e.g., 9:20). Assume that the "group emotion" determined by the "reaction recognition system" (control unit 31) at 9:20 was a negative emotion. Even at 9:22 (time t51), the emotion was still determined to be negative. Therefore, the "meeting time change function" (control unit 31) determines that it would be better to shorten the meeting time beyond the scheduled end time of 10 o'clock, and notifies the participants that the meeting or presentation should be ended.

[0092] If the "reaction detection system" (control unit 31) determines that the "group emotion" is not a negative emotion, it decides to continue the meeting as scheduled and does not implement the usage time reduction process. At this time, the control unit 31 determines that it would be better to shorten the duration of the conference from the scheduled end time, and adjusts the output from the output control unit 314 or adjusts the lighting state of the lighting fixtures of the lighting equipment 2. When adjusting the lighting state of the lighting fixtures of the lighting equipment 2, the output control unit 314 controls the output unit 34 to reduce the illuminance of the room, reduce the output of the lighting fixtures, or change the color temperature to create a suitable room environment, and notify participants of this situation. The above is an example of control for adjusting the illuminance of the room, the output of the lighting fixture, and the color temperature, and the present invention is not limited to this.

[0093] <Stage 2-B> (Emotion analysis and usage time extension processing at the end of usage time) If the above "<Stage 2-A>" processing is omitted, or if the result of the "<Stage 2-A>" processing is that it is decided to continue the conference as scheduled, a determination is made as to whether or not to perform usage time extension processing at the end of the usage time.

[0094] FIG. 8 is a flowchart of the automatic extension process of the usage time according to the embodiment.

[0095] A control unit 31 determines that the "emotion of the group" is positive when a predetermined time has elapsed since the start of the conference before the end of the conference period, and acquires reservation information for the conference room at that time.

[0096] At that time, the "holding time change function" (control unit 31) judges whether or not another meeting is scheduled immediately after the time indicated by the reservation information (SC15). If another meeting is scheduled, the control unit 31 judges that extended use of the conference room is not possible, and guides the user in the direction of ending the conference room (SC16).

[0097] If no other meetings are scheduled, the control unit 31 determines that the conference room can be used for an extended period of time, and automatically extends the conference room for a certain period of time (SC26). For example, the control unit 31 rewrites the conference room use time on the reservation system to the extended period of time accordingly. This certain period of time is preferably determined in advance. The above "another meeting" may be defined as, for example, "a meeting registered by another employee, etc."

[0098] By using such a procedure, emotion analysis processing at the end of the usage time and usage time extension processing can be realized.

[0099] (Automatic re-extension of usage time) As described above, after the usage time has been extended by the usage time extension process, further extension may be permitted. An example of this will be described. After the aforementioned step SC26, the "group emotion" is identified after a certain period of time has elapsed, and if the "group emotion" is still determined to be positive after the certain period of time has elapsed, the meeting time is automatically extended for a further certain period of time. For example, a maximum extension time for a meeting may be preset, and participants may be notified that the meeting or presentation should be ended 5 minutes before the maximum extension time is reached or 5 minutes before the end of the extended meeting time.

[0100] For example, in a meeting scheduled to be held from 9:00 to 10:00, at 9:50, the "system for knowing reactions" (control unit 31) determines whether the "emotion of the group" is a positive emotion.

[0101] As a result, the "group emotion" determined by the "system for knowing reactions" (control unit 31) was a positive emotion. At this time, when the "function for changing the conference time" (control unit 31) read the conference reservation information, it found that there was no next reservation, so it automatically rewrote the conference reservation information to extend it by 15 minutes. The conference room reservation time was automatically changed to 9:00 to 10:15. At 10:05, the "group emotion" determined by the "system for knowing reactions" (control unit 31) was a positive emotion again. When the "function for changing the conference time" read the conference reservation information, it found that there was no next reservation after 10:15, so it automatically rewrote the conference reservation information to extend it by 15 minutes. The conference room reservation time was automatically changed to 9:00 to 10:30. Since the maximum extension time was preset to 30 minutes, a wrap-up performance, which was separately applied for, was performed at 10:25.

[0102] According to the above embodiment, the performance control system 1 uses the estimation unit 313 to detect the ongoing situation within the facility and adjusts the scheduled usage time of the conference room based on the result, thereby achieving the same effect as the first embodiment.

[0103] (Third embodiment) In the third embodiment, a presentation function using light, sound, and aroma (hereinafter, referred to as "presentation function using light, sound, and aroma") will be described.

[0104] Third Scenario: In the third scenario illustrated here, as in the first scenario described above, we explain the process according to the result of the judgment of whether the "group emotion" is "negative emotion" or "positive emotion." In this process, the performance function using light, sound, and scent is provided.

[0105] <Stage 2-C> After completing the above-mentioned "Scenario 1 <Phase 1>", execute the following steps. For example, the control unit 31 periodically performs the following determination, regardless of the amount of time that has elapsed since the start of the conference or presentation. The control unit 31 identifies whether or not the time during which the "emotion of the group" is a negative emotion continues for a preset certain period of time (for example, three minutes) or more. If the "group emotion" remains negative for a certain period of time, The control unit 31 operates some or all of the following functions installed in the conference room or presentation environment.

[0106] Relaxing music is played from a speaker installed in the room. It is advisable to select the type of music to be played in advance. The speaker is an example of the output unit 34. Aroma control is performed by emitting a relaxing aroma from an aroma diffuser. It is advisable to select the type of aroma to be emitted in advance. The aroma diffuser is an example of the output unit 34. - To create an environment that encourages relaxation by gradually increasing or decreasing the illuminance and / or output of lighting fixtures in the room. For example, to create an environment that encourages relaxation by temporarily lowering the illuminance and output of lighting fixtures in the room from normal levels. It is advisable to select the illuminance and output of lighting fixtures in the room to be created in advance. - To create an environment that encourages relaxation by implementing lighting control that gradually raises and / or lowers the color temperature of the lighting. For example, to create an environment that encourages relaxation by implementing lighting control that gradually lowers the color temperature of the lighting temporarily. It is advisable to select the color temperature to be implemented here in advance. Using a projector, an image that promotes relaxation is projected onto the wall of the room. It is advisable to select the type of image to be projected in advance. This projector may be an example of the output unit 34, and may be integrated into the lighting control equipment 2.

[0107] There is no limitation on the sensing method for executing the above-mentioned environmental rendering process, and it may be based on the results detected from an image, or may be based on the results detected from sound characteristics.

[0108] For example, during the above-mentioned production control, the control unit 31 may change the environment of the venue by controlling the sound within a range that does not affect the transmission of the voice of the speaker who is the information provider. During the above-mentioned performance control, the control unit 31 may change the environment of the location by executing scent control so that the change in scent reaches the information provider. During the above-mentioned production control, the control unit 31 may change the environment of the venue by controlling the lighting using a projector so that the information provider can identify changes in the lighting of the venue.

[0109] (Fourth embodiment) In the fourth embodiment, the awakening support function will be described. In the fourth scenario illustrated here, a process will be described according to whether or not the "emotion of the group" is in a "state of low arousal."

[0110] Fourth Scenario: In this fourth scenario, a wakefulness support function is provided that takes emotions into account through lighting and scents to enable meetings to be held efficiently and comfortably.

[0111] <Stage 1-A> An initial state for classifying "group emotions" is set so as to identify "arousal states." For example, some or all of the "group emotions" are classified in advance into two types: "low arousal state" and "not low arousal state." For example, "boredom" is set to correspond to "low arousal state."

[0112] <Stage 2-E> For example, the control unit 31 periodically performs the following determination, regardless of the amount of time that has elapsed since the start of the conference or presentation. The control unit 31 identifies whether or not the time during which the "emotion of the group" is in a "state of low arousal" continues for a preset certain period of time (for example, 3 minutes) or more. If the "emotion of the group" remains in a "state of low arousal" for a certain period of time or longer, the control unit 31 will activate some or all of the following functions installed in the conference room or presentation environment.

[0113] Music that promotes awakening is played from a speaker installed in the room. It is advisable to select the type of music to be played in advance. This speaker is an example of the output unit 34. The aroma control is performed by emitting an aroma that promotes awakening from an aroma diffuser. It is advisable to select the type of aroma to be emitted in advance. The aroma diffuser is an example of the output unit 34. - To create an environment that encourages awakening by gradually increasing or decreasing the illuminance and / or output of lighting fixtures in the room. For example, to create an environment that encourages awakening by temporarily increasing the illuminance and output of lighting fixtures in the room above normal levels. It is advisable to select the illuminance and output of lighting fixtures in the room to be created in advance. - To create an environment that encourages wakefulness by gradually raising and / or lowering the color temperature of the lighting. For example, to create an environment that encourages wakefulness by temporarily raising the color temperature of the lighting. It is a good idea to select the color temperature to be used in advance. Using a projector or the like, an image that promotes awakening is projected onto the wall of the room. It is advisable to select the type of image to be projected in advance. This projector may be an example of the output unit 34, and may be integrated into the lighting control equipment 2.

[0114] There is no limitation on the sensing method for executing the above-mentioned environmental rendering process, and it may be based on the results detected from an image, or may be based on the results detected from sound characteristics.

[0115] According to the above embodiment, "emotions of the group" can contribute to resolving the problems caused by a "state of low arousal."

[0116] (Effects common to all embodiments) The "Reaction Monitoring System" allows organizers and speakers to easily know the reactions of the audience, even in meetings and presentations that use remote meetings. Even if organizers and speakers have disabilities or other reasons that make it difficult to read people's emotions, it is expected that they will be able to visually obtain the evaluation results of the audience's reactions.

[0117] The "reaction monitoring system" makes it possible to carry out objective evaluations even when the number of attendees is large and it is difficult for organizers and speakers to grasp the emotions of each individual. In the comparative example, the more attendees there were, the more difficult it became to grasp the emotions of each individual. However, by applying this embodiment, it becomes possible to carry out an objective evaluation.

[0118] - With the "change meeting time function," if a discussion or presentation is in full swing and extending it would increase the team's productivity, the discussion or presentation can be extended without being interrupted at the scheduled end time, even when the scheduled end time of the meeting or presentation has come. As a result, it is expected that the team's results for that day will improve. At the same time, it eliminates the trouble of meeting or presentation participants having to check the reservation status of the conference room or presentation environment themselves, visually confirm that there are no further reservations and that the time can be extended, and then decide to extend the time, improving efficiency.

[0119] - The "change meeting time function" can be used to shorten meetings that evoke negative emotions. In general, meetings that evoke negative emotions are assumed to be less productive. By shortening the time of such meetings, you can reduce unnecessary work by your team.

[0120] - With the "light, sound and scent production function," when one participant in a meeting yells or behaves in a way that intimidates or annoys other participants, the facilitator does not have to rely solely on their personal skills to change the atmosphere, but instead has the potential to keep the meeting comfortable, productive and creative. This reduces the chances of the meeting ending up being inefficient and unpleasant. In particular, the "light effect function" can induce the behavior of lowering the volume of one's voice by making the person see the red light effect.

[0121] The "awakening support function" is expected to help resolve states of low awakening.

[0122] The production control system 1 according to any of the above embodiments produces an environment in which an information provider provides information to an audience. The production control system includes a production control unit that controls the production of the environment in which the information provider provides information to an audience based on the result of emotion analysis that estimates the emotion of the information provider or the audience, thereby helping the information provider to grasp the emotion of the audience.

[0123] The embodiment of the present invention has been described above, but the performance control system 1 has a computer system inside. A series of processing steps related to the above-mentioned processing is stored in a computer-readable storage medium in the form of a program, and the above-mentioned processing is performed by the computer reading and executing this program. Here, the computer-readable storage medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. In addition, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program. In addition, the "computer system" here includes an OS and the like.

[0124] Although one embodiment of the present invention has been described above, the embodiment of the present invention is not limited to the above. For example, the methods exemplified in each embodiment and its modified examples may be combined in combinations other than those exemplified. In addition, the embodiment of the present invention may be a modification of the above embodiment as follows. [Explanation of symbols]

[0125] 1. Production control system 2. Lighting control equipment 21 First Lighting Control Facility 22 Second Lighting Control Equipment 3. Production control equipment 31 Control Unit 311 Facility Reservation Information Management Department 312 Production Control Unit 313 Estimation Department 32 Storage section 33 Remote Conference Monitor Unit 34 Output section 4 Terminal Equipment 5. Camera 7. Remote Conference System 8 Remote Terminal Equipment

Claims

1. A production control system for producing an environment of a presentation in which an information provider provides information to an audience, the participants of the presentation including the information provider of the presentation, a first audience member participating in the presentation online, and a second audience member participating in the presentation in a venue where the presentation is held, the system producing an environment of the venue using a result of emotion analysis of the audience members, The correspondence between the emotion analysis results and the performance of the controlled object is preset, obtaining the number of listeners who show the same emotion from the result of emotion analysis of each listener including the first listener and the second listener, and estimating the emotion of the group of listeners based on the number of listeners; A control unit that controls the presentation of the environment of the venue based on the result of emotion analysis, which is an estimation result of the emotion of the group of listeners, and the correspondence relationship. Equipped with The control unit is controlling an output from an output unit provided in the location where the information provider is present, to produce an environment of the location; Production control system.

2. The control unit is identifying emotions of the audience for two or more categories to estimate emotions of the audience; changing the environment of the venue for the listeners by the sound control of a production method determined from two or more production methods based on the estimation result of the emotion of the group and the correspondence relationship, using an output unit that outputs a result of production control of sound control; The production control system according to claim 1 .

3. The control unit is identifying emotions of the audience for two or more categories to estimate emotions of the audience; changing the environment of the venue for the listeners by scent control of a scent control method selected from two or more scent control methods based on the estimation result of the emotion of the group and the correspondence relationship, using an output unit that outputs a result of scent control; The production control system according to claim 1 .

4. The control unit is identifying emotions of the audience for two or more categories to estimate emotions of the audience; changing an environment of the place by controlling the lighting of a production method determined from two or more production methods based on the estimation result of the emotion of the group and the correspondence relationship, using an output unit that outputs a result of production control of the lighting; The production control system according to claim 1 .

5. The control unit is When the estimation result of the emotion of the group based on the result of the emotion analysis indicates a relatively positive tendency, the usage time of the place is extended; shortening a usage time of the venue when an estimation result of the emotion of the group based on the result of the emotion analysis indicates a relatively negative tendency; A performance control system according to any one of claims 1 to 4.

6. The control unit is The emotions of the listeners are identified for two classification items including items with a relatively positive tendency and items with a relatively negative tendency, and the number of people who showed the same classification item emotion is counted for each classification item, and emotions common to the listeners are identified from the counting results. When the estimation result of the emotion of the group based on the result of the emotion analysis indicates a relatively negative tendency, the environment of the place is controlled so as to promote relaxation of the place. A performance control system according to any one of claims 1 to 4.

7. The control unit is When a high tendency of "anger" or "fear" is detected from the result of the emotion analysis, the environment of the place is controlled to be an environment that promotes relaxation. A performance control system according to any one of claims 1 to 4.

8. The control unit is When it is detected from the result of the emotion analysis that the tendency of the subject to be in a "low arousal state" is high, the environment of the subject is controlled to promote arousal. A performance control system according to any one of claims 1 to 4.

9. A method for controlling presentation using a presentation control device, the method comprising the steps of: participants in a presentation in which an information provider provides information to an audience include the information provider of the presentation, a first audience participating in the presentation online, and a second audience participating in the presentation in person at a venue where the presentation is held; and the method for controlling a presentation environment using a result of emotion analysis of the audience, the method comprising the steps of: The correspondence between the emotion analysis results and the performance of the controlled object is preset, obtaining the number of listeners who show the same emotion from the result of emotion analysis of each listener including the first listener and the second listener, and estimating the emotion of the group of listeners based on the number of listeners; A step of controlling the presentation of the environment of the venue based on a result of emotion analysis, which is an estimation result of the emotion of the group of listeners, and the correspondence relationship; a step of controlling an output from an output unit provided in the location where the information provider is present, to produce an environment of the location; A production control method including:

10. Participants of a presentation in which an information provider provides information to an audience include the information provider of the presentation, a first audience member who participates in the presentation online, and a second audience member who gathers at a venue where the presentation is held, and a computer of a production control device that produces an environment of the venue using a result of emotion analysis of the audience, The correspondence between the emotion analysis results and the performance of the controlled object is preset, obtaining the number of listeners who show the same emotion from the result of emotion analysis of each listener including the first listener and the second listener, and estimating the emotion of the group of listeners based on the number of listeners; controlling the presentation of the environment of the venue based on a result of emotion analysis, which is an estimation result of the emotion of the group of listeners, and the correspondence relationship; A step of controlling an output from an output unit provided in the location where the information provider is present, to produce an environment of the location; A program for executing the above.

Images