Conference system, control method, and control program

Abstract

To provide a technology for supporting the progress of an on-line conference by a facilitator.SOLUTION: A conference system comprises a control unit. The control unit executes a process of acquiring a plurality of moving images obtained by imaging each participant participating in an on-line conference, a process of estimating a participation attitude of each participant of the on-line conference based on each of the moving images, and a process of outputting information indicating the participation attitude of each participant of the on-line conference to a user terminal of a participant registered as a facilitator of the on-line conference. The information indicating the participation attitude of each participant of the on-line conference is not output to the user terminals of the participants other than the facilitator.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a conference system, a control method, and a control program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2023-059238 (Patent Document 1) discloses a "video meeting evaluation terminal that analyzes the biological responses of participants based on video images obtained in an online session with multiple participants" (see paragraph

[0001] ). The video meeting evaluation terminal displays information indicating changes in the analyzed biological responses alongside the video images of the participants (see Figure 6). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-059238 Summary of the Invention [Problem to be solved by the invention]

[0004] Depending on the online meeting being held, a facilitator may be in charge of the online meeting. The facilitator will encourage participants to speak and summarize what they have said to ensure the online meeting proceeds smoothly.

[0005] When the number of participants in an online conference increases, it becomes difficult for the facilitator to grasp the participation attitude of each participant, making it difficult to smoothly proceed with the online conference. Therefore, there is a need for technology to support facilitators in proceeding with online conferences. [Means for solving the problem]

[0006] In one example of the present disclosure, a conference system is provided. The conference system includes a control unit. The control unit executes the following processes: acquiring a plurality of moving images obtained by photographing each participant participating in an online conference; estimating a participation posture of each participant of the online conference based on each of the plurality of moving images; and outputting information indicating the participation posture of each participant of the online conference to a user terminal of a participant registered as a facilitator of the online conference. The information indicating the participation posture of each participant of the online conference is not output to the user terminals of participants other than the facilitator.

[0007] In one example of the present disclosure, the control unit further executes a process of acquiring voice data obtained by collecting voices of participants who are speaking in the online conference with a microphone. In the estimating process, the control unit estimates the participation posture of a participant who is speaking in the online conference using the voice data of the participant and a video image of the participant, and estimates the participation posture of a participant who is not speaking in the online conference based on the video image of the participant without using the voice data of the participant.

[0008] In one example of the present disclosure, the control unit further executes a process of acquiring text data of the voice data and a process of identifying the number of the positive terms and the number of the negative terms contained in the text data based on dictionary data defining positive terms and negative terms. In the estimation process, the control unit estimates the participation posture of a participant who is speaking in the online conference using the number of the positive terms, the number of the negative terms, and a video image of the participant.

[0009] In another example of the present disclosure, a control method for a conference system is provided. The control method includes the steps of acquiring a plurality of moving images obtained by photographing each participant participating in an online conference, estimating a participation posture of each participant of the online conference based on each of the plurality of moving images, and outputting information indicating the participation posture of each participant of the online conference to a user terminal of a participant registered as a facilitator of the online conference. The information indicating the participation posture of each participant of the online conference is not output to the user terminals of participants other than the facilitator.

[0010] In another example of the present disclosure, a control program for a conference system is provided. The control program causes a computer to execute the following processes: acquire a plurality of moving images obtained by photographing each participant participating in an online conference; estimate a participation posture of each participant of the online conference based on each of the plurality of moving images; and output information indicating the participation posture of each participant of the online conference to a user terminal of a participant registered as a facilitator of the online conference. The information indicating the participation posture of each participant of the online conference is not output to the user terminals of participants other than the facilitator.

[0011] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of a device configuration of a conference system. [Figure 2] FIG. 10 is a diagram showing a portion of an output screen displayed on a facilitator's user terminal. [Figure 3] FIG. 2 illustrates an example of a hardware configuration of a server. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal. [Figure 5]It is a diagram showing an example of dictionary data. [Figure 6] It is a diagram showing an example of participant information. [Figure 7] It is a diagram showing an example of the functional configuration of the server. [Figure 8] It is a diagram showing an output screen displayed on the user terminal of the facilitator during an online meeting. [Figure 9] It is a diagram showing an output screen displayed on a user terminal other than the facilitator during an online meeting. [Figure 10] It is a flowchart showing the flow of the display control process of the server.

Modes for Carrying Out the Invention

[0013] Hereinafter, each embodiment according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. In addition, each embodiment and each modification described below may be selectively combined as appropriate.

[0014] <A. Conference System 10> First, referring to FIG. 1, the device configuration of the conference system 10 will be described. FIG. 1 is a diagram showing an example of the device configuration of the conference system 10.

[0015] As shown in FIG. 1, the conference system 10 includes a server 100 and user terminals 200. The server 100 and the user terminals 200 are configured to be able to communicate with each other through the network NW.

[0016] The server 100 is a kind of computer. The server 100 is, for example, a notebook or desktop PC (Personal Computer), a tablet terminal, a smartphone, or other communication terminal. The number of servers 100 constituting the conference system 10 may be one or two or more.

[0017] The user terminal 200 is a type of computer. The user terminal 200 is, for example, a notebook or desktop PC, a tablet terminal, a smartphone, or other communication terminal. In the example of FIG. 1, three user terminals 200A to 200C are shown, but the number of user terminals 200 constituting the conference system 10 may be one or two or more. Hereinafter, when not particularly distinguishing between the user terminals 200A to 200C, the user terminals 200A to 200C are also referred to as the user terminal 200.

[0018] The server 100 provides various services to the user terminal 200. As an example, a user can participate in an online conference by establishing a communication connection to the server 100 via his or her own user terminal 200. When holding an online conference, the user registers conference information in advance. Examples of the registered conference information include, for example, the conference date, start time of the conference, end time of the conference, conference title, conference materials, conference agenda, participant information of the conference, and the like.

[0019] In addition, when registering participants in an online conference, the user can register the role of each participant. Examples of such roles include, for example, a facilitator who is a person in charge of proceeding with the conference, a recorder of minutes, an observer, and the like. In the example of FIG. 1, the role of the facilitator is set for the participant "A" from the user terminal 200A. The facilitator smoothly proceeds with the online conference by prompting participants to speak and summarizing what the participants say when holding the online conference.

[0020] <B. Overview> As the number of participants in an online conference increases, it becomes difficult for the facilitator to read the emotions of the participants. As a result, it becomes difficult for the facilitator to smoothly proceed with the online conference. Therefore, the conference system 10 according to the embodiment provides a function for assisting the progress of an online conference to the user terminal 200 set as the facilitator.

[0021] The function for supporting the facilitator will be described below with reference to Fig. 2. Fig. 2 is a diagram showing a part of an output screen 230 displayed on the facilitator's user terminal 200A.

[0022] The output screen 230 has a display field 236. User icons IC1 to IC8 are displayed in the display field 236. The user icons IC1 to IC8 represent users "A" to "H" who have been invited to the online conference, respectively.

[0023] The conference system 10 displays information indicating the participation posture of each participant in the online conference (hereinafter also referred to as "participation posture information") on the output screen 230. More specifically, the conference system 10 first acquires a plurality of moving images obtained by photographing each participant participating in the online conference. Next, the conference system 10 estimates the participation posture of each participant in the online conference based on each of the acquired plurality of moving images. Thereafter, the conference system 10 outputs the participation posture information of each participant in the online conference to the user terminal 200A of the participant registered as the facilitator of the online conference. At this time, the participation posture information of each participant is not output to the user terminals 200B, 200C of participants other than the facilitator.

[0024] Note that "not being output to user terminals 200B, 200C of participants other than the facilitator" does not only mean always prohibiting the display of participation posture information on user terminals 200B, 200C, but also includes switching the display of participation posture information on user terminals 200B, 200C according to settings. In this way, a conference system 10 having a function for switching whether or not to output participation posture information to user terminals 200B, 200C does not depart from the spirit of the present invention.

[0025] The estimated participation postures can be distinguished by any display mode. As an example, the conference system 10 expresses the difference in the participation posture of each user by varying the display mode of each of the user icons IC1 to IC8. As an example, the participation postures can be distinguished by changing the color of the user icons IC1 to IC8. Alternatively, the participation postures can be distinguished by changing the display size of the user icons IC1 to IC8. Alternatively, the participation postures can be distinguished by characters added to the user icons IC1 to IC8.

[0026] In the example of FIG. 2, the participation attitude is distinguished into two types: a first display mode indicating a negative attitude and a second display mode indicating a positive attitude. More specifically, user icons IC1, IC3, and IC6 corresponding to users "A," "C," and "F" are displayed in the first display mode indicating a negative participation attitude. On the other hand, user icons IC2, IC4, and IC5 corresponding to users "B," "D," and "E" are displayed in the second display mode indicating a positive participation attitude. Preferably, user icons IC7 and IC8 corresponding to users "G" and "H" who are not participating in the online conference are displayed in a third display mode different from the first and second display modes.

[0027] By checking the display field 236 on the output screen 230, the facilitator can understand the participation attitude of each participant in the online conference. This allows the facilitator to ensure a smooth online conference. As one example, the facilitator can encourage participants who are less motivated to participate to actively participate in the online conference. As another example, the facilitator can encourage a more active exchange of opinions in the online conference by asking for opinions from participants who are more motivated to participate.

[0028] In the above description, an example has been described in which the participation attitude is displayed in either a positive or negative manner, but a more detailed participation attitude can be confirmed after the end of the conference.

[0029] In addition, in FIG. 1, an example in which moving images are sent from user terminals 200B and 200C to server 100 is shown. However, the moving images may also be sent from user terminal 200A to server 100.

[0030] <C. Hardware Configuration> Next, referring to FIGS. 3 and 4, the hardware configuration of server 100 shown in FIG. 1 described above and the hardware configuration of user terminal 200 shown in FIG. 1 described above will be described in order.

[0031] (C1. Server 100) FIG. 3 is a diagram showing an example of the hardware configuration of server 100.

[0032] As shown in FIG. 3, server 100 includes a control device 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a communication interface 104, an input / output interface 105, and an auxiliary storage device 120. These components are connected to bus BS1.

[0033] The control device 101 is constituted by, for example, at least one integrated circuit. The integrated circuit may be constituted by, for example, at least one CPU (Central Processing Unit), at least one GPU (Graphics Processing Unit), at least one ASIC (Application Specific Integrated Circuit), at least one FPGA (Field Programmable Gate Array), or a combination thereof.

[0034] The control device 101 controls the operation of the server 100 by executing various programs such as a control program 122. Upon receiving an execution command for one of the various programs, the control device 101 reads the program from the auxiliary storage device 120 or the ROM 102 into the RAM 103. The RAM 103 functions as a working memory and temporarily stores various data required for executing the various programs.

[0035] A LAN (Local Area Network), an antenna, etc. are connected to the communication interface 104. The server 100 exchanges data with external devices via the communication interface 104. The external devices include, for example, a user terminal 200 and other communication devices.

[0036] The input / output interface 105 is an interface for connecting the server 100 to an external device. Examples of the input / output interface 105 include a Universal Serial Bus (USB), a Digital Visual Interface (DVI), and a High-Definition Multimedia Interface (HDMI). The number of input / output interfaces 105 provided in the server 100 is arbitrary. In the example of FIG. 3, two input / output interfaces 105A and 105B are provided in the server 100.

[0037] For example, a display 106 is connected to the input / output interface 105A. The input / output interface 105A sends an image signal for displaying an image to the display 106 in accordance with a command from the control device 101 or the like. The display 106 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or other display device. The display 106 may be configured integrally with the server 100 or may be configured separately from the server 100.

[0038] To the input / output interface 105B, for example, an input device 108 is connected. The input device 108 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving user operations. The input device 108 may be configured integrally with the server 100 or may be configured separately from the server 100.

[0039] The auxiliary storage device 120 is, for example, a hard disk, a flash memory, an SSD (Solid State Drive), or other storage medium. The auxiliary storage device 120 stores a control program 122, dictionary data 124 (described later), and participant information 126 (described later). These may be stored in a storage location other than the auxiliary storage device 120, such as a storage area (for example, cache memory) of the control device 101, the ROM 102, the RAM 103, an external device, or the like.

[0040] The control program 122 may be provided not as a standalone program but as part of an arbitrary program. Even if the program does not include some of the modules, it does not deviate from the spirit of the control program 122 according to this embodiment. Furthermore, some or all of the functions provided by the control program 122 may be realized by dedicated hardware. Furthermore, the server 100 may be configured in the form of a so-called cloud service in which a server different from the server 100 executes part of the processing of the control program 122.

[0041] (C2. User terminal 200) Next, the hardware configuration of the user terminal 200 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the hardware configuration of the user terminal 200.

[0042] 4, the user terminal 200 includes a control device 201, a ROM 202, a RAM 203, a communication interface 204, an input / output interface 205, and an auxiliary storage device 220. These components are connected to a bus BS2.

[0043] The control device 201 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one CPU, at least one GPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0044] The control device 201 controls the operation of the user terminal 200 by executing various programs such as the conference program 222. Upon receiving an execution command for one of the programs, the control device 201 reads the program from the auxiliary storage device 220 or the ROM 202 into the RAM 203. The RAM 203 functions as a working memory and temporarily stores various data required for executing the programs.

[0045] A LAN, an antenna, etc. are connected to the communication interface 204. The user terminal 200 exchanges data with external devices via the communication interface 204. The external devices include, for example, the server 100, other user terminals 200, and other communication devices.

[0046] The input / output interface 205 is an interface for connecting the user terminal 200 to an external device. Examples of the input / output interface 205 include USB, DVI, and HDMI. The number of input / output interfaces 205 provided in the user terminal 200 is arbitrary. In the example of FIG. 4, four input / output interfaces 205A to 205D are provided in the user terminal 200.

[0047] For example, a display 206 is connected to the input / output interface 205A. The input / output interface 205A sends an image signal for displaying an image to the display 206 in accordance with a command from the control device 201 or the like. The display 206 is, for example, a liquid crystal display, an organic EL display, or other display device. The display 206 may be configured integrally with the user terminal 200, or may be configured separately from the user terminal 200.

[0048] To the input / output interface 205B, for example, an input device 208 is connected. The input device 208 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving user operations. The input device 208 may be configured integrally with the user terminal 200, or may be configured separately from the user terminal 200.

[0049] To the input / output interface 205C, for example, a camera 209 is connected. The camera 209 is configured to be able to capture images of users participating in an online conference. The camera 209 may be configured integrally with the user terminal 200, or may be configured separately from the user terminal 200.

[0050] For example, a microphone 210 is connected to the input / output interface 205D. The microphone 210 is configured to collect the voice of a user during an online conference and generate audio data corresponding to the voice. The microphone 210 may be configured integrally with the user terminal 200 or may be configured separately from the user terminal 200. The microphone 210 may be configured integrally with the camera 209 or may be configured separately from the camera 209.

[0051] The auxiliary storage device 220 is, for example, a hard disk, a flash memory, an SSD, and other storage media. The auxiliary storage device 220 stores the conference program 222 and the like. The storage location of the conference program 222 is not limited to the auxiliary storage device 220, and may be stored in the storage area of the control device 201 (such as a cache memory), the ROM 202, the RAM 203, an external device, or the like.

[0052] The conference program 222 may be provided not as a single program but incorporated into a part of any program. Even a program that does not include such a part does not deviate from the gist of the conference program 222 according to the present embodiment. Furthermore, some or all of the functions provided by the conference program 222 may be realized by dedicated hardware. Furthermore, the user terminal 200 may be configured in a form such as a so-called cloud service in which at least one server executes a part of the processing of the conference program 222.

[0053] <D. Data Structure> Next, referring to FIGS. 5 and 6, the dictionary data 124 shown in FIG. 3 described above and the participant information 126 shown in FIG. 3 described above will be described in order.

[0054] (D1. Dictionary Data 124) First, referring to FIG. 5, the dictionary data 124 will be described. FIG. 5 is a diagram showing an example of the dictionary data 124.

[0055] The dictionary data 124 is used when estimating the participation posture of the participants in the online conference. Details of the estimation process of the participation posture will be described later.

[0056] The dictionary data 124 defines positive terms and negative terms. In the example of FIG. 5, positive terms such as "as you said", "good", "wonderful", "I agree" are shown. Also, as negative terms, "don't understand", "can't do", "no good", "dangerous" are shown. The positive terms and negative terms may be preset or may be arbitrarily registered by the user.

[0057] (D2. Participant Information 126) Next, referring to FIG. 6, the participant information 126 will be described. FIG. 6 is a diagram showing an example of the participant information 126.

[0058] When registering new meeting information, the user can also register the participant information 126 of the online meeting. The participant information 126 defines information about the users invited to the online meeting. In the example of FIG. 6, the participant information 126 associates, in addition to the users invited to the online meeting, the user ID (Identification), the name of the company to which the user belongs, the name of the department to which the user belongs, the user's participation approval, and the role set for the user.

[0059] The user ID is an identifier for uniquely identifying the user. The user ID may be defined, for example, by a character string, a numeric string, or a symbol string, or may be defined by a combination of at least one of characters, numbers, and symbols. Alternatively, the user ID may be defined by the identifier of the user terminal 200 owned by the user (for example, computer name or IP address).

[0060] Examples of roles that can be set as the participant information 126 include, for example, a facilitator who is in charge of the progress of the meeting, a recorder of the minutes, an observer, etc. The role in the online meeting may be set at the time of registration of the online meeting, may be set afterwards, or may be changed afterwards.

[0061] <E. Functional Configuration of Server 100> Next, the functional configuration of the server 100 will be described with reference to Fig. 7 to Fig. 9. Fig. 7 is a diagram showing an example of the functional configuration of the server 100.

[0062] 7, the server 100 includes, as functional components, an image acquisition unit 152, a voice acquisition unit 154, an estimation unit 160, and an output unit 170. These functional components may be implemented in the server 100 or in the user terminal 200.

[0063] The image acquisition unit 152, the audio acquisition unit 154, the estimation unit 160, and the output unit 170 will be described below in order.

[0064] (E1. Image acquisition unit 152) The image acquisition unit 152 is configured to acquire a moving image IM obtained by photographing each participant participating in the online conference for each participant.

[0065] More specifically, the user terminal 200 continues capturing images with the camera 209 during the online conference and generates a video IM. The generated video IM is periodically transmitted to the server 100. The image acquisition unit 152 periodically acquires the video received from the user terminal 200. The acquired video IM is stored in the auxiliary storage device 120 of the server 100 after being associated with the identification information of the user terminal 200 from which it was acquired.

[0066] The image acquiring unit 152 may acquire video images IM for all participants of the online conference, or may acquire video images IM for some of the participants of the online conference. For example, the image acquiring unit 152 does not need to acquire video images IM for the facilitator.

[0067] The moving images IM acquired by the image acquisition unit 152 are continuously output to the estimation unit 160 .

[0068] (E2.Audio acquisition unit 154) Next, the function of the voice acquisition unit 154 shown in FIG. 7 will be described.

[0069] The voice acquisition unit 154 is configured to acquire voice data VD obtained by collecting the voices of participants speaking in an online conference.

[0070] More specifically, the user terminal 200 continues to collect sound with the microphone 210 during the online conference and generates audio data VD. The generated audio data VD is periodically transmitted to the server 100. The audio acquisition unit 154 periodically acquires the audio data VD received from the user terminal 200. The acquired audio data VD is stored in the auxiliary storage device 120 of the server 100 after being associated with the identification information of the user terminal 200 from which the data was acquired.

[0071] The voice acquisition unit 154 may acquire voice data VD for all participants in the online conference, or may acquire voice data VD for some of the participants in the online conference. As an example, the voice acquisition unit 154 acquires voice data VD for participants who are speaking, but does not acquire voice data VD for participants who are watching silently.

[0072] The voice data VD acquired by the voice acquisition unit 154 is continuously output to the estimation unit 160.

[0073] (E3.Estimation part 160) Next, the function of the estimation unit 160 shown in FIG. 7 will be described.

[0074] The estimation unit 160 is configured to estimate the participation posture of each participant in the online conference. At least the moving image IM is used to estimate the participation posture. Preferably, not only the moving image IM but also the audio data VD is used to estimate the participation posture.

[0075] At this time, for a participant who is speaking in the online conference, the estimation unit 160 estimates the participation posture of the participant using the voice data VD of the participant and the video image IM of the participant. On the other hand, for a participant who is not speaking in the online conference, the estimation unit 160 estimates the participation posture of the participant based on the video image IM of the participant without using the voice data VD of the participant.

[0076] Such estimation of the participation posture is realized by the image analysis unit 162, the voice analysis unit 164, the text analysis unit 166, and the evaluation unit 168. These functions will be explained below in order.

[0077] (a) Image analysis unit 162 First, the function of the image analysis unit 162 shown in FIG. 7 will be described.

[0078] The image analysis unit 162 uses a predetermined image analysis program to analyze the moving images IM acquired by the image acquisition unit 152 for each participant, and evaluates the emotions of the participants appearing in the moving images IM.

[0079] The image analysis program uses, for example, an API (Application Programming Interface) provided by an individual or a company, such as Rekognition provided by Amazon, Face API provided by Microsoft, or Google Cloud Vision API.

[0080] When the image analysis program receives input of a video IM via the API, it outputs an evaluation value indicating the emotion of the person appearing in the video IM. A trained model is used to estimate the emotion. The trained model has been trained in advance by machine learning to output the emotion of the person appearing in the video IM when the video IM is input.

[0081] The output values ​​of the image analysis program may include, for example, the degree of “happiness,” the degree of “anger,” the degree of “confusion,” the degree of “disgust,” and the degree of “sadness.” The image analysis unit 162 calculates an emotion evaluation value using at least one of these degrees.

[0082] As an example, the image analysis unit 162 multiplies the degree to which a positive emotion is indicated among the output values ​​by a positive weight, and multiplies the degree to which a negative emotion is indicated among the output values ​​by a negative weight. The image analysis unit 162 then outputs the integrated value of each multiplication as the emotion evaluation value "E1." As an example, the emotion evaluation value "E1" is calculated according to the following formula (1):

[0083] E1=w1 x1+w2 x2+w3 x3+w4 x4+w5 x5 (1) Here, "x1" in the above formula (1) indicates the degree of "happiness" output from the image analysis program. "x2" indicates the degree of "anger" output from the image analysis program. "x3" indicates the degree of "confusion" output from the image analysis program. "x4" indicates the degree of "disgust" output from the image analysis program. "x5" indicates the degree of "sadness" output from the image analysis program.

[0084] Furthermore, "w1" to "w5" indicate weights. As an example, "w1" indicates a positive value (for example, 100%), and "w2" to "w5" indicate negative values ​​(for example, -20%). The weights "w1" to "w5" may be set in advance or may be arbitrarily set by the user.

[0085] The emotion evaluation value "E1" calculated by the image analysis unit 162 is output to the evaluation unit 168, which will be described later.

[0086] The function of the image analysis unit 162 may be implemented in the server 100 or in the user terminal 200.

[0087] (b) Voice analysis section 164 Next, the function of the voice analysis unit 164 shown in FIG. 7 will be described.

[0088] The voice analysis unit 164 uses a predetermined voice analysis program to analyze the voice data VD acquired by the voice acquisition unit 154 and evaluate the emotions of the participants.

[0089] The voice analysis program uses, for example, an API provided by an individual or a company, such as the Cloud Natural Language API provided by Google or Chime Voice Tone Analysis by Amazon Transcibe.

[0090] When the voice analysis program receives voice data VD via the API, it outputs an evaluation value indicating the emotion of the speaker of the voice. A trained model is used to estimate the emotion. The trained model has been trained in advance by machine learning to output the emotion of the person who spoke the voice when it receives voice data VD.

[0091] The output values ​​of the voice analysis program may include, for example, the degree of "calmness," the degree of "happiness," the degree of "anger," etc. The voice analysis unit 164 calculates the emotion evaluation value "E2" using at least one of these degrees.

[0092] As one example, the voice analysis unit 164 outputs one of the output values ​​of the voice analysis program (for example, the degree of "calmness") as the emotion evaluation value "E2." As another example, the voice analysis unit 164 multiplies the degree of a positive emotion among the output values ​​by a positive weight, and multiplies the degree of a negative emotion among the output values ​​by a negative weight. Then, the voice analysis unit 164 outputs the integrated value of each multiplication value as the emotion evaluation value "E2."

[0093] The emotion evaluation value "E2" calculated by the voice analysis unit 164 is output to the evaluation unit 168, which will be described later.

[0094] (c) Text analysis unit 166 Next, the function of the text analysis unit 166 shown in FIG. 7 will be described.

[0095] The text analysis unit 166 converts the voice data VD acquired by the voice acquisition unit 154 into text using a predetermined voice analysis program, and evaluates the speaker's proactiveness based on the text data.

[0096] To convert the voice data VD into text, for example, an API provided by an individual or a company is used, such as Speech-to-Text provided by Google or Amazon Transcibe.

[0097] The text analysis unit 166 calculates an evaluation value "E3" indicating the speaker's assertiveness based on the text data of the voice data VD. As an example, the text analysis unit 166 counts the number of positive terms included in the voice text data with reference to the above-mentioned dictionary data 124 (see FIG. 5). The text analysis unit 166 also counts the number of negative terms included in the voice text data with reference to the above-mentioned dictionary data 124. The text analysis unit 166 then subtracts the number of negative terms from the number of positive terms, and outputs the result as the assertiveness evaluation value "E3."

[0098] As described above, the text analysis unit 166 determines the number of positive terms contained in the text data of the speech and the number of negative terms contained in the text data based on the dictionary data 124 that defines positive terms and negative terms.

[0099] The aggressiveness evaluation value "E3" calculated by the text analysis unit 166 is output to the evaluation unit 168, which will be described later.

[0100] (d) Evaluation section 168 Next, the function of the evaluation unit 168 shown in FIG. 7 will be described.

[0101] The evaluation unit 168 calculates an evaluation value "V" indicating the participation attitude of a participant in an online conference based on the above-mentioned evaluation value "E1" output from the image analysis unit 162, the above-mentioned evaluation value "E2" output from the audio analysis unit 164, and the above-mentioned evaluation value "E3" output from the text analysis unit 166.

[0102] As an example, the evaluation unit 168 multiplies each of the evaluation values ​​"E1" to "E3" by a predetermined weight, and calculates the sum of each multiplication value as the evaluation value "V" indicating the participation attitude. The evaluation value "V" is calculated for each participant of the online conference. More specifically, the evaluation value "V" is calculated according to the following formula (2).

[0103] V=w A E1+w B E2+w C ·E3···(2) Here, "w" shown in the above formula (2) A "~"w C " indicates the weight. The weight "w A "~"w C " may be set in advance or may be arbitrarily set by the user.

[0104] The evaluation unit 168 estimates that a participant whose evaluation value "V" shown in the above formula (2) is equal to or greater than a first threshold has an active participation attitude. On the other hand, the evaluation unit 168 estimates that a participant whose evaluation value "V" is equal to or less than a second threshold has a passive participation attitude. The second threshold may be the same as the first threshold or may be smaller than the first threshold. Furthermore, the first and second thresholds may be set in advance or may be arbitrarily set by the user.

[0105] The evaluation unit 168 calculates the weights "w A "~"w C " Change it.

[0106] In a certain situation, the evaluation unit 168 assigns a weight "w A "~"wC ” is set to be greater than 0. For example, the weight “w A "~"w C " are set to 33%. That is, for a participant who is speaking in an online conference, the evaluation unit 168 estimates the participation posture of the participant by using the voice data VD of the participant and the moving image IM of the participant.

[0107] On the other hand, the evaluation unit 168 assigns a weight "w B ", "w C " are set to 0, and the weight "w A " is set to 100%. That is, for a participant who is watching quietly, the evaluation unit 168 estimates the participation posture of the participant by using the moving image IM of the participant without using the voice data VD of the participant.

[0108] In addition, the weight "w A "~"w C The way of changing the weight "w" is not limited to the above example. As another example, a participant of an online conference may have the camera function turned off or may be using a user terminal 200 that does not have a camera. For such a participant, the evaluation unit 168 may change the weight "w" A " is set to 0, and the weight "w B ", "w C " are set to 50%. That is, for a participant whose moving image IM cannot be acquired, the evaluation unit 168 estimates the participation posture of the participant by using the voice data VD of the participant without using the moving image IM of the participant.

[0109] Furthermore, in the above, an example was described in which a participant's participation attitude is estimated based on both the evaluation value "E2" calculated from the audio data VD and the evaluation value "E3" calculated from the audio text data, but only one of the evaluation values ​​"E2" or "E3" may be used to estimate the participation attitude.

[0110] (E4. Output unit 170) Next, the function of the output unit 170 shown in FIG. 7 will be described with reference to FIG.

[0111] The output unit 170 outputs information indicating the participation posture of each participant estimated by the estimation unit 160 (participation posture information) to the user terminal 200 of the participant who is registered as a facilitator of the online conference.

[0112] Whether a participant is a facilitator is determined based on the participant information 126 (see FIG. 6) described above. More specifically, the output unit 170 refers to the participant information 126 to identify participants who are registered as facilitators. Then, the output unit 170 outputs participation attitude information to the user terminal 200 of the participant who is registered as a facilitator.

[0113] The facilitator's user terminal 200 displays an output screen according to the participation attitude information received from the server 100. Figure 8 shows an output screen 230A displayed on the facilitator's user terminal 200 during an online conference.

[0114] The output screen 230A for the facilitator includes display fields 232, 234, and 236, and edit fields 238 and 240.

[0115] The contents of the agenda for the online conference are displayed in the display field 232. The time allocated to the discussion of each agenda item is also displayed in the display field 232. For the agenda item currently being discussed, the remaining time is displayed. Furthermore, the display field 232 displays a progress bar indicating the progress of the online conference.

[0116] The display field 234 displays the content of comments made during the online conference as text data. As an example, the display field 234 displays the text data of the voice output during the processing of the text analysis unit 166 described above. Preferably, the name of the speaker is written alongside each comment. Furthermore, the content of comments displayed in the display field 234 is updated in real time during the online conference. Preferably, the content displayed in the display field 234 can be saved as the minutes of the online conference.

[0117] The participation posture information estimated by the estimation unit 160 is displayed for each participant in the display field 236. The display mode of the participation posture information in the display field 236 is as described with reference to FIG. 2, and therefore the description thereof will not be repeated.

[0118] Participants can edit the comment content displayed in the display field 234 in the edit field 238. More specifically, participants can select the comment content displayed in the display field 234. The selected comment content is reflected in the edit field 238. Thereafter, participants can use the above-mentioned input device 208 (see FIG. 4 ) to edit the comment content reflected in the edit field 238. Thereafter, by pressing the send button in the edit field 238, the edited comment content can be reflected in the display field 234.

[0119] Participants can input decisions and considerations for the online conference in the edit field 240. The matters input in the edit field 240 can be saved as minutes of the online conference.

[0120] The participation attitude information displayed in display field 236 is not output to the user terminals 200 of participants other than the facilitator. Figure 9 shows an output screen 230B displayed on the user terminals 200 of users other than the facilitator during an online conference.

[0121] Compared to the output screen 230A for the facilitator, the output screen 230B does not have a display field 236 for displaying the participation attitude information of each participant. In this way, the participation attitude information of each participant is not displayed on the user terminals 200 of users other than the facilitator.

[0122] <F.フローチャート> Next, the display control process of the server 100 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the display control process of the server 100.

[0123] The process shown in Fig. 10 is realized by the control device 101 of the server 100 executing the above-described control program 122. In another aspect, part of the process shown in Fig. 10 may be executed by the server 100 or another device.

[0124] In step S110, the control device 101 determines whether or not an instruction to enable the display of the participation posture has been received during the online conference. The enable instruction may be issued automatically when the online conference starts, or may be issued by a user operation by the facilitator, for example. If the control device 101 determines that the enable instruction has been received (YES in step S110), it switches control to step S112. If not (NO in step S110), the control device 101 executes the process of step S110 again.

[0125] In step S112, the control device 101 functions as the image acquisition unit 152 described above, and acquires video images of participants participating in the online conference from the user terminals 200 of the participants. The video images are acquired for each participant participating in the online conference.

[0126] In step S114, the control device 101 functions as the above-mentioned voice acquisition unit 154 and acquires voice data of a participant who is speaking in the online conference from the user terminal 200 of that participant.

[0127] In step S116, the control device 101 functions as the image analysis unit 162 described above, analyzes the video acquired in step S112, and calculates an evaluation value "E1" that indicates the emotion of the participant appearing in the video. The evaluation value "E1" is calculated for each participant participating in the online conference. The method for calculating the evaluation value "E1" is as described above, and therefore will not be described again.

[0128] In step S118, the control device 101 functions as the above-mentioned voice analysis unit 164, analyzes the voice data acquired in step S114, and calculates an evaluation value "E2" indicating the emotion of the speaker in the online conference. The evaluation value "E2" is calculated for the speaker in the online conference. The method for calculating the evaluation value "E2" is as described above, and therefore the description thereof will not be repeated.

[0129] In step S120, the control device 101 functions as the above-mentioned text analysis unit 166 and converts the voice data acquired in step S114 into text. Then, the control device 101 calculates an evaluation value "E3" indicating the speaker's proactiveness based on the text data. The evaluation value "E3" is calculated for each speaker in the online conference. The method for calculating the evaluation value "E3" is as described above, and therefore will not be described again.

[0130] In step S122, the control device 101 functions as the evaluation unit 168 described above, and calculates an evaluation value "V" indicating the participants' attitudes toward the online conference based on the evaluation value "E1" calculated in step S116, the evaluation value "E2" calculated in step S118, and the evaluation value "E3" calculated in step S120. The evaluation value "V" is calculated separately for each participant taking part in the online conference. The method for calculating the evaluation value "V" is as described above, and therefore will not be described again.

[0131] In step S124, the control device 101 functions as the above-mentioned output unit 170, and outputs information indicating the participation posture of each participant based on the evaluation value "V" calculated in step S122 to the user terminal 200. The information is output to the user terminal 200 of the participant who is registered as a facilitator of the online conference.

[0132] In step S130, the control device 101 determines whether or not an instruction to disable the display of the participation posture has been received during the online conference. The instruction to disable may be issued automatically when the online conference ends, or may be issued by a user operation by the facilitator, for example. If the control device 101 determines that the instruction to disable has been received (YES in step S130), it ends the processing shown in FIG. 10. If not (NO in step S130), the control device 101 returns the processing to step S112.

[0133] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0134] 10 Conference system, 100 Server, 101 Control device, 102 ROM, 103 RAM, 104 Communication interface, 105 Input / output interface, 105A Input / output interface, 105B Input / output interface, 106 Display, 108 Input device, 120 Auxiliary storage device, 122 Control program, 124 Dictionary data, 126 Participant information, 152 Image acquisition unit, 154 Voice acquisition unit, 160 Estimation unit, 162 Image analysis unit, 164 Voice analysis unit, 166 Text analysis unit, 168 Evaluation unit, 170 Output unit, 200 User terminal, 200A User terminal, 200B User terminal, 200C User terminal, 201 Control device, 202 ROM, 203 RAM, 204 Communication interface, 205 Input / output interface, 205A Input / output interface, 205B Input / output interface, 205C Input / output interface, 205D input / output interface, 206 display, 208 input device, 209 camera, 210 microphone, 220 auxiliary storage device, 222 conference program, 230 output screen, 230A output screen, 230B output screen, 232 display column, 234 display column, 236 display column, 238 editing column, 240 editing column, BS1 bus, BS2 bus, IC1 to IC8 user icon, IM moving image, NW network, VD voice data.

Claims

1. A conference system, comprising: A control unit is provided, The control unit A process of acquiring a plurality of moving images obtained by photographing each participant participating in an online conference; a process of estimating a participation posture of each participant of the online conference based on each of the plurality of moving images; and outputting information indicating the participation attitude of each participant of the online conference to a user terminal of a participant who is registered as a facilitator of the online conference; A conference system, wherein information indicating the participation attitude of each participant in the online conference is not output to user terminals of participants other than the facilitator.

2. The control unit further execute a process of acquiring voice data obtained by collecting voices of participants speaking in the online conference with a microphone; The control unit, in the estimation process, For a participant who is speaking in the online conference, the participation posture is estimated using the voice data of the participant and a video image of the participant; The conference system according to claim 1 , wherein, for a participant who is not speaking in the online conference, the participation posture is estimated based on a video image of the participant without using voice data of the participant.

3. The control unit further A process of obtaining text data of the voice data; performing a process of identifying the number of positive terms contained in the text data and the number of negative terms contained in the text data based on dictionary data defining positive terms and negative terms; The control unit, in the estimation process, The conference system according to claim 2 , wherein the participation attitude of a participant who is speaking in the online conference is estimated using the number of positive terms, the number of negative terms, and a video image of the participant.

4. A control method for a conference system, comprising: acquiring a plurality of moving images obtained by photographing each participant participating in an online conference; estimating a participation posture of each participant of the online conference based on each of the plurality of moving images; outputting information indicating the participation attitude of each participant of the online conference to a user terminal of a participant who is registered as a facilitator of the online conference; The control method, wherein information indicating the participation attitude of each participant in the online conference is not output to user terminals of participants other than the facilitator.

5. A control program for a conference system, The control program is configured to A process of acquiring a plurality of moving images obtained by photographing each participant participating in an online conference; a process of estimating a participation posture of each participant of the online conference based on each of the plurality of moving images; and outputting information indicating the participation attitude of each participant of the online conference to a user terminal of a participant who is registered as a facilitator of the online conference; The control program is configured such that information indicating the participation attitude of each participant in the online conference is not output to user terminals of participants other than the facilitator.

Images