Information processing system and information processing program

Abstract

To provide an information processing device and a program capable of determining a priority of a speaker of overlapped utterance when utterance overlaps during an online meeting.SOLUTION: An information processing device executes the steps of: determining whether or not there is overlapped utterance spoken by another speaker during utterance of one speaker in an online meeting; and determining a priority of a speaker who has performed overlapped utterance according to a predetermined rule when there is overlapped utterance.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to information processing system and information processing programs. [Background technology]

[0002] Patent document 1 discloses a speech management system that is characterized by the fact that in a remote conferencing system in which a management server and participant terminals are connected via a network and simultaneous conversations are possible, when multiple conference participants speak at the same time, the management server has the function of equalizing speaking opportunities.

[0003] Patent Document 2 discloses a conference support system that supports the operation of remote conferences in which participants from each location participate, characterized in that the conference support system comprises an audio / image information acquisition unit at each location that acquires audio and image information of remarks made by participants, a memory unit that stores the audio and image information acquired for each location, a speaker determination unit that determines whether multiple participants have spoken simultaneously or whether there has been overlap in their remarks, and a remark presentation unit that presents multiple remarks made simultaneously or that have overlap.

[0004] Patent Document 3 discloses a method for equalizing speaking opportunities in which a conference is held by transmitting and receiving the voices of conference participants via a network, the method comprising the steps of: capturing the voices of the conference participants using an audio capturing means; capturing video of the conference participants using an image capturing means; detecting the movements of the conference participants from the video using an action detecting means; calculating a next utterance recommendation value indicating the priority of a speech based on the order in which the conference participants spoke in a speaker turn history that lists the participants in the order in which they spoke and the movements detected in the detecting step using a next utterance recommendation value calculating means; adjusting the sound pressure levels of the voices of the conference participants based on the magnitude of the next utterance recommendation value using a sound pressure adjusting means; transmitting and receiving the voices of the conference participants with the adjusted sound pressure levels using a transmitting and receiving means; and selecting the conference participant who is currently speaking as the speaker and recording this in the speaker turn history using a speaker selecting means. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-304337 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-229903 [Patent Document 3] Patent No. 5432805 Summary of the Invention [Problem to be solved by the invention]

[0006] During online meetings, speakers may speak multiple times.

[0007] The present invention aims to provide an information processing device and a program that can determine the priority of speakers of overlapping statements when overlapping statements occur during an online meeting. [Means for solving the problem]

[0008] The information processing device of the first aspect includes a processor that determines whether or not a duplicate statement has been made by another speaker during an online meeting, and if such a duplicate statement has been made, determines the priority of the speaker who made the duplicate statement in accordance with predetermined rules.

[0009] In the information processing device of the second aspect, the processor acquires audio data of speech made by a speaker during the online meeting, generates filtered audio data by removing noise from the audio data, and determines the priority from the filtered audio data.

[0010] In the information processing device of the third aspect, the processor determines the start and end times of the speech of one speaker, and determines that the overlapping speech has occurred if the speech of another speaker begins between the start and end times.

[0011] In the information processing device according to a fourth aspect, the processor notifies the priority order.

[0012] In the information processing device according to a fifth aspect, the notification of the priority is made to predetermined participants among the participants of the online meeting.

[0013] In the information processing device according to a sixth aspect, the notification of the priority is displayed superimposed on the front side of the materials displayed on the display unit during the online meeting.

[0014] In an information processing device according to a seventh aspect, the predetermined rule is a predetermined attribute of the speaker.

[0015] In an information processing device according to an eighth aspect, the predetermined rule is a feature of the speech of the speaker.

[0016] In the information processing device according to a ninth aspect, the predetermined rule is determined by weighting predetermined attributes of the speaker and characteristics of the speaker's utterances.

[0017] In an information processing device according to a tenth aspect, the attribute is job title information of the speaker.

[0018] In an information processing device according to an eleventh aspect, the attribute is whether the speaker is a required participant or an optional participant in the online meeting.

[0019] In an information processing device according to a twelfth aspect, the characteristic is the speed of the overlapping comments.

[0020] The information processing program of the thirteenth aspect causes a computer to determine whether or not a duplicate statement has been made by another speaker during an online meeting, and if such a duplicate statement has been made, to execute a process of determining the priority of the speaker who made the duplicate statement in accordance with predetermined rules. [Effects of the Invention]

[0021] According to the first aspect, when overlapping statements occur during an online meeting, it is possible to determine the priority order of the speakers of the overlapping statements.

[0022] According to the second aspect, there is an advantage that the priority can be determined without being affected by noise, compared to when noise is not removed.

[0023] According to the third aspect, it is possible to determine whether or not there has been an overlapping statement by determining whether or not another speaker has started speaking between the start and end of a statement.

[0024] According to the fourth aspect, there is an effect that a speaker who has made an overlapping comment can know the order in which he or she can comment.

[0025] According to the fifth aspect, there is an advantage that the priority order can be notified only to necessary participants.

[0026] According to the sixth aspect, there is an effect that it is possible to make the notification of the priority easier for the person who has been notified of the priority, compared to when the notification is displayed on the back side of the document.

[0027] According to the seventh aspect, there is an effect that the priority can be determined according to the attributes of the speaker.

[0028] According to the eighth aspect, there is an effect that the priority can be determined according to the characteristics of the speaker.

[0029] According to the ninth aspect, there is an effect that the priority can be determined based on the attributes and characteristics of the speaker.

[0030] According to the tenth aspect, there is an effect that the order of priority can be determined based on the job title information.

[0031] According to the eleventh aspect, there is an effect that the priority order can be determined depending on whether the participant is a required participant or an optional participant.

[0032] According to the twelfth aspect, there is an effect that priority can be given to the speech of the person who made the speech earlier.

[0033] According to the thirteenth aspect, it is possible to provide an information processing program that can determine the priority of speakers of overlapping statements when overlapping statements occur during an online meeting. [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a schematic configuration diagram of an information processing system according to an embodiment of the present invention. [Figure 2] 1 is a schematic block diagram of a PC according to an embodiment of the present invention. [Figure 3]2 is a schematic block diagram of a server and an online meeting server according to an embodiment of the present invention; [Figure 4] 10 is a flowchart showing an example of a process flow for determining and displaying priorities of speakers of overlapping comments according to an embodiment of the present invention. [Figure 5] 10 is an example of advance information registration for determining priority levels according to an embodiment of the present invention. [Figure 6] 10 is an example of a case where the priority order according to the embodiment of the present invention is determined by combining the advance information and the characteristics of overlapping statements. [Figure 7] 10 is an example of setting a notification of priority according to an embodiment of the present invention. [Figure 8] 10 is a diagram illustrating an example of a display on a display unit of a PC when notifying a priority order according to an embodiment of the present invention. [Figure 9] 10 is another example of a display on the display unit of the PC when notifying the priority order according to the embodiment of the present invention. [Figure 10] FIG. 10 is an explanatory diagram for explaining the removal of noise from audio data according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0035] (Embodiment) An example of an embodiment of the present disclosure will be described below with reference to the drawings. The same or equivalent components and parts in each drawing are designated by the same reference numerals. The dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0036] An example of an information processing system 10 according to this embodiment will be described with reference to FIG.

[0037] FIG. 1 is a diagram showing an example of a schematic configuration of an information processing system 10 according to this embodiment. As shown in FIG. 1, an information processing system 10 according to this embodiment includes a plurality of PCs (Personal Computers) 20, a server 50, and an online meeting server 60. The PCs 20, the server 50, and the online meeting server 60 are connected via a network N. This network N may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. In this embodiment, the server 50 is an example of an information processing device. The online meeting server 60 may also function as the server 50.

[0038] The PCs 20 are terminal devices used by participants in the online meeting in the information processing system 10, and include tablet PCs and smartphones. In this embodiment, one PC is used per participant. Each PC 20 stores information identifying the participant using it, such as a user ID, and when connecting to and logging in to the online meeting server 60, the information identifying the participant is also transmitted to the server 50 or the online meeting server 60. This makes it possible to identify who is making a comment during the online meeting.

[0039] The online meeting server 60 is a server computer in the information processing system 10 that provides an online meeting service.

[0040] Furthermore, the information processing system 10 of this embodiment is a system in which, when participants use PCs 20 to connect to the online meeting server 60 and hold an online meeting, the server 50 acquires participant comments from the PCs 20 or the online meeting server 60, and, if multiple participants' comments overlap, determines the priority of the speakers of the overlapping comments. That is, in an online meeting, multiple participants may make overlapping comments, but it can take time to determine which participant's comments are overlapping and which of the participants who made the overlapping comments should be given priority, or participants may hesitate to speak in order to avoid the overlapping comments, which can make the online meeting inefficient.

[0041] FIG. 2 is a block diagram showing the hardware configuration of the PC 20 according to this embodiment.

[0042] 2, the PC 20 includes a CPU (Central Processing Unit) 21, which is an example of a processor, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, an input unit 25, a display unit 26, a camera 27, a microphone 28, a speaker 29, and a communication interface 30. Each component is connected to each other via a bus 31 so as to be able to communicate with each other.

[0043] The CPU 21 is a central processing unit that executes various programs and controls each component. That is, the CPU 21 reads programs from the ROM 22 or the storage unit 24 and executes the programs using the RAM 23 as a work area. The CPU 21 controls the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 22 or the storage unit 24. In this embodiment, the programs are stored in the ROM 22 or the storage unit 24.

[0044] The ROM 22 stores various programs and various data. The RAM 23 serves as a working area and temporarily stores programs or data. The storage unit 24 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0045] The input unit 25 includes a pointing device such as a mouse and a keyboard, and is used for various inputs. As will be described later, the input unit 25 is used to input advance information for determining the priority of overlapping messages.

[0046] The display unit 26 is, for example, a liquid crystal display, and displays various types of information under the control of the CPU 21.

[0047] The camera 27 captures images of participants in the online meeting.

[0048] The microphone 28 picks up the speech of the participants in the online meeting.

[0049] The speaker 29 uses another PC 20 connected via the network N. Outputs the comments of other participants in an online meeting.

[0050] The communication interface 30 is connected to a network N, and enables the PC 20 to communicate with other PCs 20 and other devices such as a server 50.

[0051] 3 is a block diagram showing the hardware configuration of the server 50 and the online meeting server 60. The server 50 and the online meeting server 60 basically have a general computer configuration, so the server 50 will be described as a representative.

[0052] 3, the server 50 includes a CPU 51, a ROM 52, a RAM 53, a storage unit 54, and a communication interface 55. Each component is connected to each other so as to be able to communicate with each other via a bus 56. The server 50 may also include an input unit such as a keyboard and a display unit such as a display.

[0053] The CPU 51 is a central processing unit that executes various programs and controls each component. That is, the CPU 51 reads programs from the ROM 52 or the storage unit 54 and executes the programs using the RAM 53 as a work area. The CPU 51 controls the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 52 or the storage unit 54. In this embodiment, the ROM 52 or the storage unit 54 stores programs and various data.

[0054] The ROM 52 stores various programs and various data. The RAM 53 temporarily stores programs or data as a working area. The storage unit 54 is configured with an HDD or SSD and stores various programs including an operating system and various data. The storage unit 54 of the server 50 stores various data transmitted from the PC 20 and the online meeting server 60.

[0055] The communication interface 55 is for communicating with other devices such as the PC 20 and the online meeting server 60, and uses standards such as the Internet, intranet, Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0056] Next, the operation of the information processing system 10 will be described.

[0057] 4 is a flowchart showing an example of the process flow in which the server 50 determines and notifies the priority of speakers of overlapping messages. Note that this process may be performed by the online meeting server 60 instead of the server 50.

[0058] As a premise of this flowchart, before the start of an online meeting, the administrator, facilitator, participants, etc. of the online meeting perform the predetermined settings shown in Figures 5 to 7. Note that this setting is not limited to being performed for each online meeting, and the settings of a previous online meeting may be used as is or partially reused.

[0059] First, the server 50 is connected to the online meeting server 60. Through this connection, the server 50 acquires the voice data of the speech made by the speaker from the online meeting server 60. Note that the voice data does not necessarily have to be acquired from the online meeting server 60, but may be acquired from the PC 20.

[0060] In addition, as shown in FIG. 5, the administrator or moderator of the online meeting accesses the server 50 from the PC 20 and registers advance information for determining the priority of overlapping comments on the server 50. The advance information is participant attributes, which are an example of predetermined rules for determining the priority of overlapping comments. Participant attributes include, for example, the participant's "job title," "evaluation history," "required participant," "optional participant," and "priority speaker," as shown in FIG. 5. "Job title information" is information about a position, such as a department manager or group leader, and prioritizes the comments of superiors among overlapping comments. This is adopted as a participant attribute for determining priority because superiors are more likely to make important comments. Furthermore, "evaluation history" is adopted as a participant attribute for determining priority because employees with higher evaluation records are more likely to make useful comments than employees with lower evaluation records, even if they have the same job title. Furthermore, "required participant" and "optional participant" are adopted as participant attributes for determining priority because the comments of "required participants" may be more important than those of "optional participants." In addition to the above attributes, an input field for "priority speaker" is provided, allowing employees who are particularly desired to speak to be registered in advance.

[0061] Note that the attributes of participants are not limited to those shown in FIG. 5 and may include other attributes. Furthermore, the priority of comments may be determined using only one of the attributes shown in FIG. 5, or multiple attributes may be combined to determine the priority of comments. A mode of determining the priority of comments by combining multiple attributes will be described later with reference to FIG. 6. Furthermore, if there is an existing database in which internal organization information, such as a mailer or groupware, is registered, the participant attributes may be obtained from the information in that database. In this way, obtaining participant attributes from an existing database can reduce the burden of inputting information in advance for each online meeting.

[0062] Furthermore, the priority of overlapping utterances may be determined based on characteristics of overlapping utterances other than prior information. Examples include "speed of utterance," "volume of voice," and "volume of gestures and hand movements." "Speed of utterance" prioritizes utterances that are spoken earlier among overlapping utterances. "Voice volume" prioritizes utterances that are spoken louder among overlapping utterances. "Voice volume" prioritizes utterances that are captured by camera 27 and feature louder gestures and hand movements. Because louder voices and louder gestures are presumed to indicate confidence in the content of the utterance or a strong assertion, these characteristics are used as characteristics of overlapping utterances for determining priority. Note that overlapping utterance characteristics are not limited to the above-mentioned characteristics and may include other characteristics. Furthermore, the priority of utterances may be determined based on only one of the above-mentioned characteristics, or multiple characteristics may be combined to determine the priority of utterances. Furthermore, the priority of utterances may be determined in combination with the above-mentioned prior information. A manner in which the priority of utterances is determined by combining such prior information and characteristics of overlapping utterances will be described later with reference to FIG. 6.

[0063] Using FIG. 6, we will explain how to determine the priority of statements by combining prior information and characteristics of overlapping statements. In such a case, as shown in FIG. 6, the manager or moderator of the online meeting accesses the server 50 from the PC 20 and registers the priorities (weightings) of the prior information for determining the priority of overlapping statements in the server 50. In the example shown in FIG. 6, priority "1" is "job title information," priority "2" is "priority speaker," priority "3" is "mandatory participant," priority "4" is "speed of speech," priority "5" is "volume of voice," priority "6" is "performance evaluation," priority "7" is "volume of gestures and hand movements," and priority "8" is "optional participant." For example, first, the "job title information" of the speaker of the overlapping statement with priority "1" is referenced, and the statement of the speaker with the higher job title is given priority. If the job title information is the same, the "priority speaker" with priority "2" is referenced, and the statement of the speaker with the higher rank is given priority. If the ranks are the same, the "required participant" with priority "3" is referenced, and if only one of the speakers of the overlapping message is a required participant, the message of that required participant is given priority. If there are multiple required participants, the "speed of speech" with priority "4" is referenced. In this way, the priority of the messages of the speakers of the overlapping message is determined according to the priority. This priority determination is performed in step S140, which will be described later.

[0064] 7 to 9, the manager or moderator of the online meeting accesses the server 50 from the PC 20 and sets up notification of the priority of the determined speaker. Settings for notification of priority include, for example, "notification destination settings" and "notification method settings," as shown in FIG.

[0065] "Notification settings" is a setting for which participants in an online meeting will be notified. In the example shown in Figure 7, there are "Everyone," "Only the moderator and those who have made overlapping comments," and "Specified participants." "Everyone" is a setting for notifying all participants in an online meeting. "Only the moderator and those who have made overlapping comments" is a setting for notifying only the moderator of an online meeting and those who have made overlapping comments. "Specified participants" is a setting for notifying specified participants among the participants.

[0066] The "Notification Method Settings" setting controls how notifications are sent during an online meeting. In the example shown in FIG. 7, options include "List Display" and "Mark on Face Photo." As shown in FIG. 8, "List Display" is a setting that displays the names and priorities of participants designated as notification destinations in the "Notification Destination Settings" on the display unit 26 of the PC 20. Here, the names and priorities are preferably displayed overlaid on the online meeting document S displayed on the display unit 26. Since online meeting participants often focus on the document S, displaying the names and priorities overlaid on the document S in question makes it possible to prevent participants from overlooking the priority level, compared to displaying the names and priorities in other locations. As shown in FIG. 9, the "Mark on Face Photo" setting displays a number indicating the priority level, such as "1," on a pre-prepared participant's face photo G or a participant's face image captured by the camera 27 during the online meeting. Here, the face photo G and the priorities are preferably displayed overlaid on the online meeting document S displayed on the display unit 26. Displaying the names and priorities overlaid on the document S makes it possible to prevent participants from overlooking the priority level.

[0067] After the predetermined settings shown in Figures 5 to 7 are made, the process of the flowchart shown in Figure 4 is started. Note that the predetermined settings shown in Figures 5 to 7 are not limited to those made before the start of the online meeting, but may be made during the online meeting or may be able to be changed during the online meeting.

[0068] In step S100 shown in FIG. 4, the CPU 51 of the server 50 obtains voice data uttered by any of the participants participating in the online meeting. The voice data is sequentially obtained from the online meeting server 60. The voice data also includes information identifying the participant who made the speech. Then, it is determined whether or not there has been an overlapping speech. Here, whether or not there has been an overlapping speech is determined by determining the start and end times of one speaker's speech, and if another speaker's speech starts between the start and end times, it is determined that there has been an overlapping speech. The start and end times are calculated based on the time managed by the online meeting server 60. In other words, the start and end times of one speech are obtained, and if the start time of another speech falls between the two times, the two speeches are determined to be overlapping speeches. Note that the determination of whether there is an overlapping statement is not limited to the case where it is determined from the start and end times of the statements, but the timing when the server 50 starts acquiring voice data from the online meeting server 60 may be determined to be the start of the statement, and the timing when the server 50 finishes acquiring the voice data may be determined to be the end of the statement, and if acquisition of other voice data starts between these timings, the statements may be determined to be overlapping statements. Then, proceed to the next step S110.

[0069] In step S110, if it is determined that no overlapping messages have been made in step S100, the process proceeds to the next step S120.

[0070] In step S120, the CPU 51 of the server 50 determines whether the online meeting has ended. If it is determined that the online meeting has ended, the process ends. If it is not determined that the online meeting has ended, the process returns to step S100.

[0071] If the CPU 51 of the server 50 determines in step S110 that there are overlapping comments, the process proceeds to step S130, where it generates noise-removed voice data by removing noise from the voice data. An example of noise removal will now be described with reference to FIG. 10 . As shown in FIG. 10 , noise removal involves first frequency-converting the voice data. Next, for example, pop noise, which frequently occurs in online meetings, is removed from the frequency-converted voice data. Note that the noise to be removed is not limited to pop noise, and other noises may also be used. In this way, noise-removed voice data is generated. Then, the process proceeds to step S140. Here, noise removal is performed for determining the priority order in step S140, which will be described later. However, the present invention is not limited to this, and noise removal may be performed before step S110 to determine whether there are overlapping comments in the noise-removed voice data.

[0072] In step S140, the CPU 51 of the server 50 identifies the speaker from information identifying the participant who made the comment, which is included in the voice data. Then, the CPU 51 determines the priority of the speaker who made the overlapping comment according to the predetermined rules for determining the priority of overlapping comments shown in Figures 5 and 6. Then, the process proceeds to the next step, S150.

[0073] In step S150, the CPU 51 of the server 50 notifies the speaker of the priority determined in step S140, as shown in Fig. 8 or 9, in accordance with the settings shown in Fig. 7. Then, the process proceeds to the next step S120 described above.

[0074] The present invention is not limited to the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0075] In the above embodiment, the program is pre-stored (installed) in a ROM or a storage unit, but the present invention is not limited to this. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network N.

[0076] Furthermore, in the above-described embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0077] Furthermore, the operations of the processors in the above-described embodiments may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processors is not limited to the order described in the above-described embodiments, and may be changed as appropriate. [Explanation of symbols]

[0078] 10 Information Processing Systems 20 PC 50 servers 60 Online Meeting Server

Claims

1. a processor; The processor: Determine whether there is an overlapping statement made by another speaker during an online meeting; When the overlapping message is received, the priority of the person who made the overlapping message is determined in accordance with a predetermined rule; notifying the priority; The information processing system wherein the notification of the priority is executed for participants of the online meeting who have been preset as notification targets.

2. The processor: Acquire audio data of speech made by a speaker during the online meeting; generating noise-removed audio data by removing noise from the audio data; The information processing system according to claim 1 , wherein the priority is determined from the removed audio data.

3. The processor: determining the start and end times of the speech of said one speaker; 3. The information processing system according to claim 1, wherein it is determined that the overlapping speech has occurred when another speaker starts speaking between the start time and the end time.

4. The notification of the priority is made by using the materials displayed on the display unit in the online meeting.

2. The information processing system according to claim 1, wherein the image is displayed on the front side of the image.

5. 5. The information processing system according to claim 1, wherein the predetermined rule is a predetermined attribute of the speaker.

6. 5. The information processing system according to claim 1, wherein the predetermined rule is a feature of the speech of the speaker.

7. An information processing system according to any one of claims 1 to 4, wherein the predetermined rules are determined by weighting predetermined attributes of the speaker and characteristics of the speaker's statements.

8. The predetermined rule is a combination of a predetermined attribute of the speaker and a feature of the speaker's utterance, 5. The information processing system according to claim 1, wherein the priority is determined according to a weighting registered in advance.

9. 8. The information processing system according to claim 5, wherein the attribute is job title information of the speaker.

10. 8. The information processing system according to claim 5, wherein the attribute is whether the speaker is a required participant or an optional participant in the online meeting.

11. 8. The information processing system according to claim 6, wherein the characteristic is the speed of the overlapping comments.

12. On the computer, Determine whether there is an overlapping statement made by another speaker during an online meeting; When the overlapping message is received, the priority of the person who made the overlapping message is determined in accordance with a predetermined rule; notifying the priority; The information processing program causes the notification of the priority to be executed for participants of the online meeting who have been preset as notification targets.

Images