Information storage medium, session evaluation device, and session evaluation method

Through computer programs, the speaker switching time is speculated and determined, and the conversation status is evaluated in combination with dialogue information, the bias problem of the evaluation of communication status of multiple speakers in the prior art is solved, and a more accurate and objective communication status evaluation is achieved.

CN119068908APending Publication Date: 2024-12-03KK TOSHIBA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410224907.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-02-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to independently and objectively evaluate communication status from multiple speakers, and it is easy to cause inaccurate evaluation due to bias.

Method used

Through the computer program stored in the information storage medium, the vocal start and end time of each major speaker is speculated, the speaker switch time is determined, and the conversation status is evaluated based on the dialogue information before and after the switching, including the vocal interval repetition and the length of the soundless interval.

Benefits of technology

The objective and accurate evaluation of the communication status between multiple speakers is achieved, and the independence and fairness of the evaluation is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068908A_ABST
    Figure CN119068908A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to an information storage medium, a session evaluation device and a session evaluation method. [Problem] To appropriately evaluate the state of communication between a plurality of speakers. An information storage medium according to an embodiment stores a program that causes a computer to implement an estimation function, a specification function, and an evaluation function. The estimation function estimates the start time and the end time of the utterance by each main speaker associated with the session. A determination function determines a switching timing for the respective primary speakers. The evaluation function evaluates the session on the basis of the session information before and after the switching timing. The dialogue information includes at least one of a repetition section and a soundless section, and a sound generation section.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is based on Japanese Patent Application No. 2023-091036 (filing date: 06 / 01 / 2023) and claims the benefit of priority from the aforementioned application. The present application hereby refers to the aforementioned application and the entire contents of the aforementioned application are cited herein. Technical Field

[0002] Embodiments of the present invention relate to an information storage medium, a conversation evaluation device, and a method. Background Art

[0003] Generally speaking, in order to improve labor productivity, companies need to increase the participation of each business person in the company. Therefore, companies need to evaluate the state of communication (especially the soundness) between business people.

[0004] For example, there is a technique for evaluating the communication status between the first speaker and the second speaker using the length of a repetition interval of the utterance intervals of the two speakers in a conversation between the two speakers. In this technique, the time from the start of the utterance of the second speaker to the end of the utterance of the first speaker is regarded as a repetition interval, and the impression of the second speaker on the first speaker is evaluated as "average" or "poor" according to the length of the repetition interval.

[0005] However, in the above-mentioned technology, the impression of the first speaker observed by the second speaker is evaluated, so it is impossible to independently and objectively evaluate the impression of the first speaker from the perspective of the second speaker. For example, although the second speaker who interrupted the utterance of the first speaker is actually not good, the impression of the first speaker may be inappropriately evaluated as "bad". Therefore, it is required to appropriately evaluate the state of communication between multiple speakers. Summary of the invention

[0006] An object of the present invention is to appropriately evaluate the state of communication between a plurality of speakers.

[0007] The information storage medium involved in the implementation mode stores a program that enables a computer to implement an inference function, a determination function, and an evaluation function. The inference function estimates the start time and end time of the utterance made by each main speaker based on sound data related to a conversation including each utterance made by multiple speakers. The determination function determines the switching timing of each main speaker based on the inferred start time and end time. The evaluation function evaluates the state of the conversation based on dialogue information before and after the determined switching timing. The dialogue information includes at least one of the length of a repeated interval in which the utterance intervals of each main speaker are repeated and the length of a silent interval in which the utterance intervals of each main speaker are not repeated, as well as the length of the utterance interval of each main speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a block diagram showing a functional configuration example of a conversation evaluation device according to the present embodiment.

[0009] Figure 2 This is a block diagram showing a hardware configuration example of the conversation evaluation device according to the present embodiment.

[0010] Figure 3 This is a flowchart showing an overall operation example of the conversation evaluation device according to the present embodiment.

[0011] Figure 4 Detailed Description of the Invention This is a flowchart showing a method for estimating an utterance of a main speaker according to the present embodiment.

[0012] Figure 5 It is a diagram showing an example of conversation analysis according to the present embodiment.

[0013] Figure 6 This is a flowchart showing a method for evaluating the state of a session according to the present embodiment.

[0014] Figure 7 It is a diagram showing an example of session information according to the present embodiment.

[0015] Figure 8 It is a diagram showing a display example of the evaluation result according to the present embodiment.

[0016] Fig. 9 It is a diagram showing examples of the estimation accuracy of conversation analysis related to the conventional method and the suggested method.

[0017] (Explanation of symbols)

[0018] 1: Conversation evaluation device; 11: Processing circuit; 12: Storage circuit; 13: Input IF; 14: Output IF; 15: Communication IF; 51, 52, 53, 54: Stars; 81, 510, 520: Bars; 82: Boxes; 111: Acquisition unit; 112: Inference unit; 113: Determination unit; 114: Evaluation unit; 115: System control unit; 120: Detection model; 200: Conversation voice data; 300: Evaluation result; 500A, 700, 900: Table; 500B, 800: Graph. DETAILED DESCRIPTION

[0019] Hereinafter, an information storage medium, a conversation evaluation device, and a method according to an embodiment will be described with reference to the drawings. In the following embodiments, parts with the same reference numerals are assumed to perform the same operations, and duplicate descriptions are omitted as appropriate.

[0020] The following are definitions of the terms used in the present embodiment. (1) "utterance" refers to an utterance interval (sound interval) sandwiched between two silent intervals of more than 0.2 seconds. (2) "main speaker" refers to a speaker in a conversation. (3) "non-main speaker" refers to a listener to the main speaker. The utterance of a non-main speaker often includes an echo of the utterance of the main speaker or a repetition of a fragment. (4) "main speaker switching" refers to a switch between two main speakers. (5) "main speaker's turn" refers to the set of utterances made by the main speaker from the time the main speaker is switched to the main speaker until the main speaker who is next to the main speaker is switched to the main speaker. (6) "length of the utterance interval of the main speaker" refers to the total length of all utterance intervals in a main speaker's turn.

[0021] Figure 1 1 is a block diagram showing a functional configuration example of a conversation evaluation device 1 according to the present embodiment. The conversation evaluation device 1 is a device for evaluating the state of a conversation between a plurality of speakers. The conversation evaluation device 1 includes an acquisition unit 111 , an estimation unit 112 , a determination unit 113 , and an evaluation unit 114 .

[0022] The acquisition unit 111 acquires various data or information. For example, the acquisition unit 111 acquires conversation voice data 200 related to a conversation including individual utterances made by a plurality of speakers. The conversation voice data 200 is data that records changes in electrical signals related to conversation voice in a time series. The acquisition unit 111 sends the acquired conversation voice data 200 to the estimation unit 112.

[0023] The estimation unit 112 estimates various data or information. For example, the estimation unit 112 estimates the start time and end time of the utterance made by each main speaker based on the conversation voice data 200 sent from the acquisition unit 111. The estimation unit 112 sends the estimated start time and end time to the determination unit 113 and the evaluation unit 114.

[0024] The determination unit 113 determines various data or information. For example, the determination unit 113 determines the switching timing of each main speaker (hereinafter also referred to as “speaker switching”) based on the start time and end time sent from the estimation unit 112. The determination unit 113 sends the determined switching timing to the evaluation unit 114.

[0025] The evaluation unit 114 evaluates various data or information. For example, the evaluation unit 114 evaluates the state of the conversation based on the conversation information D before and after the switching timing sent from the determination unit 113. The conversation information D includes at least one of the length of the repeated intervals of the speech intervals of each main speaker and the length of the silent intervals of each main speaker that do not overlap, and the length of the speech interval of each main speaker. The conversation information D may also include the start time and the end time sent from the estimation unit 112. The evaluation unit 114 obtains the evaluation result 300 related to the state of the conversation from the detection model 120 by inputting the conversation information D into the detection model 120 trained in advance. The evaluation unit 114 outputs the obtained evaluation result 300.

[0026] Figure 2 1 is a block diagram showing an example of the hardware configuration of the conversation evaluation device 1 according to the present embodiment. For example, the conversation evaluation device 1 is a computer (for example, a personal computer, a tablet terminal, a smart phone). The conversation evaluation device 1 includes a processing circuit 11, a storage circuit 12, an input IF 13, an output IF 14, and a communication IF 15 as each structure. Each structure is connected to each other via a bus (BUS) as a common signal communication path so as to be able to communicate with each other.

[0027] The processing circuit 11 is a circuit that controls the overall operation of the conversation evaluation device 1. The processing circuit 11 includes at least one processor. The processor means a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an application-specific integrated circuit (ASIC: Application Specific Integrated Circuit), a programmable logic device (example: a simple programmable logic device (SPLD: Simple Programmable Logic Device), a complex programmable logic device (CPLD: Complex Programmable Logic Device), and a field programmable gate array (FPGA: Field Programmable Gate Array)). When the processor is a CPU, the CPU implements each function by reading and executing each program stored in the storage circuit 12. When the processor is an ASIC, each function is directly embedded in the ASIC as a logic circuit. The processor can be configured as a single circuit or a combination of multiple independent circuits. The processing circuit 11 implements an acquisition unit 111, an inference unit 112, a determination unit 113, an evaluation unit 114, and a system control unit 115. The processing circuit 11 is an example of a processing unit.

[0028] The system control unit 115 controls various operations performed by the processing circuit 11. For example, the system control unit 115 provides an operating system (OS) for the processing circuit 11 to realize each unit (acquisition unit 111, estimation unit 112, determination unit 113, evaluation unit 114).

[0029] The storage circuit 12 is a circuit that stores various data or information. The storage circuit 12 can be a storage medium that can be read by the processor (for example: a magnetic storage medium, an electromagnetic storage medium, an optical storage medium, a semiconductor memory), or a drive device that reads and writes data or information between the storage medium. The storage circuit 12 stores various programs that enable the processing circuit 11 to implement various parts (acquisition part 111, estimation part 112, determination part 113, evaluation part 114, system control part 115). The storage circuit 12 can also store the detection model 120. The storage circuit 12 is an example of a storage unit.

[0030] The input IF 13 is an interface for receiving various inputs from the user. The input IF 13 converts the received input into an electrical signal and sends the electrical signal to the processing circuit 11. The input IF 13 may also be a mouse, a keyboard, a button, a panel switch, a slider switch, a trackball, an operation panel, or a touch screen. The input IF 13 may also be provided outside the conversation evaluation device 1. The input IF 13 is an example of an input unit.

[0031] The output IF 14 is an interface for outputting various data or information to the user. The output IF 14 outputs various data or information based on the electrical signal sent from the processing circuit 11. The output IF 14 may also be a display device (for example, a monitor) or an audio device (for example, a speaker). The output IF 14 may also be provided outside the conversation evaluation device 1. The output IF 14 is an example of an output unit, a display unit, or an audio unit.

[0032] The communication IF 15 is an interface for communicating various data or information with an external device. The communication IF 15 is an example of a communication unit.

[0033] Figure 3 1 is a flowchart showing an overall operation example of the conversation evaluation device 1 according to the present embodiment. According to the operation example, the conversation evaluation device 1 analyzes the conversation voice data 200 and outputs the evaluation result 300. In particular, the conversation evaluation device 1 may start the operation example according to an instruction input by the user using the input IF 13. The conversation evaluation device 1 may also obtain the conversation voice data 200 from the communication IF 15 and present the evaluation result 300 to the user using the output IF 14.

[0034] (Step S1) First, the conversation evaluation device 1 obtains the conversation voice data 200 using the acquisition unit 111. The conversation voice data 200 is obtained using at least one sound pickup device. For example, the conversation voice data 200 is voice data related to a telephone conference, a web conference, or a video conference conducted by multiple speakers using their own sound pickup devices. The sound pickup device may also be a microphone of each speaker, or a built-in microphone mounted on a smartphone or a personal computer. Alternatively, the sound pickup device may also be an earphone microphone or a desktop microphone connected to the personal computer of each speaker.

[0035] (Step S2) Next, the conversation evaluation device 1 uses the acquisition unit 111 to determine whether the conversation voice data 200 acquired in step S1 has been separated from the sound source. Specifically, the acquisition unit 111 determines whether the conversation voice data 200 has been separated into the voice of each speaker (separated sound). If the conversation voice data 200 has been separated from the sound source (step S2-"Yes"), the process proceeds to step S4. If the conversation voice data 200 has not been separated from the sound source (step S2-"No"), the process proceeds to step S3.

[0036] First, assume that conversation audio data 200 is collected from sound collection devices of multiple speakers. In this case, each sound collection device collects audio data from a single speaker, so the conversation audio data 200 includes the voice of each speaker independently. Therefore, the conversation evaluation device 1 determines that the conversation audio data 200 is separated by sound source.

[0037] Second, assume that the conversation voice data 200 is collected from a single sound collection device. In this case, the sound collection device collects voices from multiple speakers at the same time, so the conversation voice data 200 includes a sound (mixed sound) in which the voices of the speakers are mixed. Therefore, the conversation evaluation device 1 determines that the conversation voice data 200 is not separated by sound source.

[0038] (Step S3) Next, the conversation evaluation device 1 performs sound source separation on the conversation voice data 200 determined not to be sound source separated in step S2 by the acquisition unit 111. Thus, the conversation voice data 200 independently includes the voice of each speaker. Sound source separation can be achieved by a known technique (see Non-Patent Document 1).

[0039] (Step S4) Next, the conversation evaluation device 1 uses the estimation unit 112 to estimate the utterance interval of each speaker based on the voice of each speaker independently included in the conversation voice data 200. Specifically, the estimation unit 112 estimates the start time and end time of the utterance made by each speaker, and the length of the utterance interval. The length of the utterance interval is the duration from the start time to the end time of the utterance. The estimation of the utterance interval can be achieved by a well-known technique (see non-patent document 2).

[0040] (Step S5) Next, the conversation evaluation device 1 uses the estimation unit 112 to estimate the utterance of each main speaker based on the utterance interval of each speaker estimated in step S4 (see Figure 4 ).

[0041] (Step S6) Next, the conversation evaluation device 1 uses the determination unit 113 to determine the speaker switching of each main speaker based on the utterance of each main speaker estimated in step S5. Specifically, the determination unit 113 determines whether the two main speakers are different from each other in the utterance intervals of two main speakers that are adjacent in time. If the two main speakers are different from each other, the determination unit 113 determines that there is a speaker switching between the utterance intervals of the two main speakers (see Figure 5 ).

[0042] (Step S7) Finally, the conversation evaluation device 1 uses the evaluation unit 114 to output the evaluation result 300 (see FIG. 1 ) regarding the state of the conversation in the conversation voice data 200 based on the conversation information D before and after the speaker switching determined in step S6. Figure 6 ). After step S7, the conversation evaluation device 1 ends a series of operations.

[0043] Figure 4 1 is a flowchart showing a method for estimating an utterance of a main speaker according to the present embodiment. According to the estimating method, the estimating unit 112 estimates whether an utterance is an utterance of a main speaker or an utterance of a non-main speaker for each utterance interval to be processed.

[0044] (Step S51) First, the estimation unit 112 selects a utterance interval to be processed from among all utterance intervals estimated in step S4. For example, the estimation unit 112 selects the utterance interval with the earliest start time of utterance from among all unprocessed utterance intervals. That is, each time step S51 is executed, one utterance interval to be processed is selected.

[0045] (Step S52) Next, the estimation unit 112 determines whether the length of the utterance interval selected in step S51 is greater than a threshold value. The threshold value is determined empirically (for example, 0.5 seconds). When the length of the utterance interval is greater than the threshold value (step S52-"Yes"), the process proceeds to step S53. When the length of the utterance interval is not greater than the threshold value (step S52-"No"), the process proceeds to step S54A. That is, the estimation unit 112 estimates that the utterance interval whose length is not greater than the threshold value is a shorter utterance such as an echo, and is a utterance of a non-main speaker.

[0046] (Step S53) Next, the estimation unit 112 determines whether the utterance interval determined in step S52 to be longer than the threshold is included in other utterance intervals. Specifically, the estimation unit 112 determines whether the start time and end time of the utterance interval are included in the period from the start time to the end time of other utterance intervals. When the utterance interval is included in other utterance intervals (step S53-"Yes"), the processing proceeds to step S54A. When the utterance interval is not included in other utterance intervals (step S53-"No"), the processing proceeds to step S54B.

[0047] (Step S54A) In this case, the estimation unit 112 estimates that the utterance section selected in step S51 is the utterance of the “non-main speaker.” After step S54A, the process proceeds to step S55.

[0048] (Step S54B) In this case, the estimation unit 112 estimates that the utterance section selected in step S51 is the utterance of the “main speaker.” After step S54B, the process proceeds to step S55.

[0049] (Step S55) Next, the estimation unit 112 determines whether all the utterance intervals estimated in step S4 have been processed from step S51 to step S54A or S54B. If all the utterance intervals have been processed (step S55-"Yes"), the process proceeds to step S6 (refer to Figure 3 ). When all utterance intervals have not been processed (step S55-No), the process returns to step S51.

[0050] In addition, the method of estimating the utterance of the main speaker is not limited to the above method. The estimation unit 112 may also use an external sound recognition system to identify the utterance content of each utterance. When the identified utterance content includes only echoes, the estimation unit 112 estimates that the utterance is the utterance of the "non-main speaker". On the contrary, the estimation unit 112 estimates that the utterance that is not estimated as the utterance of the non-main speaker is the utterance of the "main speaker".

[0051] Figure 5 It is a diagram showing an example of conversation analysis according to the present embodiment. Figure 5 Table 500A of (A) shows the analysis results of the conversation in the conversation voice data 200 . Figure 5 Graph 500B in (B) shows the analysis result of the conversation in table 500A. Table 500A or graph 500B may be displayed on output IF 14 as a display device.

[0052] Table 500A shows a conversation between three speakers (A, B, C). Table 500A shows utterances from each speaker in the row direction and shows items related to each utterance in the column direction. The items are "main speaker" in the first column, "speaker switching" in the second column, "start time" in the third column, "end time" in the fourth column, "speaker" in the fifth column, and "utterance content" in the sixth column.

[0053] For column 1 “Main speaker”, indicate by checking the box that you Figure 3 Step S5 (refer to Figure 4 ) is the utterance of the main speaker inferred from the table 500A. Specifically, among the 11 utterances included in the table 500A, the 7 utterances, the 1st, 4th to 7th, 9th and 11th from the top, are inferred to be the utterances of the "main speaker". On the contrary, the 4 utterances, the 2nd to 3rd, 8th and 10th from the top, are inferred to be the utterances of the "non-main speaker". That is, the utterances including the echo ("yes") or the repetition of the fragment ("chocolate") as the utterance content are inferred to be the utterances of the "non-main speaker".

[0054] Regarding the second column "Speaker Switch", the stars 51, 52, 53 and 54 indicate that Figure 3 Specifically, among the 11 utterances included in table 500A, four utterances, the fifth, sixth, ninth, and eleventh utterances from the top, are given stars 51, 52, 53, and 54. Stars 51, 52, 53, and 54 are given to the utterances after the speaker switching.

[0055] Graph 500B shows three speakers (A, B, C) on the vertical axis and the utterance interval of each speaker on the horizontal axis. The oblique line bar 510 shows the utterance of the "main speaker" in each utterance interval, and the white bar 520 shows the utterance of the "non-main speaker" in each utterance interval. Each of the bars 510 and 520 corresponds to each of the 11 utterances included in Table 500A. The four bars 510 with stars 51, 52, 53, and 54 in Graph 500B correspond to the four utterances with stars 51, 52, 53, and 54 in Table 500A.

[0056] Figure 6 2 is a flowchart showing a method for evaluating the state of a conversation according to the present embodiment. According to the present evaluation method, the evaluation unit 114 outputs an evaluation result 300 based on the dialogue information D before and after the speaker switches.

[0057] (Step S71) First, the evaluation unit 114 extracts the dialogue information D before and after the speaker switch for each speaker switch identified in step S6.

[0058] For example, focus on Figure 5 The first speaker switch with star 51 added in . As can be understood from Table 500A and Figure 500B, this speaker switch is a speaker switch from main speaker A to main speaker B. The length of the utterance interval in the turn of main speaker A before the speaker switch is the length of the utterance interval of the first and fourth utterances from the top in Table 500A. This length is calculated as (6.976-5.672)+(8.568-7.408)=2.464 (seconds). On the other hand, the length of the utterance interval in the turn of main speaker B after the speaker switch is the length of the utterance interval of the fifth utterance from the top in Table 500A. This length is calculated as (9.576-8.728)=0.848 (seconds).

[0059] Furthermore, in the case of the first speaker switching with the star 51 attached, the speech interval of the main speaker A and the speech interval of the main speaker B do not overlap with each other. Specifically, according to the end time "8.568" of the 4th speech interval from the top and the start time "8.728" of the 5th speech interval from the top in Table 500A, the speech intervals of the two do not overlap with each other. Therefore, the presence or absence of overlapping intervals is "no", and the length of the overlapping interval is "0.000" (seconds). On the contrary, there is a silent interval between the speech intervals of the two. Therefore, the presence or absence of silent intervals is "yes", and the length of the silent interval is calculated as (8.728-8.568)=0.160 (seconds).

[0060] Next, focus on Figure 5 The second speaker switch with star 52 added in the table 500A. As can be understood from Table 500A and Figure 500B, this speaker switch is a speaker switch from main speaker B to main speaker C. The length of the utterance interval in the turn of main speaker B before the speaker switch is the length of the utterance interval of the fifth utterance from the top in Table 500A. This length is calculated as (9.576-8.728)=0.848 (seconds). On the other hand, the length of the utterance interval in the turn of main speaker C after the speaker switch is the length of the utterance interval of the second utterance, which is the sixth and seventh utterances from the top in Table 500A. This length is calculated as (9.757-8.800)+(11.829-10.821)=1.965 (seconds).

[0061] Furthermore, in the case of the second speaker switching with the star 52 attached, the speech interval of the main speaker B and the speech interval of the main speaker C overlap with each other. Specifically, according to the end time "9.576" of the fifth speech interval from the top and the start time "8.800" of the sixth speech interval from the top in Table 500A, the speech intervals of the two overlap with each other. Therefore, the presence or absence of the overlapping interval is "yes", and the length of the overlapping interval is (9.576-8.800)=0.776 (seconds). On the contrary, there is no silent interval between the speech intervals of the two. Therefore, the presence or absence of the silent interval is "no", and the length of the silent interval is "0.000" (seconds).

[0062] Similarly, the evaluation unit 114 extracts the dialogue information D (see Figure 7 ).

[0063] (Step S72) Next, the evaluation unit 114 applies the trained detection model 120 to the dialogue information D of each speaker switch extracted in step S71. Specifically, the evaluation unit 114 detects the predetermined state or probability of each speaker switch as the evaluation result 300 by inputting the dialogue information D of each speaker switch into the trained detection model 120.

[0064] The detection model 120 is trained by using the dialogue information D when the speaker switches as input data and the labels indicating the states of the conversation before and after the speaker switches as training data of the correct answer data. For example, the detection model 120 is trained by using the dialogue information D as input data and the probability of a predetermined event occurring when the dialogue information D is provided as training data of the correct answer data.

[0065] First, the detection model 120 is trained by pairing data of the conversation information D and a label indicating whether a strong speaker switch has been made. In this case, the trained detection model 120 can detect the "probability of a strong speaker switch". Second, the detection model 120 is trained by pairing data of the conversation information D and a label indicating the name of the main speaker who made a strong utterance. In this case, the trained detection model 120 can detect the "probability of each main speaker making a strong utterance". Third, the detection model 120 is trained by pairing data of the conversation information D and a label indicating a value related to whether the conversation is active. In this case, the trained detection model 120 can detect the "activity of the conversation".

[0066] For example, focus on Figure 5 In the figure, four speaker switches are added with stars 51, 52, 53 and 54. Here, the dialogue information D under the Nth speaker switch is assumed to be X n, the probability of a predetermined event occurring when the Nth speaker switches is assumed to be Y n (n: a natural number from 1 to N). Furthermore, it is assumed that the dialogue information D includes (1) the length of the speech interval of the main speaker before the speaker is switched, (2) the length of the speech interval of the main speaker after the speaker is switched, (3) the length of the silent interval, and (4) the length of the repeated interval. In this case, the dialogue information X1 under the first speaker switch is expressed as X1 = {2.464, 0.848, 0.160, 0.000}. Similarly, the dialogue information X2, X3, and X4 under the second, third, and fourth speaker switches are expressed as X2 = {0.848, 1.965, 0.000, 0.776}, X3 = {1.965, 1.269, 0.851, 0.000}, and X4 = {1.269, 2.259, 1.176, 0.000}.

[0067] The evaluation unit 114 inputs the dialogue information X1 to the trained detection model 120, and the trained detection model 120 outputs the probability Y1 that the predetermined event occurs at the first speaker switching. Similarly, the evaluation unit 114 inputs the dialogue information X2, X3, and X4 to the trained detection model 120, and the trained detection model 120 outputs the probabilities Y2, Y3, and Y4 that the predetermined event occurs at the second, third, and fourth speaker switching.

[0068] In addition, the detection model 120 can also be a machine learning model (for example, a regression model, a support vector machine, a decision tree, a neural network). n Output the probability Y of the expected event occurring n It is also possible to output the probability of a predetermined event occurring for each of a plurality of consecutive dialogue messages.

[0069] Furthermore, the input data input to the detection model 120 may include feature quantities other than the dialogue information D. Specifically, the input data may include acoustic feature quantities (for example, mean, variance) related to the interval or strength of the voice of the main speaker before and after the speaker switch. Alternatively, the input data may include text information indicating the utterance content of each main speaker obtained by voice recognition.

[0070] (Step S73) Finally, the evaluation unit 114 calculates the value of the session information X for each session information X in step S72. n (Speaker switching) Predetermined event (probability Y n ), generates the evaluation result 300. For example, the evaluation unit 114 generates the evaluation result 300 in the probability Y n If the value is above the threshold, the dialogue information X nThe evaluation unit 114 generates warning information by combining the information for each session information X n The evaluation result 300 is outputted by generating a warning message or the like. For example, the evaluation result 300 is displayed on the output IF 14 (see Figure 8 ).

[0071] Figure 7 It is a diagram showing an example of the session information D according to the present embodiment. Figure 7 Table 700 shows the Figure 5 The conversation information D is extracted for each of the four speakers switching in the table 500A and the graph 500B.

[0072] Table 700 shows speaker switching with stars 51, 52, 53, and 54 in the row direction, and shows items related to speaker switching in the column direction. The items are column 1 "speaker switching", column 2 "length of the main speaker's utterance interval before speaker switching", column 3 "length of the main speaker's utterance interval after speaker switching", column 4 "presence of silent interval", column 5 "length of silent interval", column 6 "presence of repeated interval", and column 7 "length of repeated interval".

[0073] Figure 8 3 is a diagram showing an example of display of the evaluation result 300 according to the present embodiment. The conversation evaluation device 1 may also be configured to perform a check after the end of the telephone conference, web conference, or video conference and after the end of the audio recording of the conversation. Figure 3 Alternatively, the conversation evaluation device 1 may also perform the operation by acquiring the conversation voice data 200 in real time during a telephone conference, a web conference, or a video conference. Figure 3 The evaluation result 300 is output in real time (real-time action).

[0074] (Example of offline operation) For example, suppose that three speakers (A, B, C) are having a meeting in a conference room. Furthermore, suppose that during the meeting, speaker B starts speaking in a way that interrupts speaker A's speech and then speaker B unilaterally speaks for a long time. After the meeting, the conversation evaluation device 1 performs an off-line operation based on the conversation voice data 200 collected by the sound pickup device installed in the conference room. Figure 3 action.

[0075] The graph 800 shows three speakers (A, B, C) on the vertical axis and the utterance interval of each speaker on the horizontal axis. The diagonal bars 81 show the utterance intervals. In particular, around the time "05m00s", the speaker switches from speaker A to speaker B. When a strong speaker switch is detected based on the dialogue information D before and after the speaker switch, a warning message is displayed in association with the speaker switch. For example, the warning message is expressed by a box 82.

[0076] (Real-time Operation Example 1) Similarly, assume that three speakers (A, B, C) are having a meeting in a conference room. The conversation evaluation device 1 performs a conversation evaluation based on the conversation voice data 200 collected in real time by a sound collection device installed in the conference room. Figure 3 When the conversation evaluation device 1 detects that the number of times that the strong speaker switches exceeds the threshold value in the meeting, it outputs a warning message indicating that "there is a possibility that a strong speaker switches in the meeting". For example, the conversation evaluation device 1 sends an email including the warning message to the terminal of the boss or the human resources department who manages the three speakers (A, B, and C).

[0077] (Example 2 of Real-time Operation) For example, suppose that three speakers (A, B, C) are holding an online conference in the presence of a moderator F. The conversation evaluation device 1 performs an online conference based on the conversation audio data 200 collected in real time during the conference. Figure 3 The conversation evaluation device 1 sends the evaluation result 300 indicating the activity of the meeting to the terminal of the host F in real time. Furthermore, when the activity of the meeting is below the threshold value within a predetermined time or number of times, the conversation evaluation device 1 outputs a warning message indicating that the meeting is not active. For example, the conversation evaluation device 1 sends an email including the warning message to the terminal of the host F.

[0078] Fig. 9 This is a diagram showing an example of the estimation accuracy of conversation analysis related to the previous method and the proposed method. Table 900 shows the estimation accuracy of the activity of the conversation of three people using two detection models 120 trained by different training methods. As the detection model 120, XGboost (eXtreme Gradient Boosting), which is a method of decision tree, is used.

[0079] In order to compare the conventional method with the proposed method, the audio data of 10 small talk sessions with 3 speakers were prepared as a data set. The audio data included 6280 speaker switchings of the main speaker, and a label indicating "active" or "inactive" was manually added to each speaker switching.

[0080] In the conventional method, for each speaker switch, (1) the length of the repetition interval is added as the dialogue information D. In the proposed method, for each speaker switch, (1) the length of the repetition interval, (2) the length of the silent interval, (3) the length of the speech interval of the main speaker before the speaker switch, and (4) the length of the speech interval of the main speaker after the speaker switch are added as the dialogue information D. As a result, paired data including the dialogue information D and the label are generated for each of the 6280 speaker switches of the main speaker.

[0081] Next, 80% of all paired data is divided into training data, and the remaining 20% ​​is divided into evaluation data. The detection model 120 is trained using the training data. The trained detection model 120 infers the label corresponding to each conversation information D contained in the evaluation data as "active" or "inactive". By comparing the inferred label with the correct label, the inference accuracy of the label inferred by the trained detection model 120 is calculated. In the evaluation scale of the inference accuracy of the label, the F-measure is used. The closer the F-value is to "1.0", the higher the inference accuracy of the label.

[0082] According to Table 900, under the detection model 120 trained by the previous method, the F value of the label "active" is "0.46", and the F value of the label "inactive" is "0.76". On the other hand, under the detection model 120 trained by the suggested method, the F value of the label "active" is "0.66", and the F value of the label "inactive" is "0.88". That is, it can be considered that the estimation accuracy of the labels "active" and "inactive" is higher under the detection model 120 trained by the suggested method than that of the previous method.

[0083] According to the present embodiment described above, the conversation evaluation device 1 can appropriately evaluate the state of communication between a plurality of speakers. In particular, the conversation evaluation device 1 can objectively evaluate the state of conversation between a plurality of speakers.

[0084] First, when two utterance intervals are repeated when the main speaker switches, the conversation evaluation device 1 uses the length of the utterance interval of the first speaker before the repetition and the length of the utterance interval of the second speaker after the repetition. Thus, the conversation evaluation device 1 can evaluate which of the first speaker and the second speaker is uttering unilaterally. In other words, the conversation evaluation device 1 can evaluate which of the first speaker and the second speaker is uttering more strongly.

[0085] Second, the conversation evaluation device 1 can evaluate the activity of the conversation between the two speakers by using the length of the utterance interval of the two speakers and the length of the silent interval when the speakers switch between the two speakers. For example, when the silent interval is short and the length of the utterance interval of the two speakers is long, the conversation evaluation device 1 can evaluate that the conversation between the two speakers is active. On the contrary, when the silent interval is long and the length of the utterance interval of the two speakers is short, the conversation evaluation device 1 can evaluate that the conversation between the two speakers is inactive.

[0086] Although several embodiments of the present invention have been described, these embodiments are merely illustrative and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their variations are included in the scope and gist of the invention, and are also included in the invention described in the claims and their equivalents.

Claims

1. An information storage medium storing a program that enables a computer to: An estimation function for estimating a start time and an end time of an utterance made by each main speaker based on audio data related to a conversation including individual utterances made by a plurality of speakers; A determination function, for determining the switching timing of each main speaker according to the estimated start time and end time; and an evaluation function, evaluating the state of the session based on the conversation information before and after the determined switching timing, in, The dialogue information includes at least one of the length of a repeated section where the utterance sections of the main speakers overlap and the length of a silent section where the utterance sections of the main speakers do not overlap, and the length of the utterance section of the main speakers.

2. The information storage medium according to claim 1, wherein: The estimation function estimates, among the utterances made by the plurality of speakers, an utterance whose utterance section has a length equal to or longer than a threshold and whose utterance section is not included in other utterance sections as an utterance made by the main speaker.

3. The information storage medium according to claim 1, wherein: The evaluation function evaluates the state of the conversation using a trained model trained by using the dialog information as input data and a label indicating the state of the conversation before and after the switching timing as training data for correct answer data.

4. The information storage medium according to claim 3, wherein: The label is the name of the primary speaker making the dominant utterance, The evaluation function outputs, as an evaluation result regarding the state of the conversation, a probability that each of the main speakers will make a strong utterance.

5. The information storage medium according to claim 3, wherein: The tag is a value related to whether the session is active. The evaluation function outputs the activity of the session as an evaluation result related to the state of the session.

6. The information storage medium according to claim 1, wherein: When the evaluation function detects a strong speaker switch based on the dialogue information, the evaluation function outputs warning information in association with the switch timing at which the strong speaker switch is detected.

7. The information storage medium according to claim 1, wherein: The evaluation function outputs warning information when the switching of the dominant speaker in the conversation is detected a number of times exceeding a threshold value based on the dialogue information.

8. The information storage medium according to claim 1, wherein: The evaluation function outputs warning information when detecting, based on the conversation information, that the session is inactive a number of times greater than a threshold value in the session.

9. A conversation evaluation device comprising: an estimating unit that estimates a start time and an end time of an utterance made by each main speaker based on audio data related to a conversation including utterances made by a plurality of speakers; a determination unit that determines the switching timing of each main speaker according to the estimated start time and end time; and an evaluation unit that evaluates the state of the session based on the conversation information before and after the determined switching timing, The dialogue information includes at least one of the length of a repeated section where the utterance sections of the main speakers overlap and the length of a silent section where the utterance sections of the main speakers do not overlap, and the length of the utterance section of the main speakers.

10. A conversation evaluation method, comprising: inferring a start time and an end time of an utterance made by each main speaker based on audio data related to a conversation including utterances made by a plurality of speakers; Determining the switching timing of each main speaker according to the estimated start time and end time; and evaluating the state of the session based on the conversation information before and after the determined switching timing, The dialogue information includes at least one of the length of a repeated section where the utterance sections of the main speakers overlap and the length of a silent section where the utterance sections of the main speakers do not overlap, and the length of the utterance section of the main speakers.