Description assistance device, description assistance method, and program product

Abstract

The explanation assistance device (2) displays information related to the explanation items (C1 to C10) of the inspection object in the speech of the user (4). The explanation assistance device includes an acquisition unit (26), a control unit (20), and a display unit (23). The acquisition unit acquires input information representing a speech sentence based on the speech. The control unit generates information representing the inspection result of the explanation items related to the speech sentence. The display unit displays the information generated by the control unit. The display unit displays an inspection list (50) indicating whether the explanation items are explained in the speech sentences shown in the input information sequentially acquired by the acquisition unit. The display unit displays display information (55, 70) including the speech sentences based on the likelihood of each speech sentence of the inspection result of the explanation items in the specified inspection list.

Description

Technical Field

[0001] The present disclosure relates to an explanation assisting device and an explanation assisting method. Background Art

[0002] Patent document 1 discloses an explanation assistance system for assisting explanations using a computer terminal. In the explanation assistance system of patent document 1, when a control unit of a computer terminal detects a keyword included in a check list in a sound recognition result of a collected sound, a message including the detected keyword is displayed on a display. Furthermore, the control unit extracts the sound recognition result of the keyword during the collection period and sends the explanation status to a background terminal.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-25609 Summary of the invention

[0006] -Problems to be solved by the invention-

[0007] An object of the present disclosure is to provide an explanation support device and an explanation support method that can easily check the support of a user's explanation of an explanation matter through information processing.

[0008] -Methods for solving the problem-

[0009] The explanation assistance device involved in one mode of the present disclosure is a device that displays information related to the explanation items of the inspection object in the speech of the user. The explanation assistance device includes an acquisition unit, a control unit, and a display unit. The acquisition unit acquires input information representing a speech sentence based on the speech. The control unit generates information representing the inspection result of the explanation items related to the speech sentence. The display unit displays the information generated by the control unit. The display unit displays a check list indicating whether the explanation items are explained in the speech sentences shown in the input information sequentially acquired by the acquisition unit. The display unit displays the display information including the speech sentence based on the likelihood of each speech sentence that specifies the inspection result of the explanation items in the inspection list.

[0010] The explanation assisting method involved in one aspect of the present disclosure is a method for displaying information related to the explanation items of the inspection object in the speech of the user. The method includes: a step in which an acquisition unit acquires input information representing a speech sentence based on the speech; a step in which a control unit generates information representing the inspection result of the explanation items related to the speech sentence; and a step in which a display unit displays the information generated by the control unit. The display unit displays a check list indicating whether the explanation items in the speech sentences shown in the input information sequentially acquired by the acquisition unit are explained. The display unit displays the display information including the speech sentence based on the likelihood of each speech sentence that specifies the inspection result of the explanation items in the check list.

[0011] -Effects of the Invention-

[0012] According to the explanation support device and the explanation support method of the present disclosure, it is possible to easily support the user in explaining the explanation matters by checking the user through information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a diagram showing an overview of the explanation support system according to the first embodiment of the present disclosure.

[0014] Figure 2 This is a block diagram illustrating an example of the configuration of an explanation supporting device in the explanation supporting system.

[0015] Figure 3 This is a block diagram illustrating an example of the structure of a language processing server in the description assistance system.

[0016] Figure 4 This is a diagram showing a display example in the explanation support device according to the first embodiment.

[0017] Figure 5 Yes means Figure 4 The following are diagrams showing display examples in the explanation support device.

[0018] Figure 6 This is a flowchart for explaining the detection operation of the explanation supporting system according to the first embodiment.

[0019] Figure 7 This is a flowchart for explaining the inspection display processing by the explanation supporting device.

[0020] Figure 8 This is a diagram for explaining the history data in the explanation support device.

[0021] Fig. 9 It is a diagram showing a display example of a sales conversation list in the explanation support device.

[0022] Fig.10This is a diagram showing a display example of a speech history screen in the explanation support device.

[0023] Fig.11 It is a diagram showing a display example of the inspection history screen in the explanation support device.

[0024] Fig.12 This is a flowchart for explaining the processing based on the detection history of the explanation supporting device.

[0025] Fig.13 This is a flowchart for explaining the detection operation of the explanation support system involved in the second embodiment.

[0026] Fig.14 This is a diagram showing a display example in the explanation support device according to the second embodiment. DETAILED DESCRIPTION

[0027] The following describes the embodiments in detail with reference to the accompanying drawings as appropriate. Unnecessary detailed descriptions may be omitted. For example, detailed descriptions of known matters and repeated descriptions of substantially the same structures may be omitted. This is to avoid the following description from becoming unnecessarily lengthy and to make it easy for those skilled in the art to understand.

[0028] In addition, the applicant provides the drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and does not intend to limit the subject matter described in the claims by these.

[0029] (Implementation Method 1)

[0030] Hereinafter, Embodiment 1 of the present disclosure will be described with reference to the drawings.

[0031] 1. Structure

[0032] 1-1. System Overview

[0033] use Figure 1 The explanation support system according to the first embodiment will be described. Figure 1 It is a diagram showing an overview of the explanation support system 1 according to the present embodiment.

[0034] like Figure 1 As shown, the system 1 includes an explanation support device 2, a language processing server 3, and a voice recognition server 11. The system 1 automatically detects, for example, whether a user 4 who performs customer reception business has properly said important matters (i.e., explanation matters) in a business conversation with a customer 40, in the explanation of a product, or in the explanation when concluding a contract, and visualizes the inspection results of the business conversation.

[0035] like Figure 1As shown, the explanation support device 2 according to the present embodiment communicates with various customer terminals 41 held by customers 40 of the user 4, or communicates with various servers 3 and 11 via a communication network 10 such as a public telephone network and the Internet. The present system 1 can be applied to information support when a user 4 such as an operator provides various explanations to a customer 40, for example, in a call center or a remote customer service system.

[0036] Next, the configurations of the explanation support device 2 and the various servers 3 and 11 in the present system 1 will be described.

[0037] 1-2. Description of the structure of the auxiliary device

[0038] Reference Figure 2 The structure of the explanation support device 2 in the present system 1 will be described. Figure 2 This is a block diagram for explaining an example of the structure of the support device 2.

[0039] The explanation support device 2 includes, for example, an information terminal such as a personal computer, a tablet terminal, or a smartphone. Figure 2 The explanation support device 2 shown in the example includes a control unit 20, a storage unit 21, an operation unit 22, a display unit 23, a device interface 24, and a network interface 25. Hereinafter, the interface is abbreviated as “I / F”. In addition, the explanation support device 2 includes a microphone 26 and a speaker 27, for example.

[0040] The control unit 20 includes, for example, a CPU or MPU that cooperates with software to realize the specified functions, and controls the overall operation of the explanation auxiliary device 2. The control unit 20 reads the data and programs stored in the storage unit 21 and performs various calculations to realize various functions. For example, the control unit 20 executes a program including a command group for realizing various processes of the explanation auxiliary device 2 in the present system 1. The above-mentioned program is, for example, an application program, which can be provided from the communication network 10 or the like, or can be stored in a portable recording medium.

[0041] In addition, the control unit 20 may also be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to implement a specified function. The control unit 20 may also include various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcomputer, DSP, FPGA, and ASIC.

[0042] The storage unit 21 is a storage medium that stores programs and data required to realize the functions of the explanation support device 2. Figure 2 As shown, the storage unit 21 includes a saving unit 21 a and a temporary storage unit 21 b.

[0043] The storage unit 21a stores parameters, data, and control programs for realizing predetermined functions. The storage unit 21a includes, for example, a HDD or an SSD. For example, the storage unit 21a stores the above-mentioned programs and data indicating explanatory matters to be inspected by the system 1.

[0044] Temporary storage unit 21b includes RAM such as DRAM or SRAM, and temporarily stores (ie, holds) data. For example, temporary storage unit 21b may function as a work area of ​​control unit 20 or may be a storage area in an internal memory of control unit 20.

[0045] The operation unit 22 is a user interface device operated by a user. The operation unit 22 includes, for example, a keyboard, a mouse, a touch pad, a touch panel, buttons, switches, and a combination of these. The operation unit 22 is an example of an acquisition unit that acquires various information input by a user's operation.

[0046] The display unit 23 includes, for example, a liquid crystal display or an organic EL display. The display unit 23 displays, for example, information indicating the inspection results based on the system 1. In addition, the display unit 23 may also display various information such as various icons for operating the operation unit 22 and information input from the operation unit 22.

[0047] The device I / F 24 is a circuit for connecting the explanation support device 2 to an external device. The device I / F 24 is an example of a communication unit that communicates according to a predetermined communication standard. The predetermined standards include USB, HDMI (registered trademark), IEEE1395, WiFi, Bluetooth (registered trademark), etc. The device I / F 24 may also constitute an acquisition unit that receives various information from an external device in the explanation support device 2.

[0048] The network I / F 25 is a circuit for connecting the explanation support device 2 to the communication network 10 via a wireless or wired communication line. The network I / F 25 is an example of a communication unit that performs communication based on a predetermined communication standard. The predetermined communication standards include IEEE802.3, IEEE802.11a / 11b / 11g / 11ac, 3G or 4G for portable communication, and the like. The network I / F 25 may also constitute an acquisition unit that receives various information via the communication network 10 in the explanation support device 2.

[0049] The microphone 26 is an input device for collecting sound and acquiring sound data of the collected sound. The microphone 26 is an example of an acquisition unit in this embodiment. The microphone 26 and the speaker 27 are, for example, Figure 1 As shown in the example, the earphones used by the user 4 are configured.

[0050] The speaker 27 is an output device that outputs audio data, and is an example of an output unit in the present embodiment. The microphone 26 and the speaker 27 may be provided externally to the information terminal constituting the explanation support device 2, or may be built in the information terminal.

[0051] The above-mentioned structure of the explanation auxiliary device 2 is an example, and the structure of the explanation auxiliary device 2 is not limited to this. In addition, the explanation auxiliary device 2 may also include various computers that are not limited to information terminals. In addition, the acquisition unit in the explanation auxiliary device 2 may also be implemented by cooperating with various software in the control unit 20, etc. The acquisition unit in the explanation auxiliary device 2 may also acquire various information by reading various information stored in various storage media (such as the storage unit 21a) into the working area (such as the temporary storage unit 21b) of the control unit 20.

[0052] 1-3. Server structure

[0053] As an example of the hardware configuration of the various servers 3 and 11 in the present system 1, refer to Figure 3 The structure of the language processing server 3 will be described. Figure 3 This is a block diagram illustrating the structure of the language processing server 3 in the present system 1 .

[0054] Figure 3 The language processing server 3 illustrated in the figure includes a calculation processing unit 30, a storage unit 31, and a communication unit 32. The language processing server 3 includes one or more computers.

[0055] The calculation processing unit 30 includes, for example, a CPU and a GPU that realize predetermined functions in cooperation with software, and controls the operation of the language processing server 3. The calculation processing unit 30 reads data and programs stored in the storage unit 31 and performs various calculation processes to realize various functions.

[0056] For example, the operation processing unit 30 executes the program of the learning model 35 as a program for executing natural language processing for detecting the explanatory items described later. The learning model 35 includes various neural networks such as a forward propagation type neural language model, and has an input layer, one or more intermediate layers, and an output layer. For example, the output layer of the learning model 35 includes multiple nodes corresponding to multiple explanatory items, and outputs the likelihood of each explanatory item.

[0057] Furthermore, the operation processing unit 30 may also perform word embedding for generating an input vector to be input to the learning model 35, for example, by word2vec or the like. In addition, the learning model 35 may also include word embedding. In addition, the operation processing unit 30 may also execute a program for performing machine learning of the learning model 35 or the like. The above-mentioned various programs may be provided from the communication network 10 or the like, or may be stored in a portable recording medium.

[0058] In addition, the operation processing unit 30 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to implement a predetermined function. The operation processing unit 30 may also include various semiconductor integrated circuits such as a CPU, a GPU, a TPU, an MPU, a microcomputer, a DSP, an FPGA, and an ASIC.

[0059] The storage unit 31 is a storage medium for storing programs and data required to realize the functions of the language processing server 3, and includes, for example, an HDD or an SSD. In addition, the storage unit 31 includes, for example, a DRAM or an SRAM, and can also function as a working area of ​​the operation processing unit 30. The storage unit 31 stores, for example, various dictionaries related to terms and expressions in natural language processing based on the learning model 35, and various parameter groups and programs that define the learning model 35. The parameter group includes, for example, various weighting parameters of a neural network. The storage unit 31 can also store training data and programs for machine learning of the learning model 35.

[0060] The communication unit 32 is an I / F circuit for communicating according to a predetermined communication standard, and connects the communication network 10 or an external device to the language processing server 3. The predetermined communication standards include IEEE802.3, IEEE802.11a / 11b / 11g / 11ac, USB, HDMI, IEEE1395, WiFi, Bluetooth, etc.

[0061] The voice recognition server 11 has the same configuration as the above-mentioned language processing server 3, but includes a voice recognition model that realizes the voice recognition function instead of the learning model 35. The voice recognition model can be configured in various ways, and may include various neural networks that perform machine learning, for example.

[0062] The various servers 3 and 11 in the present system 1 are not limited to the above-mentioned structure, and may have various structures. The present system 1 may also be implemented in cloud computing. In addition, hardware resources for implementing the functions of the various servers 3 and 11 may be shared. In addition, the functions of one or both of the various servers 3 and 11 may be installed in the explanation auxiliary device 2.

[0063] 2. Action

[0064] Next, the operation of the explanation support system 1 and the explanation support device 2 configured as above will be described.

[0065] 2-1. Overview of the operation

[0066] use Figure 1 to Figure 5 An overview of the operations of the explanation supporting system 1 and the explanation supporting device 2 according to the present embodiment will be described.

[0067] This system 1 is for example Figure 1 As shown, the explanation support device 2 performs a detection operation to check the content of the speech of the user 4 when performing voice communication for the conversation between the user 4 and the customer 40. The explanation support device 2 displays information visualized to the user 4 regarding the detection result of the system 1. Figure 4 2 is an example of a display of the assistance device 2.

[0068] Figure 4 The display example of FIG. 4 is an example of a real-time display of the detection action for the conversation of the user 4. In this display example, the display unit 23 of the auxiliary device 2 displays various operation buttons 5, a check list 50, and a speech list 55. The operation buttons 5 include, for example, a voice recognition button that receives an operation by clicking the operation unit 22, a reset button, a setting button, and an application exit button.

[0069] The check list 50 includes a plurality of explanation items C1 to C10 and a check box 51 corresponding to each of the explanation items C1 to C10. The explanation items C1 to C10 are items to be checked in the speech of the user 4 and are preset. The number of the explanation items C1 to C10 is not particularly limited and can be set appropriately. Hereinafter, the explanation items C1 to C10 may be collectively referred to as "explanation items C". The check box 51 has a selected (ON) state with a check mark 52 and a cleared (OFF) state without the check mark 52. The check list 50 indicates whether the explanation item C corresponding to each check box 51 is explained according to whether the check box 51 is selected / cleared.

[0070] The speech list 55 displays information related to speech sentences from the latest voice recognition result to the specified number of minutes in the past in order, for example. The speech list 55 includes a number column 56, a speech sentence column 57, and a remark column 58. The number column 56 indicates the order of speech recognized by voice in the present system 1. The speech sentence column 57 indicates the speech sentence of the voice recognition result. The remark column 58 indicates the precautions corresponding to the speech sentence, etc. The speech list 55 is an example of display information in this embodiment.

[0071] exist Figure 4, a display example is shown in which the user 4 says "You can join the ABC card". At this time, the system 1 performs voice recognition on the speech sentence 53 of the above content, and detects the explanation item C1 of the speech sentence 53 explaining the "card membership guide". The explanation auxiliary device 2 changes the check box 51 of the explanation item C1 from the clear state to the selected state, such as Figure 4 The display is shown on the display unit 23.

[0072] The present system 1 repeats the above-mentioned detection operation every time the user 4 speaks, for example, and updates the display of the display unit 23 in real time. Figure 5 2 shows an example of the display of the display unit 23 after the user 4 repeatedly speaks. According to the present system 1, the user 4 can confirm the explanation items C1 to C7, C9, and C10 explained by his / her own speech in the conversation with the customer 40 and the explanation item C8 not explained, so that the sales activities of the user 4 can be assisted.

[0073] When performing the above detection operation, the system 1 applies natural language processing based on the learning model 35 to the speech sentence, and calculates the likelihood for each explanatory item C. The likelihood indicates the degree to which the corresponding speech sentence is detected as explaining the explanatory item C, and has a value in the range of 0 to 1, for example. It is considered that making the information related to the process of such detection visible to the user 4 is useful in operating the system 1 in order to more appropriately assist the user 4.

[0074] Therefore, in the present embodiment, the explanation support device 2 displays information corresponding to the likelihood together with the corresponding speech sentence in the speech list 55, for example. Figure 4 The "Card Membership Guide [99%]" in the remarks column 58 indicates that the likelihood of the explanation item C1 "Card Membership Guide" is "0.99" for the corresponding speech sentence 53. Thus, the user 4 can confirm to what extent each speech is detected as explaining the explanation items C1 to C10 in the process of obtaining the check results of the check list 50 in the present system 1.

[0075] In addition, the description support device 2 of this embodiment can also display the history of the confirmation of the detection result of the present system 1 by the above user 4 after the real-time detection action. The confirmation result based on the user 4 is useful, for example, to improve the detection accuracy of the present system 1. The following describes the details of the operation of the present system 1 and the description support device 2.

[0076] 2-2. Explanation of the detection operation of the auxiliary system

[0077] use Figure 6 The detection operation of the explanation supporting system 1 according to the present embodiment will be described. Figure 6This is a flowchart for explaining the detection operation of the present system 1 .

[0078] Figure 6 Each process shown in the flowchart is executed by the control unit 20 of the explanation support device 2 in the present system 1. This flowchart starts when, for example, an operation for executing voice recognition is performed on the operation button 5 displayed on the display unit 23 via the operation unit 22 (see Figure 4 ). In addition, at the beginning of this flowchart, for example, all the check boxes 51 in the check list 50 are set to a clear state.

[0079] First, the control unit 20 of the support device 2 acquires audio data representing the audio of the user 4 speaking from the microphone 26 (S1). The microphone 26 collects audio during the conversation of the user 4 and generates audio data. The audio data of the collected audio is an example of input information in this embodiment.

[0080] Next, the control unit 20 acquires a speech sentence indicating a result of speech recognition by communicating with the speech recognition server 11 ( S2 ). At this time, the control unit 20 transmits the input speech data to the speech recognition server 11 via the network I / F 25 .

[0081] The voice recognition server 11 performs processing based on the voice recognition model based on the voice data from the explanation support device 2, generates text data of the speech sentence, and sends it to the explanation support device 2. The processing based on the voice recognition model includes speech segmentation and various voice recognition processing for the voice data. When the control unit 20 of the explanation support device 2 receives the speech sentence from the voice recognition server 11 via the network I / F 25 (S2), the received speech sentence and the corresponding voice data are recorded in the storage unit 21.

[0082] Next, the control unit 20 acquires likelihood information including likelihood of each explanatory item C with respect to the acquired speech sentence by communicating with the language processing server 3 ( S3 ). At this time, the control unit 20 transmits the acquired speech sentence to the language processing server 3 via the network I / F 25 .

[0083] When the language processing server 3 receives a speech sentence from the explanation support device 2, it performs natural language processing based on the learning model 35, generates likelihood information, and sends it to the explanation support device 2. For example, in the natural language processing, the received speech sentence is converted into an input vector by word embedding, and input to the learning model 35. The learning model 35 performs machine learning so that the likelihood representation of each explanation item C output based on the input vector detects the degree to which each explanation item C is predicted to be explained by the corresponding speech sentence.

[0084] Next, the control unit 20 of the explanation support device 2 performs a check display process (S4) based on the acquired speech sentence and likelihood information. The check display process checks whether each explanation item C1 to C10 is explained for each speech sentence of the speech recognition result based on the likelihood information, for example Figure 4 The inspection result display process is described in detail later.

[0085] The control unit 20 determines whether the detection operation of the system 1 is completed (S5) based on the operation of the operation button 5, for example. When the control unit 20 determines that the detection operation is not completed (No in S5), it returns to step S1 and performs the processing after step S1 for the new speech. The user 4 performs an operation to complete the detection operation when, for example, the conversation with the customer 40 is completed.

[0086] When the control unit 20 determines that the detection operation is completed (Yes in S5), it generates history data indicating the history of the detection operation and stores it in the storage unit 21 (S6). The history data will be described later (see Figure 8 ).

[0087] When the control unit 20 stores the historical data in the storage unit 21 ( S6 ), the processing based on this flowchart ends.

[0088] According to the above processing, each time the user 4 speaks (S1), the likelihood of the speech sentence of the voice recognition result is calculated (S2, S3), and the inspection results for various explanation items C1 to C10 are displayed in real time (S4).

[0089] If the length of the speech sentence acquired in the above step S2 is shorter than the specified value, the control unit 20 may also omit the processing of step S3. The specified value can be set to the number of characters or sentences that can be considered as not including the description of the various explanatory items C1 to C10 in the speech sentence. In this way, the processing load for speech that is irrelevant to the description of the explanatory items C1 to C10, such as the responsive words in the conversation, can be reduced.

[0090] 2-2-1. Check display processing

[0091] use Figure 7 To illustrate the inspection display processing ( Figure 6 Details of S4).

[0092] Figure 7 A flowchart for explaining the examination display process by the explanation supporting device 2 is shown. Figure 7 The flowchart is in Figure 6 It starts from a state where a speech sentence is obtained in step S2 and likelihood information for the speech sentence is obtained in step S3.

[0093] First, the control unit 20 of the explanation support device 2 selects one explanation item C as an inspection target from a plurality of explanation items C1 to C10 set in advance ( S11 ). Figure 6 In order to check all the explanation items C1 to C10 for one speech sentence, the explanation items C are selected one by one in sequence in step S11.

[0094] Next, the control unit 20 determines whether the likelihood in the acquired likelihood information exceeds the detection threshold V1 (S12) for the selected explanatory item C. The detection threshold V1 is a threshold indicating a reference for detecting that the corresponding explanatory item C is explained, and is set, for example, in consideration of the likelihood of a speech sentence explaining the explanatory item C.

[0095] If the control unit 20 determines that the likelihood of the selected explanatory item C exceeds the detection threshold V1 (Yes in S12), it determines whether the check box 51 corresponding to the explanatory item C in the check list 50 is in a selected state (S13). For example, if the user 4 has not explained the selected item in the conversation and the corresponding check box 51 is cleared, the control unit 20 proceeds to No in step S13.

[0096] If the control unit 20 determines that the check box 51 of the selected explanation item C is not in the selected state (No in S13), the control unit 20 changes the check box 51 from the clear state to the selected state, and updates the display of the check list 50 in the display unit 23 (S14). In addition, the display update in step S14 can also be performed simultaneously with step S18.

[0097] Furthermore, the control unit 20 stores the likelihood of the selected explanation item C as a candidate to be displayed in the remarks column 58 of the speech list 55 (S15). Specifically, the control unit 20 associates the selected explanation item C with the likelihood and stores them in the storage unit 21 as display candidates.

[0098] On the other hand, if the check box 51 of the selected explanatory item C is in a checked state (Yes in S13 ), the control unit 20 proceeds to step S15 without performing the process of step S14 .

[0099] In addition, if the control unit 20 determines that the likelihood of the selected explanatory item C does not exceed the detection threshold V1 (No in S12), for example, it determines whether the likelihood exceeds the display threshold V2 (S16). The display threshold V2 is set to a value smaller than the detection threshold V1, for example, a value indicating a predetermined width near the detection threshold V1. The display threshold V2 is a threshold value indicating that although the likelihood of the speech sentence does not reach the detection threshold V1, it may be related to the explanatory item C and is considered to be displayed.

[0100] If the control unit 20 determines that the likelihood of the selected explanatory item C exceeds the display threshold value V2 (Yes in S16), the likelihood is maintained as a display candidate (S15). On the other hand, if the control unit 20 determines that the likelihood does not exceed the display threshold value V2 (No in S16), the process proceeds to step S17 without performing step S15.

[0101] The control unit 20 determines whether all the explanation items C1 to C10 are selected as the inspection object (S17). If all the explanation items C1 to C10 are not selected (No in S17), the control unit 20 performs the processing from step S11 onwards for the explanation items C that are not selected.

[0102] After selecting and checking all the explanation items C1 to C10 (Yes in S17), the control unit 20 controls the display unit 23 to update and display the speech list 55 (S18). Specifically, the control unit 20 displays the speech sentence in the speech sentence column 57 of the speech list 55 (see Figure 4 ). In addition, when the display candidates of the remark column 58 are retained (S15), the control unit 20 additionally displays the retained information in the remark column 58.

[0103] If the control unit 20 controls the display unit 23 to update the speech list 55 and the like (S18), the process ends. Figure 6 After completing the processing of step S4, proceed to step S5.

[0104] Through the above processing, it is possible to check the speech sentences of the voice recognition results corresponding to one speech of the user 4 based on the likelihood information and the related explanation items C. At this time, the explanation support device 2 changes the display method of the speech list 55 according to the likelihood. The user 4 can confirm the inspection results of his own speech in real time by checking the list 50 and the speech list 55.

[0105] For example, for speech sentences whose likelihood exceeds the detection threshold value V1, the likelihood is displayed in the remarks column 58 of the speech list 55 together with the check box 51 in the check list 50. Thus, the user 4 can confirm to what extent the speech is detected to explain the explanation item C, or whether the speech in the conversation after the check is also a sufficient explanation, etc.

[0106] Even if the likelihood does not reach the detection threshold V1, the likelihood of the speech sentence exceeding the display threshold V2 may be displayed in the remark column 58. The user 4 can understand that the description of the description item C in his / her speech is insufficient in the remark column 58 when the check box 51 is cleared.

[0107] Also, for speech sentences having a likelihood less than the display threshold value V2, the likelihood is not displayed in the remark column 58 of the speech list 55. This allows the display of the remark column 58 to be omitted for speech such as small talk that is not related to any of the explanation items C1 to C10.

[0108] Furthermore, through the above processing, when the likelihood of multiple explanation items C for a speech sentence exceeds the detection threshold V1 (Yes in S12), multiple check boxes 51 can be updated to a selected state according to the speech sentence (S14). When the likelihood of multiple explanation items C for a speech sentence exceeds the display threshold V2 (Yes in S16), multiple likelihoods are annotated in the remarks column 58 (S16, S18).

[0109] 2-3. About historical data

[0110] The explanation support device 2 according to the present embodiment accumulates the history data in the storage unit 21 ( Figure 6 S6). Use Figure 8 To explain the resume data.

[0111] Figure 8 1 is a diagram for explaining the history data D1 in the explanation support device 2. The history data D1 is managed, for example, for each "business session ID". The "business session ID" is an ID for identifying the session in which the detection operation of the present system 1 is performed. The history data D1 is managed, for example, for each "business session ID". Figure 8 As shown, "speech number", "sound data", "speech sentence", "likelihood information" and "user evaluation information" are associated and recorded.

[0112] Furthermore, the history data D1 and the detection threshold value V1 used when detecting the action may be associated and managed in the storage unit 21. The detection threshold value V1 may be managed separately for each of the explanatory items C1 to C10.

[0113] In the history data D1, the "speech number" indicates the order of the speech that is the object of voice recognition in the conversation identified by the business conversation ID. The "voice data" is the voice data of the speech that is the object of voice recognition, and is divided into individual files for each speech. The "speech sentence" indicates the text data of the voice recognition result of the speech corresponding to the voice data of the file of each speech number. The "likelihood information" includes the likelihood of each explanatory item C for the speech sentence. As described later, the "user evaluation information" indicates the evaluation of the user 4 on the detection result of the present system 1.

[0114] exist Figure 6In the flowchart, it is explained that the control unit 20 of the auxiliary device 2 associates the speech sentences, sound data and likelihood information obtained by each repetition of steps S1 to S5, assigns speech numbers in sequence and records them in the history data D1 (S6). In addition, at the moment of step S6, the user evaluation information is not specially recorded and is a null value "-".

[0115] 2-4. Confirmation display of history

[0116] The explanation support device 2 of this embodiment can perform various displays for the user 4 to confirm the detection results based on the above-mentioned history data D1. Figures 9 to 12 Next, the confirmation display of the history in the explanation support device 2 will be described.

[0117] Fig. 9 The following is a display example of the sales conversation list 6 on the display unit 23 of the explanation support device 2. The sales conversation list 6 is displayed in response to an operation of confirming a setting using the operation button 5, for example.

[0118] The sales conversation list 6 manages information on the history of the detection operation performed by the system 1, for example, according to the sales conversation ID of each history data D1. Fig. 9 In the example of FIG. 6 , the business conversation list 6 includes a person in charge column 61 , a time column 62 , a customer column 63 , an inspection record icon 64 , and a speech record icon 65 .

[0119] In the business conversation list 6, the person-in-charge column 61 indicates the user 4 in the business conversation during the detection operation based on the present system 1. The time column 62 indicates the time when the business conversation was performed. The customer column 63 indicates the customer 40 in the business conversation during the detection operation. The inspection history icon 64 accepts an operation for displaying the inspection history screen. The inspection history screen displays the final inspection list 50 during the detection operation based on the present system 1. The speech history icon 65 accepts an operation for displaying the speech history screen.

[0120] Fig.10 FIG. 2 shows an example of a display of a speech history screen on the display unit 23. The speech history screen displays each speech sentence as a speech history in the history data D1 of the business conversation corresponding to the operated speech history icon 65 and a playback column 66 for playing back the voice data. Fig.10 In the display example, a search bar 67 is displayed on the speech history screen. The explanation support device 2 performs a keyword search for, for example, a speech sentence according to the operation of the search bar 67. The search range of the search bar 67 may be specified in units of lines of the speech sentence.

[0121] Fig.11The display example of the inspection history screen on the display unit 23 is shown. In the explanation support device 2 of this embodiment, for example, by double-clicking any text portion of the explanation items C1 to C10 in the inspection list 50 of the inspection history screen, the inspection history list 70 for the explanation item C operated is displayed in a pop-up. Fig.11 In FIG. 1 , the state of the detection history list 70 for the explanatory item C2 of “confirming contact” is displayed as an example.

[0122] The detection history list 70 is a list containing speech sentences that were detected as explaining a specific explanation item C during the business conversation or as possibly explaining it. According to the detection history list 70, not only the speech sentences of the speech in which the check box 51 was set to a checked state during the detection operation of the system 1 can be confirmed, but also the speech sentences detected as explaining the explanation item C in the subsequent speech can be confirmed. The detection history list 70 is an example of display information in this embodiment.

[0123] exist Fig.11 In the display example, the detection history list 70 establishes and displays the reproduction column 71, the speech sentence, the system detection box 72, and the user evaluation box 73. The system detection box 72 has a check / clear state indicating whether the corresponding speech sentence is detected during the detection operation of the system 1 to explain the explanation item.

[0124] The user evaluation box 73 has a check / clear state indicating, for example, a correct / incorrect evaluation of the detection result shown in the system detection box 72. The check / clear state of the user evaluation box 73 can be changed by an operation of the user 4 such as clicking.

[0125] The description support device 2 of this embodiment stores the user evaluation information based on the user evaluation frame 73 in the history data D1. The user evaluation information in the history data D1 can be used to improve the detection accuracy of the present system 1. For example, the detection threshold V1 in the present system 1 can be adjusted, or it can be used as teaching data in machine learning such as active learning of the learning model 35.

[0126] use Fig.12 The processing of the above detection history list 70 by the description support device 2 will be described. Fig.12 This is a flowchart for explaining the processing based on the detection history of the explanation support device 2.

[0127] Fig.12 Each process shown in the flowchart is executed by the control unit 20 of the support device 2. Fig.12 The flowchart is started when an operation of designating the explanation item C in the inspection list 50 is input via the operation unit 22 on the inspection history screen.

[0128] First, the control unit 20 searches for speech sentences associated with the description item C specified by the operation of the user 4 in the history data D1 with a likelihood exceeding the search threshold V3 (S21). The search threshold V3 is a threshold used as a reference for searching for a specific description item, and is set to V3 = V1, for example. The search threshold V3 is not limited to this, and can be appropriately set within a range of greater than V2 and less than V1, for example.

[0129] Next, the control unit 20 generates a detection history list 70 based on the search result of the history data D1 so as to include the searched speech sentence, and displays the detection history list 70 on the display unit 23, for example, by popping up (S22). At this time, the system detection box 72 in the detection history list 70 is set to be selected or cleared according to whether the likelihood exceeds the detection threshold value V1. The user evaluation box 73 is set to be cleared or selected in the initial state, for example.

[0130] Next, the control unit 20 receives the operation in the detection history list 70 and performs control corresponding to various operations (S23). For example, when the user evaluation box 73 is operated, the control unit 20 controls the display unit 23 to switch the display of the selected state or the cleared state of the operated user evaluation box 73. In addition, when the playback bar 71 is operated, the control unit 20 controls the speaker 27 to reproduce the sound based on the operated playback bar 71.

[0131] The control unit 20 determines whether the operation of the detection history list 70 is completed (S24) based on the operation of the pop-up close button 75 attached to the detection history list 70. The control unit 20 executes step S23 until the operation of the detection history list 70 is completed (No in S24).

[0132] If the operation of detecting the history list 70 is completed (Yes in S24), the control unit 20 updates the history data D1 (S25) according to the state of the user evaluation box 73 at the end of the operation. The control unit 20 records "Y" or "N" in the user evaluation information record according to the selected state or cleared state of the user evaluation box 73 of each speech sentence at the end of the operation of the detection history list 70 in the history data D1 stored in the storage unit 21 (see Figure 8 ) At this time, the column that is not the evaluation object in the user evaluation information is maintained as "-".

[0133] Furthermore, the control unit 20 deletes the pop-up display of the detection history list 70 (S26), and ends the processing based on this flowchart. The processing order of steps S25 and S26 is not particularly limited.

[0134] Through the above processing, the control unit 20 causes the display unit 23 to display the detection history list 70 (S22) including the speech sentences in the history data D1 according to the likelihood related to the explanatory item C specified by the user 4. In the detection history list 70, the user 4 can evaluate whether the detection result of the speech sentence related to the specified explanatory item C is appropriate.

[0135] 3. Summary

[0136] As described above, in the present embodiment, the explanation support device 2 displays information related to the explanation items C1 to C10 to be checked in the speech of the user 4. The explanation support device 2 includes the microphone 26 as an example of an acquisition unit, the control unit 20, and the display unit 23. The microphone 26 acquires voice data as input information representing a speech sentence based on the speech (S1). The control unit 20 generates information representing the inspection result of the explanation item related to the speech sentence (S4). The display unit 23 displays the information generated by the control unit 20 (S14, S18, S22). The display unit 23 displays the inspection list 50 indicating whether the explanation items C1 to C10 are explained in the speech sentence indicated by the input information sequentially acquired by the microphone 26. The display unit 23 displays the speech list 55 or the detection history list 70 as display information including the speech sentence according to the likelihood of each speech sentence defining the inspection result of the explanation item C in the inspection list 50 (S18, S22).

[0137] The explanation support device 2 described above displays the inspection list 50 related to the explanation items C of the inspection object and display information including the utterance sentence according to the likelihood. This makes it possible to easily support the user 4 in explaining the explanation items C by detecting the information processing.

[0138] In this embodiment, the check list 50 indicates whether the explanation item C is explained based on the likelihood of each utterance. The user 4 can confirm that the inspection result of the check list 50 is available based on the likelihood in the displayed information, and the information support of the user 4 is easy.

[0139] In the present embodiment, the display unit 23 updates the speech list 55 (S18) every time the display unit 23 acquires input information from the microphone 26 (S1). Thus, the user 4 can confirm the detection result of the system 1 in real time during speech.

[0140] In this embodiment, the speech list 55 as display information includes a speech text field 57 showing speech texts and a remark field 58 showing the likelihood of the speech texts. The user 4 can confirm that the speech texts and likelihoods are compared in the speech list 55.

[0141] In the present embodiment, the explanation assisting device 2 further includes a storage unit 21 for recording the history data D1 that associates the speech sentence with the likelihood. The control unit 20 generates a detection history list 70 (S22) as display information based on the history data D1 recorded in the storage unit 21. Thus, the user 4 can confirm the detection result of the present system 1 afterwards. The detection history list 70 may also include a reproduction column 71 for reproducing the sound data for the selected explanation item C, and a display of the speech sentence for the selected explanation item C.

[0142] In this embodiment, the detection history list 70 includes a system detection box 72 indicating whether the likelihood of the associated speech sentence in the history data D1 exceeds a predetermined detection threshold value V1. The system detection box 72 allows the user 4 to easily confirm the detection result of the system 1 in the detection history list 70.

[0143] In the present embodiment, an operation unit 22 is further provided, and the operation unit 22 inputs the user's operation of evaluating the inspection result of the explanatory item C in the user evaluation box 73 for each speech sentence in the detection history list 70. Thus, information indicating the evaluation of the user 4 on the detection result of the present system 1 can be obtained, and the present system 1 can be easily used.

[0144] In this embodiment, the acquisition unit of the explanation support device 2 includes a microphone 26 that acquires voice data as input information. The speech sentence represents the voice recognition result of the voice data. The detection operation of the system 1 can be performed based on the voice spoken by the user 4.

[0145] The explanation assisting method in the present embodiment is a method for displaying information related to the explanation items C1 to C10 of the inspection object in the speech of the user 4. The method includes: step S1 in which the acquisition unit acquires input information representing a speech sentence based on the speech; step S4 in which the control unit 20 generates information representing the inspection result of the explanation item related to the speech sentence; and steps S14, S18, and S22 in which the display unit 23 displays the information generated by the control unit 20. The display unit 23 displays a check list 50 indicating whether the explanation items C1 to C10 are explained in the speech sentences indicated by the input information sequentially acquired by the acquisition unit. The display unit 23 displays the display information including the speech sentence based on the likelihood of each speech sentence that specifies the inspection result of the explanation item C in the check list 50.

[0146] In this embodiment, a program for causing the control unit 20 of the computer to execute the above-mentioned explanation support method is provided. The explanation support method of this embodiment can easily support the user 4 to check the explanation of the explanation item C through information processing.

[0147] (Implementation Method 2)

[0148] Embodiment 2 is described below using the drawings. In Embodiment 1, the explanation support system 1 for detecting whether the explanation item is explained in the speech of the user 4 is described. In Embodiment 2, the explanation support system 1 for further detecting whether the speech of the user 4 contains an NG phrase is described.

[0149] Hereinafter, description of the same configuration and operation as those of the explanation support system 1 according to the first embodiment will be appropriately omitted, and the explanation support system 1 and the explanation support device 2 according to the present embodiment will be described.

[0150] Fig.13 is a flowchart for explaining the detection operation of the explanation support system 1 according to the second embodiment. Figure 6 The same treatment is also Fig.13 As shown, the process for detecting the NG phrase is performed (S31 to S33).

[0151] The control unit 20 of the explanation auxiliary device 2 in this embodiment detects whether the speech sentence is a pre-set NG phrase (i.e., a prohibited phrase) (S31) based on the speech sentence or likelihood information obtained in steps S2 and S3. The judgment of step S31 can also be performed by performing keyword detection on the specified NG phrase in the speech sentence. In addition, machine learning can also be performed through the learning model 35 so that the likelihood indicating the degree of prediction of the speech sentence as an NG phrase is output together with the likelihood of various explanation items C1 to C10.

[0152] If the control unit 20 detects that the speech sentence is not an NG phrase (No in S31), the control unit 20 transmits the audio data corresponding to the speech sentence from the explanation support device 2 to the customer terminal 41 (S32). For example, the control unit 20 buffers the audio data acquired in step S1 until the determination in step S31.

[0153] On the other hand, if the detected speech sentence is an NG phrase (Yes in S31), the network I / F 25 is controlled, for example, so that the transmission of the voice data from the explanation support device 2 to the customer terminal 41 is cut off (S33). Thus, if it is detected that the user 4 utters an NG phrase, the voice of the uttered NG phrase can be prevented from being recognized by the customer 40.

[0154] Fig.14 The following is a display example of the explanation support device 2 according to the present embodiment. When the above NG phrase is detected (Yes in S31), the explanation support device 2 may display the NG phrase to warn the user 4 in the check display process (S4). Fig.14, an example of display when the speech sentence 54 "can agree on a greater benefit" is detected as an NG phrase. In this display example, the display unit 23 displays "Warning" in the remark column 58 corresponding to the above speech sentence 54. Thus, the user 4 can be prompted to pay attention when an NG phrase is detected.

[0155] As described above, in this embodiment, the explanation assistance device 2 further includes a communication unit, namely, a network I / F 25 or a machine I / F 24, which transmits information indicating a speech sentence to the outside. When the control unit 20 detects a predetermined prohibited phrase, namely, an NG phrase, in the speech sentence, the communication unit is controlled so that the transmission of the information indicating that the speech sentence in which the NG phrase is detected is cut off. Thus, the information indicating the NG phrase is not selectively transmitted to the outside, and information assistance can be provided to the user 4.

[0156] (Other embodiments)

[0157] As described above, embodiments 1 to 2 are described as examples of the technology disclosed in this application. However, the technology disclosed in this application is not limited thereto, and can also be applied to embodiments in which changes, substitutions, additions, omissions, etc. are appropriately made. In addition, the various structural elements described in the above embodiments can also be combined to form new embodiments. Therefore, other embodiments are exemplified below.

[0158] In the above-mentioned embodiments, the explanation support device 2 of the explanation support system 1 performs voice communication with the customer terminal 41. The explanation support device 2 of this embodiment is not limited to voice communication, and may perform various data communications.

[0159] In addition, in each of the above-mentioned embodiments, the explanation support device 2 of the explanation support system 1 communicates with the customer terminal 41, but the explanation support device 2 of this embodiment may not particularly communicate with the customer terminal 41. The present system 1 may also be applied to various face-to-face customer receptions such as a window of a financial institution. In this case, the explanation support device 2 may be configured to appropriately recognize the speech of the user 4 and the speech of the customer 40.

[0160] In the above-mentioned embodiments, speech data is exemplified as input information of the explanation support device 2. In the present embodiment, the input information of the explanation support device 2 may be text data instead of speech data. The present system 1 can also be applied to various electronic conferences, etc.

[0161] As described above, the embodiments are described as examples of the technology in the present disclosure. For this purpose, the drawings and detailed description are provided.

[0162] Therefore, the structural elements described in the drawings and detailed descriptions include not only the structural elements required for solving the problem, but also the structural elements that are not required for solving the problem in order to illustrate the above-mentioned technology. Therefore, even if these non-essential structural elements are recorded in the drawings and detailed descriptions, they should not be directly identified as essential.

[0163] In addition, the above-mentioned embodiments are for illustrating the technology in the present disclosure, and therefore various changes, substitutions, additions, omissions, etc. can be made within the scope of the claims or their equivalents.

[0164] Industrial Applicability

[0165] The present disclosure can be applied to information assistance when a user makes various explanations, for example, can be applied to a call center, a remote customer reception system, or various face-to-face customer receptions.

Claims

1. An explanation assisting device for displaying information related to explanation items of an inspection object in a user's speech, the explanation assisting device comprising: an acquisition unit, which acquires input information representing a speech sentence based on the speech; A control unit that generates information indicating a result of checking the explanatory matters related to the speech sentence; and a display unit for displaying information generated by the control unit; The display unit displays a check list indicating whether the explanation items are explained in the speech sentences indicated by the input information sequentially obtained by the obtaining unit, The display unit displays display information including each speech sentence based on the likelihood of each speech sentence defining the inspection result of the description item in the inspection list.

2. The explanation aid according to claim 1, in, The check list indicates whether the explanation item is explained based on the likelihood of each utterance sentence.

3. The explanation aid according to claim 1 or 2, in, The display unit updates the display information every time the display unit acquires the input information from the acquisition unit.

4. The explanation aid according to claim 1 or 2, in, The display information includes information indicating the utterance sentence and the magnitude of the likelihood of the utterance sentence.

5. The explanation aid according to claim 1 or 2, in, The explanation support device further includes: a storage unit that records history data that associates the utterance sentence with the likelihood; The control unit generates the display information based on the history data recorded in the storage unit.

6. The explanation aid according to claim 5, in, The display information includes information indicating whether the likelihood of being associated exceeds a predetermined threshold value for each utterance in the history data.

7. The explanation aid according to claim 5, in, The display information includes a reproduction column for reproducing audio data related to the selected explanatory matter, and display of a speech sentence related to the selected explanatory matter.

8. The explanation aid according to claim 1 or 2, in, The explanation support device further includes an operation unit that inputs a user's operation for evaluating a check result of the explanation matter for each speech sentence in the display information.

9. The explanation aid according to claim 1 or 2, in, The explanation support device further comprises: a communication unit that transmits information representing the speech sentence to the outside; The control unit controls the communication unit so that, when a predetermined prohibited phrase is detected in the utterance, transmission of information indicating the utterance in which the prohibited phrase is detected is cut off.

10. The explanation aid according to claim 1 or 2, in, The acquisition unit includes a microphone for acquiring audio data as the input information. The utterance sentence represents a result of voice recognition of the voice data.

11. A method for assisting explanation, comprising: displaying information related to an explanation item of an inspection object in a user's speech; the method for assisting explanation comprising: The step of an acquisition unit acquiring input information representing a speech sentence based on the speech; A step of a control unit generating information indicating a result of checking the explanatory matters related to the speech sentence; and A step of displaying information generated by the control unit on a display unit, The display unit displays a check list indicating whether the explanation items are explained in the speech sentences indicated by the input information sequentially obtained by the obtaining unit, The display unit displays display information including each speech sentence based on the likelihood of each speech sentence defining the inspection result of the description item in the inspection list.

12. A program product comprising a program for causing a computer to execute the method according to claim 11.

Images