Display processing device, display processing method, program, and recording medium

Abstract

To effectively utilize a limited display space while making it possible to perform utterance identification of participants / non-participants based on user operations by displaying utterance text data.SOLUTION: A display processing device of the present invention includes: a participant number setting unit for setting the number of participants of an interview based on a user operation; and a display processing unit for performing processing for displaying utterance text data obtained by converting utterance sounds in the interview into text based on a sound signal collected by a microphone arranged at a place where the interview is to be held on a display unit. The display processing unit displays only utterance text data corresponding to the number of participants set by the participant number setting unit on the display unit.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display processing device and method, a program, and a recording medium, and in particular to a technique for distinguishing between speech by interview participants and speech by non-participants. [Background technology]

[0002] Conventionally, the content of interviews has been widely recorded. By recording the content of an interview as audio data, the content of the interview can be saved without the need to take notes on the content of the interview. There are also known techniques for converting recorded interview data into text and displaying it, and even for analyzing the content of the interview, such as creating a summary of the interview based on the converted text (see, for example, Patent Document 1 below).

[0003] As a related prior art, the following Patent Document 2 can also be cited. Patent Document 2 discloses a technique for obtaining a plurality of pieces of voice data by collecting a conversation between a plurality of people at a plurality of sound collection positions having different positional relationships with each of the plurality of people, extracting speech sections from the voice data, determining waveform similarity, attenuation characteristics, and dissimilarity in the speech sections between the pieces of voice data, and identifying the speaker of the speech section based on the similarity, attenuation characteristics, and dissimilarity. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223469 [Patent Document 2] JP 2018-013742 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, the interview may take place in a private space isolated from people other than the participants, such as a conference room or a private room, or in a public space not isolated from people other than the participants, such as a coffee shop or a lobby. For example, a business negotiation for insurance solicitation may take place in a public space such as a coffee shop.

[0006] When an interview is held in a public space, the speech of people other than the interview participants may also be picked up by the microphone. Therefore, if it is desired to process only the text data of the participants' speech, it is possible to carry out a process to distinguish between speech by participants and speech by non-participants. However, such processing is relatively difficult and, at present, it is considered difficult to put into practical use.

[0007] Therefore, it is conceivable to display the speech text data of all persons whose speech sounds have been converted into text, and to allow the user to specify from there the speech of persons who are participants or persons who are not participants.

[0008] However, in a limited display space, it is not efficient to display the speech text data of all people, including non-participants in the interview. In particular, in an environment where there are many non-participants, it is expected that the speech text data of more non-participants than the number of participants will have to be displayed, which is undesirable.

[0009] The present invention has been made in consideration of the above-mentioned circumstances, and aims to make effective use of limited display space while enabling speech identification of participants / non-participants based on user operation by displaying speech text data. [Means for solving the problem]

[0010] The display processing device of the present invention comprises a participant number setting unit that sets the number of participants in an interview based on user operation, and a display processing unit that performs processing to display on a display unit spoken text data obtained by converting spoken sounds in the interview into text based on a signal picked up by a microphone placed at the interview venue, and the display processing unit displays on the display unit only the spoken text data for the number of participants set by the participant number setting unit. According to the above configuration, when the speech of any of the set number of people is displayed and the user designates the speech of any of the people as a non-participant speech, the speech of another person is displayed instead, and the user can designate the speech of the other person again as whether it is a participant or not. By repeating such processing, it is finally possible to identify the speech of all participants, and as a result, it is possible to identify the speech of participants / non-participants. And, according to the above configuration, the speech of people other than the number set by the user is not displayed, so that the limited display space can be used effectively. Effect of the Invention

[0011] According to the present invention, by displaying spoken text data, it is possible to distinguish between utterances of participants and non-participants based on user operations, while also making effective use of limited display space. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the configuration of a minutes-taking support system including a display processing device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a perspective view showing a schematic external appearance example of an information processing device configured as a smartphone. [Diagram 3] FIG. 13 is a diagram for explaining an example of how an information processing device is placed when recording an interview. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Diagram 5] 2 is a block diagram illustrating an example of a hardware configuration of a server device (display processing device) according to an embodiment. [Figure 6] FIG. 2 is a functional block diagram showing functions of a display processing device according to an embodiment. [Figure 7] FIG. 11 is a diagram showing an example of a start screen for recording in the embodiment. [Figure 8] 13 is a diagram illustrating an example of a state in which a sound collection device is placed on a support surface when recording an interview. [Figure 9] 11 is an explanatory diagram of an initial setting process for a person to be displayed in the embodiment. FIG. [Figure 10] FIG. 11 is an explanatory diagram of the initial setting process of a person to be displayed in the embodiment. [Figure 11] FIG. 11 is an explanatory diagram of the initial setting process of a person to be displayed in the embodiment. [Figure 12] FIG. 11 is an explanatory diagram of the initial setting process of a person to be displayed in the embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a GUI for setting important people and for making exclusion settings. [Figure 14] 13 is a flowchart of a basic process for setting a person to be displayed. [Figure 15] 13 is a flowchart of a process related to setting a person to be displayed based on a speech overlap rate. [Figure 16] 13 is a flowchart of a process related to a participant designation operation and a non-participant designation operation. [Figure 17] 13 is a diagram showing an example of a registration screen for interview participant information in an embodiment. FIG. [Figure 18] FIG. 13 is a diagram showing an example of a recording list screen in the embodiment. [Figure 19] FIG. 13 is a diagram showing an example of an editing screen (before speaker / group setting and analysis processing are performed) in the embodiment. [Figure 20] FIG. 13 is a diagram showing an example of a setting screen for setting speaker groups in the embodiment. [Figure 21]13 shows an example of information displayed in the speaker / group setting area after information has been set. FIG. [Figure 22] FIG. 13 is a diagram showing an example of an editing screen (after setting speakers and groups) in the embodiment. [Figure 23] 13 is a diagram showing an example of display of analysis result text data on an editing screen. FIG. [Figure 24] 11A and 11B are diagrams illustrating an example of a change operation of already-set group information. [Diagram 25] 25 is a diagram illustrating an example of a display state of the spoken text display area in response to the change operation shown in FIG. 24 being performed. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described in the following order. <1. System configuration example> <2. Device configuration example> <3. Display-related processing as an embodiment> <4. Steps for creating minutes and example of display screen> <5. Modifications> <6. Summary of the embodiment>

[0014] In the embodiment, the display process of the present invention is performed in real time in parallel with recording of the interview, and the recorded interview data is used to create minutes of the interview.

[0015] <1. System configuration example> FIG. 1 is a block diagram showing an example of the configuration of a minutes-taking support system 100 including a display processing device according to an embodiment of the present invention. The minutes-taking support system 100 according to the embodiment includes at least an information processing device 1 and a server device 20.

[0016] The information processing device 1 is a computer device used by a user who records an interview and creates minutes, and in this example, a GUI (Graphical User Interface) related to recording an interview and creating minutes is displayed on this information processing device 1. That is, the user performs various operational inputs related to recording an interview and creating minutes to this information processing device 1.

[0017] In this example, the interview is recorded using a microphone included in the information processing device 1. In this example, the information processing device 1 is configured as a smartphone, and therefore, in the minutes creation support system 100 of this example, the smartphone is used as a sound collection device for recording the interview. Specifically, the information processing device 1 of this example is configured with multiple microphones arranged at different positions on the device body and a display unit that displays information, and one of the multiple microphones is designated as an other-side microphone that is intended to pick up the other person's speech during the interview, and the other is designated as a self-side microphone that is intended to pick up the self-side speech during the interview.

[0018] FIG. 2 is an oblique view showing an example of the general appearance of an information processing device 1 configured as a smartphone, where FIG. 2A is an oblique view of the information processing device 1 seen from the front side, and FIG. 2B is an oblique view of the information processing device 1 seen from the back (rear) side. Here, the "front" of the information processing device 1 means the side on which the display surface of the display unit is present. 2A, a display screen of the display unit 7 is disposed on the front side of the information processing device 1, and a call speaker 8a is disposed above the display screen. A first microphone (microphone) 13a used for NC (noise canceling) is disposed near the call speaker 8a. Further, a second microphone 14a used for picking up sound during a call is disposed on the lower part of the information processing device 1, specifically, on the lower surface of the information processing device 1 in this example. In this example, as the second microphone 14a disposed on the lower part of the information processing device 1, two microphones are disposed to enable stereo sound collection. Here, with respect to the information processing device 1 configured as a smartphone, the upper and lower parts are concepts in which the side on which the call speaker 8a is arranged is the upper side when the sound collection device is viewed from the front.

[0019] 2B, a camera unit 16 is disposed at one of the left and right ends of the upper part on the back side of the information processing device 1. The camera unit 16 provided on the back side enables a user to capture an image of a subject while checking the captured image displayed on the display unit 7. In the information processing device 1 of this embodiment, a first microphone 13b used for NC is also disposed near the camera unit 16.

[0020] As can be understood from the above description, the information processing device 1 of this example is provided with first microphones 13a, 13b for NC arranged on the upper part of the device body, and second microphones 14a, 14b capable of picking up stereo sound arranged on the lower part of the device body. In this example, at least one of the first microphones 13a, 13b arranged at the top is defined as the above-mentioned local microphone, and the second microphones 14a, 14b arranged at the bottom is defined as the above-mentioned remote microphone. In this example, the first microphone 13a is defined as the local microphone.

[0021] Here, since the second microphones 14a and 14b are capable of collecting stereo sound, they can be said to be microphones with higher performance than the first microphone 13a (or the first microphone 13b).

[0022] In this example, it is assumed that directional microphones are used for the first microphone 13a (your own microphone) and the second microphones 14a and 14b (other party microphones).

[0023] FIG. 3 is a diagram for explaining an example of how the information processing device 1 is placed when recording an interview. In this embodiment, a "business negotiation" between a sales representative Hs and a customer Hc is taken as an example of an "interview." The sales representative Hs is a person who participates in business negotiations, performs sales activities, and prepares minutes of the business negotiations. The sales representative Hs can be considered as the host in a business meeting, while the customer Hc can be considered as the guest in the meeting. The business negotiation here is premised on a face-to-face format in which a sales representative Hs and a customer Hc actually meet face to face.

[0024] In many cases, business negotiations are conducted in a room such as a conference room, with the sales representative Hs seated on one side of a table or desk and the customer Hc seated on the other side. Here, the participants in the interview are assumed to be multiple sales representatives Hs and multiple customers Hc (specifically, two of each is used as an example), but a business negotiation may be conducted with the participation of at least one sales representative Hs and one customer Hc.

[0025] In this case, the information processing device 1 is a smartphone used by the sales representative Hs. When recording a business negotiation, the sales representative Hs places the information processing device 1 on a support surface such as the top surface of a table as shown in the figure. As described above, in this example, the second microphones 14a and 14b are defined as the other party's microphones, so when the information processing device 1 is placed on the support surface for recording in this way, the second microphones 14a and 14b should be directed toward the customer Hc as the other party. Here, when recording an interview, the object on which the information processing device 1 is placed is assumed to be a stationary object such as a table.

[0026] As described above, in this example, directional microphones are used for the first microphone 13a (your microphone) and the second microphones 14a, 14b (the other party's microphones). As a result, in this example, the first microphone 13a, which serves as the your microphone, picks up sound (speech) from a sound source (person) located on your side, and the second microphones 14a, 14b, which serve as the other party's microphones, pick up sound (speech) from a sound source (person) located on the other party's side. In this case, the speech sound picked up by the first microphone 13a can be identified as the speech sound from the participant's own side, i.e., the speech sound from a sound source located on the side where the participant from the other side is located, and the speech sound picked up by the second microphones 14a, 14b can be identified as the speech sound from the other side, i.e., the speech sound from a sound source located on the side where the participant from the other side is located, from the participant from the other side.

[0027] It is not necessary to use the microphone that the information processing device 1 has to record the interview; for example, it is also possible to use an external sound recording device connected to the information processing device 1 (whether wired or wireless) to record the interview.

[0028] 1, the information processing device 1 and the server device 20 are configured with a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The information processing device 1 and the server device 20 are capable of mutual data communication via a network NT, which is a predetermined communication network such as the Internet.

[0029] The minutes creation support system 100 of the embodiment has a function of analyzing the contents of the interview based on the recorded data of the interview, and a function of presenting text data showing the results of the analysis (hereinafter referred to as "analysis result text data") to the user on a screen. The user can use the analysis result text data presented on the screen to create minutes of the interview.

[0030] The information processing device 1 has a function of recording an interview (a business negotiation in this example). Specifically, the information processing device 1 records the audio signals picked up by the first microphone 13a and the second microphones 14a and 14b, thereby recording the interview. The recorded data of the interview is transmitted to the server device 20 via the network NT, and is stored (accumulated) in the server device 20.

[0031] In this embodiment, the function of recording the interview is realized by a "recording application." In this embodiment, the recording application is assumed to be in the form of a so-called SaaS (Software as a Service) type in which software (software program) as the recording application is stored on the server device 20 side and the software is provided as a service. In this case, various processes of the information processing device 1 related to recording, such as recording of the picked-up signal by the first microphone 13a and the second microphones 14a and 14b, transmission of the recorded data to the server device 20, and display of the spoken text data during recording, which will be described later, are executed by the information processing device 1 under control from the server device 20.

[0032] The server device 20 has a function of converting the recorded data of the interview into text, i.e., a function of generating text data by transcribing (writing down) the spoken portion of the interview, and an analysis function of performing an analysis process related to the contents of the interview based on the text data obtained by the text conversion function (hereinafter referred to as "speech text data"). Specifically, the analysis function in this embodiment is a function of performing a summary creation process that is a process of creating a summary of the interview, a main point list creation process that is a process of creating a list of main points of the interview, and a question and answer identification process that is a process of identifying the question portion and the answer portion of the interview.

[0033] In the minutes-taking support system 100, text data showing the analysis results obtained by the analysis function of the server device 20 as described above (hereinafter referred to as "analysis result text data") is displayed on a screen to the sales representative Hs as a user. Specifically, the analysis result text data showing the analysis results as a summary, a list of key points, questions and answers, etc. as described above is displayed on a screen and presented to the user. In this example, the screen display of such analysis result text data is performed on an information processing device 1 configured as a smartphone.

[0034] The user can efficiently create minutes by using the analysis result text data of the interview presented as described above. Specifically, the user can transcribe (copy) text data such as a summary presented as the analysis result text data and use it to create minutes, which eliminates the need to create minutes from scratch, improving the efficiency of creating minutes.

[0035] In this embodiment, the analysis result text data is presented by displaying one editing screen (editing screen Ge, described later) including an analysis result display area for displaying the analysis result text data and a text editing area for creating minutes. Here, the text editing area is an area in which the analysis result text data can be transcribed and the transcribed analysis result text data can be edited.

[0036] Since an editing screen including not only an analysis result display area but also a text editing area is displayed, it is possible to eliminate the need to open a separate editing screen when creating minutes using the analysis result text data. Therefore, the effort required for the user to create minutes of an interview can be reduced, and the ease of creating minutes can be improved. In particular, when the interview is a business meeting as in this example, the improved ease of creating minutes can improve the efficiency of business negotiation activities and contribute to an improvement in the success rate.

[0037] <2. Device configuration example> FIG. 4 is a block diagram showing an example of a hardware configuration of the information processing device 1. As shown in FIG. 4, the information processing device 1 includes a CPU 1a, a ROM 2, and a RAM 3. The CPU 1a functions as an arithmetic processing unit that performs various processes, and executes the various processes according to a program stored in the ROM 2 or a program loaded from the storage unit 9 to the RAM 3. The RAM 3 also stores data and the like necessary for the CPU 1a to execute the various processes, as appropriate.

[0038] The CPU 1a, the ROM 2, and the RAM 3 are connected to one another via a bus 4. To this bus 4, an input / output interface (I / F) 5 is also connected.

[0039] To the input / output interface 5, an input section 6 including an operator and an operating device is connected. For example, various types of operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. are assumed as the input unit 6. In this example, the touch panel in the input unit 6 is formed on the display screen of the display unit 7, and the user can perform a touch operation on the display screen. An operation by a user is detected by the input unit 6, and a signal corresponding to the input operation is interpreted by the CPU 1a.

[0040] Further, the input / output interface 5 is connected, either integrally or separately, to a display unit 7 formed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 8 formed of a speaker, etc. The display unit 7 is used to display various types of information, and is configured as a display device provided in the housing of the information processing device 1, for example.

[0041] The display unit 7 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 1a. The display unit 7 also displays various operation menus, icons, messages, etc., i.e., GUI, based on instructions from the CPU 1a.

[0042] The input / output interface 5 may be connected to a storage unit 9 and a communication unit 10 . The storage unit 9 is configured with a hard disk drive (HDD) or a solid state drive (SSD) and stores various types of information.

[0043] The communication unit 10 performs communication processing via a transmission path such as the Internet, downloads applications, and performs communication with various devices via wired / wireless communication, bus communication, and the like.

[0044] A drive 12 is also connected to the input / output interface 5 as required, and a removable recording medium 11 such as a memory card or an optical disk is appropriately attached thereto.

[0045] The drive 12 allows data files such as programs used in each process to be read from the removable recording medium 11. The read data files are stored in the storage unit 9, and images and sounds contained in the data files are output on the display unit 7 and the audio output unit 8. In addition, the computer programs and the like read from the removable recording medium 11 are installed in the storage unit 9 as necessary.

[0046] In addition, a first sound collection unit 13 and a second sound collection unit 14 are connected to the input / output interface 5 . The first sound collection unit 13 is a comprehensive representation of the sound collection unit for NC including the first microphones 13a and 13b described above, and is configured to have at least the first microphones 13a and 13b, an amplifier unit that amplifies the collected sound signals, and an A / D conversion unit that A / D converts the amplified collected sound signals. The second sound collection unit 14 is a comprehensive reference to a stereo sound collection unit including the second microphones 14a and 14b described above, and is configured to have at least the second microphones 14a and 14b, an amplifier unit that amplifies the collected sound signals, and an A / D conversion unit that A / D converts the amplified collected sound signals. The audio signals collected by the first audio collection unit 13 and the second audio collection unit 14 are at least temporarily recorded in a storage device such as the memory unit 9, thereby enabling recording of the interview.

[0047] Further, a sensor unit 15 is connected to the input / output interface 5. The sensor unit 15 collectively indicates sensors provided in the information processing device 1 as a smartphone, and the image sensor provided in the above-mentioned camera unit 16 is included in this sensor unit 15. The sensor unit 15 also includes an IMU (Inertial Measurement Unit) provided with an acceleration sensor, an angular velocity sensor, an orientation sensor, and the like.

[0048] FIG. 5 is a block diagram showing an example of the hardware configuration of the server device 20. As shown in FIG. In the following description, parts that are similar to parts that have already been described will be given the same reference numerals and description thereof will be omitted.

[0049] The difference from the information processing device 1 shown in FIG. 4 is that the first sound collecting unit 13, the second sound collecting unit 14, and the sensor unit 15 are not provided, and an analysis unit 20b is added. For convenience of explanation, the CPU 1a included in the server device 20 will be referred to as CPU 20a as shown in the figure.

[0050] The analysis unit 20b is configured as, for example, a signal processor or an AI (Artificial Intelligence) processor, and performs various analytical processes related to the contents of the interview based on the recorded data (sound pickup signal) of the interview. Specifically, the analysis unit 20b in this example performs a text conversion process to convert the spoken portion of the recorded data into text to obtain spoken text data, and also performs at least the summary creation process, main point list creation process, and question and answer identification process described above.

[0051] Here, the analysis process by the analysis unit 20b may be executed by the CPU 20a, particularly when the analysis process is performed as a rule-based process rather than an AI process.

[0052] The server device 20 is not limited to being configured as a single computer device as shown in Fig. 5, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0053] <3. Display-related processing as an embodiment> As mentioned above, it is assumed that an interview will take place in a public space such as a coffee shop or a lobby that is not isolated from people other than the participants, and if processing is to be performed only on the text data of the participants' speech, it is necessary to perform processing to distinguish between speech by the participants and speech by people other than the participants. However, it is relatively difficult to perform such a discrimination process by analyzing recorded data of the interview, and it is currently considered difficult to put it into practical use.

[0054] Therefore, it is conceivable to display the speech text data of all persons whose speech sounds have been converted into text, and to allow the user to specify from there the speech of persons who are participants or persons who are not participants.

[0055] However, in a limited display space, it is not efficient to display the speech text data of all people, including non-participants in the interview. In particular, in an environment where there are many non-participants, it is expected that the speech text data of more non-participants than the number of participants will have to be displayed, which is undesirable.

[0056] In consideration of the above circumstances, the embodiment aims to make effective use of limited display space while enabling speech identification of participants / non-participants based on user operation by displaying speech text data.

[0057] FIG. 6 is a functional block diagram for explaining various functions as an embodiment for achieving the above object. As mentioned above, in this example, the direction guidance display in the information processing device 1 is realized as a function of the recording application, and the recording application is assumed to be provided as a SaaS type service. Therefore, in this example, various functions related to the display processing as an embodiment are realized by software processing of the CPU 20a in the server device 20.

[0058] As shown in the figure, the CPU 20a of the server device 20 has the functions of a number-of-participants setting section F1 and a display processing section F2. The number-of-participants setting section F1 sets the number of participants of the interview based on a user operation.

[0059] FIG. 7 shows an example of a start screen Gk for recording using a recording application. The start screen Gk is displayed on the display unit 7 under the control of the CPU 20a. In this example, the GUI for inputting the number of participants in the interview (hereinafter sometimes abbreviated as "participant number") is displayed on this start screen Gk. Specifically, on the start screen Gk, a number input box bi is displayed as the GUI for inputting the number of participants, and the user can input the number of participants into this number input box bi.

[0060] A recording start button Bs is displayed on the start screen Gk, and the user can instruct the information processing device 1 to start recording by operating the recording start button Bs. In this example, the number of participants input in the number of participants input box bi is finalized in response to the operation of the recording start button Bs. That is, the number of participants setting unit F1 sets the number of participants to the numerical value input in the number of participants input box bi in response to the operation of the recording start button Bs.

[0061] The start screen Gk of this example also displays direction guidance information Id for guiding the direction in which the other-side microphone (the second microphones 14a, 14b in this example) can be pointed toward the other side. As the direction guidance information Id, at least information indicating the side on which the other-side microphone (the second microphones 14a, 14b) is placed is displayed. In the figure, an example is shown in which not only information indicating the side on which the other-side microphone is placed ("other-side microphone presentation information"), but also information indicating the side on which the self-side microphone (the first microphone 13a in this example) is placed ("self-side microphone presentation information") is displayed as the direction guidance information Id. Specifically, in FIG. 7, the other-side microphone presentation information is displayed in an area on the display screen that is closer to the position of second microphones 14a, 14b than the position of first microphone 13a, and the self-side microphone presentation information is displayed in an area on the display screen that is closer to the position of first microphone 13a than the positions of second microphones 14a, 14b, and further, the other-side microphone presentation information includes the words “other side” and the self-side microphone presentation information includes the words “your side”. Here, when both the information presented by the remote microphone and the information presented by the user's microphone are displayed as shown in Fig. 7, the information presented by the remote microphone can be displayed more emphasized than the information presented by the user's microphone. Various examples of the emphasis can be considered. For example, emphasis by color, emphasis by display size such as character size, emphasis by adding a specific figure or pattern, etc. can be considered.

[0062] 6, the display processing unit F2 performs a process of displaying on the display unit 7 spoken text data obtained by converting speech sounds during the interview into text based on a signal picked up by a microphone placed at the interview venue. In particular, the display processing unit F2 of the embodiment performs a process of displaying on the display unit 7 only spoken text data for the number of people set by the number-of-participants setting unit F1. The display processing unit F2 in this example sets the set number of people for which speech text data is to be displayed, based on the speech volume of each person. Participants in an interview are expected to be located fairly close to the microphone that picks up the speech, so by setting the speech volume as described above, people who are likely to be participants can be set as the people to be displayed with speech text data.

[0063] The specific display processing by the display processing unit F2 will be described below. In this example, when the display processing unit F2 first sets the above-mentioned number of people to be displayed in the speech text data after the interview begins, it sets people whose speech volume is equal to or greater than a predetermined reference volume as people to be displayed in the order in which their speech is detected.

[0064] The initial setting process for such a set number of persons to be displayed will be described with reference to Figs. 8 to 11.

[0065] First, as a premise, in this example, while the interview is being recorded, the recorded data is transmitted from the information processing device 1 to the server device 20. In other words, the recorded data of the interview is transmitted from the information processing device 1 to the server device 20 in approximately real time. Then, the analysis unit 20b of the server device 20 in this example performs a process of converting the spoken portion of the recorded data received from the information processing device 1 into text.

[0066] Furthermore, the analysis unit 20b of this embodiment performs a voiceprint analysis on the speech portion of the recorded data and associates the speech portion with the voiceprint, i.e., performs a so-called speaker separation process. The speaker separation process here refers to the process of identifying which speaker each part of a speech is from. In the speaker separation process, the voiceprints detected as a result of performing voiceprint analysis on the speech parts of the recorded data are linked to speaker IDs. For this speaker ID, an ID such as speaker A, speaker B, speaker C, etc. is assigned according to the detected voiceprint. This type of speaker separation process allows a speaker ID to be assigned (associated) with each spoken part of the recorded data, making it possible to identify the correspondence between the spoken part and the speaker for the spoken text data obtained by converting the recorded data into text. Note that voiceprint analysis is not essential for speaker separation processing. For example, if an interview is not only recorded but also video recorded, the person who is speaking and the timing of the speech can be identified based on the recorded image, for example, by whether or not the mouth is moving, so the speaker can be identified by temporally matching the speech portion detected in the recorded data with the speech timing detected in the image.

[0067] Based on the above premise, the initial setting process for a person to be displayed will be described. After the interview begins, when the set number of people to be displayed is less than the number set by the participant number setting unit F1, each time the analysis unit 20b detects the speech of a new person (voiceprint), the display processing unit F2 determines whether the speech volume, which is the volume of the speech part, is equal to or greater than the standard volume, and if the speech volume is equal to or greater than the standard volume, sets the new person as a person to be displayed. Here, the speech volume may be an average value of the volume in the target speech portion, or a maximum value of the volume in the target speech portion.

[0068] The display processing unit F2 sets the persons to be displayed based on the judgment result using the reference volume as described above until the number of persons to be displayed reaches the number set by the number-of-participants setting unit F1. In this example, the number set by the number-of-participants setting unit F1 is set to "4" on the premise that the number of participants in the interview is four, as illustrated in FIG. 3 and the like.

[0069] 8 to 11 show examples of display of utterance text data on the display screen of the display unit 7 when four display target persons are set in order after the start of the interview. 8 shows a display example in response to the first display target person being set. In this example, the speech text data of speaker A is displayed in response to the first person whose speech is detected after the start of the interview having a speech volume equal to or higher than the reference volume, that is, the first person detected is set as the display target person.

[0070] FIG. 9 shows an example of the display when a second person to be displayed has been set, FIG. 10 shows an example of the display when a third person to be displayed has been set, and FIG. 11 shows an example of the display when a fourth person to be displayed has been set, which is the maximum number of people. Here, an example is shown of a case where the speech volumes of the person whose speech was detected second, third, and fourth after the start of the interview are all above the reference volume, and these people are set in order as people to be displayed.In this case, the transition of the display content on the display unit 7 is as shown in the transition from Figure 9 to Figure 11, where the speech text data of speaker B, the speech text data of speaker C, and the speech text data of speaker D are gradually added and displayed. In addition, Figure 11 shows an example in which speaker A's utterance text data is displayed between speaker C's utterance text data and speaker D's utterance text data, in response to speaker A's second utterance after speaker C's utterance and before speaker D's first utterance.

[0071] By performing the above-described initial setting process for display target persons, it is possible to determine the set number of people who are estimated to be highly likely to be participants in terms of speech volume as display target persons in the order in which their speech is detected. Therefore, the set number of people to be displayed can be set with a certain degree of accuracy and as quickly as possible.

[0072] In this example, when speech from more people than the number set by the participant number setting unit F1 is detected, the display processing unit F2 ranks the people based on the speech volume of each person, and sets the top number of people as the people to be displayed in the speech text data. Specifically, after the display processing unit F2 performs the initial setting process of the display target person as described above, every time a speech of a new person other than the display target person is detected, the display processing unit F2 performs ranking based on the speech volume for all people whose speech has already been detected. Then, as a result of this ranking, the top number of people set as a result of this ranking, that is, the top n people when the number of people set by the number-of-participants setting unit F1 is n, are set as the display target person of the speech text data.

[0073] FIG. 12 is an explanatory diagram of setting of persons to be displayed based on the ranking of speech volume. FIG. 12A illustrates a state in which, after four speakers A, B, C, and D have been set as persons to be displayed in the initial setting, a new utterance by a person designated as speaker E has been detected. In this case, the speech volume for each speaker is assumed to be speaker A=10, speaker D=8, speaker B=7, speaker C=5, and speaker E=7, as shown in the figure.

[0074] 12B is an explanatory diagram of the ranking in this case, and as shown in the figure, the ranking results in this case are speaker A=1st place, speaker D=2nd place, speaker B=3rd place, speaker E=4th place, and speaker C=5th place. In this case, the speech volume of the newly detected speaker E exceeds the speech volume of speaker C who was set as the person to be displayed, so the person to be displayed is replaced.

[0075] As described above, by resetting the display target person according to the ranking based on the speech volume after the initial setting of the display target person, if the person who speaks for the first time after the initial setting is a participant, the person can be appropriately set as the display target person. In other words, in this respect, the setting accuracy of the display target person is improved.

[0076] Furthermore, the display processing unit F2 in this example accepts a non-participant designation operation that designates a person other than the participants of the interview from among the persons to be displayed, and excludes the person designated by the non-participant designation operation from the persons to be displayed. Furthermore, the display processing unit F2 of this example accepts a participant designation operation that designates a person as a participant in the interview from among the persons to be displayed, and sets the person designated by the participant designation operation as the person to be displayed regardless of the speech volume.

[0077] FIG. 13 is an explanatory diagram of an example of a GUI for performing the non-participant designation operation and participant designation operation as described above. In this example, the non-participant designation operation and the participant designation operation can be performed on a selection window Ws as exemplified in the figure. Specifically, in this example, when a user wishes to perform a non-participant designation operation or a participant designation operation, the user performs an operation to select a display portion of the utterance text data of a person to be designated as a designation target on a display screen of the utterance text data of each speaker (the utterance text data of each person set as a display target person) displayed in real time during recording on the display unit 7. Then, a selection window Ws for performing a non-participant designation operation or a participant designation operation is displayed near the display portion of the selected utterance text data. Note that specific examples of the operation for selecting the display portion of the spoken text data include a long press on the display portion, a double click (double tap) operation, and a right click operation when using a mouse.

[0078] As shown in the figure, the selection window Ws has the options "Important Person," "Remove," and "Cancel" available. The "important person" item is an item for designating a selected person as an important person. An important person here means a participant in the interview who the user considers important. Therefore, the "important person" item can be said to be an item for designating a person as a participant in the interview from among the displayed persons. The "Exclude" item is an item for specifying that the selected person is to be excluded from the persons to be displayed. In this case, it is assumed that a person other than the participants in the interview is mistakenly set as a person to be displayed and that the person is to be excluded, so the "Exclude" item corresponds to an item for specifying a person other than the participants in the interview from among the persons to be displayed. The "cancel" item is an item for instructing cancellation of the non-participant designation operation and the participant designation operation.

[0079] When an operation is performed to select the "Important Person" item, the display processing unit F2 sets the selected person (the person whose displayed portion of the speech text data is selected) as the person to be displayed regardless of the speech volume, specifically, in this example, regardless of the result of ranking based on the speech volume. As will be understood from the previous explanation, in this example, the process of setting the persons to be displayed according to the ranking based on speech volume is executed each time the speech of a new person is detected, but when an "important person" is designated, the display processing unit F2 sets the "important person" as the person to be displayed in the subsequent process of setting the persons to be displayed, regardless of the result of the ranking based on speech volume.

[0080] By setting "important people" as described above, it becomes unnecessary to carry out a process for ranking people as participants. Therefore, the processing load required for identifying utterances of participants / non-participants can be reduced.

[0081] Furthermore, when an operation for selecting the "exclusion" item is performed, the display processing unit F2 excludes the selected person from persons to be displayed. This makes it possible to correct the person to be displayed based on a user operation when there is an error in the person to be displayed that has been set based on the speech volume. By performing this type of correction until all participants are set as people to be displayed, it becomes possible to identify the speech of all participants and to distinguish between participants and non-participants.

[0082] Here, when a person currently set as a display target person is removed from the display target persons, replacement of the display target persons occurs. In this case, the display processing unit F2 selects a person (substitute person) to be newly set as a display target person in place of the person to be removed from the display target persons, according to the ranking based on the speech volume. Specifically, for example, it is considered to select the person with the highest ranking among the people who were not set as display target persons as the substitute person. In addition, the display processing unit F2 excludes a person who has been designated as an "excluded" person from the targets of ranking based on speech volume. This makes it possible for a person who is a participant and has a relatively low speech volume and has not been set as a display target person to be more likely to be set as a display target person.

[0083] Furthermore, when the "Cancel" item is selected, the display processing unit F2 performs processing to close the selection window Ws.

[0084] Here, the display processing unit F2 in this example excludes from the persons whose speech sounds have been detected those whose speech overlap rate with respect to a reference person set according to a predetermined criterion is equal to or greater than a predetermined threshold value from the persons to be displayed in the speech text data, or makes them more likely to be excluded from the persons to be displayed.

[0085] Here, it is assumed that a person who is estimated to be a participant is set as the reference person. In this example, the display processing unit F2 sets the person who is ranked high (for example, first) based on the speech volume among the people set as the display target people as the reference person. Alternatively, when a person is designated as a participant by the above-mentioned participant designation operation, the display processing unit F2 sets the person designated by the participant designation operation as the reference person.

[0086] Then, the display processing unit F2 calculates the speech overlap rate with respect to the reference person for the person currently set as the display target person (or for all people whose speech sounds have been detected from the recorded interview data). In other words, it calculates the rate at which the speech overlaps. In conversations between participants, there tends to be relatively little overlap in speech, but since conversations between non-participants are conducted independently of the conversations of participants, it can be said that the speech overlap rate with participants is relatively high. Therefore, a person whose speech overlap rate with respect to the reference person is equal to or greater than a predetermined threshold value is excluded from the display target persons, or is made more likely to be excluded from the display target persons. This can reduce the possibility that a person other than a participant is mistakenly set as the person to be displayed.

[0087] Here, like the case of a person designated as "excluded" above, a person who is excluded based on the speech overlap rate is also excluded from the ranking based on the speech volume. This makes it possible to prevent a person who is estimated to be a non-participant from being erroneously set as a person to be displayed.

[0088] In addition, specific processes that may be considered to make a person more likely to be excluded from display include, for example, when ranking based on speech volume, applying a negative offset to the calculated speech volume, or applying a ranking offset that lowers the ranking by a specified position.

[0089] Further, the display processing unit F2 in this example performs processing to make it easier for a person who has uttered a remote party's speech sound to be set as a display target person for utterance text data than for a person who has uttered a self party's speech sound. For clarity, as mentioned above, the speech sound from the other party means the speech sound from a sound source located on the side where the other party is located as viewed from the participant on the first side; specifically, in this example, it refers to the speech sound picked up by the second microphones 14a, 14b, which are the microphones on the other side. In addition, the speech sound from your side means the speech sound from a sound source located on the side where the your side participant is located as viewed from the other participant, and specifically, in this example, it refers to the speech sound picked up by the first microphone 13a, which is the microphone on your side.

[0090] Generally, in an interview, the statements made by the other party are considered more important than those made by oneself. Therefore, by making it easier for a person on the other side to be set as a display target person as described above, it is possible to make it easier for a person who makes an important statement to be set as a display target person, which is suitable for performing speech content analysis such as summarization on a display target person as described later.

[0091] Here, specific methods for making it easier for a person who has made a speech sound from the other party to be set as a person to be displayed could, for example, be to give a positive offset to the calculated speech volume when ranking the person based on the speech volume, or to give a ranking offset that raises the ranking by a specified number.

[0092] Note that, here, it is assumed that the process of making it easier for a person who has emitted a speech sound from the other party to be set as a display target person is performed when setting a display target person based on ranking performed after the initial setting of the display target person, but it is also possible that the process of making it easier for a person who has emitted a speech sound from the other party to be set as a display target person is performed when the initial setting of the display target person is performed. For example, the display target person may be set in order starting from the person whose speech volume is equal to or higher than the reference volume and who has emitted a speech sound from the other party.

[0093] Next, a specific example of a processing procedure for implementing the display-related processing according to the embodiment described above will be described with reference to the flowcharts of FIGS. In this example, the processes shown in FIGS. 14 to 16 are executed by the CPU 20a of the server device 20 based on a program stored in a predetermined storage device such as the storage unit 9 or the ROM 2.

[0094] 14 is a flowchart of a basic process for setting a person to be displayed, in which a person to be displayed is set according to a ranking based on the speech volume. This basic process is started, for example, when recording of an interview is started.

[0095] First, the CPU 20a waits for either the detection of a new person or the end of recording by the processes of steps S101 and S102. In step S101, in this example, the determination as to whether or not a new person has been detected is performed by determining whether or not a new speaker ID has been assigned in the speaker separation process of the analysis unit 20b. In this example, recording is ended based on an operation to end recording by the user, and the determination process in step S102 is performed as a process to determine whether or not the operation to end recording has been performed.

[0096] When it is determined in step S101 that a new person is detected, the CPU 20a proceeds to step S102 and determines whether the set number of people is equal to or greater than the set number of people. That is, it determines whether the number of people set as persons to be displayed is equal to or greater than the number of participants set based on an input operation to the number input box bi.

[0097] If it is determined in step S103 that the set number of people is not equal to or greater than the set number of people, the CPU 20a proceeds to step S104 and determines whether or not the speech volume of the newly detected person is equal to or greater than the reference volume. If the speech volume of the newly detected person is equal to or greater than the reference volume, the CPU 20a proceeds to step S105, sets the newly detected person as a person to be displayed, and returns to step S101. On the other hand, if the speech volume of the newly detected person is not equal to or greater than the reference volume, the CPU 20a skips the process of step S105 and returns to step S101. This allows for the implementation of an initial setting process in which, after the start of an interview, a set number of people whose speech volume is equal to or greater than the reference volume are set.

[0098] On the other hand, if it is determined in step S103 that the number of people being set is equal to or greater than the set number, the CPU 20a proceeds to step S106 and ranks the detected persons based on their speech volumes. That is, the CPU 20a ranks each person whose speech portion has been detected based on the speech volume. Then, in the next step S107, the CPU 20a sets the number of people equal to the set number of top ranking people as display target people, and returns to step S101. As a result, after the above-mentioned initial setting process is performed, each time a new person is detected, a person to be displayed is set according to the ranking based on the speech volume.

[0099] In response to determining in step S102 that recording has ended, the CPU 20a ends the series of processes shown in FIG.

[0100] 15 is a flowchart of a process for setting a person to be displayed based on a speech overlap rate. This process is started after the initial setting process for the person to be displayed is performed and after the above-mentioned reference person is set.

[0101] In step S201, the CPU 20a calculates the speech overlap rate between the reference person and each person within the display range other than the reference person. Here, the person within the display range means a person currently set as a person to be displayed.

[0102] In step S202 following step S201, the CPU 20a determines whether or not there is a person whose speech overlap rate is equal to or greater than a predetermined threshold value. When it is determined that there is a person whose speech overlap rate is equal to or greater than the predetermined threshold, the CPU 20a excludes the person from the persons to be displayed in step S203. Here, as described above, when a person currently set as a display target person is removed from the display target people, the display target people are replaced. In this case, the CPU 20a selects a substitute person to be newly set as a display target person in place of the person to be removed from the display target people, according to the ranking based on the speech volume. For example, the CPU 20a selects the person with the highest ranking among the people who were not set as display target people as the substitute person. As described above, in this example, a person who is excluded from the persons to be displayed is also excluded from the persons to be ranked based on the speech volume.

[0103] In step S204 following step S203, the CPU 20a performs a rank update process, that is, a process of moving up the ranks of people lower than the excluded person.

[0104] The CPU 20a ends the series of processes shown in FIG. 15 in response to the ranking update process in step S204.

[0105] Moreover, if the CPU 20a determines in step S202 that there is no person whose speech overlap rate is equal to or greater than the predetermined threshold, it skips steps S203 and S204 and ends the series of processes shown in FIG.

[0106] It is also possible that the process shown in Fig. 15 is performed multiple times on the time axis. For example, the process shown in Fig. 15 may be performed at predetermined time intervals, or each time a person to be displayed is set according to the ranking based on the speech volume (each time the process of step S107 is performed).

[0107] Fig. 16 is a flowchart of the process related to the participant designation operation and non-participant designation operation described above. The process of Fig. 16 is started in response to the start of displaying the utterance text data on the display unit 7 after the start of recording. The process shown in Fig. 16 is repeated until the end of recording.

[0108] In step S301, the CPU 20a waits for an operation to instruct display of the selection window Ws. Specifically, in this example, the CPU 20a waits for an operation to select a display portion of any of the utterance text data being displayed on the display unit 7.

[0109] If an instruction to display the selected window Ws has been given, the CPU 20a proceeds to step S302, performs a display process for the selected window Ws, and proceeds to step S303.

[0110] The CPU 20a waits for any of the VIP setting operation, the exclusion setting operation, and the cancel operation through the processes of steps S303, S304, and S305. That is, the CPU 20a waits for any of the selection operation of the above-mentioned "VIP" item displayed in the selection window Ws, the selection operation of the "Exclusion" item, and the selection operation of the "Cancel" item.

[0111] If it is determined in step S303 that a VIP setting operation has been performed, the CPU 20a proceeds to step S306, and performs VIP setting processing for the relevant person, i.e., processing for setting the person whose display portion of the speech text data has been selected as the person to be displayed regardless of the speech volume, and then ends the series of processes shown in Figure 16. In this example, the process of setting a person to be displayed according to the ranking based on speech volume is executed each time a new person's speech is detected, but as described above, if an "IMPORTANT PERSON" is specified, the "IMPORTANT PERSON" is set as the person to be displayed in the process of setting a person to be displayed thereafter, regardless of the result of the ranking based on speech volume. In the process of step S306, a setting process is performed so that the "IMPORTANT PERSON" is continuously set as the person to be displayed.

[0112] Also, if it is determined in step S304 that an exclusion setting operation has been performed, the CPU 20a proceeds to step S307, and performs a process of excluding the person from the display targets, i.e., a process of excluding the person for whom the display portion of the speech text data has been selected, from the persons to be displayed, and then ends the series of processes shown in Figure 16. In this case, too, the substitute person to be set as the person to be displayed in place of the excluded person is selected according to a ranking based on the speech volume, for example, by selecting the person with the highest ranking among the people who were not set as people to be displayed. Furthermore, the CPU 20a excludes a person designated as an "excluded" person from the ranking based on the speech volume. This makes it possible for a person who is a participant and has a relatively low speech volume and has not been set as a display target person to be more likely to be set as a display target person.

[0113] Moreover, if it is determined in step S305 that a cancel operation has been performed, the CPU 20a performs a process of closing the selection window Ws in step S308, and ends the series of processes shown in FIG.

[0114] <4. Steps for creating minutes and example of display screen> The above mainly describes the process of recording an interview using a recording application and displaying spoken text data based on the recorded data. Although not illustrated, in this example, after recording starts, the aforementioned stop recording button is displayed in the GUI of the recording application, and the user can instruct the end of recording by operating the stop recording button. In this example, in response to the operation to end recording, a participant information registration screen Gr as shown in FIG. 17 is displayed as the GUI of the recording application.

[0115] As shown in FIG. 17, the registration screen Gr has input boxes b1, b2, and b3 for inputting the company name, name, and job title of the customer Hc (client) in the business negotiation, and an input box b4 for inputting the name of the sales representative Hs himself. Input boxes b1, b2, and b3 for customer Hc can be added by operating an add button B1 in the figure, making it possible to input information for a plurality of customers Hc. The registration screen Gr also has a registration button B2 for issuing an instruction to register input information.

[0116] In this registration screen Gr, the sales representative Hs inputs the company name, name, and job title of the customer Hc, as well as his or her own name, and operates the registration button B2. For confirmation, in this example, the reason why participant information is registered in response to recording is to display the participant information on a screen (recording list screen Gi) showing a list of recording data, which will be described later.

[0117] It is also possible to register participant information before the start of recording.

[0118] In addition, the CPU 1a of the information processing device 1 performs processing to transmit to the server device 20 the interview participant information registered via the registration screen Gr as described above, specifically, the name and job title information of the customer Hc, and the name information of the sales representative Hs.

[0119] In the server device 20, the CPU 20a associates the participant information transmitted from the information processing device 1 with the recorded interview data also transmitted from the information processing device 1 (in this example, transmitted approximately in real time during recording) and stores it in a specified storage device, such as the memory unit 9. The server device 20 assigns a negotiation ID, which serves as an identifier for the negotiation, to each recording data of the negotiation.

[0120] In the server device 20, an analysis process for analyzing the contents of the business negotiation is carried out based on the recorded data of the business negotiation. This analysis process is carried out by the above-mentioned analysis unit 20b.

[0121] As described above, the analysis unit 20b performs a process of converting the audio recording data of the business negotiation into text as a process related to the analysis of the contents of the business negotiation. Here, the technology for converting audio data as recorded data into text, that is, the technology for converting audio data into text data, is a known technology, and detailed description thereof will be omitted here.

[0122] The analysis unit 20b also performs the above-mentioned speaker separation process based on the recorded data.

[0123] Furthermore, the analysis unit 20b also performs the above-mentioned summary creation process, main point list creation process, and question and answer identification process based on the text data obtained by converting the audio recording data of the business negotiation into text.

[0124] In addition, in the server device 20 in this embodiment, the CPU 20a performs a group determination process to determine which of multiple groups a speaker in a business negotiation belongs to, based on the recording data of the business negotiation (the audio pickup signals of each audio pickup channel) recorded (stored) in the server device 20. In this embodiment, the speech sound of the participant on the other side is picked up by the microphone on the other side, and the speech sound of the participant on your side is picked up by the microphone on your side, so the speaker whose speech sound is picked up by the second microphones 14a, 14b can be determined to be a speaker in the other side group, and the speaker whose speech sound is picked up by the first microphone 13a can be determined to be a speaker in the own side group.

[0125] Here, in the minutes-taking support system 100, it is possible to present a list of recording data of business negotiations stored in the server device 20 to the user as the sales representative Hs. Specifically, in this example, it is assumed that the above-mentioned recording application is provided with a function for accessing such a list of recorded data. In other words, in this case, the user as the sales representative Hs can be presented with a recording list screen Gi, which is a list screen of recorded data, by operating an access button (not shown) to the recording data list displayed on the screen of the recording application.

[0126] FIG. 18 shows an example of the recording list screen Gi. As shown in the figure, the recording list screen Gi displays, for each business negotiation ID, date and time information of the business negotiation, a play button Bp, participant information, and an input memo field. In this example, the date and time information of the business negotiation includes the start time (recording start time) and end time (recording end time). The playback button Bp is a button for issuing an instruction to play back the recording data of the business negotiation associated with the business negotiation ID. The participant information is displayed as the information entered on the registration screen Gr described above. If there are any items on the registration screen Gr that have not been entered, a message to that effect is displayed. In the input memo field, memo information after the business negotiation can be input as text. The information input in the input memo field is stored in association with the business negotiation ID.

[0127] In addition, the recording list screen Gi can also display links to data on materials used in business negotiations (see "Main Materials" in the figure).

[0128] On the recording list screen Gi, a minutes creation button B3 for instructing the start of software as a minutes editor is displayed for each business negotiation ID. The user as the sales representative Hs operates the minutes creation button B3 displayed on the recording list screen Gi for the business negotiation for which the minutes are to be created.

[0129] In response to an operation of the minutes creation button B3, a GUI of a minutes editor is displayed in the information processing device 1. The minutes editor is software that displays the above-mentioned editing screen Ge, i.e., a screen for creating minutes including an analysis result display area and a text editing area, and performs processing related to creating minutes.

[0130] In this example, the minutes editor is also assumed to be of the SaaS type, that is, the software is stored on the server device 20 side and the software is provided to the user of the information processing device 1 as a service. In this case, the processing for creating minutes using the editing screen Ge, such as display control processing for determining what information to display on the editing screen Ge and processing for generating text data as minutes in response to operational input on the editing screen Ge, will be performed by the server device 20 (e.g., CPU 20a).

[0131] In the above example, the recording application is given the ability to access the recording list screen Gi, but access to the recording list screen Gi can be done by specifying the URI (Uniform Resource Identifier) ​​of the recording list screen Gi, and any software can be used.

[0132] Also, although an example in which the recording application and the minutes editor are separate applications have been given here, it is also possible for the functions of the recording application and the minutes editor to be provided by the same application.

[0133] In response to the operation of the minutes creation button B3, the CPU 20a of the server device 20 performs a process of displaying an editing screen Ge shown in FIG.

[0134] The editing screen Ge shown in FIG. 19 is the editing screen Ge before an instruction to execute an analysis process is given. As will be described later, in the editing screen Ge of this example, a display area for analysis result text data (analysis result display area Aa, which will be described later) is displayed on the screen in response to an instruction to execute analysis processing.

[0135] The editing screen Ge in Figure 19 includes an utterance text display area At, a recording period display area Ap, a speaker / group setting area Ah, and a text editing area Ae, as well as an input box b5, a play button Bp, copy (transcription) buttons B10 and B11, an analysis execution button B13, a speaker setting instruction button B14, an output button B15, and a clipboard copy button B16.

[0136] As shown in the figure, in the editing screen Ge of this example, the speech text display area At, the speaker / group setting area Ah, and the text editing area Ae are arranged from left to right on the screen in the following order: utterance text display area At, speaker / group setting area Ah, text editing area Ae. In this example, the recording period display area Ap is disposed below the spoken text display area At.

[0137] The spoken text display area At is an area in which text data of spoken portions in an interview are displayed for each speaker. As described above, in this embodiment, the text data of the speech portion is generated by the server device 20 (the analysis unit 20b) converting the recorded data of the interview into text.

[0138] In this example, the analysis unit 20b divides the speech portion into a predetermined block unit and manages it. The block unit here is, for example, a unit of the duration of the speech. In other words, the timing when the speech is interrupted is recognized as the break of the block. As described above, the analysis unit 20b in this example also performs speaker separation processing based on the recorded data, and links each detected voiceprint to a speaker ID, such as speaker A, speaker B, or speaker C. Furthermore, processing is also performed to identify speech sections for each speaker in the recorded data, based on the voiceprint information for each speaker. By identifying speech sections for each speaker in the recorded data in this manner, it is possible to identify the correspondence between speech portions and speakers for the spoken text data obtained by converting the recorded data into text.

[0139] The CPU 20a displays the text data of the speech part for each speaker in the speech text display area At based on the text data of each speech part obtained by the speaker separation processing in the analysis unit 20b and information indicating the correspondence between each speech part and the speaker ID. In this example, the text data of the speech portions of each speaker are displayed in chronological order from the top to the bottom of the screen. In this example, the speech text display area At displays speaker icons as identifiers of speech parts for each speaker. At the stage of Fig. 19, speaker icons display speaker IDs such as speakers A, B, and C that are assigned in the speaker separation process.

[0140] In this example, the spoken text display area At displays only the spoken text data of the person set as the display target person by the display-related process as the embodiment described above. Specifically, the CPU 20a displays only the spoken text data of the person set as the display target person at the time when the recording ends in the spoken text display area At.

[0141] Furthermore, in this example, the result of the speaker separation process is reflected in the utterance text display area At, and the utterance text data of the participant on one side and the utterance text data of the participant on the other side are displayed close to different ends of the area. Specifically, in the editing screen Ge of this example, the utterance text data of the participant on one side is moved close to one of the left and right ends of the utterance text display area At, and the utterance text data of the participant on the other side is moved close to the other left and right end of the utterance text display area At. Note that the figure shows an example in which the speech text data of the participant on one side is shifted to the right end of the speech text display area At, and the speech text data of the participant on the other side is shifted to the left end of the speech text display area At, but the shifting direction may be reversed.

[0142] Here, it is also conceivable that the non-participant designation operation described above can be accepted after the fact on a screen that displays the spoken text data to the user after the interview recording is completed, such as the editing screen Ge. In that case, a GUI capable of accepting a non-participant designation operation such as the aforementioned selection window Ws may be displayed in the utterance text display area At of the editing screen Ge, for example, in response to a selection operation of the display portion of the utterance text data.

[0143] In response to the non-participant designation operation in this case, the CPU 20a performs a process of excluding the person designated by the non-participant designation operation from the persons to be displayed. In this case, the CPU 20a performs a process of displaying the speech text data of a substitute person for the excluded person (for example, also in this case, selected based on the result of ranking based on the speech volume) in the speech text display area At. This makes it possible to correct the person to be displayed after the interview has been recorded.

[0144] In addition, in the editing screen Ge, the recording period display area Ap is an area for displaying recording period display information that graphically represents the recording period of the interview. As shown in the figure, in the recording period display area Ap in this example, information showing the recording period of the interview in the form of a bar extending in the left-right direction is displayed as recording period display information. In the recording period display area Ap, a seek bar Sb for indicating the playback position of the recording data is displayed in association with the recording period display information. This seek bar Sb can be moved left and right, and the user can operate the play button Bp after moving the seek bar Sb to instruct the information processing device 1 to play the recording data from the playback position indicated by the position of the seek bar Sb.

[0145] In this embodiment, the recording period display information displays information showing the distribution of the speech period for each speaker. Specifically, the recording period display information in this case displays a bar indicating the recording period for each speaker, and displays the information on the speech period for each speaker in a different display format for each speaker. For example, it is possible to display the color or pattern of the speech period for each speaker differently. In this example, the bar for each speaker indicating the recording period displays the speaker icon of the corresponding speaker.

[0146] By displaying information showing the distribution of speaking periods for each speaker as recording period display information as described above, the user can easily grasp information such as which speaker spoke for how long during what period during the interview, and the amount of speech each speaker made, allowing the user to easily understand which speaker's speech they should pay attention to when creating the minutes. In addition, by displaying the speech period information for each speaker in a different display format for the recording period display information, the user can intuitively identify the speech period for each speaker. When a user wants to check the recording data for a specific speech portion of a specific speaker during the process of creating minutes, the ease of searching for that speech portion can be improved.

[0147] In addition, in the recording period display area Ap of this embodiment, the negotiation process period information Is is displayed in association with the recording period display information. Here, the negotiation process period information Is is information indicating the name and period of each negotiation process when the negotiation is divided into processes according to a predetermined criterion.

[0148] The analysis unit 20b in this embodiment performs a process of identifying negotiation steps in a negotiation based on the audio recording data of the negotiation. The negotiation steps referred to here are stages in the progress of the negotiation, and examples of such information include "ice-breaking (small talk)", "needs analysis", "proposal and explanation", and "negotiation". Specifically, the analysis unit 20b performs a process of identifying the negotiation process based on text data obtained by converting the audio recording data of the negotiation into text. This process may be performed by an AI model generated by performing machine learning such as deep learning using the text data as learning input data and the correct answer data of the negotiation process as teacher data. Alternatively, it may be realized as a rule-based process in which frequently occurring words for each negotiation process are predetermined as process identification keywords, and the negotiation process is identified based on the results of matching with the process identification keywords.

[0149] The CPU 20a performs display processing of the negotiation process period information Is based on the period information (information indicating the start time and end period of a process) for each negotiation process obtained as a result of the process of identifying the negotiation process by the analysis unit 20b. The figure shows an example of the negotiation process period information Is when five processes, ST1 to ST5, are identified as the negotiation process.

[0150] By displaying the negotiation process period information Is, i.e., the chronological information of the negotiation process, in association with the recording period display information as described above, when a user wishes to check the recording data of a specific speech portion of a specific speaker during the process of creating minutes, it is possible to improve the ease of searching for that speech portion.

[0151] In the above, an example of the recording period display information in which the recording period is displayed in the shape of a bar has been given, but the recording period may be displayed in other shapes such as an arc. When the recording period is displayed in the shape of an arc, a rotational operator may be used as the playback position indication operator, rather than a sliding operator such as the seek bar Sb.

[0152] The speaker / group setting area Ah is an area for setting the name of each speaker and the group for each speaker. Group setting here means setting which group each speaker belongs to, with respect to two groups: the self-group, which is the group on the sales representative Hs (user)'s side, and the other-side group, which is the group on the sales representative Hs's other side (i.e., the customer Hc's side).

[0153] In this example, since the group determination process described above has already been performed before the editing screen Ge is displayed, the speaker / group setting area Ah reflects the results of the group determination process and displays speakers belonging to the other party's group and speakers belonging to the user's group separately, as shown in the figure. Specifically, the speaker / group setting area Ah in this example is provided with an area for displaying speakers belonging to the other party's group and an area for displaying speakers belonging to the user's group, and the icon of the corresponding speaker is displayed in each area. Here, the speakers displayed in the speaker / group setting area Ah are only those persons who are set as persons to be displayed.

[0154] In the editing screen Ge of this example, when the user wishes to set the speaker's name and group, he or she operates the speaker setting instruction button B14. Then, a setting screen Gs for setting speakers and groups as shown in FIG. 20 is displayed on the speaker / group setting area Ah. As shown in the figure, the setting screen Gs is provided with an input box b6 for inputting the speaker's name, a selection operator s1 for selecting one's own group / other party group, a save button B20, and a cancel button B21. Here, an example of the setting screen Gs for speaker B is shown, but in this example, in response to the operation of the speaker setting instruction button B14 described above, the setting screen Gs is displayed in order starting from speaker A, and the user can input name information in the input box b6 in order starting from speaker A, set a group by operating the selection operator s1 as necessary, and further perform a save operation using the save button B20, thereby inputting the name and setting a group for each speaker. When the cancel button B21 is operated on the setting screen Gs, the CPU 20a ends the acceptance of the setting operation of the speaker name and group, and performs processing to return to the screen display state of FIG. 19, for example. Furthermore, the method of setting the name and group for each speaker is not limited to the above method, and other methods may be adopted, such as allowing arbitrary selection of speakers to be set.

[0155] Here, in this embodiment, since group determination (determination of the other party's group / your own group) has already been performed for each speaker by the group determination process described above, it is not essential to have the user perform group setting operations using the setting screen Gs as described above. In this case, it is sufficient to have the user input and set the name information for each speaker on the setting screen Gs. By allowing the user to perform group setting operations while reflecting the results of the group determination process, even if an error occurs in the group determination process, it is possible to correct the error.

[0156] FIG. 21 shows an example of display information in the speaker / group setting area Ah when the name and group of each speaker are set on the setting screen Gs described above. As shown in the figure, the set name information is displayed for each speaker. Also, as shown in Figure 19 above, in the speaker / group setting area Ah, before the name of each speaker is set, information such as A, B, C, etc. indicating the speaker ID is displayed as the speaker icon, but after the name is set, the speaker icon displays the initial letter of the name, as shown in Figure 21.

[0157] In this example, the settings of the name and group for each speaker in the speaker / group setting area Ah are also reflected in the speech text display area At. Specifically, as in the editing screen Ge shown in FIG. 22, the speaker icon for each speaker in the spoken text display area At is changed to a notation showing the initials of the set name.

[0158] Furthermore, in the spoken text display area At, if a group setting different from the result of the group determination process is performed by the user through a group setting operation for each speaker, a display reflecting the group setting is performed.

[0159] Here, in the speech text display area At, the speech text data of the other party and the user is displayed separately, so that the user can intuitively distinguish between the utterances of the user's group and the other party's group. In particular, when the interview is a business meeting as in this embodiment, the user as the sales representative Hs can refer to the utterances of the customer Hc as the other party's group all at once, which increases convenience.

[0160] In this example, the setting information of the name of each speaker in the speaker / group setting area Ah is also reflected in the display information in the recording period display area Ap. Specifically, the display of the speaker icon displayed in each bar for each speaker will be changed from the speaker ID to the initials of the speaker's name.

[0161] In the edit screen Ge, the analysis execution button B13 is a button for instructing the execution of analysis processes related to the contents of the interview, such as the creation of a summary, etc. Specifically, in this example, it is a button for instructing the execution of the summary creation process, the main points list creation process, and the question and answer identification process.

[0162] As described above, in this example, the analysis processing such as the summary creation processing, the main point list creation processing, and the question and answer identification processing are performed by the analysis unit 20b. Therefore, when the analysis execution button B13 is operated, the CPU 20a issues an instruction to the analysis unit 20b to execute the analysis processing. At this time, the CPU 20a issues an instruction to execute analysis processes such as a summary creation process, a main point creation process, and a question and answer identification process, targeting only the person set as the display target person. That is, the analysis unit 20b in this example performs analysis process of the speech content targeting only the person set as the display target person.

[0163] The CPU 20a displays the analysis result text data obtained as a result of the analysis process performed by the analysis unit 20b on the editing screen Ge.

[0164] FIG. 23 shows an example of the analysis result text data displayed on the editing screen Ge. In the editing screen Ge, the analysis result text data is displayed in the analysis result display area Aa. As shown in the figure, the analysis result display area Aa in this example is disposed in a position between the speaker / group setting area Ah and the text editing area Ae in the left-right direction of the screen.

[0165] The analysis result display area Aa is provided with a summary display area A1, a main points list display area A2, and a question and answer display area A3, as well as copy buttons B25, B26, and B27. The summary display area A1 is an area for displaying text data of the summary created in the summary creation process. The main point list display area A2 is an area for displaying text data of the main point list created in the main point list creation process, that is, text data for each main point. The question and answer display area A3 is an area for displaying text data of the question portion and text data of the answer portion identified in the question and answer identification process.

[0166] The copy button B25 is a button for instructing to copy the text data displayed in the summary display area A1 to the text editing area Ae. In the summary display area A1 of this example, it is possible to perform a range designation operation of the text data, and the user can designate the range of the text data to be copied in the summary display area A1 and then operate the copy button B25 to instruct to copy the text data of the designated range to the text editing area Ae.

[0167] In the gist list display area A2, a check box cb for enabling selection of text data on a gist unit basis is provided for each gist. Further, a copy button B26 is provided for the main point list display area A2. The user can check any one of the check boxes cb for each key point and then operate the copy button B26 to instruct the text data of the key point corresponding to the checked check box cb to be copied to the text editing area Ae.

[0168] In the present example, the text data for each main point is arranged in the main point list display area A2 in the chronological order of the original utterance. Specifically, in the present example, the text data for a main point whose original utterance is older is arranged higher in the main point list display area A2. In this main point list display area A2, the main points can be arranged in chronological order for each group, i.e., your own group and the other party's group. In the figure, the main points list for your own group and the other party's group are displayed in separate columns.

[0169] In the question and answer display area A3, text data showing the question portion and text data showing the answer portion are displayed. In the question and answer display area A3 of this example, text data showing the question portion and text data showing the answer portion are arranged in the chronological order of the original utterances.

[0170] The copy button B27 is a button for instructing copying of the text data in the question and answer display area A3 to the text editing area Ae. In question and answer display area A3, it is possible to perform an operation to specify the text data of the question portion and answer portion to be copied. After performing an operation to specify the range of the text data of the question portion and answer portion to be copied in question and answer display area A3, the user can issue an instruction to copy the specified text data to text editing area Ae by operating copy button B27.

[0171] In the question and answer display area A3 of this example, the text data showing the question portion and the text data showing the answer portion are displayed so that it is possible to identify whether the comment is made by the user's group or the other party's group. Specifically, in this example, the question and answer portions of your group are displayed in text data toward the right edge of question and answer display area A3, and the question and answer portions of the other group are displayed in text data toward the left edge of question and answer display area A3. This makes it easier for a user to understand specific questions or answers from his or her own or the other party when creating minutes of a meeting. As illustrated in the figure, it is also possible to provide separate columns in the question and answer display area A3 for displaying questions from your side and the other person's answers to those questions, and for displaying questions from the other person's side and your own answers to those questions.

[0172] Next, the text editing area Ae will be described. The text editing area Ae is an area in which the analysis result text data in the analysis result display area Aa can be transcribed and the transcribed analysis result text data can be edited. The user can use this text editing area Ae to create minutes of the interview. Since the analysis result text data in the analysis result display area Aa, such as a list of key points or a summary, can be copied and used in the sentences that make up the minutes, there is no need to input the sentences for creating the minutes from scratch, and the efficiency of creating the minutes can be improved.

[0173] In the text editing area Ae in this embodiment, if the transcribed analysis result text data is corrected, the corrected text portion is displayed in a different display mode from the uncorrected text portion. Although not shown, if the analysis result text data copied from the analysis result display area Aa is corrected by a user operation in the text editing area Ae, for example, the corrected text portion is displayed in a different color from the uncorrected text portion. Alternatively, only the corrected text portion may be displayed in bold or underlined.

[0174] In addition, in this embodiment, when the transcribed analysis result text data is corrected in the text editing area Ae, the CPU 20a performs a process of recording the text data before and after correction in a memory device in association with each other. The pre-correction / post-correction text data is recorded in a memory device within the server device 20, such as the storage unit 9 of the server device 20.

[0175] Here, the text data before and after the correction in the text editing area Ae can be used for training the minutes creator. Specifically, for example, by having a low performer who is inexperienced in minutes creation refer to the text data before and after the correction by a high performer who is skilled in minutes creation, it becomes possible for the low performer to learn the skills related to minutes creation, which can contribute to the development of the low performer creator. In particular, if the interview is a business meeting, it becomes possible for the low performer to learn the skills of a high performer sales representative, which can contribute to the improvement of the sales representative's skills. In addition, the above-mentioned pre- and post-correction text data can also be used to improve and enhance the functionality of the analysis process for obtaining analysis result text data from the recorded interview data. Specifically, for example, when the analysis process is performed using an AI model, the pre- and post-correction text data can be used as learning data for machine learning to improve and enhance the analysis process. Or, even when the analysis process is performed using rule-based processing, the pre- and post-correction text data can be referred to in updating the analysis process, which can contribute to improving and enhancing the functionality of the analysis process.

[0176] In the editing screen Ge, an output button B15 and a clipboard copy button B16 are provided as operators related to the text editing area Ae. The output button B15 is a button for instructing output of the text (text as minutes) created in the text editing area Ae in a predetermined file format such as Word. The clipboard copy button B16 is a button for instructing copying of the text created in the text editing area Ae to the clipboard. By providing the output button B15 and the clipboard copy button B16, the user can save the created text data as the minutes in the information processing device 1 in a predetermined file format.

[0177] Furthermore, in the editing screen Ge of this example, an upload button B17 and a submission button B18 are also provided as operators related to the text editing area Ae. The upload button B17 is a button for instructing uploading of the text created in the text editing area Ae to a cloud server of a predetermined sales information management service (for example, Salesforce: registered trademark). For example, when the upload button B17 is operated, a dialog is displayed, and when the user specifies his / her company name on the dialog and performs an operation to instruct uploading, the text created in the text editing area Ae is uploaded to the above-mentioned cloud server.

[0178] The post button B18 is a button for instructing to post the text created in the text editing area Ae as shared information in a specified sales communication service (e.g., Slack: registered trademark). For example, when the post button B18 is operated, a dialog is displayed, and when the user designates a channel related to information sharing on the dialog and performs an operation to instruct to execute posting, the text created in the text editing area Ae is posted as shared information in the designated channel in the above-mentioned sales communication service.

[0179] In addition, in the editing screen Ge of this example, a copy button B10 and a copy button B11 are also provided as operators for issuing a copy instruction to the text editing area Ae. The copy button B10 is a button for instructing to copy the text data in the speech text display area At to the text editing area Ae. The copy button B11 is a button for instructing to copy the speaker name information and group setting information set in the speaker / group setting area Ah to the text editing area Ae. By providing these copy buttons, the efficiency of creating minutes can be further improved.

[0180] Here, in the editing screen Ge in this embodiment, it is possible to change the group information that has already been set in the speaker / group setting area Ah. Specifically, as shown in Fig. 24, the group information that has already been set can be changed by dragging the display part of the speaker whose group setting is to be changed in the speaker / group setting area Ah. The illustrated example shows an example in which the group setting of a speaker that belonged to the user's group is changed so that the speaker now belongs to the other party's group.

[0181] In this embodiment, when the group setting is changed as described above, the display position of the utterance text data in the utterance text display area At is changed according to the changed group setting information. Specifically, according to the changed group setting information, the display position of the utterance text data for the speaker whose group setting has been changed is switched between left edge alignment and right edge alignment (see the transition from FIG. 24 to FIG. 25).

[0182] Although not shown in the drawings, when the group setting is changed, the question and answer display area A3 of the analysis result display area Aa may also be similarly switched between left edge alignment and right edge alignment.

[0183] Although not specifically mentioned above, it is conceivable that the summary creation process, the main points list creation process, and the question and answer identification process can each be realized as a process using an AI model. For example, it is conceivable that the summary creation process can use an AI model trained by machine learning using deep learning or the like so that text data can be used as input data and a summary can be obtained as output data. Similarly, it is conceivable that the main points list creation process and the question and answer identification process can use an AI model trained by machine learning using deep learning or the like so that text data can be used as input data and a main points list, question portion, and answer portion can be obtained as output data.

[0184] Alternatively, the summarization process, the main point list creation process, and the question and answer identification process may be realized as rule-based processes. For example, the summary creation process may be realized by a processing method of identifying unnecessary sentences in spoken language and deleting the unnecessary sentences. Also, the main point list creation process may be realized by a processing method of dividing the summary created in the summary creation process according to a predetermined rule, for example by dividing it by periods, and identifying each divided sentence as a main point. Also, the question / answer identification process may be realized by a processing method of identifying the question part in the utterance based on the recorded data, on the premise that the ending of the question part is raised, and identifying the part of the utterance next to the question part as the answer part.

[0185] <5. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, in the above, it was explained that when initially setting people to be displayed, a set number of people are set in the order in which their speech sounds are detected, but instead, it is also possible to set a set number of people in the order of loudest speech volume.

[0186] In addition, although the above description refers to continuing to set a person set as an "important person" as a person to be displayed regardless of the speech volume, priority processing from another perspective can also be performed on a person set as an "important person". For example, when a speech section of an "important person" overlaps with that of another person, it is possible to give priority to converting the speech of the "important person" to text. In this case, the conversion to text can be performed by generating a signal in which the speech components of the other person are removed from the audio signal of the overlapping speech section, and using this signal as the target.

[0187] Also, for a person set as an "important person," the volume of speech may be monitored, and an alert may be issued if the volume of speech falls below a predetermined volume. In this case, the alert may be accompanied by the presentation of information encouraging the "important person" to move closer to the information processing device 1 (a voice recording device for an interview). By issuing the above-mentioned alert, it is possible to prevent the speech volume of a person set as an "important person" from decreasing, which would make it impossible to perform analytical processing such as generating speech text data and creating summaries.

[0188] It is also possible to accept changes to the number of participants in an interview once it has been set. When the number of participants is changed while the speech text data for the previously set number of participants is being displayed, the speech text data for the new number of participants may be displayed. In this case, the people to be displayed may be selected according to the ranking based on the speech volume. Specifically, when the new number of participants is m, the top m people may be set as the people to be displayed. The change in the number of participants can be accepted during recording, or on a screen that displays the speech text data after recording, such as the above-mentioned editing screen Ge. The change in the number of participants during recording is preferably accepted when the information processing device 1 is not a smartphone but a device such as a notebook PC whose screen is difficult for the other participants to see.

[0189] In addition, in the above example, the start of recording of an interview is triggered by operating the recording start button Bs, but it is also possible that recording can start automatically when the information processing device 1 is placed on a support such as a table. Alternatively, recording of the interview may start automatically when a recording app is launched. In this case, it is assumed that there will be a period of silence before the interview starts after the recording starts. In this case, the recording data for the period of silence before the interview starts may be automatically deleted.

[0190] Also, when recording starts, it is possible to notify the participants that recording is about to start. For example, this notification can be performed by playing and outputting a message sound indicating that recording is about to start in response to the operation of the recording start button Bs. By notifying in this way, the other participants can be made aware that recording is about to start.

[0191] Regarding the end of recording, an example was given in which the recording stop button displayed as a GUI was operated, but various patterns regarding the end of recording are conceivable. For example, the change in the orientation of the information processing device 1 to a vertical orientation (approximately upright orientation) could be used as a trigger to display a message such as "Is the interview over?" along with "Yes" and "No" buttons on the display unit 7, and recording could end when the "Yes" button is pressed.

[0192] Also, it is considered that recording may be ended in response to the operation of a predetermined physical button (for example, a power button, etc.) on the information processing device 1. In this case, when the predetermined physical button is operated, a question screen may be displayed asking whether or not to end recording. While asking whether to end recording, recording may be automatically paused. At this time, a message such as "Recording is paused. Do you want to end recording?" may be displayed along with "Yes" or "No" buttons, in which case recording ends when the "Yes" button is pressed. Alternatively, the operation to end recording in this case may be performed by pressing a specific physical button again.

[0193] In addition, in the explanation so far, an example has been given of the interview being a business negotiation, but interviews to which the technology of the present invention can be applied include various opportunities for people to communicate with each other, such as business negotiations, as well as interviews in the fields of medicine, counseling, lectures, speeches, education, and entertainment.

[0194] In the above explanation, the server device 20 (CPU 20a) performs all of the processes described as the number-of-participants setting unit F1 and the display processing unit F2, but the information processing device 1 (CPU 1a) may perform some or all of the processes. In this case, the information processing device 1 may store a program for implementing the processes to be executed by the information processing device 1.

[0195] In the above example, the microphone used to record the voice during the interview is a directional microphone, but it is also possible to use an omnidirectional microphone. Even when an omnidirectional microphone is used, it is possible to distinguish between the self-side speech sound and the other party's speech sound based on the characteristics of the recorded voice signal. For example, if microphones placed at least in different positions are used as the other party's microphone and the self-side microphone, a difference will occur in the characteristics of the recorded voice signals of the self-side and other party's speech sounds, and it will be possible to distinguish between the self-side speech sound and the other party's speech sound based on this difference. In addition, when distinguishing between your own speech sounds and the other party's speech sounds, it is possible to identify the positions of your own and the other party's sound sources using beamforming technology with an array microphone, and the method is not limited to using the other party's microphone and your own microphone.

[0196] Also, during recording (interview), it is conceivable to perform support processing related to the progress of the interview.As a specific example, on the premise that the required items to be discussed during the interview are set in advance, a process is performed based on the converted text data to determine whether or not all the required items have been discussed, and if there is a required item that has not been discussed, a support process is performed to notify the user of that fact. Alternatively, a support process may be performed in which, in response to a specific user operation during recording, review information on the topic of the topic is presented to the user based on the text data of the recorded data up to that point. In this case, the review information may be presented by playing back a voice reading out the topic of the topic. By carrying out the interview support process as described above, it is possible to make the interview more meaningful.

[0197] <6. Summary of the embodiment> As described above, the display processing device (server device 20, or information processing device 1) as an embodiment includes a participant number setting unit (F1) that sets the number of participants in the interview based on user operation, and a display processing unit (F2) that performs processing to display on the display unit (7) spoken text data obtained by converting the speech sounds in the interview into text based on a signal picked up by a microphone placed at the interview venue, and the display processing unit displays on the display unit only the spoken text data for the number of participants set by the participant number setting unit. According to the above configuration, when the speech of any of the set number of people is displayed and the user designates the speech of any of the people as a non-participant speech, the speech of another person is displayed instead, and the user can designate the speech of the other person again as whether it is a participant or not. By repeating such processing, it is finally possible to identify the speech of all participants, and as a result, it is possible to identify the speech of participants / non-participants. And, according to the above configuration, the speech of people other than the number set by the user is not displayed, so that the limited display space can be used effectively. Therefore, by displaying the speech text data, it is possible to distinguish between speech of participants and non-participants based on user operations, while also making effective use of the limited display space. In addition, while the interview is in progress, it is difficult for the user to check the speech content of all people for which speech text data is being generated and specify the speech of participants / non-participants. However, with the above configuration, it is sufficient for the user to specify the speech of participants / non-participants from the speech displayed which is narrowed down to the number of participants, thereby reducing the operational burden on the user when having the device identify the speech of participants / non-participants.

[0198] Furthermore, in the display processing device as the embodiment, the display processing unit sets the set number of people for which the spoken text data is to be displayed, based on the speech volume of each person. Participants in an interview are expected to be located fairly close to the microphone that picks up the speech, so by setting the speech volume as described above, people who are likely to be participants can be set as the people to be displayed with speech text data.

[0199] Furthermore, in a display processing device as an embodiment, when the display processing unit sets the number of people to be displayed in the speech text data for the first time after the interview begins, it sets people whose speech volume is equal to or greater than a predetermined reference volume as people to be displayed in the order in which their speech is detected. This makes it possible to determine the set number of people who are estimated to be highly likely to be participants in terms of their speech volume as display target people in the order in which their speech was detected. It becomes possible to set the set number of people to be displayed as quickly as possible with a certain degree of accuracy, and it is possible to shorten the time required to identify the speech of participants / non-participants.

[0200] Furthermore, in a display processing device as an embodiment, when speech from at least more people than the number set by the participant number setting unit is detected, the display processing unit ranks each person based on the speech volume of each person, and sets the top number of people as the people to be displayed in the speech text data. There are rare circumstances in which a non-participant is closer to the microphone than the participant. Therefore, by setting the set number of display target persons according to the ranking based on the speech volume as described above, it is possible to increase the possibility that a person who is a participant will be set as a display target person.

[0201] In addition, in a display processing device as an embodiment, the display processing unit accepts a non-participant designation operation that designates a person other than the participants in the interview from among the persons to be displayed, and excludes the person designated by the non-participant designation operation from the persons to be displayed. This makes it possible to correct the person to be displayed based on a user operation when there is an error in the person to be displayed that has been set based on the speech volume. By performing this type of correction until all participants are set as people to be displayed, it becomes possible to identify the speech of all participants and to distinguish between participants and non-participants.

[0202] Furthermore, in a display processing device as an embodiment, the display processing unit accepts a participant designation operation that designates a person as a participant in the interview from among the persons to be displayed, and sets the person designated by the participant designation operation as the person to be displayed regardless of the speech volume. This eliminates the need to perform a process for ranking people as participants. Therefore, the processing load required for identifying utterances of participants / non-participants can be reduced.

[0203] Furthermore, in a display processing device as an embodiment, the display processing unit excludes from among the people from whom speech sounds have been detected, people whose speech overlap rate with a reference person set according to a predetermined criterion is equal to or greater than a predetermined threshold value, from the people to be displayed in the speech text data, or makes them more likely to be excluded from the people to be displayed. It is assumed that the reference person is a person who is presumed to be a participant, such as a person who has a loud voice or a person designated by the user as a participant. Since a person who has a high speech overlap rate with such a reference person is likely to be a person other than a participant, it is possible to reduce the possibility that a person other than a participant will be mistakenly set as a display target person by excluding the person from the display target person or making it easier to exclude the person from the display target person as described above. By making it less likely that a person other than a participant will be mistakenly set as the person to be displayed, the need for the user to take action to point out the mistake is reduced, thereby reducing the operational burden on the user when having the device distinguish between participant and non-participant speech.

[0204] Furthermore, in a display processing device as an embodiment, the participants in the interview include a self-side participant including the user, and an opposite-side participant who is the interview partner of the self-side participant, and the speech sounds in the interview are picked up so that the opposite-side speech sounds, which are speech sounds from a sound source located on the side where the opposite-side participant is located as viewed from the self-side participant, and the self-side speech sounds, which are speech sounds from a sound source located on the side where the self-side participant is located as viewed from the opposite-side participant, can be distinguished, and the display processing unit makes it easier for a person who has emitted the opposite-side speech sounds to be set as a person to be displayed for speech text data than a person who has emitted the self-side speech sounds. Generally, in an interview, the statements made by the other party are considered more important than those made by oneself. Therefore, by making it easier for a person on the other side to be set as a display target person as described above, it is possible to make it easier for a person who makes an important statement to be set as a display target person, which is suitable for performing speech content analysis such as summarization on a display target person.

[0205] Furthermore, the display processing device according to the embodiment includes an analysis unit (20b) that performs analysis processing of the speech content only for the person set as the person to be displayed in the speech text data. This makes it possible to prevent unnecessary analysis processing from being performed on people who are presumed to be non-participants.

[0206] A display processing method as an embodiment includes a computer device performing a number-of-participants setting process for setting the number of participants in the interview based on user operation, and a display process for displaying spoken text data on a display unit, the spoken text data being obtained by converting the speech sounds in the interview into text based on a signal picked up by a microphone placed at the interview venue, and in the display process, only the spoken text data for the number of participants set by the number-of-participants setting process is displayed on the display unit. With such a display processing method, it is possible to obtain the same functions and effects as those of the display processing device according to the above-described embodiment.

[0207] The program as the embodiment is a program that causes a computer device such as the CPU 20a to execute all or part of the display-related processing as the embodiment described above with reference to FIGS. In other words, the program as an embodiment is a program readable by a computer device, and causes the computer device to execute a number-of-participants setting process for setting the number of participants in the interview based on user operation, and a display process for displaying on a display unit spoken text data obtained by converting the speech sounds in the interview into text based on a signal picked up by a microphone placed at the interview venue, and in the display process, the program causes the computer device to realize a process for displaying on the display unit only the spoken text data for the number of participants set by the number-of-participants setting process.

[0208] Such a program can realize a computer device that realizes the functions of the number-of-participants setting unit F1 and the display processing unit F2 described in the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, or the like, the personal computer can be made to function as a computer device that performs processing as the number-of-participants setting unit F1 and the display processing unit F2.

[0209] Such a program can be pre-recorded in a HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU, etc. Also, the program can be provided by being temporarily or permanently recorded on a removable recording medium such as a memory card or an optical disk. Such programs can also be downloaded from a download site via a network such as a LAN or the Internet. [Explanation of symbols]

[0210] 1. Information processing device 20 Server device Hs Sales Representative Hc customer 1a,20a CPU 2 ROM 3 RAM 6 Input section 7 Display section 13 First sound collection section 13a, 13b First microphone 14 Second recording section 14a, 14b Second microphone 20b Analysis Department F1 Number of participants setting section F2 Display processing section Id Direction Information Gk start screen bi Number of people input box Bs Recording start button Ws Selection window Gi Recording list screen B3 Minutes creation button Ge Editing Screen At Speech text display area Ah Speaker / Group Settings Area Ae Text Editing Area Aa Analysis result display area

Claims

1. A participant number setting unit that sets the number of participants of the interview based on a user operation; a display processing unit that performs processing to display on a display unit speech text data obtained by converting speech sounds during the interview into text based on a signal picked up by a microphone disposed at the interview venue, The display processing unit is Regarding the display of the utterance text data, only the utterance text data for the number of people set by the number-of-participants setting unit is displayed on the display unit. Display processing device.

2. The display processing unit is The set number of people for whom the spoken text data is to be displayed are set based on the speech volume of each person. The display processing device according to claim 1 .

3. The display processing unit is After the interview starts, when setting the number of people to be displayed as the spoken text data for the first time, people whose speech volume is equal to or greater than a predetermined reference volume are set as the people to be displayed in order of detection of the speech. The display processing device according to claim 2 .

4. The display processing unit is When speeches of at least more people than the number set by the participant number setting unit are detected, ranking is performed based on the speech volume of each person, and the top set number of people are set as people to be displayed with the speech text data. The display processing device according to claim 2 .

5. The display processing unit is A non-participant designation operation is accepted for designating a person other than the participants of the interview from among the persons to be displayed, and the person designated by the non-participant designation operation is excluded from the persons to be displayed. The display processing device according to claim 4.

6. The display processing unit is A participant designation operation for designating a person as a participant of the interview from among the display target persons is accepted, and the person designated by the participant designation operation is set as the display target person regardless of the speech volume. The display processing device according to claim 4.

7. The display processing unit is Among the persons whose speech sounds have been detected, a person whose speech overlap rate with a reference person set according to a predetermined criterion is equal to or greater than a predetermined threshold is excluded from the persons to be displayed in the speech text data, or is made more likely to be excluded from the persons to be displayed. The display processing device according to claim 4.

8. The participants of the interview include a self-side participant including a user and a counterpart-side participant who is the other party in the interview with the self-side participant, The speech sounds in the interview are collected so that the other party's speech sounds, which are speech sounds from a sound source located on the side where the other party is present as viewed from the participant on the other side, and the participant's own speech sounds, which are speech sounds from a sound source located on the side where the participant on the other side is present as viewed from the participant on the other side, can be distinguished from each other party's speech sounds, The display processing unit is The person who has uttered the other party's speech sound is more likely to be set as a display target person for the speech text data than the person who has uttered the self-side speech sound. The display processing device according to claim 2 .

9. An analysis unit that performs analysis processing of the contents of speech only on persons set as display target persons of the speech text data.

9. The display processing device according to claim 2.

10. A computer device comprising: A participant number setting process for setting the number of participants of the interview based on a user operation; a display process for displaying on a display unit spoken text data obtained by converting spoken sounds during the interview into text based on a signal picked up by a microphone disposed at the interview venue; In the display process, Regarding the display of the utterance text data, only the utterance text data for the number of people set by the number-of-participants setting process is displayed on the display unit. Display processing method.

11. A computer readable program, A participant number setting process for setting the number of participants of the interview based on a user operation; a display process for displaying on a display unit speech text data obtained by converting speech sounds during the interview into text based on a signal picked up by a microphone disposed at the interview venue, and In the display process, Regarding the display of the utterance text data, a process is realized in which only the utterance text data for the number of people set by the number-of-participants setting process is displayed on the display unit. program.

12. A recording medium having a computer-readable program recorded thereon, A participant number setting process for setting the number of participants of the interview based on a user operation; a display process for displaying on a display unit speech text data obtained by converting speech sounds during the interview into text based on a signal picked up by a microphone disposed at the interview venue, and In the display process, Regarding the display of the utterance text data, a program for causing the computer device to realize a process of displaying only the utterance text data for the number of people set by the number-of-participants setting process on the display unit is recorded. Recording medium.

Images