Control device, control method, program, recording medium

Abstract

To reduce burden of a user for pointing a sound pickup device in an appropriate direction when recording an interview.SOLUTION: A control device includes: a plurality of microphones arranged at different positions of a device body; and a display part for displaying information. One of the microphones is an interviewee's microphone to record interviewee's utterance at an interview, and the other is an interviewer's microphone to record interviewer's utterance. The display part has a display processing unit which displays direction guide information showing a direction in which the interviewee's microphone can be directed towards the interviewee.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a control device and method, a program and a recording medium, and in particular to a control technique for a sound collection device used for recording an interview. [Background technology]

[0002] Conventionally, the content of interviews has been widely recorded. By recording the content of an interview as audio data, the content of the interview can be saved without the need to take notes on the content of the interview. There is also known technology for converting recorded interview data into text and displaying it, and even for analyzing the contents of the interview, such as creating a summary of the interview based on the converted text (see Patent Document 1 below). Furthermore, voiceprint analysis technology can be used to perform speaker separation of spoken utterances based on recorded data. Speaker separation here means identifying which of the participants in a conversation has spoken a given utterance.

[0003] As a related prior art, the following Patent Document 2 can also be mentioned. The following Patent Document 2 discloses a technology for recording an interview using a smartphone and creating a summary based on the recorded data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7223469 [Patent Document 2] Patent Publication No. 2022-87521 Summary of the Invention [Problem to be solved by the invention]

[0005] A device with multiple microphones can be used as a recording device for recording an interview. For example, by using one microphone to record the other person's speech and the other microphone to record one's own speech, it becomes easy to group the recorded data (speech data) into those that are from the other group and those that are from one's own group.

[0006] Here, when using a sound collection device equipped with multiple microphones to record an interview, currently there is no means of guiding the user as to which microphone should be appropriately pointed in which direction. This requires the user to remember which microphone should face which side, which places a burden on the user.

[0007] The present invention has been made in consideration of the above circumstances, and aims to reduce the burden on a user when recording an interview by pointing a sound collection device in an appropriate direction. [Means for solving the problem]

[0008] The control device of the present invention is a sound collection device equipped with a plurality of microphones arranged at different positions on the device body and a display unit for displaying information, one of the plurality of microphones being designated as an other-side microphone for picking up the other person's speech in an interview, and the other being designated as a self-side microphone for picking up the self-side speech in the interview, and the control device is equipped with a display processing unit for processing the display unit to display direction guidance information guiding the other-side microphone in the direction in which it can be pointed toward the other person. This makes it possible to guide the user as to which microphone to point in which direction when using a sound collection device equipped with multiple microphones to record an interview. Effect of the Invention

[0009] According to the present invention, it is possible to reduce the burden on the user when recording an interview by pointing the sound collection device in an appropriate direction. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of a minutes-taking support system including a control device according to an embodiment of the present invention; [Diagram 2] 1 is a perspective view showing a schematic external appearance example of a sound collecting device configured as a smartphone. [Diagram 3] FIG. 13 is a diagram for explaining an example of placement of a sound pickup device when recording an interview. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a sound collection device according to an embodiment. [Diagram 5] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a server device according to an embodiment. [Figure 6] FIG. 2 is a functional block diagram showing functions of a control device according to an embodiment. [Figure 7] FIG. 11 is a diagram showing an example of a display of direction guidance information. [Figure 8] 13 is a diagram illustrating an example of a state in which a sound collection device is placed on a support surface when recording an interview. [Figure 9] 11 is an explanatory diagram of an example of information used for determining a direction change in the embodiment. [Figure 10] FIG. 13 is a diagram showing an example of an alert display. [Figure 11] 13 is a flowchart of a process related to a display processing unit. [Figure 12] 13 is a flowchart of a process related to a placement time processing unit. [Figure 13] 13 is a flowchart of a process related to a direction change response processing unit. [Figure 14] 13A and 13B are diagrams showing a display example of correction direction guidance information. [Figure 15] 13 is a diagram showing an example of a registration screen for interview participant information in an embodiment. FIG. [Figure 16]FIG. 13 is a diagram showing an example of a recording list screen in the embodiment. [Figure 17] FIG. 13 is a diagram showing an example of an editing screen (before speaker / group setting and analysis processing are performed) in the embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a setting screen for setting speaker groups in the embodiment. [Figure 19] 13 shows an example of information displayed in the speaker / group setting area after information has been set. FIG. [Figure 20] FIG. 13 is a diagram showing an example of an editing screen (after setting speakers and groups) in the embodiment. [Figure 21] 13 is a diagram showing an example of display of analysis result text data on an editing screen. FIG. [Figure 22] 11A and 11B are diagrams illustrating an example of a change operation for set group information. [Diagram 23] 23 is a diagram illustrating an example of a display state of the spoken text display area in response to the change operation shown in FIG. 22 being performed. FIG. [Figure 24] 1 is an explanatory diagram of a problem that occurs when the sound collection device 1 is disposed offset. [Diagram 25] FIG. 11 is a diagram showing a display example of movement suggestion information. [Figure 26] FIG. 13 is a functional block diagram showing an example of a functional configuration of a control device as a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described in the following order. <1. System configuration example> <2. Device configuration example> <3. Guidance method for placement of sound pickup devices during recording> <4. Steps for creating minutes and example of display screen> <5. Modifications> <6. Summary of the embodiment>

[0012] In the embodiment, it is assumed that an interview is recorded using a sound collection device according to the present invention. In addition, it is assumed that the recorded data of the interview is used to create minutes of the interview.

[0013] <1. System configuration example> FIG. 1 is a block diagram showing an example of the configuration of a minutes-taking support system 100 including a control device according to an embodiment of the present invention. The minutes-taking support system 100 according to the embodiment includes at least a sound collection device 1 and a server device 20.

[0014] The sound collection device 1 is a sound collection device equipped with multiple microphones arranged at different positions on the device body and a display unit that displays information, one of the multiple microphones being designated as an other-side microphone for picking up the speech of the other party during the interview, and the other being designated as a self-side microphone for picking up the speech of the self-side during the interview. In this example, the sound collection device 1 is configured as a smartphone, and the minutes creation support system 100 of this example employs a configuration in which the smartphone is used as a sound collection device for recording the interview.

[0015] FIG. 2 is an oblique view showing an example of the general appearance of a sound collection device 1 configured as a smartphone, where FIG. 2A is an oblique view of the sound collection device 1 seen from the front side, and FIG. 2B is an oblique view of the sound collection device 1 seen from the back (rear) side. Here, the "front" of the sound collection device 1 means the side on which the display surface of the display unit is located. 2A, the display screen of the display unit 7 is disposed on the front side of the sound collection device 1, and a call speaker 8a is disposed above the display screen. A first microphone (microphone) 13a used for NC (noise canceling) is disposed near the call speaker 8a. Further, a second microphone 14a used for collecting voice during a call is disposed on the lower part of the sound collection device 1, specifically, on the lower surface of the sound collection device 1 in this example. In this example, as the second microphone 14a disposed on the lower part of the sound collection device 1, two microphones are disposed to enable stereo sound collection. Here, with respect to the sound collection device 1 configured as a smartphone, the upper and lower parts are concepts in which the side on which the call speaker 8a is arranged is the upper side when the sound collection device is viewed from the front.

[0016] 2B, a camera unit 16 is disposed at one of the left and right ends of the upper part on the back side of the sound collection device 1. The camera unit 16 disposed on the back side allows the user to capture an image of a subject while checking the captured image displayed on the display unit 7. In the sound collection device 1 of this example, a first microphone 13b used for NC is also disposed near the camera unit 16.

[0017] As can be understood from the above description, the sound collection device 1 in this example is provided with first microphones 13a, 13b for NC arranged on the upper part of the device body, and second microphones 14a, 14b capable of collecting stereo sound arranged on the lower part of the device body. In this example, at least one of the first microphones 13a, 13b arranged at the top is defined as the above-mentioned local microphone, and the second microphones 14a, 14b arranged at the bottom is defined as the above-mentioned remote microphone. In this example, the first microphone 13a is defined as the local microphone.

[0018] Here, since the second microphones 14a and 14b are capable of collecting stereo sound, they can be said to be microphones with higher performance than the first microphone 13a (or the first microphone 13b).

[0019] In this example, it is assumed that directional microphones are used as the first microphones 13a and 13b and the second microphones 14a and 14b.

[0020] FIG. 3 is a diagram for explaining an example of placement of the sound collection device 1 when recording an interview. In this embodiment, a "business negotiation" between a sales representative Hs and a customer Hc is taken as an example of an "interview." The sales representative Hs is a person who participates in business negotiations, performs sales activities, and prepares minutes of the business negotiations. The sales representative Hs can be considered as the host in a business meeting, while the customer Hc can be considered as the guest in the meeting. The business negotiation here is premised on a face-to-face format in which a sales representative Hs and a customer Hc actually meet face to face.

[0021] In many cases, business negotiations are conducted in a room such as a conference room, with the sales representative Hs seated on one side of a table or desk and the customer Hc seated on the other side. Here, the participants in the interview are assumed to be multiple sales representatives Hs and multiple customers Hc (specifically, two of each is used as an example), but a business negotiation may be conducted with the participation of at least one sales representative Hs and one customer Hc.

[0022] In this case, the sound collection device 1 is a smartphone used by the sales representative Hs. When recording a business negotiation, the sales representative Hs places the sound collection device 1 on a support surface such as the top surface of a table as shown in the figure. As described above, in this example, the second microphones 14a and 14b are determined as the other party's microphones, so when the sound collection device 1 is placed on the support surface for recording, the second microphones 14a and 14b should be directed toward the customer Hc as the other party. Here, when recording an interview, the object on which the sound collection device 1 is placed is assumed to be a stationary object such as a table.

[0023] 1, the sound collection device 1 and the server device 20 are configured with a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The sound collection device 1 and the server device 20 are capable of mutual data communication via a network NT, which is a predetermined communication network such as the Internet.

[0024] The minutes creation support system 100 of the embodiment has a function of analyzing the contents of the interview based on the recorded data of the interview, and a function of presenting text data showing the results of the analysis (hereinafter referred to as "analysis result text data") to the user on a screen. The user can use the analysis result text data presented on the screen to create minutes of the interview.

[0025] The sound collection device 1 has a function of recording a meeting (a business negotiation in this example). Specifically, the sound collection device 1 records the sound signals collected by the first microphone 13a and the second microphones 14a and 14b, thereby achieving recording of the meeting. The recorded data of the interview is transmitted to the server device 20 via the network NT, and is stored (accumulated) in the server device 20.

[0026] In this embodiment, the function of recording the interview is realized by a "recording application." In this embodiment, the recording application is assumed to be in the form of a so-called SaaS (Software as a Service) type in which software (software program) as the recording application is stored on the server device 20 side and the software is provided as a service. In this case, various processes of the sound collection device 1 related to recording, such as recording of the picked-up signal by the first microphone 13a and the second microphones 14a and 14b, transmission of the recorded data to the server device 20, and display of the direction guidance information Id described later, are executed by the sound collection device 1 under control from the server device 20.

[0027] The server device 20 has a function of converting the recorded data of the interview into text, i.e., a function of generating text data by transcribing (writing down) the spoken portion of the interview, and an analysis function of performing an analysis process related to the contents of the interview based on the text data obtained by the text conversion function (hereinafter referred to as "interview text data"). Specifically, the analysis function in this embodiment is a function of performing a summary creation process that is a process of creating a summary of the interview, a main point list creation process that is a process of creating a list of main points of the interview, and a question and answer identification process that is a process of identifying the question portion and the answer portion of the interview.

[0028] In the minutes creation support system 100, text data showing the analysis results by the analysis function of the server device 20 as described above (hereinafter referred to as "analysis result text data") is displayed on a screen to the sales representative Hs as a user. Specifically, the analysis result text data showing the analysis results as a summary, a list of main points, questions and answers, etc. as described above is displayed on a screen and presented to the user. In this example, the screen display of such analysis result text data is performed on the sound collection device 1 configured as a smartphone.

[0029] The user can efficiently create minutes by using the analysis result text data of the interview presented as described above. Specifically, the user can transcribe (copy) text data such as a summary presented as the analysis result text data and use it to create minutes, which eliminates the need to create minutes from scratch, improving the efficiency of creating minutes.

[0030] In this embodiment, the analysis result text data is presented by displaying one editing screen (editing screen Ge, described later) including an analysis result display area for displaying the analysis result text data and a text editing area for creating minutes. Here, the text editing area is an area in which the analysis result text data can be transcribed and the transcribed analysis result text data can be edited.

[0031] Since an editing screen including not only an analysis result display area but also a text editing area is displayed, it is possible to eliminate the need to open a separate editing screen when creating minutes using the analysis result text data. Therefore, the effort required for the user to create minutes of an interview can be reduced, and the ease of creating minutes can be improved. In particular, when the interview is a business meeting as in this example, the improved ease of creating minutes can improve the efficiency of business negotiation activities and contribute to an improvement in the success rate.

[0032] <2. Device configuration example> FIG. 4 is a block diagram showing an example of the hardware configuration of the sound collection device 1. As shown in FIG. 4, the sound collection device 1 includes a CPU 1a, a ROM 2, and a RAM 3. The CPU 1a functions as an arithmetic processing unit that performs various processes, and executes various processes according to a program stored in the ROM 2 or a program loaded from the storage unit 9 to the RAM 3. The RAM 3 also stores data and the like required for the CPU 1a to execute various processes as appropriate.

[0033] The CPU 1a, the ROM 2, and the RAM 3 are connected to one another via a bus 4. To this bus 4, an input / output interface (I / F) 5 is also connected.

[0034] To the input / output interface 5, an input section 6 including an operator and an operating device is connected. For example, various types of operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. are assumed as the input unit 6. In this example, the touch panel in the input unit 6 is formed on the display screen of the display unit 7, and the user can perform a touch operation on the display screen. An operation by a user is detected by the input unit 6, and a signal corresponding to the input operation is interpreted by the CPU 1a.

[0035] Further, the input / output interface 5 is connected, either integrally or separately, to a display unit 7 formed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 8 formed of a speaker, etc. The display unit 7 is used to display various types of information, and is configured as a display device provided on the housing of the sound collection device 1, for example.

[0036] The display unit 7 executes display of images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 1a. Also, the display unit 7 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), based on instructions from the CPU 1a.

[0037] The input / output interface 5 may be connected to a storage unit 9 and a communication unit 10 . The storage unit 9 is configured with a hard disk drive (HDD) or a solid state drive (SSD) and stores various types of information.

[0038] The communication unit 10 performs communication processing via a transmission path such as the Internet, downloads applications, and performs communication with various devices via wired / wireless communication, bus communication, and the like.

[0039] A drive 12 is also connected to the input / output interface 5 as required, and a removable recording medium 11 such as a memory card or an optical disk is appropriately attached thereto.

[0040] The drive 12 allows data files such as programs used in each process to be read from the removable recording medium 11. The read data files are stored in the storage unit 9, and images and sounds contained in the data files are output on the display unit 7 and the audio output unit 8. In addition, the computer programs and the like read from the removable recording medium 11 are installed in the storage unit 9 as necessary.

[0041] In addition, a first sound collection unit 13 and a second sound collection unit 14 are connected to the input / output interface 5 . The first sound collection unit 13 is a comprehensive representation of the sound collection unit for NC including the first microphones 13a and 13b described above, and is configured to have at least the first microphones 13a and 13b, an amplifier unit that amplifies the collected sound signals, and an A / D conversion unit that A / D converts the amplified collected sound signals. The second sound collection unit 14 is a comprehensive reference to a stereo sound collection unit including the second microphones 14a and 14b described above, and is configured to have at least the second microphones 14a and 14b, an amplifier unit that amplifies the collected sound signals, and an A / D conversion unit that A / D converts the amplified collected sound signals. The audio signals collected by the first audio collection unit 13 and the second audio collection unit 14 are at least temporarily recorded in a storage device such as the memory unit 9, thereby enabling recording of the interview.

[0042] Further, a sensor unit 15 is connected to the input / output interface 5. The sensor unit 15 collectively indicates sensors provided in the sound collection device 1 serving as a smartphone, and the image sensor provided in the camera unit 16 described above is included in this sensor unit 15. The sensor unit 15 also includes an IMU (Inertial Measurement Unit) provided with an acceleration sensor, an angular velocity sensor, an orientation sensor, and the like.

[0043] FIG. 5 is a block diagram showing an example of the hardware configuration of the server device 20. As shown in FIG. In the following description, parts that are similar to parts that have already been described will be given the same reference numerals and description thereof will be omitted.

[0044] The difference from the sound collection device 1 shown in FIG. 2 is that the first sound collection section 13, the second sound collection section 14, and the sensor section 15 are not provided, and an analysis section 20b is added. For convenience of explanation, the CPU 1a included in the server device 20 will be referred to as CPU 20a as shown in the figure.

[0045] The analysis unit 20b is configured as, for example, a signal processor or an AI (Artificial Intelligence) processor, and performs various analysis processes related to the contents of the interview based on the recorded data of the interview. Specifically, in this example, the analysis unit 20b performs at least the summary creation process, the main point list creation process, and the question and answer identification process described above.

[0046] Here, the analysis process by the analysis unit 20b may be executed by the CPU 20a, particularly when the analysis process is performed as a rule-based process rather than an AI process.

[0047] The server device 20 is not limited to being configured as a single computer device as shown in Fig. 5, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0048] <3. Guidance method for placement of sound pickup devices during recording> Here, when using a sound collection device 1 equipped with multiple microphones to record an interview, if there are regulations regarding the microphone on your side and the microphone on the other side, as in this embodiment, the sound collection device 1 should be positioned so that these regulations are observed. Therefore, in this embodiment, the sound collection device 1 is configured to be able to guide the user as to which microphone should be appropriately oriented in which direction.

[0049] FIG. 6 is a functional block diagram for explaining various functions, including such a direction guidance function, according to an embodiment. As mentioned above, in this example, the direction guidance display in the sound collection device 1 is realized as a function of the recording application, and the recording application is assumed to be provided as a SaaS type service. Therefore, in this example, various functions as an embodiment including the direction guidance are realized by software processing of the CPU 20a in the server device 20.

[0050] As shown in the figure, the CPU 20a of the server device 20 has functions as a display processing section F1, a placement time response processing section F2, and a direction change response processing section F3. The display processing unit F1 performs processing to display, on the display unit 7 of the sound collection device 1, direction guidance information Id that guides the direction in which the other party's microphone (in this example, the second microphones 14a, 14b) can be pointed toward the other party. In this example, it is assumed that the direction guidance information Id is displayed in response to the start of a recording application.

[0051] FIG. 7 is a diagram showing a display example of the direction guidance information Id. As the direction guide information Id, information indicating at least the side on which the microphones of the other party (the second microphones 14a, 14b) are located is displayed on the display screen of the display unit 7. 7A shows an example in which, as the direction guidance information Id, not only information indicating the side on which the other-side microphone is arranged (hereinafter referred to as "other-side microphone presentation information") but also information indicating the side on which the own-side microphone (in this example, first microphone 13a) is arranged (hereinafter referred to as "own-side microphone presentation information"). Specifically, in FIG. 7A, the other-side microphone presentation information is displayed in an area on the display screen that is closer to the arrangement positions of second microphones 14a and 14b than the arrangement position of first microphone 13a, and the own-side microphone presentation information is displayed in an area on the display screen that is closer to the arrangement position of first microphone 13a than the arrangement positions of second microphones 14a and 14b, and further, the other-side microphone presentation information includes the characters "other side" and the own-side microphone presentation information includes the characters "own side".

[0052] Here, when both the other party's microphone presented information and the self-side microphone presented information are displayed as shown in Fig. 7A, the other party's microphone presented information can be displayed more emphasized than the self-side microphone presented information. Various examples of the emphasis here are possible. For example, emphasis by color, emphasis by display size such as character size, emphasis by adding a specific figure or pattern, etc. are possible examples.

[0053] Fig. 7B shows an example of displaying only the other party's microphone presentation information as the direction guide information Id. Specifically, Fig. 7B shows an example of displaying the characters "other party" and an arrow indicating the side where the other party's microphone is located (in other words, an arrow indicating the direction to face the other party) as the other party's microphone presentation information.

[0054] In the above, only the example in which the information presented by the microphone on the other side is displayed as the direction guidance information Id has been given, but it is also possible to display only the information presented by the microphone on the own side as the direction guidance information Id. By pointing the local microphone toward the user in accordance with the local microphone presentation information, the other user's microphone is correctly pointed toward the other user's side, so the display of the local microphone presentation information can also guide the user to the correct orientation of the sound collection device 1.

[0055] For confirmation, in the sound collection device 1, the positional relationship of the first microphone 13a and the second microphones 14a, 14b to the display screen of the display unit 7 remains unchanged, so the display process of the direction guidance information Id can be realized as a process of displaying a predetermined image on the display unit 7 in response to the launch of the recording application.

[0056] In Figure 6, the placement response processing unit F2 performs a placement direction determination, which is a determination as to whether or not the sound collection device 1 is placed with the remote microphone facing the remote side, based on the sensor detection signal in the sound collection device 1 when the sound collection device 1 is placed, and controls the display unit 7 of the sound collection device 1 to display a predetermined alert if a negative result is obtained by the placement direction determination.

[0057] To be clear, it is assumed here that the user, who is sales representative Hs, will launch the recording app while holding the sound collection device 1 in his / her hand, and then place the sound collection device 1 on a supporting surface such as a table top and start recording. The placement response processing section F2 performs a response process when the sound collection device 1 is placed after the recording application is started in this manner.

[0058] The placement response processor F2 in this example performs the placement direction determination based on a detection signal of a motion sensor provided in the sound collection device 1 or a detection signal of a touch sensor formed on the display screen of the display unit .

[0059] Figure 8 illustrates the state when the sound recording device 1 is placed on a supporting surface such as the top surface of a table when recording an interview. Figure 8A illustrates the state when the user is holding the sound recording device 1 in preparation for placing it, and Figure 8B illustrates the state after it has been placed.

[0060] During an interview, when placing the sound collection device 1 on a supporting surface such as the top surface of a table, it is considered normal for the user to place the sound collection device 1 not by dropping it vertically onto the supporting surface, but by pointing it out towards the other person, so as to bring it as close as possible to the other person, and in a sense, sliding it towards the other person. Therefore, based on the detection signal of a motion sensor when the sound collection device 1 is placed, specifically, for example, a detection signal from an acceleration sensor, it is determined whether the direction of movement when the sound collection device 1 is placed coincides with the direction on the side where the second microphones 14a, 14b as the counterpart microphones are placed (i.e., the downward direction of the sound collection device 1), thereby making it possible to determine whether the sound collection device 1 is placed with the counterpart microphone facing toward the counterpart side. In the example of Figure 8A, the direction of movement when the sound collection device 1 is placed matches the direction of placement of the first microphone 13a, which is the local microphone, rather than the other party's microphone, so the placement direction determination will yield a negative result.

[0061] Also, when placing the sound collection device 1 as a smartphone on a placement surface such as a table top, it is considered that the user usually supports the base part (the end part facing the user's side) of the sound collection device 1 with his / her fingers, not the tip part (the end part facing the other party's side) (see FIG. 8A). Therefore, based on the detection signal of the touch sensor when the sound collection device 1 is placed, it is possible to determine whether the touch area when the sound collection device 1 is placed is within the area on the side of the display screen where the user's microphone (first microphone 13a) is located (i.e., the upper half of the screen in this example), thereby realizing a determination of whether the sound collection device 1 is placed with the other party's microphone facing the other party's side. In the example of FIG. 8A, the touch area when the sound collection device 1 is placed is the lower half of the upper and lower half of the display screen, so that the placement direction determination returns a negative result.

[0062] In this way, the placement direction can be determined based on at least one of a detection signal from a motion sensor provided in the sound collection device 1 and a detection signal from a touch sensor formed on the display screen of the display unit . It is also possible to determine the placement direction based on both the detection signal of the motion sensor and the detection signal of the touch sensor. For example, a negative result may be considered valid only when a negative result is obtained in both of the determinations based on the respective detection signals.

[0063] When a negative result is obtained from the above-mentioned placement direction determination, the placement time response processor F2 controls the display unit 7 of the sound collection device 1 to display a predetermined alert. In this case, the alert display may be, for example, a message notifying that the placing direction is incorrect, displayed on the display unit 7. As an example, a form may be considered in which, while continuing to display the direction guidance information Id, information notifying that the placing direction is incorrect is additionally displayed (see FIG. 10).

[0064] By carrying out the above-described processing by the placement response processor F2, it is possible to warn the user that recording will be carried out in a state where the microphone on the other side is not correctly facing the other side.

[0065] In addition, in this example, when a negative result is obtained from the placement direction determination, the placement time response processor F2 performs a recording start restriction process for controlling so that recording is not started.

[0066] In this example, the recording of the interview is started when the sound collection device 1 is placed after the recording application is started. In other words, the CPU 20a judges whether the sound collection device 1 is placed after the recording application is started, and performs a process of making the sound collection device 1 start recording in response to a positive result of the judgment. Whether or not the sound collection device 1 has been placed can be determined based on detection signals from an acceleration sensor and an angular velocity sensor that the sound collection device 1 has.

[0067] In this example, when a negative result is obtained from the placement direction determination, the placement time response processor F2 disables the automatic sound recording start function, thereby preventing sound recording from starting.

[0068] Note that the example of the recording start restriction process is not limited to disabling the automatic recording start function. For example, if a recording app displays a recording start button on the display screen to instruct the user to start recording, it is possible to make it impossible to start recording by graying out the recording start button or by changing it to a hidden state.

[0069] By performing the above-described recording start restriction process, it is possible to prevent recording from starting when the other party's microphone is not correctly pointed toward the other party, and it is possible to ensure the accuracy of summaries and minutes when they are prepared based on the recorded data.

[0070] In addition, if the device does not have the function of automatically starting recording in response to placement as described above, the placement direction determination may be performed after recording has started. In other words, a negative result may be obtained in the placement direction determination after recording has started. In this case, the placement response processor F2 does not execute the recording start restriction process, but performs control so as to display the above-mentioned alert.

[0071] In Figure 6, the direction change response processing unit F3 performs a direction change determination, which is a determination of whether or not the orientation of the sound collection device has become incorrect, based on the results of voiceprint analysis of the sound collection signal from the local microphone or the remote microphone after recording has started, and controls the display unit 7 to display a predetermined alert if a positive result is obtained from the direction change determination.

[0072] In order to realize the function of the direction change response processing unit F3, in this example, the recording data is transmitted from the sound collection device 1 to the server device 20 during recording of the interview. That is, the recording data of the interview is transmitted from the sound collection device 1 to the server device 20 in approximately real time. Then, in this example, in response to the start of recording and the recording data being received by the server device 20, the direction change response processing unit F3 generates initial correspondence information indicating the correspondence between the voiceprints detected by the local microphone, the remote microphone, and each microphone at the initial stage after recording has started.

[0073] FIG. 9A shows an example of the initial correspondence relationship information. Here, an example is shown of initial correspondence information for a case in which there are two sales representatives Hs and two customers Hc, and in the initial period after recording begins, voiceprints A and B are detected by the other party's microphone, and voiceprints C and D are detected by the customer's microphone.

[0074] Here, the above-mentioned initial period may be determined as a certain period after the start of recording. Since it is usually expected that each interview participant will introduce himself / herself at the start of the interview, the length of the initial period may be determined so as to include such a self-introduction portion. In this example, the condition for starting recording is that the microphone on the other side is correctly pointed toward the other side due to the recording start restriction function described above, so the accuracy of the initial correspondence information is guaranteed.

[0075] After generating the initial correspondence information as described above, the direction change response processing unit F3 performs a voiceprint analysis on the received recorded data (sound pickup signal), and performs the above-mentioned direction change judgment based on the results of the voiceprint analysis and the initial correspondence information. In this example, the direction change determination is performed in response to the establishment of a predetermined trigger condition (determination execution condition), for example, at regular time intervals after the generation of the initial correspondence information.

[0076] FIG. 9B shows an example of voiceprints for each microphone detected at the timing when a direction change determination is performed after the generation of the initial correspondence information. As shown in the example of FIG. 9B, voiceprints A and C are detected by the other party's microphone, and voiceprints B and D are detected by the own party's microphone.

[0077] According to the initial correspondence information in FIG. 9A, the correct voiceprint detected by the other party's microphone is either voiceprint A or B, and the correct voiceprint detected by the own party's microphone is either voiceprint C or D. 9B is obtained, it can be determined that the orientation of the sound collection device 1 is incorrect because the other party's microphone detects voiceprint C other than voiceprints A and B. It can also be determined that the orientation of the sound collection device 1 is incorrect because the own party's microphone detects voiceprint B other than voiceprints C and D.

[0078] In this example, the direction change response processing unit F3 performs a voiceprint analysis on each of the collected signals of the self-side microphone (first microphone 13a) and the other-side microphone (second microphones 14a, 14b) received as recording data in response to the establishment of the above-mentioned judgment execution condition, and judges whether or not an erroneous voiceprint other than the correct voiceprint indicated by the initial correspondence relationship information is detected for each of the self-side microphone and the other-side microphone. If an erroneous voiceprint is detected in at least one of the self-side microphone and the other-side microphone, a positive result is obtained as the judgment result of the direction change judgment, that is, a judgment result that the direction of the sound collection device 1 has become incorrect, and if an erroneous voiceprint is not detected in either the self-side microphone or the other-side microphone, a negative result is obtained as the judgment result of the direction change judgment, that is, a judgment result that the direction of the sound collection device 1 has not become incorrect.

[0079] Although the above provides an example of determining a change in direction based on the signals picked up by both the local microphone and the remote microphone, as can be understood from the above principle, the change in direction can be determined based on the signals picked up by either the local microphone or the remote microphone.

[0080] The direction change response processing unit F3 controls the display unit 7 of the sound collection device 1 to display a predetermined alert display AL when a positive result is obtained from the above-mentioned direction change determination.

[0081] FIG. 10 shows an example of an alert display AL. As shown in the figure, the alert display AL displays, for example, a message such as "Please correct the orientation", indicating that the orientation of the sound collection device 1 is incorrect, in other words, that the orientation of the sound collection device 1 should be corrected. This makes it possible to display an alert to the user if the sound collection device is moved for some reason after recording has started and is now facing the wrong direction. Therefore, the user can be warned that recording will continue when the microphone on the other side is not correctly pointed at the other side.

[0082] Here, as shown in FIG. 10, in this example, the display unit 7 displays the direction guidance information Id together with the alert display AL. In this way, by displaying the direction guidance information Id on the display unit 7 together with the alert display AL, it is possible to appropriately guide the user in which direction to move the sound collection device 1.

[0083] Next, a specific example of a processing procedure for implementing the arrangement guidance method according to the embodiment described above will be described with reference to the flowcharts of FIGS. In this example, the processes shown in FIGS. 11 to 13 are executed by the CPU 20a of the server device 20 based on a program stored in a predetermined storage device such as the storage unit 9 or the ROM 2.

[0084] FIG. 11 is a flowchart of the process related to the display processing unit F1. First, in step S101, the CPU 20a waits for an application to start, that is, the above-mentioned recording application to start.

[0085] When it is determined in step S101 that the recording application has been started, the CPU 20a performs a process of displaying the direction guidance information Id in step S102. That is, the CPU 20a performs a process of displaying the direction guidance information Id as exemplified in FIG. 7 on the display unit 7 of the sound collection device 1.

[0086] The CPU 20a ends the series of processes shown in FIG. 11 in response to execution of the process of step S102.

[0087] FIG. 12 is a flowchart of the process related to the placement time processing unit F2. The process shown in FIG. 12 is started in response to the start of a recording application. In step S201, the CPU 20a waits until the sound collection device 1 is placed on a surface such as a table top. In this example, the CPU 20a starts inputting a detection signal from the acceleration sensor provided in the sensor unit 15 from the sound collection device 1 in response to the start of the recording application, and determines whether the sound collection device 1 is placed on the table based on the detection signal from the acceleration sensor.

[0088] When it is determined in step S201 that the sound collection device 1 is placed, the CPU 20a performs a process of estimating the orientation at the time of placement in step S202. Specifically, when the motion sensor described above is used, it is determined whether or not the direction of movement when the sound collection device 1 is placed matches the direction of the side where the second microphones 14a and 14b as the counterpart microphones are arranged, based on the detection signal of the acceleration sensor within a predetermined time from the timing when the sound collection device 1 is placed (hereinafter, referred to as "first placement orientation determination"). Alternatively, in the case of using a touch sensor, it is determined whether or not the touch area when the sound collection device 1 is placed is within the area on the side of the display screen where the self-side microphone (first microphone 13a) is located, based on the detection signal of the touch sensor within a predetermined time from the time when the sound collection device 1 is placed (hereinafter referred to as "second placement direction determination").

[0089] In step S203 following step S202, the CPU 20a judges whether the orientation is correct. That is, when the first placement direction judgment is performed in step S202, the CPU 20a judges whether a positive result is obtained in the first placement direction judgment. Also, when the second placement direction judgment is performed in step S202, the CPU 20a judges whether a positive result is obtained in the second placement direction judgment.

[0090] In step S203, when it is determined that the orientation is correct, the CPU 20a ends the series of processes shown in Fig. 12. That is, when the sound collection device 1 is placed with the remote microphone correctly facing the remote party, no particular alert display is performed.

[0091] On the other hand, if it is determined in step S203 that the orientation is not correct, the CPU 20a executes the processes in and after step S204. In step S204, the CPU 20a executes the recording start restriction process. Note that a specific example of the recording start restriction process has already been described, so a duplicate description will be avoided.

[0092] In step S205 following step S204, the CPU 20a executes an orientation alert display process, that is, a process of displaying information notifying that the placement orientation is incorrect on the display unit 7, as exemplified above.

[0093] In step S206 following step S205, the CPU 20a performs a waiting process for a change in orientation, that is, a process of waiting for a change in the orientation of the sound collection device 1. This waiting process can be considered as a process of waiting until the amount of rotation of the sound collection device 1 about the thickness direction axis of the sound collection device 1 becomes a predetermined amount or more. In this case, the CPU 20a performs a process of waiting until the amount of rotation becomes a predetermined amount or more based on a detection signal of an angular velocity sensor included in the sensor unit 15 of the sound collection device 1.

[0094] If it is determined in step S206 that the orientation of the sound collection device 1 has changed, the CPU 20a proceeds to step S207 to display an alert and release the recording start restriction. That is, the display of the alert information displayed in step S205 is released, and the recording start restriction imposed by the processing in step S204 is released. The process of releasing the recording start restriction, for example, is to start recording if automatic recording start in response to placement has been restricted, and to change the recording button from an inactive display to an active display, or from a hidden state to a displayed state, if a restriction method of displaying the recording button in an inactive or hidden state is adopted.

[0095] The CPU 20a ends the series of processes shown in FIG. 12 in response to execution of the process of step S207.

[0096] FIG. 13 is a flowchart of the process related to the direction change adaptation processor F3. The process shown in FIG. 13 is also started in response to the start of a recording application, similar to the process shown in FIG.

[0097] In step S301, the CPU 20a waits until recording starts, and if it is determined that recording has started, the process proceeds to step S302, where it executes a process of generating initial correspondence information. That is, the CPU 20a performs a voiceprint analysis for the initial period described above on the recording data (sound pickup signals of each microphone) received from the sound collection device 1 in response to the start of recording, and generates initial correspondence information (see FIG. 9A) indicating the correspondence between the detected voiceprint and the microphone of the other party and the microphone of the user.

[0098] In step S303 following step S302, the CPU 20a determines whether or not a determination execution condition is satisfied. That is, the CPU 20a determines whether or not a determination execution condition (for example, at regular intervals) for the above-described direction change determination is satisfied. If it is determined in step S303 that the determination execution condition is not satisfied, the CPU 20a proceeds to step S304 to determine whether or not a termination condition is satisfied. That is, it determines whether or not a predetermined processing termination condition for terminating the processing shown in FIG. 13, such as an operation to terminate a recording application, is satisfied. If it is determined in step S304 that the end condition is not satisfied, the CPU 20a returns to step S303. That is, the processes in steps S303 and S304 form a loop process of waiting for either the satisfaction of the determination execution condition or the satisfaction of the end condition.

[0099] If it is determined in step S303 that the determination execution condition is satisfied, the CPU 20a proceeds to step S305 to execute a voiceprint analysis of the signals picked up by each microphone. That is, the voiceprint analysis is performed on the signals picked up by the microphone on the user's side (the first microphone 13a) and the microphones on the other side (the second microphones 14a and 14b) received as recording data.

[0100] Then, in step S306 following step S305, the CPU 20a judges whether there is a change from the initial correspondence relationship. Specifically, as a result of the voiceprint analysis in step S305, it is judged whether an incorrect voiceprint other than the correct voiceprint indicated by the initial correspondence relationship information is detected for each of the self-side microphone and the other-side microphone. If an incorrect voiceprint is detected in at least one of the self-side microphone and the other-side microphone, a judgment result is obtained that there is a change from the initial correspondence relationship (i.e., a judgment result that the orientation of the sound collection device 1 has become incorrect). On the other hand, if no incorrect voiceprint is detected in either the self-side microphone or the other-side microphone, a judgment result is obtained that there is no change from the initial correspondence relationship (i.e., a judgment result that the orientation of the sound collection device 1 has not become incorrect).

[0101] In step S306, if it is determined that there is no change from the initial correspondence relationship, the CPU 20a returns to step S303. In other words, if it is determined that the direction of the sound collection device 1 is not incorrect by the direction change determination, no particular alert display is performed.

[0102] On the other hand, if it is determined in step S306 that there is a change from the initial correspondence relationship, the CPU 20a proceeds to step S307 to execute an alert display process. Specifically, in this example, as illustrated in Fig. 10, information indicating that the orientation of the sound collection device 1 should be corrected to the correct orientation is displayed on the display unit 7 as an alert display AL. Note that, as illustrated in Fig. 10, direction guidance information Id can also be displayed on the display unit 7 together with the alert display AL.

[0103] In step S308 following step S307, the CPU 20a performs a process of waiting for a change in orientation, that is, a process of waiting for a change in the orientation of the sound collection device 1. The process of step S308 is similar to the process of step S206, and therefore a duplicated description will be avoided.

[0104] In step S307, when it is determined that the orientation of the sound collection device 1 has changed, the CPU 20a proceeds to step S308, and performs a process of canceling the alert display AL in the previous step S307 as a process of canceling the alert display. The CPU 20a returns to step S303 in response to execution of the process of step S307. If the determination execution condition is satisfied again, the processes of steps S305 and onward are executed.

[0105] In response to determining that the end condition is established in step S304, the CPU 20a ends the series of processes shown in FIG.

[0106] In the above, it has been mentioned that an alert is displayed when it is determined that the orientation of the sound collection device 1 has changed from the correct orientation. However, when it is determined that the orientation of the sound collection device 1 has changed from the correct orientation, it is also possible to display on the display unit 7 correction direction guidance information Gu indicating in which direction the sound collection device 1 should be rotated (whether the orientation should be changed).

[0107] FIG. 14 shows an example of how the correction direction guide information Gu is displayed. Here, an example is shown in which corrected direction guidance information Gu is displayed together with an alert display AL. As shown in the figure, the correction direction guide information Gu may be an arrow-shaped display object that indicates the direction in which the sound collection device 1 should be rotated to achieve the correct orientation (to point the other party's microphone toward the other party). Hereinafter, the direction in which the sound collection device 1 should be rotated to achieve the correct orientation is referred to as the "error correction direction."

[0108] The error-resolving direction may be calculated based on, for example, a detection signal of a direction sensor included in the sound collection device 1. For example, similar to the initial correspondence relationship information, initial direction information may be acquired, and a direction in which the error between the direction indicated by the initial direction information and the current direction is eliminated may be calculated as the error-resolving direction.

[0109] <4. Steps for creating minutes and example of display screen> The above mainly explains the process up to the start of recording an interview using a recording app. Although not illustrated, in this example, after recording starts, a recording stop button is displayed in the GUI of the recording application, and the user can operate the recording stop button to instruct recording to stop. In this example, in response to the operation to stop recording, a participant information registration screen Gr as shown in FIG. 15 is displayed as the GUI of the recording application.

[0110] As shown in FIG. 15, the registration screen Gr has input boxes b1, b2, and b3 for inputting the company name, name, and job title of the customer Hc (client) in the business negotiation, and an input box b4 for inputting the name of the sales representative Hs himself. Input boxes b1, b2, and b3 for customer Hc can be added by operating an add button B1 in the figure, making it possible to input information for a plurality of customers Hc. The registration screen Gr also has a registration button B2 for issuing an instruction to register input information.

[0111] On this registration screen Gr, the sales representative Hs inputs the company name, name, and job title of the customer Hc, as well as his or her own name, and operates the registration button B2. For confirmation, in this example, the reason why participant information is registered in response to recording is to display the participant information on a screen (recording list screen Gi) showing a list of recording data, which will be described later.

[0112] It is also possible to register participant information before the start of recording.

[0113] In addition, the CPU 1a of the sound collection device 1 performs processing to transmit to the server device 20 the interview participant information registered via the registration screen Gr as described above, specifically, the name and job title information of the customer Hc, and the name information of the sales representative Hs.

[0114] In the server device 20, the CPU 20a associates the participant information transmitted from the sound collection device 1 with the recorded interview data also transmitted from the sound collection device 1 (in this example, transmitted approximately in real time during recording) and stores it in a specified storage device, such as the memory unit 9. The server device 20 assigns a negotiation ID, which serves as an identifier for the negotiation, to each recording of the negotiation data.

[0115] In the server device 20, an analysis process for analyzing the contents of the business negotiation is carried out based on the recorded data of the business negotiation. This analysis process is carried out by the above-mentioned analysis unit 20b.

[0116] The analysis unit 20b performs a process of converting the audio recording data of the business negotiation into text as a process related to the analysis of the contents of the business negotiation. Here, the technology for converting audio data as recorded data into text, that is, the technology for converting audio data into text data, is a known technology, and detailed description thereof will be omitted here.

[0117] The analysis unit 20b also performs speaker separation processing based on the recorded data. The speaker separation process is a process for identifying which speaker each speech portion is from. Specifically, for example, a voiceprint analysis process is performed on the recorded data, and a speaker ID is associated with each detected voiceprint. For example, speaker A, speaker B, speaker C, etc. are assigned as this speaker ID. Then, a process for identifying the speech section for each speaker is performed on the recorded data based on the voiceprint information for each speaker. By identifying the speech section for each speaker in this way for the recorded data, it is possible to identify the correspondence between the speech portion and the speaker for the speech text data obtained by converting the recorded data into text.

[0118] The analysis unit 20b also performs the above-mentioned summary creation process, main point list creation process, and question and answer identification process based on the text data obtained by converting the audio recording data of the business negotiation into text.

[0119] In addition, in the server device 20 in this embodiment, the CPU 20a performs a group determination process to determine which of multiple groups a speaker in a business negotiation belongs to, based on the recording data of the business negotiation (the audio pickup signals of each audio pickup channel) recorded (stored) in the server device 20. In this embodiment, the speech sound of the participant on the other side is picked up by the microphone on the other side, and the speech sound of the participant on your side is picked up by the microphone on your side, so the speaker whose speech sound is picked up by the second microphones 14a, 14b can be determined to be a speaker in the other side group, and the speaker whose speech sound is picked up by the first microphone 13a can be determined to be a speaker in the own side group.

[0120] Here, in the minutes-taking support system 100, it is possible to present a list of recording data of business negotiations stored in the server device 20 to the user as the sales representative Hs. Specifically, in this example, it is assumed that the above-mentioned recording application is provided with a function for accessing such a list of recorded data. In other words, in this case, the user as the sales representative Hs can be presented with a recording list screen Gi, which is a list screen of recorded data, by operating an access button (not shown) to the recording data list displayed on the screen of the recording application.

[0121] FIG. 16 shows an example of the recording list screen Gi. As shown in the figure, the recording list screen Gi displays, for each business negotiation ID, date and time information of the business negotiation, a play button Bp, participant information, and an input memo field. In this example, the date and time information of the business negotiation includes the start time (recording start time) and end time (recording end time). The playback button Bp is a button for issuing an instruction to play back the recording data of the business negotiation associated with the business negotiation ID. The participant information is displayed as the information entered on the registration screen Gr described above. If there are any items on the registration screen Gr that have not been entered, a message to that effect is displayed. In the input memo field, memo information after the business negotiation can be input as text. The information input in the input memo field is stored in association with the business negotiation ID.

[0122] In addition, the recording list screen Gi can also display links to data on materials used in business negotiations (see "Main Materials" in the figure).

[0123] On the recording list screen Gi, a minutes creation button B3 for instructing the start of software as a minutes editor is displayed for each business negotiation ID. The user as the sales representative Hs operates the minutes creation button B3 displayed on the recording list screen Gi for the business negotiation for which the minutes are to be created.

[0124] In response to the operation of the minutes creation button B3, a GUI of a minutes editor is displayed in the sound collection device 1. The minutes editor is software that displays the above-mentioned editing screen Ge, that is, a screen for creating minutes including an analysis result display area and a text editing area, and performs processing related to creating minutes.

[0125] In this example, the minutes editor is also assumed to be of the SaaS type, that is, the software is stored on the server device 20 side and the software is provided to the user of the sound collection device 1 as a service. In this case, the processing for creating minutes using the editing screen Ge, such as display control processing for determining what information to display on the editing screen Ge and processing for generating text data as minutes in response to operational input on the editing screen Ge, will be performed by the server device 20 (e.g., CPU 20a).

[0126] In the above example, the recording application is given the ability to access the recording list screen Gi, but access to the recording list screen Gi can be done by specifying the URI (Uniform Resource Identifier) ​​of the recording list screen Gi, and any software can be used.

[0127] Also, although an example in which the recording application and the minutes editor are separate applications have been given here, it is also possible for the functions of the recording application and the minutes editor to be provided by the same application.

[0128] In response to the operation of the minutes creation button B3, the CPU 20a of the server device 20 performs a process of displaying the editing screen Ge shown in FIG.

[0129] The editing screen Ge shown in FIG. 17 is the editing screen Ge before an instruction to execute the analysis process is given. As will be described later, in the editing screen Ge of this example, a display area for analysis result text data (analysis result display area Aa, which will be described later) is displayed on the screen in response to an instruction to execute analysis processing.

[0130] The editing screen Ge in Figure 17 has an utterance text display area At, a recording period display area Ap, a speaker / group setting area Ah, and a text editing area Ae, as well as an input box b5, a play button Bp, copy (transcription) buttons B10 and B11, an analysis execution button B13, a speaker setting instruction button B14, an output button B15, and a clipboard copy button B16.

[0131] As shown in the figure, in the editing screen Ge of this example, the speech text display area At, the speaker / group setting area Ah, and the text editing area Ae are arranged from left to right on the screen in the following order: utterance text display area At, speaker / group setting area Ah, text editing area Ae. In this example, the recording period display area Ap is disposed below the spoken text display area At.

[0132] The spoken text display area At is an area in which text data of spoken portions in an interview are displayed for each speaker. As described above, in this embodiment, the text data of the speech portion is generated by the server device 20 (the analysis unit 20b) converting the recorded data of the interview into text.

[0133] In this example, the analysis unit 20b divides the speech portion into a predetermined block unit and manages it. The block unit here is, for example, a unit of the duration of the speech. In other words, the timing when the speech is interrupted is recognized as the break of the block. As described above, the analysis unit 20b in this example also performs speaker separation processing based on the recorded data, and links each detected voiceprint to a speaker ID, such as speaker A, speaker B, or speaker C. Furthermore, processing is also performed to identify speech sections for each speaker in the recorded data, based on the voiceprint information for each speaker. By identifying speech sections for each speaker in the recorded data in this manner, it is possible to identify the correspondence between speech portions and speakers for the spoken text data obtained by converting the recorded data into text.

[0134] The CPU 20a displays the text data of the speech part for each speaker in the speech text display area At based on the text data of each speech part obtained by the speaker separation processing in the analysis unit 20b and information indicating the correspondence between each speech part and the speaker ID. In this example, the text data of the speech portions of each speaker are displayed in chronological order from the top to the bottom of the screen. In this example, the speech text display area At displays speaker icons as identifiers of speech parts for each speaker. At the stage of Fig. 17, speaker icons display speaker IDs such as speakers A, B, and C that are assigned in the speaker separation process.

[0135] 17, all of the text data of the utterance portion for each speaker is displayed close to either the left or right end of the utterance text display area At. Specifically, in this example, the text data is displayed close to the left end of the utterance text display area At as shown in the figure.

[0136] The recording period display area Ap is an area for displaying recording period display information that graphically represents the recording period of the interview. As shown in the figure, in the recording period display area Ap in this example, information showing the recording period of the interview in the form of a bar extending in the left-right direction is displayed as recording period display information. In the recording period display area Ap, a seek bar Sb for indicating the playback position of the recording data is displayed in association with the recording period display information. This seek bar Sb can be moved left and right, and the user can operate the play button Bp after moving the seek bar Sb to instruct the sound collection device 1 to play the recording data from the playback position indicated by the position of the seek bar Sb.

[0137] In this embodiment, the recording period display information displays information showing the distribution of the speech period for each speaker. Specifically, the recording period display information in this case displays a bar indicating the recording period for each speaker, and displays the information on the speech period for each speaker in a different display format for each speaker. For example, it is possible to display the color or pattern of the speech period for each speaker differently. In this example, the bar for each speaker indicating the recording period displays the speaker icon of the corresponding speaker.

[0138] By displaying information showing the distribution of speaking periods for each speaker as recording period display information as described above, the user can easily grasp information such as which speaker spoke for how long during what period during the interview, and the amount of speech each speaker made, allowing the user to easily understand which speaker's speech they should pay attention to when creating the minutes. In addition, by displaying the speech period information for each speaker in a different display format for the recording period display information, the user can intuitively identify the speech period for each speaker. When a user wants to check the recording data for a specific speech portion of a specific speaker during the process of creating minutes, the ease of searching for that speech portion can be improved.

[0139] In addition, in the recording period display area Ap of this embodiment, the negotiation process period information Is is displayed in association with the recording period display information. Here, the negotiation process period information Is is information indicating the name and period of each negotiation process when the negotiation is divided into processes according to a predetermined criterion.

[0140] The analysis unit 20b in this embodiment performs a process of identifying negotiation steps in a negotiation based on the audio recording data of the negotiation. The negotiation steps referred to here are stages in the progress of the negotiation, and examples of such information include "ice-breaking (small talk)", "needs analysis", "proposal and explanation", and "negotiation". Specifically, the analysis unit 20b performs a process of identifying the negotiation process based on text data obtained by converting the audio recording data of the negotiation into text. This process may be performed by an AI model generated by performing machine learning such as deep learning using the text data as learning input data and the correct answer data of the negotiation process as teacher data. Alternatively, it may be realized as a rule-based process in which frequently occurring words for each negotiation process are predetermined as process identification keywords, and the negotiation process is identified based on the results of matching with the process identification keywords.

[0141] The CPU 20a performs display processing of the negotiation process period information Is based on the period information (information indicating the start time and end period of a process) for each negotiation process obtained as a result of the process of identifying the negotiation process by the analysis unit 20b. The figure shows an example of the negotiation process period information Is when five processes, ST1 to ST5, are identified as the negotiation process.

[0142] By displaying the negotiation process period information Is, i.e., the chronological information of the negotiation process, in association with the recording period display information as described above, when a user wishes to check the recording data of a specific speech portion of a specific speaker during the process of creating minutes, it is possible to improve the ease of searching for that speech portion.

[0143] In the above, an example of the recording period display information in which the recording period is displayed in the shape of a bar has been given, but the recording period may be displayed in other shapes such as an arc. When the recording period is displayed in the shape of an arc, a rotational operator may be used as the playback position indication operator, rather than a sliding operator such as the seek bar Sb.

[0144] The speaker / group setting area Ah is an area for setting the name of each speaker and the group for each speaker. Group setting here means setting which group each speaker belongs to, with respect to two groups: the self-group, which is the group on the sales representative Hs (user)'s side, and the other-side group, which is the group on the sales representative Hs's other side (i.e., the customer Hc's side).

[0145] In the speaker / group setting area Ah of this example, when group setting has not been performed, each speaker is displayed to indicate that she belongs to the other party's group, as shown in the figure. Note that the display mode when group setting has not been performed is not limited to this, and other display modes may be adopted, such as a display indicating that each speaker belongs to the user's group.

[0146] In the editing screen Ge of this example, when the user wishes to set the speaker's name and group, he or she operates the speaker setting instruction button B14. Then, a setting screen Gs for setting speakers and groups as shown in FIG. 18 is displayed on the speaker / group setting area Ah. As shown in the figure, the setting screen Gs is provided with an input box b6 for inputting the speaker's name, a selection operator s1 for selecting one's own group / other party group, a save button B20, and a cancel button B21. Here, an example of the setting screen Gs for speaker B is shown, but in this example, in response to the operation of the speaker setting instruction button B14 described above, the setting screen Gs is displayed in order starting from speaker A, and the user can input name information in the input box b6 in order starting from speaker A, set a group by operating the selection operator s1 as necessary, and further perform a save operation using the save button B20, thereby inputting the name and setting a group for each speaker. When the cancel button B21 is operated on the setting screen Gs, the CPU 20a ends the acceptance of the setting operation of the speaker name and group, and performs processing to return to the screen display state of FIG. 17, for example.

[0147] Here, in this embodiment, since group determination (determination of the other party's group / your own group) has already been performed for each speaker by the group determination process described above, it is not essential to have the user perform group setting operations using the setting screen Gs as described above. In this case, it is sufficient to have the user input and set the name information for each speaker on the setting screen Gs.

[0148] FIG. 19 shows an example of display information in the speaker / group setting area Ah when the name and group for each speaker are set on the setting screen Gs described above. As shown in the figure, the set name information is displayed for each speaker. Also, as shown in Figure 17 above, in the speaker / group setting area Ah, before the name of each speaker is set, information such as A, B, C, etc. indicating the speaker ID is displayed as the speaker icon, but after the name is set, the speaker icon displays the initial letter of the name, as shown in Figure 19. In this case, in the speaker / group setting area Ah, the speaker set to the other party's group is displayed in the other party's group area, and the speaker set to your own group is displayed in your own group area, according to the group settings for each speaker.

[0149] In this example, the settings of the name and group for each speaker in the speaker / group setting area Ah are also reflected in the speech text display area At. Specifically, as in the editing screen Ge shown in FIG. 20, the speaker icon for each speaker in the spoken text display area At is changed to a notation showing the initials of the set name.

[0150] Furthermore, in the utterance text display area At, in accordance with the group setting for each speaker, the utterance text data of the speaker belonging to the self-group is moved toward one of the left and right ends, and the utterance text data of the speaker belonging to the other-group is moved toward the other left and right end. Specifically, in this example, the utterance text data of the speaker belonging to the self-group is moved toward the right end of the utterance text display area At, and the utterance text data of the speaker belonging to the other-group is moved toward the left end of the utterance text display area At.

[0151] This allows the user to intuitively distinguish between the utterances of the user's group and the other party's group in the interview text data displayed in the utterance text display area At. In particular, when the interview is a business negotiation as in this embodiment, the user as the sales representative Hs can refer to the utterances of the person on the other party's group, the customer Hc, all at once, which increases convenience.

[0152] In this example, the setting information of the name of each speaker in the speaker / group setting area Ah is also reflected in the display information in the recording period display area Ap. Specifically, the display of the speaker icon displayed in each bar for each speaker will be changed from the speaker ID to the initials of the speaker's name.

[0153] In the edit screen Ge, the analysis execution button B13 is a button for instructing the execution of analysis processes related to the contents of the interview, such as the creation of a summary, etc. Specifically, in this example, it is a button for instructing the execution of the summary creation process, the main points list creation process, and the question and answer identification process.

[0154] As described above, in this example, the analysis processing such as the summary creation processing, the main point list creation processing, and the question and answer identification processing are performed by the analysis unit 20b. Therefore, when the analysis execution button B13 is operated, the CPU 20a issues an instruction to the analysis unit 20b to execute the analysis processing.

[0155] The CPU 20a displays the analysis result text data obtained as a result of the analysis process performed by the analysis unit 20b on the editing screen Ge.

[0156] FIG. 21 shows an example of the analysis result text data displayed on the editing screen Ge. In the editing screen Ge, the analysis result text data is displayed in the analysis result display area Aa. As shown in the figure, the analysis result display area Aa in this example is disposed in a position between the speaker / group setting area Ah and the text editing area Ae in the left-right direction of the screen.

[0157] The analysis result display area Aa is provided with a summary display area A1, a main points list display area A2, and a question and answer display area A3, as well as copy buttons B25, B26, and B27. The summary display area A1 is an area for displaying text data of the summary created in the summary creation process. The main point list display area A2 is an area for displaying text data of the main point list created in the main point list creation process, that is, text data for each main point. The question and answer display area A3 is an area for displaying text data of the question portion and text data of the answer portion identified in the question and answer identification process.

[0158] The copy button B25 is a button for instructing to copy the text data displayed in the summary display area A1 to the text editing area Ae. In the summary display area A1 of this example, it is possible to perform a range designation operation of the text data, and the user can designate the range of the text data to be copied in the summary display area A1 and then operate the copy button B25 to instruct to copy the text data of the designated range to the text editing area Ae.

[0159] In the gist list display area A2, a check box cb for enabling selection of text data on a gist unit basis is provided for each gist. Further, a copy button B26 is provided for the main point list display area A2. The user can check any one of the check boxes cb for each key point and then operate the copy button B26 to instruct the text data of the key point corresponding to the checked check box cb to be copied to the text editing area Ae.

[0160] In the present example, the text data for each main point is arranged in the main point list display area A2 in the chronological order of the original utterance. Specifically, in the present example, the text data for a main point whose original utterance is older is arranged higher in the main point list display area A2. In this main point list display area A2, the main points can be arranged in chronological order for each group, i.e., your own group and the other party's group. In the figure, the main points list for your own group and the other party's group are displayed in separate columns.

[0161] In the question and answer display area A3, text data showing the question portion and text data showing the answer portion are displayed. In the question and answer display area A3 of this example, text data showing the question portion and text data showing the answer portion are arranged in the chronological order of the original utterances.

[0162] The copy button B27 is a button for instructing copying of the text data in the question and answer display area A3 to the text editing area Ae. In question and answer display area A3, it is possible to perform an operation to specify the text data of the question portion and answer portion to be copied. After performing an operation to specify the range of the text data of the question portion and answer portion to be copied in question and answer display area A3, the user can issue an instruction to copy the specified text data to text editing area Ae by operating copy button B27.

[0163] In the question and answer display area A3 of this example, the text data showing the question portion and the text data showing the answer portion are displayed so that it is possible to identify whether the comment is made by the user's group or the other party's group. Specifically, in this example, the question and answer portions of your group are displayed in text data toward the right edge of question and answer display area A3, and the question and answer portions of the other group are displayed in text data toward the left edge of question and answer display area A3. This makes it easier for a user to understand specific questions or answers from his or her own or the other party when creating minutes of a meeting. As illustrated in the figure, it is also possible to provide separate columns in the question and answer display area A3 for displaying questions from your side and the other person's answers to those questions, and for displaying questions from the other person's side and your own answers to those questions.

[0164] Next, the text editing area Ae will be described. The text editing area Ae is an area in which the analysis result text data in the analysis result display area Aa can be transcribed and the transcribed analysis result text data can be edited. The user can use this text editing area Ae to create minutes of the interview. Since the analysis result text data in the analysis result display area Aa, such as a list of key points or a summary, can be copied and used in the sentences that make up the minutes, there is no need to input the sentences for creating the minutes from scratch, and the efficiency of creating the minutes can be improved.

[0165] In the text editing area Ae in this embodiment, if the transcribed analysis result text data is corrected, the corrected text portion is displayed in a different display mode from the uncorrected text portion. Although not shown, if the analysis result text data copied from the analysis result display area Aa is corrected by a user operation in the text editing area Ae, for example, the corrected text portion is displayed in a different color from the uncorrected text portion. Alternatively, only the corrected text portion may be displayed in bold or underlined.

[0166] In addition, in this embodiment, when the transcribed analysis result text data is corrected in the text editing area Ae, the CPU 20a performs a process of recording the text data before and after correction in a memory device in association with each other. The pre-correction / post-correction text data is recorded in a memory device within the server device 20, such as the storage unit 9 of the server device 20.

[0167] Here, the text data before and after the correction in the text editing area Ae can be used for training the minutes creator. Specifically, for example, by having a low performer who is inexperienced in minutes creation refer to the text data before and after the correction by a high performer who is skilled in minutes creation, it becomes possible for the low performer to learn the skills related to minutes creation, which can contribute to the development of the low performer creator. In particular, if the interview is a business meeting, it becomes possible for the low performer to learn the skills of a high performer sales representative, which can contribute to the improvement of the sales representative's skills. In addition, the above-mentioned pre- and post-correction text data can also be used to improve and enhance the functionality of the analysis process for obtaining analysis result text data from the recorded interview data. Specifically, for example, when the analysis process is performed using an AI model, the pre- and post-correction text data can be used as learning data for machine learning to improve and enhance the analysis process. Or, even when the analysis process is performed using rule-based processing, the pre- and post-correction text data can be referred to in updating the analysis process, which can contribute to improving and enhancing the functionality of the analysis process.

[0168] In the editing screen Ge, an output button B15 and a clipboard copy button B16 are provided as operators related to the text editing area Ae. The output button B15 is a button for instructing output of the text (text as minutes) created in the text editing area Ae in a predetermined file format such as Word. The clipboard copy button B16 is a button for instructing copying of the text created in the text editing area Ae to the clipboard. By providing the output button B15 and the clipboard copy button B16, the user can save the created text data as the minutes in the sound collection device 1 in a predetermined file format.

[0169] Furthermore, in the editing screen Ge of this example, an upload button B17 and a submission button B18 are also provided as operators related to the text editing area Ae. The upload button B17 is a button for instructing uploading of the text created in the text editing area Ae to a cloud server of a predetermined sales information management service (for example, Salesforce: registered trademark). For example, when the upload button B17 is operated, a dialog is displayed, and when the user specifies his / her company name on the dialog and performs an operation to instruct uploading, the text created in the text editing area Ae is uploaded to the above-mentioned cloud server.

[0170] The post button B18 is a button for instructing to post the text created in the text editing area Ae as shared information in a specified sales communication service (e.g., Slack: registered trademark). For example, when the post button B18 is operated, a dialog is displayed, and when the user designates a channel related to information sharing on the dialog and performs an operation to instruct to execute posting, the text created in the text editing area Ae is posted as shared information in the designated channel in the above-mentioned sales communication service.

[0171] In addition, in the editing screen Ge of this example, a copy button B10 and a copy button B11 are also provided as operators for issuing a copy instruction to the text editing area Ae. The copy button B10 is a button for instructing to copy the text data in the speech text display area At to the text editing area Ae. The copy button B11 is a button for instructing to copy the speaker name information and group setting information set in the speaker / group setting area Ah to the text editing area Ae. By providing these copy buttons, the efficiency of creating minutes can be further improved.

[0172] Here, in the editing screen Ge in this embodiment, it is possible to change the group information that has already been set in the speaker / group setting area Ah. Specifically, as shown in Fig. 22, the group information that has already been set can be changed by dragging the display part of the speaker whose group setting is to be changed in the speaker / group setting area Ah. The illustrated example shows an example in which the group setting of a speaker that belonged to the user's group is changed so that the speaker now belongs to the other party's group.

[0173] In this embodiment, when the group setting is changed as described above, the display position of the utterance text data in the utterance text display area At is changed according to the changed group setting information. Specifically, according to the changed group setting information, the display position of the utterance text data for the speaker whose group setting has been changed is switched between left edge alignment and right edge alignment (see the transition from FIG. 22 to FIG. 23).

[0174] Although not shown in the drawings, when the group setting is changed, the question and answer display area A3 of the analysis result display area Aa may also be switched between left edge / right edge alignment in a similar manner.

[0175] Although not specifically mentioned above, it is conceivable that the summary creation process, the main points list creation process, and the question and answer identification process can each be realized as a process using an AI model. For example, it is conceivable that the summary creation process can use an AI model trained by machine learning using deep learning or the like so that text data can be used as input data and a summary can be obtained as output data. Similarly, it is conceivable that the main points list creation process and the question and answer identification process can use an AI model trained by machine learning using deep learning or the like so that text data can be used as input data and a main points list, question portion, and answer portion can be obtained as output data.

[0176] Alternatively, the summarization process, the main point list creation process, and the question and answer identification process may be realized as rule-based processes. For example, the summary creation process may be realized by a processing method of identifying unnecessary sentences in spoken language and deleting the unnecessary sentences. Also, the main point list creation process may be realized by a processing method of dividing the summary created in the summary creation process according to a predetermined rule, for example by dividing it by periods, and identifying each divided sentence as a main point. Also, the question / answer identification process may be realized by a processing method of identifying the question part in the utterance based on the recorded data, on the premise that the ending of the question part is raised, and identifying the part of the utterance next to the question part as the answer part.

[0177] <5. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, in the above description, guidance regarding the direction in which the sound collection device 1 should be pointed has been mainly described, but it is also conceivable to provide guidance regarding distance.

[0178] 24, when there are many participants (customers Hc) on the other side, if the sound collection device 1 is positioned in any direction away from the center in the direction in which the participants are lined up, there is a risk that the speech of some of the participants on the other side far from the sound collection device 1 will not be captured at a sufficient sound pressure level. In the illustrated example, the sound collection device 1 is positioned to the left of the center as viewed from the user's side (sales representative Hs's side), so there is a risk that the speech of the participant on the right side will not be captured at a sufficient sound pressure level.

[0179] Therefore, it is considered that the server device 20A (CPU 20aA) as a modified example has a function as a low sound pressure speech corresponding processing unit F4 as follows: That is, a low sound pressure speech detection process is performed for the sound signal collected by the microphones of the other party (second microphones 14a, 14b) to detect low sound pressure speech, which is speech with a sound pressure level (signal amplitude level) below a predetermined level, and when a low sound pressure speech is detected, movement suggestion information Im suggesting movement of the sound collection device 1 is displayed on the display unit 7 of the sound collection device 1. The process by the low sound pressure speech handling processor F4 may start in response to the start of a recording application, or in response to the start of recording.

[0180] FIG. 25 shows an example of how movement suggestion information Im is displayed. As an example of the movement suggestion information Im, it is possible to display information suggesting that the sound collection device 1 be moved closer to the speaker who is speaking at a low sound pressure, such as the message information illustrated in the figure, "It is difficult to hear the customer's voice. Please move closer to the person who just spoke."

[0181] FIG. 26 is a functional block diagram showing an example of the functional configuration of a CPU 20aA as a modified example having such a low sound pressure speech handling processor F4. As shown in the figure, the functional configuration of the CPU 20aA may include, in addition to the display processing unit F1, the placement response processing unit F2, and the direction change response processing unit F3 described above, a low sound pressure speech response processing unit F4. Regarding the functional configuration of the CPU 20aA, the placement response processing unit F2 and the direction change response processing unit F3 are not essential components, and may be omitted.

[0182] Regarding the low sound pressure speech handling processing unit F4, in a noisy environment such as a cafe where speakers other than the interview participants are present, there is a possibility that speech other than that of the interview participants will be erroneously detected as low sound pressure speech. Therefore, the low sound pressure speech corresponding processing unit F4 may be configured not to execute the low sound pressure speech detection process when the usage environment of the sound collection device 1 is determined to be a noisy environment. In this case, it is conceivable that the determination as to whether the environment in which the sound collection device 1 is used is a noisy environment or not can be made based on whether the user inputs whether the sound collection device 1 is used in a noisy environment or not through a user operation. Alternatively, the determination of whether or not the environment is noisy may be performed by image analysis of an image captured by a camera, or audio analysis of a signal picked up by a microphone.

[0183] By not executing the low-sound pressure speech detection process in a noisy environment as described above, the accuracy of the low-sound pressure speech detection process can be improved, and the movement suggestion information Im can be used to appropriately suggest movement of the sound collection device 1.

[0184] Alternatively, for a noisy environment, a measure may be taken in which low sound pressure detection processing is performed only on a predetermined number of voiceprints with high sound pressure levels. Specifically, in this case, the low sound pressure speech processing unit F4 performs voiceprint analysis on the signal picked up by the other party's microphone, and performs low sound pressure speech detection processing on only the speech sounds with the top x voiceprints having the highest sound pressure levels among the speech sounds detected by the other party's microphone, where x (a natural number) is the total number of participants on the other side.

[0185] In this case, the total number x of the other participants may be obtained based on a user operation. For example, the total number x may be obtained from the number of inputs of the name information of the other participants through the setting screen Gs (see FIG. 18) described above. Alternatively, the total number x may be assumed to be fixed, for example, as 1 or 2, and in such a case, it is not necessary for the user to input information to obtain the total number x.

[0186] By taking the measures described above, it is possible to prevent the speech sounds of people other than the interview participants from being subject to low sound pressure detection processing in a noisy environment where there are speakers other than the interview participants, such as in a cafe. Therefore, even in a noisy environment, the movement suggestion information Im can appropriately suggest the movement of the sound collection device 1.

[0187] Here, the technique for identifying the voiceprint (utterance) of the other participant based on the sound pressure level as described above can be suitably applied to processes other than the low sound pressure detection process. For example, it is conceivable that, among the speech sounds detected by the microphone of the other party, the speech sounds of the top x voiceprints with the highest sound pressure levels are identified as the speech sounds of the other party, and text conversion processing is performed only on the identified speech sounds of the other party. This makes it possible to prevent speech sounds of people other than the interview participants from being converted into text for recording data of an interview conducted in a noisy environment such as a cafe, thereby reducing the burden of the text conversion process. At the same time, it is possible to prevent the creation of summaries of speeches of people other than the interview participants that are not required for the minutes, thereby preventing unnecessary summary information from being presented to the user when creating the minutes. In other words, it is possible to improve performance in supporting the creation of minutes.

[0188] In the explanation so far, it has been mentioned that recording of the interview starts automatically when the sound collection device 1 is placed, but it is also possible that recording of the interview starts automatically when a recording app is launched. At this time, it is assumed that there will be a period of silence before the interview begins after the start of recording. In that case, it is conceivable to automatically delete the recording data of the silent period before the interview begins.

[0189] Regarding the end of recording, an example was given in which the recording stop button displayed as a GUI was operated, but various patterns regarding the end of recording are conceivable. For example, the change in the attitude of the sound recording device 1 to a vertical attitude (approximately upright attitude) could be used as a trigger to display a message such as "Is the interview over?" along with "Yes" and "No" buttons on the display unit 7, and recording could end when the "Yes" button is pressed. It is possible to use the sound collection device 1 being lifted from its placed position as a trigger, but since it is also expected that the sound collection device 1 will be lifted when moving it in accordance with the above-mentioned alert display AL or movement suggestion information Im, it is preferable to use the change to a vertical position as described above as a trigger.

[0190] Also, recording may be ended in response to the operation of a predetermined physical button on the sound collection device 1 (for example, a power button, etc.). At this time, when a predetermined physical button is operated, a question screen may be displayed asking whether or not to end recording. While asking whether to end recording, recording may be automatically paused. At this time, a message such as "Recording is paused. Do you want to end recording?" may be displayed along with "Yes" or "No" buttons, in which case recording ends when the "Yes" button is pressed. Alternatively, the operation to end recording in this case may be performed by pressing a specific physical button again.

[0191] In the explanation so far, a determination method using initial correspondence information has been exemplified for the direction change determination by the direction change response processing unit F3. However, for example, if the voiceprint of the sales representative Hs is known in advance, it is also possible to determine whether or not the voiceprint of the sales representative Hs is detected by the microphone on the other side as the direction change determination.

[0192] In addition, in the explanation so far, an example has been given of the interview being a business negotiation, but interviews to which the technology of the present invention can be applied include various opportunities for people to communicate with each other, such as business negotiations, as well as interviews in the fields of medicine, counseling, lectures, speeches, education, and entertainment.

[0193] In the above description, the display processing unit F1, the placement processing unit F2, the direction change processing unit F3, and the low sound pressure speech processing unit F4 are all processed by the server device 20 (CPU 20a), but it is also possible to configure the sound collection device 1 (CPU 1a) to perform some or all of these processes. In this case, it is possible to store a program for implementing the processes to be executed by the sound collection device 1 in the sound collection device 1.

[0194] In the above, a smartphone is exemplified as the device form of the sound collection device 1, but the device form of the sound collection device in the present invention is not limited to this. In the present invention, the sound collection device may be configured as a device having at least a plurality of microphones arranged at different positions on the device body and a display unit that displays information.

[0195] In the above example, the microphone of the sound collection device 1 is a directional microphone, but it is also possible to use an omnidirectional microphone. Even if an omnidirectional microphone is used, it is possible to determine whether the sound collection device 1 has changed from the correct orientation based on the collected sound signal. For example, for a certain speaker A on the other side, the sound characteristics such as the sound pressure level and phase of the speech sound picked up by the microphone on the other side when the orientation of the sound collection device 1 is correct will be different from the sound characteristics picked up by the microphone on the other side when the orientation of the sound collection device 1 is incorrect. Therefore, for example, for each speaker on the other side, the sound characteristics of the speech sound picked up by the microphone on the other side when the orientation of the sound collection device 1 is correct are stored as initial sound characteristics, and it is determined for each speaker whether or not the sound characteristics have changed from the initial sound characteristics. If there is a speaker whose sound characteristics have changed as a result of such determination for each speaker on the other side, it can be determined that the orientation of the sound collection device 1 has changed from the correct orientation.

[0196] <6. Summary of the embodiment> As described above, the control device (server device 20, or sound collection device 1) as an embodiment is a sound collection device (1) equipped with a plurality of microphones (first microphones 13a, 13b, second microphones 14a, 14b) arranged at different positions on the device body and a display unit (7) that displays information, and is equipped with a display processing unit (F1) that performs processing to display direction guidance information (Id) that guides the direction in which the other microphone can be pointed toward the other side, for the sound collection device in which one of the plurality of microphones is designated as the other side microphone (second microphone 14a or 14b) that should pick up the speech of the other side in the interview, and the other is designated as the self side microphone (first microphone 13a or 13b) that should pick up the speech of the self side in the interview. This makes it possible to guide the user as to which microphone to point in which direction when using a sound collection device equipped with multiple microphones to record an interview. Therefore, the burden on the user when recording an interview can be reduced by pointing the sound collection device in an appropriate direction.

[0197] In addition, in the control device as an embodiment, the display processing unit displays, as direction guidance information, other-side microphone presentation information, which is information indicating the side on which the other-side microphone is located in the sound collection device (see Figures 7A and 7B). This makes it possible to guide the sound collection device in the correct placement orientation.

[0198] Furthermore, in the control device as the embodiment, the display processing unit displays self-side microphone presentation information, which is information indicating the side on which the self-side microphone is placed in the sound collection device, as direction guidance information (see FIG. 7A). By pointing the local microphone toward the user in accordance with the local microphone presentation information, the other user's microphone is correctly pointed toward the other user's side, so the display of the local microphone presentation information can also guide the user to the correct orientation of the sound collection device.

[0199] Furthermore, in the control device as an embodiment, the display processing unit displays, as direction guidance information, the other party's microphone presentation information and one's own microphone presentation information, which is information indicating the side on which the one's own microphone is located in the sound collection device, and displays the other party's microphone presentation information more highlighted than the one's own microphone presentation information (see Figure 7A). Generally, when summarizing an interview or creating minutes, the information provided by the other party tends to be more important than the information provided by the user, and this tendency is particularly pronounced in business negotiations. For this reason, it is preferable to highlight the information provided by the other party's microphone more than the information provided by the user's microphone, as described above.

[0200] In addition, in the control device according to the embodiment, the microphone on the other side is a microphone with higher performance than the microphone on the own side. High performance here means high performance in terms of sound collection. For example, the distance at which sound can be collected, the number of sound collection channels, sensitivity, SNR (signal-to-noise ratio), etc. can be cited as examples of the performance referred to here. In the embodiment, an example was given in which the first microphone 13a (or the first microphone 13b) as the self-side microphone is a microphone for monophonic sound collection, while the second microphones 14a and 14b as the other-side microphones are microphones for stereophonic sound collection, but the other-side microphone can be considered to be a microphone with higher performance than the self-side microphone according to the above definition. As described above, in general, when making a summary or minutes of an interview, the information importance of the other party's remarks tends to be higher than that of one's own side, and this tendency is particularly noticeable in business negotiations. By making it possible to point the other party's microphone, which has a higher performance, toward the other party, it is possible to improve the accuracy of converting the other party's remarks into text, and it is possible to improve the accuracy of making a summary of the other party's remarks, thereby improving the usefulness of the minutes made based on the summary.

[0201] Furthermore, in a control device as an embodiment, the sound collection device is configured as a smartphone, the local microphone is a microphone located on the top of the smartphone, and the other party's microphone is a microphone located on the bottom of the smartphone. The present invention can be suitably applied by using a smartphone as a sound collecting device in which a microphone arranged at the bottom has higher performance than a microphone arranged at the top.

[0202] Furthermore, in the control device as an embodiment, a placement response processing unit (F2) is provided which performs a placement direction determination, which is a determination as to whether or not the sound collection device is placed with the other party's microphone facing the other party, based on a sensor detection signal in the sound collection device when the sound collection device is placed, and controls the display unit to display a predetermined alert if a negative result is obtained from the placement direction determination. This makes it possible to determine whether the sound collection device is placed in the correct orientation according to the direction guidance information, and to display an alert to the user if the sound collection device is not placed in the correct orientation. Therefore, the user can be warned that recording will occur when the microphone on the other side is not correctly pointed at the other side.

[0203] In the control device according to the embodiment, the placement response processing section determines the placement direction based on a detection signal from a motion sensor provided in the sound collection device. Usually, when a sound collection device is placed on a support surface such as the top surface of a table for recording, the sound collection device is slid toward the other side. Therefore, by determining whether the sliding direction is the same as the direction on the side where the other side microphone is placed based on the detection signal of the motion sensor, it is possible to determine whether the sound collection device is placed with the other side microphone facing the other side.

[0204] Furthermore, in the control device as an embodiment, the sound collection device is configured as a smartphone, and the placement response processing unit determines the placement direction based on a detection signal of a touch sensor formed on the display screen of the smartphone. Typically, when a user places a smartphone as a sound collection device on a support surface such as the top surface of a table, the user holds the front side of the smartphone (the side opposite to the direction in which it is presented to the other party).Therefore, by determining whether the touched portion when the smartphone is placed is within the area on the display surface on the side where the user's microphone is located based on the detection signal from the touch sensor, it is possible to determine whether the smartphone is placed with the other party's microphone facing toward the other party.

[0205] Furthermore, in the control device according to the embodiment, the placement response processing section performs a recording start restriction process for controlling so that recording is not started when a negative result is obtained from the placement direction determination. This makes it possible to prevent recording from starting when the other party's microphone is not correctly pointed toward the other party, and it is possible to ensure the accuracy of summaries and minutes when they are prepared based on the recorded data.

[0206] In addition, in the control device as an embodiment, a direction change response processing unit (F3) performs a direction change determination, which is a determination as to whether or not the orientation of the sound collection device has become incorrect, based on the results of voiceprint analysis of the sound collection signal from the local microphone or the other party's microphone after recording has started, and if a positive result is obtained from the direction change determination, controls the display unit to display a predetermined alert (AL). This makes it possible to display an alert to the user if the sound collection device is moved for some reason after recording has started and is now facing the wrong direction. Therefore, the user can be warned that recording will continue when the microphone on the other side is not correctly pointed at the other side.

[0207] Furthermore, in the control device as an embodiment, a low-pressure speech detection process is performed on the sound pickup signal of the opposing microphone to detect low-pressure speech, which is speech whose sound pressure level is below a predetermined level, and a low-pressure speech response processing unit (F4) is provided which displays movement suggestion information (Im) on the display unit to suggest moving the sound pickup device when low-pressure speech is detected. For example, when there are many participants on the other side, if the sound collection device is positioned in a direction offset from the center of the line of participants on the other side, there is a risk that the speech of some of the participants on the other side who are far from the sound collection device will not be captured at a sufficient sound pressure level. According to the above configuration, in response to such a case, it is possible to suggest moving the sound collection device closer to the speaker who is speaking at a low sound pressure. Therefore, it is possible to prevent the volume of the speech recording data of some of the other participants from continuing to be low, and it is possible to ensure that the speech recordings of all the other participants are performed appropriately.

[0208] Furthermore, in the control device as an embodiment, the low sound pressure speech corresponding processing section does not execute the low sound pressure speech detection process when the usage environment of the sound collection device is determined to be a noisy environment. In other words, in a noisy environment such as a cafe where there are speakers other than the interview participants, the low sound pressure speech detection process is not performed because there is a possibility that speech other than that of the interview participants will be erroneously detected as low sound pressure speech. This makes it possible to improve the accuracy of the low sound pressure speech detection process, and makes it possible to appropriately suggest movement of the sound collection device based on the movement suggestion information.

[0209] In addition, in the control device as an embodiment, the low sound pressure speech processing unit performs voiceprint analysis on the signal picked up by the other party's microphone, and performs low sound pressure speech detection processing on only the speech sounds with the top x voiceprints having the highest sound pressure levels among the speech sounds detected by the other party's microphone, where x (a natural number) is the total number of participants on the other side. This makes it possible to prevent the speech sounds of people other than the interview participants from being subject to low sound pressure detection processing in a noisy environment, such as a cafe, where there are speakers other than the interview participants. Therefore, even in a noisy environment, the movement suggestion information can appropriately suggest the movement of the sound collection device.

[0210] In one embodiment, the control method is a control method in which a control device is a sound collection device equipped with a plurality of microphones arranged at different positions on the device body and a display unit that displays information, one of the plurality of microphones being designated as an other-side microphone for picking up the other party's speech in an interview, and the other being designated as a self-side microphone for picking up the self-side speech in an interview, and the control device performs processing to display on the display unit direction guidance information that guides the other-side microphone in the direction in which it can be pointed toward the other party. With such a control method, it is possible to obtain the same functions and effects as those of the control device according to the above embodiment.

[0211] The program as the embodiment is a program that causes a computer device such as the CPU 20a to execute all or part of the processing shown in FIGS. In other words, the program as an embodiment is a program readable by a computer device, and is a sound collection device having a plurality of microphones arranged at different positions on the device body and a display unit that displays information, one of the plurality of microphones being designated as an other party's microphone for picking up the other party's speech in an interview, and the other being designated as a self-side microphone for picking up the self-side speech in an interview, and is a program that causes the computer device to realize a function of performing processing to display direction guidance information on the display unit that guides the other party's microphone in the direction in which it can be pointed toward the other party.

[0212] Such a program can realize a computer device that realizes the functions of the display processing unit F1 described in the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, or the like, the personal computer or the like can be made to function as a computer device that performs processing as the display processing unit F1.

[0213] Such a program can be pre-recorded in a HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU, etc. Also, the program can be provided by being temporarily or permanently recorded on a removable recording medium such as a memory card or an optical disk. Such programs can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet. [Explanation of symbols]

[0214] 1. Sound recording device 20 Server device NT Network Hs Sales Representative Hc customer 1a,20a CPU 8a Speaker for calls 13 First sound collection section 13a, 13b First microphone 14 Second recording section 14a, 14b Second microphone 15 Sensor section 16 Camera unit 20b Analysis Department F1 Display processing section F2 Loading Response Processing Unit F3 Direction change response processing part Id Direction Information AL Alert Display F4 Low sound pressure speech processing unit Im Movement Suggestion Information

Claims

1. A sound collection device having a plurality of microphones arranged at different positions on a device body and a display unit for displaying information, wherein one of the plurality of microphones is a partner microphone for picking up speech from the partner in an interview, and the other is a self microphone for picking up speech from the self in the interview, A display processing unit that performs processing for displaying, on the display unit, direction guidance information that guides a direction in which the other party's microphone can be pointed toward the other party. Control device.

2. The display processing unit is As the direction guidance information, a counterpart microphone presentation information which is information indicating a side on which the counterpart microphone is arranged in the sound collecting device is displayed. The control device according to claim 1 .

3. The display processing unit is As the direction guidance information, self-side microphone presentation information, which is information indicating the side on which the self-side microphone is arranged in the sound collection device, is displayed. The control device according to claim 1 .

4. The display processing unit is As the direction guidance information, the other party's microphone presentation information and self-side microphone presentation information which is information indicating the side on which the self-side microphone is arranged in the sound collecting device are displayed, The information presented by the microphone on the other side is displayed with greater emphasis than the information presented by the microphone on the self side. The control device according to claim 2.

5. The microphone on the other side is a microphone with higher performance than the microphone on the own side. The control device according to claim 1 .

6. The sound collecting device is configured as a smartphone, The local microphone is a microphone arranged at the top of the smartphone, and the remote microphone is a microphone arranged at the bottom of the smartphone. The control device according to claim 5.

7. A placement response processing unit performs a placement direction determination, which is a determination as to whether or not the sound collection device is placed with the other party's microphone facing the other party, based on a sensor detection signal in the sound collection device when the sound collection device is placed, and controls the display unit to display a predetermined alert when a negative result is obtained by the placement direction determination. The control device according to claim 1 .

8. The placement time response processing unit includes: The placement direction is determined based on a detection signal from a motion sensor provided in the sound collection device. The control device according to claim 7.

9. The sound collecting device is configured as a smartphone, The placement time response processing unit includes: The placing direction is determined based on a detection signal of a touch sensor formed on a display screen of the smartphone. The control device according to claim 7.

10. The placement time response processing unit includes: If a negative result is obtained from the placement direction determination, a recording start restriction process is performed to prevent recording from starting. The control device according to claim 7.

11. A direction change response processing unit performs a direction change determination, which is a determination as to whether or not the direction of the sound collection device has become incorrect, based on a result of a voiceprint analysis performed on the sound collection signal of the self-side microphone or the other-side microphone after the start of recording, and controls the display unit to display a predetermined alert when a positive result is obtained by the direction change determination. The control device according to claim 1 .

12. a low-sound-pressure speech detection processing unit that detects low-sound-pressure speech, which is speech with a sound pressure level equal to or lower than a predetermined level, for a sound signal collected by the microphone of the other party, and displays, on the display unit, movement suggestion information that suggests the movement of the sound collection device when the low-sound-pressure speech is detected; The control device according to claim 1 .

13. The low sound pressure speech corresponding processing unit includes: When the environment in which the sound collecting device is used is determined to be a noisy environment, the low sound pressure speech detection process is not executed. The control device according to claim 12.

14. The low sound pressure speech corresponding processing unit includes: A voiceprint analysis is performed on the signal picked up by the microphone of the other party, When the total number of participants on the other side is x (a natural number), Among the speech sounds detected by the microphone of the other party, only the speech sounds having the top x voiceprints with the highest sound pressure levels are subjected to the low sound pressure speech detection process. The control device according to claim 12.

15. The control device, A sound collection device having a plurality of microphones arranged at different positions on a device body and a display unit for displaying information, in which one of the plurality of microphones is a remote microphone for collecting the speech of the other party in an interview, and the other is a self-side microphone for collecting the speech of the self-side in the interview, the display unit is caused to display direction guidance information for guiding the direction in which the remote microphone can be pointed toward the other party. Control methods.

16. A computer readable program, The computer device realizes a function of performing a process for displaying, on the display unit, direction guidance information that guides a direction in which the other-side microphone can be pointed toward the other side, for the sound collection device having a plurality of microphones arranged at different positions on the device body and a display unit that displays information, in which one of the plurality of microphones is a remote microphone for collecting the other side's speech in an interview and the other is a self-side microphone for collecting the self-side's speech in the interview, on the display unit. program.

17. A recording medium having a computer-readable program recorded thereon, A sound collection device equipped with a plurality of microphones arranged at different positions on the device body and a display unit for displaying information, one of the plurality of microphones being a remote microphone for collecting the speech of the other person in an interview and the other being a self-side microphone for collecting the speech of the self-side in the interview, the sound collection device having a program recorded thereon to cause the computer device to execute a process of displaying on the display unit direction guidance information for guiding the direction in which the remote microphone can be pointed toward the other person. Recording medium.

Images