Display device and display method

Abstract

To make it possible to enhance convenience of annotation work of a user by easily presenting a voice section subjected to machine learning to the user.SOLUTION: A voice learning support device includes a processor, a memory, and a monitor. The processor accepts a specifying operation of a specified section by a user relative to voice data after displaying a signal waveform the voice data on a monitor, determines each of one or more learning target sections used for machine learning among the specified sections, and generates a screen on which a frame line showing each of one or more learning target sections determined on the signal waveform is superimposed to output it to the monitor.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a pronunciation learning support device and a pronunciation learning support method. [Background technology]

[0002] Patent Document 1 discloses an apparatus for discovering and outputting partial shapes or combinations of time series data from time series data, which is a series of numerical values ​​recorded according to time, and which includes a function that enables a user to input the shape of the time series data envisioned by the user using a pointing device and a means for specifying how the shape should be combined. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-61733 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been devised in consideration of the above-described conventional situation, and aims to provide a speech learning support device and a speech learning support method that present to a user in an easy-to-understand manner the speech sections that are the subject of machine learning, and help improve the convenience of the user's annotation work. [Means for solving the problem]

[0005] The present disclosure relates to a display device connected to a monitor that displays audio data, the display device including a processor and a memory, the processor displays a signal waveform of the audio data on the monitor, accepts a user's operation to designate a designated section for the audio data, and determines at least a first target section and a second target section to be displayed on the monitor from the designated designated section, determines a position shifted a first predetermined section from a start position of the first target section as an end position of the first target section, determines a position shifted a second predetermined section from the start position of the first target section as a start position of the second target section, and determines a position shifted a first predetermined section from the start position of the second target section as an end position of the second target section, and determines the position shifted a first predetermined section from the start position of the second target section as an end position of the second target section, and adjusts the position of the first target section so that the second target section overlaps with the first target section. 2 and indicating the first target section including a start point position and an end point position of the first target section. 1. The frame line and the start and end positions of the second target section 2 Indicates the target section Second A screen in which the frame line is superimposed on the signal waveform is generated and output to the monitor. The first frame line and the second frame line are in a shape other than a rectangle. The present invention provides a display device.

[0006] The present disclosure also provides a method for determining at least a first target section and a second target section to be displayed on the monitor from the designated designated section, determining a position shifted a first predetermined section from a start position of the first target section as an end position of the first target section, determining a position shifted a second predetermined section from the start position of the first target section as a start position of the second target section, and determining a position shifted a first predetermined section from the start position of the second target section as an end position of the second target section, and adjusting the position so that the second target section overlaps with the first target section. 2 and indicating the first target section including a start point position and an end point position of the first target section. 1. The frame line and the start and end positions of the second target section2 Indicates the target section Second and a processor for generating a screen in which a frame line is superimposed on the signal waveform and outputting the screen to the monitor. The first frame line and the second frame line are in a shape other than a rectangle. The present invention provides a display device.

[0007] The present disclosure also relates to a display method performed by a terminal device, which includes: displaying a signal waveform of audio data on a monitor; accepting a user's operation to designate a designated section for the audio data; determining at least a first target section and a second target section to be displayed on the monitor from the designated designated section; determining a position shifted a first predetermined section from a start position of the first target section as an end position of the first target section; determining a position shifted a second predetermined section from the start position of the first target section as a start position of the second target section; determining a position shifted a first predetermined section from the start position of the second target section as an end position of the second target section; and determining the first predetermined section from the start position of the second target section as an end position of the second target section, and adjusting the first and second target sections so that the second target section overlaps the first target section. 2 and indicating the first target section including a start point position and an end point position of the first target section. 1. A frame line and the first target section including a start point position and an end point position of the second target section are shown. Second A screen is generated in which the frame line is superimposed on the signal waveform and output. The first frame line and the second frame line are in a shape other than a rectangle. Provide a method for displaying Effect of the Invention

[0008] According to the present disclosure, speech segments to be subjected to machine learning can be presented to a user in an easy-to-understand manner, thereby helping to improve the convenience of the user's annotation work. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of an internal configuration of a terminal device according to an embodiment; [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of annotation editing software in a terminal device according to an embodiment; [Diagram 3] 1 is a flowchart showing an example of an operation procedure in a user operation reception unit; [Figure 4] A flowchart showing an example of a procedure for automatically determining a learning section in an automatic learning section determination unit. [Diagram 5] FIG. 1 is a diagram for explaining a specified section specified by a user and each of a plurality of learning target sections. [Figure 6] FIG. 1 is a diagram for explaining an example of a learning section; [Figure 7] A flowchart showing an example of a procedure for excluding a learning section in an automatic learning section correction unit. [Figure 8] A flowchart showing an example of a correction process procedure for a learning section in a learning section automatic correction unit. [Figure 9] FIG. 13 is a diagram showing an example of a learning section after exclusion processing and correction processing; [Figure 10] A diagram showing an example of an annotation editing screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Background to the embodiment) In recent years, there are voice recognition applications that use AI (Artificial Intelligence). Voice recognition applications identify specific sounds (e.g., sounds occurring in urban areas, abnormal sounds, etc.) or human emotions based on the voices picked up through a microphone. However, in order to make it possible to identify the target voices, such voice recognition applications require annotation processing to indicate the target voices among the voices picked up as machine learning data.

[0011] Here, annotation methods for voice recognition include associating voice with text, associating one label (e.g., a label indicating the object of recognition) with one voice file, or associating one learning target section based on an arbitrarily selected start point and end point on a time axis in one voice file as one label. The annotation method for associating voice with text is done manually by the user, and is therefore laborious and time-consuming.

[0012] However, if the learning target section associated with the label includes a section inappropriate for learning (for example, a silent section of a predetermined time or more), the voice recognition application may not be able to perform effective learning. Specifically, the voice recognition process using AI is performed on a voice of a certain time section (for example, 100 ms, 1 s, etc.), and when learning a learning target section of an arbitrary length, the selected learning target section is divided into certain time sections, and learning and estimation of a recognition target are performed for each divided certain time section. If the divided certain time section is inappropriate for learning, the voice recognition application learns this inappropriate section as a recognition target, so that learning may not be performed effectively. Furthermore, since the learning of this voice recognition application is performed as an internal process, the user cannot know whether the learning target section includes a section inappropriate for learning.

[0013] Hereinafter, with reference to the drawings as appropriate, an embodiment that specifically discloses the configuration and operation of the speech learning support device and speech learning support method according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0014] Here, the terms used in the following description are illustrative and not limiting. For example, the terms "section" and "position" include playback times on the audio data 12B.

[0015] First, the internal configuration of a terminal device P1 as an example of a pronunciation learning support device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the internal configuration of a terminal device P1 according to an embodiment.

[0016] The terminal device P1 can accept user operations and generates learning data (so-called teacher data) for machine learning to identify a specific voice from any voice data 12B using AI (Artificial Intelligence). The terminal device P1 can support annotation work on voice data by user operations, and executes a learning section selection process to divide any voice section (machine learning section) designated as a learning section by user operations into one or more learning section suitable for machine learning, or to correct the learning section to a learning section suitable for machine learning. The terminal device P1 also generates an annotation editing screen SC (see FIG. 10) in which each of the one or more learning section determined on the voice data is indicated by a frame line and displays it on the monitor 14, thereby presenting each of the one or more learning sections to the user.

[0017] The terminal device P1 can accept user operations and is realized by, for example, a smartphone, a tablet terminal, a PC (Personal Computer), a notebook PC, or the like. The terminal device P1 includes a processor 11, a memory 12, an input unit 13, a monitor 14, and a speaker 15. In the following description, the terminal device P1 will be described as having the voice data 12B stored in the memory 12 in advance. However, the terminal device P1 may acquire the voice data 12B from an external storage medium such as a CD-ROM (Compact Disc Read Only Memory), a USB memory, an SD (registered trademark) card, a smartphone, or a voice recorder, or may acquire the voice data 12B from a device capable of collecting sound such as a microphone (not shown) connected to the terminal device P1 so as to be capable of data communication. Furthermore, the terminal device P1 may include a communication unit (not shown) and acquire the voice data 12B from an external terminal (for example, a server, another terminal device, or the like) connected to the terminal device P1 so as to be capable of data communication via the Internet (not shown) by the communication unit.

[0018] The processor 11, which is an example of an output unit, is configured using, for example, a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 refers to the programs and data stored in the memory 12, and executes the programs to realize the functions of each unit and the functions of the annotation editing software 11A.

[0019] The processor 11 may also generate training data for identifying a specific voice from any voice data 12B using AI based on the edited data 12A after annotation work generated by the annotation editing software 11A. The training for generating the training data may be performed using one or more statistical classification techniques. Examples of statistical classification techniques include Linear Classifiers, Support Vector Machines, Quadratic Classifiers, Kernel Estimation, Decision Trees, Artificial Neural Networks, Bayesian Techniques and / or Networks, Hidden Markov Models, Binary Classifiers, Multi-Class Classifiers, Clustering Technique, Random Forest Technique, Logistic Regression Technique, Linear Regression Technique, Gradient Boosting Technique, etc. However, the statistical classification techniques used are not limited to these.

[0020] The memory 12 has a storage device including a semiconductor memory such as a random access memory (RAM) and a read only memory (ROM), and a storage device such as a solid state drive (SSD) or a hard disk drive (HDD). The memory 12 stores edited data 12A and voice data 12B. When the processor 11 generates learning data, the memory 12 may store the generated learning data. The edited data 12A here is data generated by the annotation editing software 11A, and is data in which information on the voice data 12B, information on a designated section to be subjected to machine learning in the voice data 12B (specifically, information on the start and end positions of the designated section), information on the start and end points of one or more learning sections determined for the designated section, and the label name of the designated section are associated with each other.

[0021] The input unit 13 is a user interface capable of accepting user operations and configured using, for example, a mouse, a keyboard, a touch panel, etc. The input unit 13 converts the accepted user operations into electrical signals (control commands) and outputs them to the processor 11.

[0022] The monitor 14 is configured using a display such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display, etc. The monitor 14 displays the annotation editing screen SC (see FIG. 10) output from the processor 11.

[0023] When a user performs an operation to reproduce the audio data 12B, the speaker 15 outputs the audio of the audio data 12B.

[0024] Next, a functional configuration of the annotation editing software 11A will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of a functional configuration of the annotation editing software 11A of the terminal device P1 according to the embodiment.

[0025] The annotation editing software 11A includes a user operation receiving unit 11B, a user-specified section determination unit 11C, a learning section automatic determination unit 11D, a learning section automatic correction unit 11E, a learning section data management unit 11F, a learning section display unit 11G, a voice data selection unit 11H, and a voice data display unit 111. Note that the configuration of the learning section automatic correction unit 11E in the annotation editing software 11A is not essential and may be omitted, or may be added as an optional function according to the user's request.

[0026] The user operation receiving unit 11B receives a user's designation operation for a section to be machine-learned in any of the voice data 12B selected as a target for annotation editing by the user. The user operation receiving unit 11B receives an operation to designate each of a start point UR1 and an end point UR2 of the designated section UR designated by the user operation, and outputs each of information on the start point UR1 and the end point UR2 to the user-designated section determination unit 11C.

[0027] The user-specified section determination unit 11C determines the specified section UR based on the information on the start point UR1 and the end point UR2 of the specified section UR output from the user operation reception unit 11B. The user-specified section determination unit 11C outputs information on the determined specified section UR to the learning section automatic determination unit 11D.

[0028] The automatic learning section determination unit 11D determines one or more learning sections based on the information of the specified section UR output from the user-specified section determination unit 11C. The automatic learning section determination unit 11D outputs information of the determined learning section to the automatic learning section correction unit 11E. Note that here, if the automatic learning section correction unit 11E is not included in the configuration of the annotation editing software 11A, the automatic learning section determination unit 11D may output information of the determined learning section to the learning section data management unit 11F. The automatic learning section determination unit 11D may also output information of the determined learning section to the automatic learning section correction unit 11E and the learning section data management unit 11F.

[0029] The learning target section automatic correction unit 11E judges whether or not each of the one or more learning target sections output from the learning target section automatic determination unit 11D is a learning target section that is valid for executing machine learning. When the learning target section automatic correction unit 11E judges that the learning target section is not valid for executing machine learning, it executes a process of removing the learning target section from the target of machine learning (i.e., a process of excluding the learning target section) or a process of correcting the section of the learning target section. Note that each process executed by the learning target section automatic correction unit 11E may be executed in its entirety, or only one of the processes specified by the user may be executed. The learning target section automatic correction unit 11E outputs information on each of the one or more learning target sections after the exclusion process or the correction process to the learning target section data management unit 11F.

[0030] The learning target section data management unit 11F manages information on the designated section UR designated by the user (i.e., information on the start point UR1 and end point UR2 of the designated section UR), information on the start point and end point of each of one or more learning target sections determined for this designated section UR, and a label name entered in the label input field LB (see FIG. 10) in association with each other, and outputs the information to the learning target section display unit 11G. The learning target section data management unit 11F may generate edited data 12A based on the information on the designated section UR, information on the start point and end point of each of one or more learning target sections, and the label name, and output the edited data 12A to the memory 12 for registration.

[0031] The learning target section display unit 11G generates an annotation editing screen SC (see FIG. 10) in which a frame line indicating each of the one or more registered learning sections is superimposed on at least one of the signal waveform data WF1 or the frequency spectrum data SP1 of the voice data 12B selected by the user, based on the information of the designated section UR output from the learning target section data management unit 11F and the information of the start point and end point of each of the one or more learning target sections. The learning target section display unit 11G outputs the generated annotation editing screen SC to the monitor 14 for display.

[0032] The voice data selection unit 11H acquires the voice data 12B by referring to the memory 12 based on the information of the voice data 12B output from the user operation acceptance unit 11B. The voice data selection unit 11H outputs the acquired voice data 12B to the voice data display unit 11I.

[0033] The voice data display unit 11I generates an annotation editing screen (not shown) including the signal waveform data WF1 and the frequency spectrum data SP1 of the voice data 12B based on the voice data 12B output from the voice data selection unit 11H, and outputs and displays it on the monitor 14. The annotation editing screen (not shown) generated by the voice data display unit 11I is a screen that is displayed on the monitor 14 before accepting a designation operation of a designated section UR by the user.

[0034] First, the operation procedure of the user operation acceptance unit 11B will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the operation procedure of the user operation acceptance unit 11B in the terminal device P1 according to the embodiment. Note that, as an example of the operation procedure of the user operation acceptance unit 11B described with reference to Fig. 3, an example of accepting a user operation by a mouse will be described, but it goes without saying that the present invention is not limited to this.

[0035] First, the processor 11 starts the annotation editing software 11A based on a user operation. The user operation accepting unit 11B accepts a selection operation of the voice data 12B to be the target of annotation editing based on the user operation accepted by the input unit 13. The user operation accepting unit 11B outputs information on the selected voice data 12B to the voice data selecting unit 11H.

[0036] The voice data selection unit 11H refers to the memory 12 and acquires the voice data 12B based on the information of the voice data 12B output from the user operation acceptance unit 11B. The voice data selection unit 11H outputs the acquired voice data 12B to the voice data display unit 11I. The voice data display unit 11I generates an annotation editing screen (not shown) including signal waveform data WF1 of the voice data 12B and frequency spectrum data SP1 of the voice data 12B based on the voice data 12B output from the voice data selection unit 11H, and outputs it to the monitor 14 for display. The signal waveform data WF1 has a vertical axis indicating sound pressure level and a horizontal axis indicating time. Moreover, the frequency spectrum data SP1 has a vertical axis indicating frequency and a horizontal axis indicating time.

[0037] The user operation accepting unit 11B judges whether or not the position of the cursor linked to the mouse operated by the user is within the waveform display area based on a control command transmitted from the input unit 13 capable of accepting user operations (St11). Note that the waveform display area here is an area including at least one of the display area AR1 of the signal waveform data WF1 and the display area AR2 of the frequency spectrum data SP1 on the annotation editing screen.

[0038] When the user operation acceptance unit 11B determines in the process of step St11 that the position of the cursor linked to the mouse operated by the user is within the waveform display area (St11, YES), it determines whether or not the user clicked the mouse while the cursor was at any position within the waveform display area (St12). On the other hand, when the user operation acceptance unit 11B determines in the process of step St11 that the position of the cursor linked to the mouse operated by the user is not within the waveform display area (St11, NO), it returns to the process of step St11 again.

[0039] When the user operation acceptance unit 11B determines in the process of step St12 that the user clicked the mouse while the cursor was at any position in the waveform display area (St12, YES), it accepts the designation operation of the start point UR1 in the designated section UR used for machine learning (St13) and outputs the time of the voice data 12B corresponding to the cursor position where this operation was performed to the user-designated section determination unit 11C. On the other hand, when the user operation acceptance unit 11B determines in the process of step St12 that the user did not click the mouse while the cursor was at any position in the waveform display area (St12, YES), it returns to the process of step St12.

[0040] The user operation acceptance unit 11B judges whether the state where the user clicked the mouse is being held (maintained) (St14). When the user operation acceptance unit 11B judges in the process of step St14 that the state where the user clicked (selected) the mouse is being held (maintained) (St14, YES), the process returns to the process of step St14. On the other hand, when the user operation acceptance unit 11B judges in the process of step St14 that the state where the user clicked (selected) the mouse is ended (St14, NO), the user operation acceptance unit 11B accepts the designation operation of the end point UR2 in the designated section UR used for machine learning (St15), and outputs the time of the voice data 12B corresponding to the cursor position where this operation was performed to the user-designated section determination unit 11C.

[0041] The user-specified section determination unit 11C determines one specified section UR specified by the user by associating the start point UR1 and the end point UR2 of the specified section UR output from the user operation reception unit 11B with each other. The user-specified section determination unit 11C outputs information on the determined specified section UR to the learning section automatic determination unit 11D.

[0042] The user operation acceptance unit 11B may accept the operation of specifying each of the start point UR1 and the end point UR2 of the specified section UR by inputting the time corresponding to the start point UR1 and the time corresponding to the end point UR2, respectively. For example, in such a case, the user operation acceptance unit 11B accepts the input operation of the time corresponding to each of the start point and the end point in the annotation editing screen SC (see FIG. 10) displayed on the monitor 14. When the user operation acceptance unit 11B determines that the time corresponding to each of the start point and the end point has been input in an input field SF1 capable of accepting the input operation of the time corresponding to each of the start point and the end point, it accepts the input operation of one specified section by the user. The user-specified section determination unit 11C determines one specified section based on the time corresponding to each of the start point and the end point input in the input field SF1.

[0043] Furthermore, the user operation reception unit 11B may automatically correct the times of the specified start and end points to predetermined time intervals (for example, 0.1 seconds, 0.5 seconds, etc.) when setting the start point UR1 and end point UR2 of the specified section UR.

[0044] Next, the operation procedure of the learning section automatic determination unit 11D will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a flow chart showing an example of the procedure for automatically selecting a learning section in the learning section automatic determination unit 11D. Fig. 5 is a diagram explaining a specified section UR specified by a user and each of a plurality of learning sections. Fig. 6 is a diagram explaining an example of a learning section.

[0045] In addition, the frame line FR1 indicating the designated section UR and the frame lines r11, r12, r13, r14, r15, r16, and r17 indicating each of the multiple learning sections shown in FIG. 5 are superimposed only on the signal waveform data WF1, but they may be superimposed on the frequency spectrum data SP1, or may be superimposed on each of the signal waveform data WF1 and the frequency spectrum data SP1. In the example shown in FIG. 5, the shapes of the frame lines FR1 and r11 to r17 are all elliptical, but it goes without saying that this is not limited to this. The shapes of the frame lines FR1 and r11 to r17 may be any shape other than rectangular (for example, triangular, rhombus, etc.). In addition, the shape of the frame line FR1 indicating the designated section and the shape of the frame lines r11 to r17 indicating each of the learning sections may not be the same shape. Other examples of the shape of the frame lines will be described below.

[0046] The shape of the frame line may be any shape formed by one or more straight lines and one or more curved lines (e.g., a semicircle, a shape obtained by cutting an ellipse at any position and angle, etc.), or any shape formed by multiple curved lines. For example, a frame line having an elliptical shape may be a shape formed by two curved lines, or two curved lines and two straight lines. The shape of the frame line may also be a shape having one or more acute or obtuse angles. Furthermore, the shape of the frame line may be a shape having one or more curved lines and one or more acute or obtuse angles, such as a sector shape.

[0047] The shape of the frame line may be a shape formed by an upper side portion and a lower side portion, and the upper side portion and the lower side portion may be a shape that is non-parallel to each other. Each of the upper side portion and the lower side portion here includes one or more straight lines, one or more curved lines, or one or more straight lines and one or more curved lines. For example, when the shape of the frame line is a triangle, the frame line is formed by an upper side portion including any two straight lines out of three straight lines that form the triangle, and a lower side portion including one straight line. Note that the one or more straight lines or the one or more curved lines included in the upper side portion and the lower side portion are non-parallel to the horizontal axis (i.e., the time axis) of the signal waveform data WF1 and the frequency spectrum data SP1.

[0048] Furthermore, the shape of the frame may be such that, at the center point of any shape formed by the frame, the length in the direction corresponding to the horizontal axis of the signal waveform data WF1 and the frequency spectrum data SP1 is different from the length in the direction corresponding to the vertical axis of the signal waveform data WF1 and the frequency spectrum data SP1. This allows the terminal device P1 to improve the visibility of each of the adjacent frame lines.

[0049] In FIG. 6, only the start point and end point of the first learning section are shown, and the start points and end points of the second and subsequent learning sections are not shown.

[0050] The automatic learning section determination unit 11D acquires information on the designated section UR output from the user-designated section determination unit 11C (St21). The automatic learning section determination unit 11D starts the process of determining the first learning section based on the acquired information on the designated section UR. The automatic learning section determination unit 11D determines the start point UR1 of the designated section UR to be the start point bx1 of the first learning section (St22).

[0051] The automatic learning section determination unit 11D determines the position of the predetermined processing section width PR1 (i.e., the time range to be learned) from the start point bx1 of the first learning section to the end point ex1 of the first learning section (St23). The number of samples included in the predetermined processing section width PR1 is, for example, 1500 samples or 1600 samples. The predetermined processing section width PR1 may be a width (number of samples) larger or smaller than the shift sample number A3 described later, and may be set to any value (number of samples) in advance by the user, or may be set to a predetermined value based on the size of the specified section UR specified by the user. If the predetermined processing section width PR1 is a width smaller than the shift sample number A3, the automatic learning section determination unit 11D determines the learning section while skipping some sections.

[0052] The automatic learning section determination unit 11D newly registers the section [bx1, ex1] indicated by the start point bx1 and end point ex1 of the determined first learning section as the first learning section (St24). Note that the registration process here is a process in which the automatic learning section determination unit 11D associates information on one specified section UR with information on the determined learning section, and outputs the information to the learning section data management unit 11F for storage.

[0053] The automatic learning section determination unit 11D determines the start point bx2 (not shown) of the second learning section to be at a position shifted by the shift sample number A3 from the start point bx1 of the first learning section (St25). Note that the number of samples in the shift sample number A3 referred to here is, for example, 30% or 40% of the processing section width PR1, and any number of samples may be set by the user. For example, the number of samples in the shift sample number A3 is set to a smaller number when the learning section is set to a smaller section, and is set to a larger number when the learning section is set to a larger section.

[0054] The automatic learning section determination unit 11D repeatedly executes the process of determining the start and end points of the learning section shown in steps St23 to St25, and the process of registering each of the determined one or more learning sections. If the automatic learning section determination unit 11D determines in the process of step St24 that the end point ex(N+1) of the (N+1)th learning section (N: an integer equal to or greater than 1) falls outside the designated section UR designated by the user, it registers each of the N learning sections from the first learning section to the Nth learning section in the designated section UR, and ends the learning section determination process.

[0055] Specifically, in the example shown in Figure 5, the automatic learning target section determination unit 11D newly registers the seventh learning target section, then determines that the end point of the eighth learning target section extends beyond the end point UR2 of the designated section UR specified by the user, and registers seven learning target sections from the first learning target section to the seventh learning target section to the designated section UR.

[0056] The automatic learning target section determination unit 11D associates information on each of the start point UR1 and end point UR2 of a specified section UR with information on each of the determined one or more learning target sections, and outputs the information to the automatic learning target section correction unit 11E and the learning target section data management unit 11F.

[0057] The learning target section display unit 11G superimposes a frame line FR1 surrounding the start point UR1 to the end point UR2 of a specified section UR output from the learning target section data management unit 11F on at least one of the signal waveform data WF1 and the frequency spectrum data SP1 based on the information on the start point UR1 and the end point UR2 of the specified section UR.

[0058] The learning section display unit 11G also superimposes frame lines r11-r17 surrounding the start point to the end point of each of the one or more learning sections on at least one of the signal waveform data WF1 and the frequency spectrum data SP1 based on the information on the start point and end point of each of the one or more learning sections output from the learning section data management unit 11F. The learning section display unit 11G generates an annotation editing screen on which the frame lines FR1, r11-r17 indicating the specified section and one or more learning sections are superimposed, and outputs the annotation editing screen to the monitor 14.

[0059] Here, in the example shown in FIG. 5 and FIG. 6, a frame line r11 indicates the first learning target section, and surrounds the first learning target section from the start point bx1 to the end point ex1. Similarly, a frame line r12 surrounds the second learning target section from the start point bx2 (not shown) to the end point ex2 (not shown). A frame line r13 surrounds the third learning target section from the start point bx3 (not shown) to the end point ex3 (not shown). A frame line r13 surrounds the fourth learning target section from the start point bx4 (not shown) to the end point ex4 (not shown). A frame line r13 surrounds the fifth learning target section from the start point bx5 (not shown) to the end point ex5 (not shown). A frame line r13 surrounds the sixth learning target section from the start point bx6 (not shown) to the end point ex6 (not shown). A frame line r13 surrounds the seventh learning target section from the start point bx7 (not shown) to the end point ex7 (not shown).

[0060] Next, an exclusion process performed by the learning target section automatic correcting unit 11E will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of an exclusion process for the learning target section in the learning target section automatic correcting unit 11E.

[0061] The learning target section automatic correcting unit 11E acquires information on one of the one or more learning target sections determined by the learning target section automatic determining unit 11D (St31). Here, as an example, a case will be described in which the learning target section automatic correcting unit 11E acquires information on the k-th learning target section and corrects the k-th learning target section.

[0062] The learning section automatic correction unit 11E calculates the average volume L of the acquired k-th learning section (St32), and judges whether the calculated average volume L is less than the volume specified value A1 (St33). Note that the volume specified value A1 here may be a fixed value determined based on a pre-set condition, such as -50 dB full scale, when the audio data 12B is a 16-bit digital sound. The volume specified value A1 may be a value obtained by adding a predetermined sound pressure level (for example, 6 dB, 8 dB, etc.) to the minimum sound pressure level of the audio data 12B, or may be a value obtained by determining a sound pressure level to be added based on the value of the minimum sound pressure level of the audio data 12B and adding the determined predetermined sound pressure level to the minimum sound pressure level.

[0063] If the automatic learning section correction unit 11E determines in step St33 that the calculated average volume L is less than the volume specified value A1 (St33, YES), it excludes the kth learning section from the machine learning targets (St34) and ends the correction process for the kth learning section. On the other hand, if the automatic learning section correction unit 11E determines in step St33 that the calculated average volume L is not less than the volume specified value A1 (St33, NO), it determines that the deletion process for the kth learning section is unnecessary and omits the deletion process.

[0064] The learning target section automatic correcting unit 11E executes the process shown in steps St31 to St34 for each of all learning target sections determined by the learning target section automatic determining unit 11D. If the learning target section automatic correcting unit 11E determines that the process shown in steps St31 to St34 has been executed for each of all learning target sections, it ends the deletion process shown in FIG.

[0065] Next, a correction process performed by the learning target section automatic correcting unit 11E will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of a correction process for the learning target section in the learning target section automatic correcting unit 11E.

[0066] The learning target section automatic correcting unit 11E acquires information on one of the one or more learning target sections determined by the learning target section automatic determining unit 11D (St41). Here, as an example, a case will be described in which the learning target section automatic correcting unit 11E acquires information on the k-th learning target section and corrects the k-th learning target section.

[0067] The learning section automatic correction unit 11E calculates the total time T1 of the sections exceeding the volume specified value A2 from the acquired k-th learning section (St42). Note that the volume specified value A2 here may be a fixed value determined based on a pre-set condition, such as -50 dB full scale when the audio data 12B is a 16-bit digital sound. The volume specified value A2 may be a value obtained by adding a predetermined sound pressure level (for example, 6 dB, 8 dB, etc.) to the minimum sound pressure level of the audio data 12B, or may be a value obtained by determining a sound pressure level to be added based on the value of the minimum sound pressure level of the audio data 12B and adding the determined predetermined sound pressure level to the minimum sound pressure level. Furthermore, the volume specified value A2 may be the same value as the volume specified value A1.

[0068] The learning section automatic correction unit 11E determines whether the calculated total time T1 is less than a predetermined time B (St43). Note that the predetermined time B is determined based on the time from the start point bxk to the end point exk of the k-th learning section, and is, for example, 40% or 50% of the time from the start point bxk to the end point exk.

[0069] If the learning section automatic correcting unit 11E determines in the process of step St43 that the calculated total time T1 is less than the predetermined time B (St43, YES), it extracts a section that exceeds the volume specified value A2 from this k-th learning section, and obtains information on the first position xk (time) of the extracted section (St44). On the other hand, if the learning section automatic correcting unit 11E determines in the process of step St44 that the calculated total time T1 is not less than the predetermined time B (St44, NO), it determines that correction processing for this k-th learning section is unnecessary, and omits the correction processing.

[0070] The learning section automatic correction unit 11E calculates the difference section (deviation) between the acquired position xk and the start point bxk of the k-th learning section. The learning section automatic correction unit 11E determines whether the calculated difference section (deviation) is less than the number of shift samples A3 (St45).

[0071] If the learning section automatic correcting unit 11E determines in the process of step St45 that the calculated difference section (deviation) is less than the number of shift samples A3 (St45, YES), it updates (changes) the start point of this k-th learning section to position xk (St46). On the other hand, if the learning section automatic correcting unit 11E determines in the process of step St45 that the calculated difference section (deviation) is not less than the number of shift samples A3 (St45, NO), it determines that correction processing for this k-th learning section is unnecessary and omits the correction processing.

[0072] The learning target section automatic correcting unit 11E executes the correction process shown in steps St41 to St46 for each of all learning target sections determined by the learning target section automatic determining unit 11D. If the learning target section automatic correcting unit 11E determines that the correction process shown in steps St41 to St46 has been executed for each of all learning target sections, it ends the correction process shown in FIG.

[0073] Here, an example of the learning target section after the exclusion process and correction process by the learning target section automatic correction unit 11E will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the learning target section after the exclusion process and correction process. Fig. 9 is a diagram showing a part of the annotation editing screen after each of the seven learning target sections shown in Fig. 5 has been corrected to each of the five learning target sections by the exclusion process and correction process by the learning target section automatic correction unit 11E.

[0074] In Fig. 9, the five learning target sections are indicated by five elliptical frame lines r21, r22, r23, r24, and r25. Of the five learning target sections shown in Fig. 9, the first learning target section indicated by frame line r21 corresponds to the first learning target section indicated by frame line r11 shown in Fig. 5, the second learning target section indicated by frame line r22 corresponds to the third learning target section indicated by frame line r13 shown in Fig. 5, the third learning target section indicated by frame line r23 corresponds to the fourth learning target section indicated by frame line r14 shown in Fig. 5, the fourth learning target section indicated by frame line r24 corresponds to the fifth learning target section indicated by frame line r15 shown in Fig. 5, and the fifth learning target section indicated by frame line r25 corresponds to the sixth learning target section indicated by frame line r16 shown in Fig. 5.

[0075] Here, in the example shown in Fig. 9, the second learning target section indicated by frame line r12 in Fig. 5 and the seventh learning target section indicated by frame line r17 have been deleted by being excluded from the machine learning targets through processing by the learning target section automatic correction unit 11E (specifically, the processing of step St34 shown in Fig. 7). Also, in the example shown in Fig. 9, the position of the start point of the fifth learning target section indicated by frame line r15 in Fig. 5 has been changed through processing by the learning target section automatic correction unit 11E (specifically, the processing of step St46 shown in Fig. 8).

[0076] In this way, the learning target section automatic correcting unit 11E can exclude (delete) a learning target section that is determined to be ineffective by machine learning from among the learning target sections determined by the learning target section automatic determining unit 11D. This allows the learning target section automatic correcting unit 11E to exclude a silent section or a learning target section that has a low volume and is ineffective for machine learning from among the determined learning target sections.

[0077] Furthermore, the learning target section automatic correcting unit 11E can correct the learning target section by changing the start position of the learning target section determined by the learning target section automatic determination unit 11D to be ineffective by machine learning. In this way, the learning target section automatic correcting unit 11E can correct the determined learning target section so that it includes more sections that are equal to or greater than the volume specified value A2, and therefore can determine an effective learning target section by machine learning.

[0078] Next, the annotation editing screen SC displayed on the monitor 14 will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the annotation editing screen SC.

[0079] The annotation editing screen SC is generated to include at least the signal waveform data WF2 of the voice data 12B, the frequency spectrum data SP2, and the label input field LB. When the annotation editing screen SC receives input of the start point UR3 and the end point UR4 of the designated section by a user operation, a frame line FR2 indicating the designated section and each of frame lines r31, r32, r33, r34, r35, and r36 indicating one or more learning sections determined based on the designated section are superimposed on either the signal waveform data WF2 or the frequency spectrum data SP2.

[0080] 10, the shapes of the frame lines FR2, r31 to r36 are all elliptical, but needless to say, are not limited to this. The shapes of the frame lines FR2, r31 to r36 may be any shape other than rectangular (e.g., triangular, diamond, etc.). The shape of the frame line FR2 indicating the specified section and the shape of the frame lines r31 to r36 indicating each of the learning sections do not have to be the same shape.

[0081] Furthermore, the user operation acceptance unit 11B may automatically correct the time of the specified start point and end point to a predetermined time interval (e.g., 0.1 seconds, 0.5 seconds, etc.) when setting the start point UR1 and end point UR2 of the specified section UR. For example, in the input field SF1 shown in FIG. 10, the position (time) of the start point UR3 of the specified section is input as "0:02.266" and the position (time) of the end point UR4 is input as "0:06.102". In such a case, the user operation acceptance unit 11B may automatically correct the specified start point UR3 to "0:02" and the end point UR4 to "0:06" based on the contents input in the input field SF1.

[0082] As a result, the annotation editing software 11A can assist the user in specifying the start and end points of a specified section not only by inputting the start and end points into the input field SF1 described above, but also by automatically correcting the position (time) of the start and end points of the specified section to round times even if the user shakes their hand when specifying the section using a user interface such as a mouse or touch panel.

[0083] The Add button BT1 is a button for performing a process of adding a new specified section. When the Add button BT1 is pressed (selected) by a user operation, the annotation editing software 11A accepts the addition of a new specified section.

[0084] The update button BT2 is a button that updates (changes) the specified section based on the input contents of the times corresponding to the start and end points of the specified section entered in the input field SF1, and registers (records) the label name of the specified section entered in the label input field LB, etc., in association with the specified section.

[0085] The Delete button BT3 is a button for deleting any of the designated sections or any one or more study sections designated by a user operation. When the Delete button BT3 is pressed (selected) by a user operation in a state where any of the designated sections or any one or more study sections are selected (designated), the annotation editing software 11A deletes the selected (designated) designated sections or study sections.

[0086] The Play button BT4 is a button for playing back the audio data 12 B. When the Play button BT4 is pressed (selected) by a user operation, the annotation editing software 11A plays back the audio data 12 B being edited.

[0087] The Stop button BT5 is a button for stopping the playback of the voice data 12 B. When the Stop button BT5 is pressed (selected) by a user operation, the annotation editing software 11A stops the playback of the voice data 12 B being edited.

[0088] The input field SF1 is an input field for receiving the times corresponding to the start point and the end point of the specified section. When the time corresponding to the start point or the end point of the specified section is input into the input field SF1 by a user operation, the annotation editing software 11A determines the time period from the input start point to the end point as the specified section.

[0089] The label input field LB is an input field for accepting input of a label name to be set for each designated section. When a label name that the user wants to set for the designated section is input into the label input field LB by a user operation, the annotation editing software 11A associates the input label name with the information of the designated section and each of the information of the one or more determined learning sections, and outputs and registers the data as edited data 12A to the memory 12.

[0090] As described above, the terminal device P1 (an example of a speech learning support device) according to the embodiment includes a processor 11, a memory 12, and a monitor 14. The processor 11 displays the signal waveform of the speech data 12B (for example, the signal waveform data WF2 and the frequency spectrum data SP2 shown in FIG. 10) on the monitor 14, accepts a user's designation operation of a designated section (specifically, each of the start point UR3 and the end point UR4 of the designated section) for the speech data 12B, determines each of one or more learning target sections to be used for machine learning among the designated designated sections, generates an annotation editing screen SC (an example of a screen) in which a frame line (for example, each of the frame lines r31 to r36 shown in FIG. 10) indicating each of the determined one or more learning target sections is superimposed on the signal waveform, and outputs the generated annotation editing screen SC to the monitor 14.

[0091] As a result, the terminal device P1 in the embodiment automatically determines one or more learning target sections to be subject to machine learning for the specified sections specified by the user, and displays an annotation editing screen SC in which the determined one or more learning target sections are superimposed on the signal waveform data WF2 or frequency spectrum data SP2 of the audio data 12B, thereby presenting each of the learning target sections as audio sections to be subject to machine learning in an easy-to-understand manner to the user, and helping to improve the convenience of the user's annotation work.

[0092] In addition, as described above, the frame lines showing each of the one or more learning target sections are polygonal shapes other than rectangular. As a result, the terminal device P1 according to the embodiment can further improve the visibility of each of the one or more learning target sections displayed on the annotation editing screen SC, since the shape of the monitor 14, which has a rectangular shape, is different from the shape of the superimposed frame lines. In addition, the terminal device P1 can further improve the visibility of each of the one or more learning target sections displayed on the annotation editing screen SC, since the shape of the display areas AR1, AR2 of the signal waveform data WF2 and the frequency spectrum data SP2 displayed on the monitor 14 (i.e., rectangular) is different from the shape of the superimposed frame lines.

[0093] In addition, as described above, the frame lines showing each of the one or more learning target sections are circular in shape other than a perfect circle. As a result, the terminal device P1 according to the embodiment can further improve the visibility of each of the one or more learning target sections displayed on the annotation editing screen SC because the shape of the monitor 14 having a rectangular shape or the shape of the display areas AR1 and AR2 of the signal waveform data WF2 and the frequency spectrum data SP2 (i.e., rectangular) is different from the shape of the superimposed frame lines. In addition, the terminal device P1 can further improve the visibility of each of the one or more learning target sections displayed on the annotation editing screen SC because the frame lines are not parallel to the four sides of the monitor 14 formed in a rectangular shape, the four sides of the display areas AR1 and AR2 of the signal waveform data WF2 and the frequency spectrum data SP2, or the straight lines showing the vertical and horizontal axes of the signal waveform data WF2 and the frequency spectrum data SP2. In addition, the terminal device P1 can improve the visibility even if adjacent frame lines overlap each other by superimposing the frame lines in a circular shape other than a perfect circle.

[0094] As described above, each of the one or more learning target sections determined by the terminal device P1 according to the embodiment is superimposed with a frame line having an ellipse, a triangle, or a diamond shape. As a result, the terminal device P1 according to the embodiment shows each of the one or more learning target sections with a frame line having a shape other than a rectangle, and therefore, since any one of the four sides of the monitor 14 formed in a rectangular shape is not parallel to the superimposed frame line, the visibility of each of the one or more learning target sections displayed on the annotation editing screen SC can be further improved. In addition, the terminal device P1 can further improve the visibility of each of the one or more learning target sections displayed on the annotation editing screen SC because the sides of the rectangular display areas AR1 and AR2 of the signal waveform data WF2 and the frequency spectrum data SP2 displayed on the monitor 14, or the vertical or horizontal axes of the signal waveform data WF2 and the frequency spectrum data SP2, and the superimposed frame line are not parallel to each other (i.e., non-parallel).

[0095] As described above, the processor 11 in the terminal device P1 according to the embodiment calculates the average volume L for each of one or more learning target sections, and excludes from the machine learning the learning target section for which it is determined that the calculated average volume L is less than the specified volume value A1 as a threshold value. This allows the terminal device P1 according to the embodiment to exclude from the determined learning target sections silent sections or learning target sections that have a low volume and are not effective for machine learning.

[0096] As described above, the processor 11 in the terminal device P1 according to the embodiment corrects the time at which the volume first reaches or exceeds the specified volume value A2 in a learning target section determined to have a total time T1 of sections that are equal to or greater than the specified volume value A2 as a predetermined volume within each of one or more learning target sections that is less than the specified time B, to the start point of the learning target section. This allows the terminal device P1 according to the embodiment to correct the position of the start point so that a silent section or a section with a low volume that is not valid by machine learning is not included in the learning target section. Thus, the processor 11 can determine a learning target section that has been automatically corrected from a section included in the learning target section to a section that is valid by machine learning.

[0097] As described above, the processor 11 in the terminal device P1 according to the embodiment excludes the learning target section designated by the user operation from the one or more learning target sections from the machine learning target. As a result, the terminal device P1 according to the embodiment can determine and register each of the one or more learning target sections that are valid by machine learning by excluding the learning target section that is not intended by the user.

[0098] As described above, the processor 11 in the terminal device P1 according to the embodiment generates and outputs the annotation editing screen SC (an example of a screen) including the signal waveform data WF2 and the frequency spectrum data SP2 (an example of spectrum data) of the voice data 12B. As a result, the terminal device P1 according to the embodiment can synchronously display the signal waveform data WF2 and the frequency spectrum data SP2 of the voice data 12B.

[0099] As described above, the processor 11 in the terminal device P1 according to the embodiment generates an annotation editing screen SC (an example of a screen) in which a frame line (for example, each of the frame lines r31 to r36 shown in FIG. 10) indicating the range of one or more learning sections is superimposed on either one of the signal waveform data WF2 of the voice data 12B or the frequency spectrum data SP2 (an example of spectrum data) designated by a user operation. As a result, the terminal device P1 according to the embodiment can further improve usability in annotation editing work by a user. As a result, the annotation editing software 11A can support the user's operation of designating the start point and end point of a designated section by automatically correcting the position (time) of the input start point and the position (time) of the end point to a round time, not only when the user shakes his / her hand during the designation operation using a user interface such as a mouse or a touch panel, but also when the user shakes his / her hand during the designation operation using a user interface such as a mouse or a touch panel.

[0100] As described above, the processor 11 in the terminal device P1 according to the embodiment divides the audio data 12B into predetermined time intervals (e.g., 0.1 seconds, 0.5 seconds, etc.) and corrects the time indicated by the start point or end point of the designated section to the nearest predetermined time among the divided predetermined times. As a result, the annotation editing software 11A in the terminal device P1 according to the embodiment can support the user's operation of specifying the start point and end point of the designated section by automatically correcting the position (time) of the start point and the position (time) of the end point of the designated section to a round number not only when the user shakes his / her hand during the designation operation using a user interface such as a mouse or a touch panel, but also when the user shakes his / her hand during the designation operation using the input field SF1.

[0101] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the various embodiments described above may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0102] INDUSTRIAL APPLICABILITY The present disclosure is useful as a speech learning support device and a speech learning support method that present speech sections that are the subject of machine learning to a user in an easy-to-understand manner and help improve the convenience of the user's annotation work. [Explanation of symbols]

[0103] 11 Processors 11A Annotation editing software 11B User operation reception section 11C User-specified section determination unit 11D Automatic learning section determination unit 11E Automatic correction of learning section 11F Learning Section Data Management Section 11G Learning section display 11H Audio data selection section 11I Audio data display section 12. Memory 12A Editing Data 12B Audio data 13 Input section 14 Monitor P1 terminal equipment FR1,FR2,r11,r12,r13,r14,r15,r16,r17,r21,r22,r23,r24,r25 Border SC annotation editing screen SP1, SP2 frequency spectrum data UR designated section UR1,UR3 start point UR2,UR4 End point WF1, WF2 signal waveform data

Claims

1. A display device connected to a monitor for displaying audio data, The display device includes: A processor; A memory, The processor, displaying a signal waveform of the audio data on the monitor, accepting a user's operation to designate a designated section of the audio data, and determining at least a first target section and a second target section to be displayed on the monitor from among the designated designated sections; determining a position shifted a first predetermined distance from the start position of the first target section as an end position of the first target section, a position shifted a second predetermined distance from the start position of the first target section as the start position of the second target section, and a position shifted a first predetermined distance from the start position of the second target section as the end position of the second target section, and determining the second predetermined distance so that the second target section overlaps with the first target section; generating a screen in which a first frame line indicating the first target section including a start point position and an end point position of the first target section and a second frame line indicating the second target section including a start point position and an end point position of the second target section are superimposed on the signal waveform, and outputting the screen to the monitor; The first frame line and the second frame line are in a shape other than a rectangle. Display device.

2. The shape other than a rectangle is a polygonal shape other than a rectangle or a circular shape other than a perfect circle. The display device according to claim 1 .

3. The polygonal shape other than a rectangle is a triangle or a rhombus, The circular shape other than a perfect circle is an ellipse. The display device according to claim 2 .

4. the processor generates a screen in which a frame line indicating the designated section is superimposed on the signal waveform, and outputs the screen to the monitor. The display device according to claim 1 .

5. The target section is a learning target section used for machine learning. The display device according to claim 4.

6. a monitor for displaying the audio data; an input unit that accepts a user's operation to designate a designated section of the audio data while a signal waveform of the audio data is displayed on the monitor; a processor which determines at least a first target section and a second target section to be displayed on the monitor from the designated designated section, determines a position shifted a first predetermined section from a start position of the first target section as an end position of the first target section, determines a position shifted a second predetermined section from the start position of the first target section as a start position of the second target section, and determines a position shifted a first predetermined section from the start position of the first target section as an end position of the second target section, and determines the second predetermined section so that the second target section overlaps with the first target section, generates a screen in which a first frame line indicating the first target section including the start position and end position of the first target section and a second frame line indicating the second target section including the start position and end position of the second target section are superimposed on the signal waveform and outputs the screen to the monitor, The first frame line and the second frame line are in a shape other than a rectangle. Display device.

7. A display method performed by a terminal device, comprising: displaying a signal waveform of the audio data on a monitor, accepting a user's operation to designate a designated section of the audio data, and determining at least a first target section and a second target section to be displayed on the monitor from the designated designated section; determining a position shifted a first predetermined distance from the start position of the first target section as an end position of the first target section, a position shifted a second predetermined distance from the start position of the first target section as the start position of the second target section, and a position shifted a first predetermined distance from the start position of the second target section as the end position of the second target section, and determining the second predetermined distance so that the second target section overlaps with the first target section; generating and outputting a screen in which a first frame line indicating the first target section including a start point position and an end point position of the first target section and a second frame line indicating the second target section including a start point position and an end point position of the second target section are superimposed on the signal waveform; The first frame line and the second frame line are in a shape other than a rectangle. Display method.

Images