Processing system, program and processing method

Abstract

To provide a processing system that makes it possible to associate audio with lines of dialogue with a low workload.SOLUTION: A processing system includes: a speech data storage section 172 that stores speech data from a series of utterances by a speaker; a line of dialogue data storage section 174 that stores multiple lines of dialogue data; a decomposition section 102 that decomposes the speech data into multiple utterances in multiple utterance sections; and a matching section 104 that associates the decomposed utterance speech data with the lines of dialogue data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a processing system, a program, a processing method, and the like. [Background technology]

[0002] Patent Document 1 discloses a file system in which an orderer orders audio files containing lines from a script from a supplier, and checks and inspects the delivered audio files. Specifically, the file system generates a recording script, which is an order form for requesting voice recording by a voice actor. The supplier delivers delivery data including a plurality of audio files, each of which has a voice label set for each line. The orderer then performs a check by clicking a play button while looking at a check support screen as shown in FIG. 6 of Patent Document 1, and comparing the audio played back with the lines displayed on the check support screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-028787 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology of Patent Document 1, the worker had to record the voice in units of lines set in advance and convert it into data manually. This resulted in a large time cost for the worker, and when there were a huge number of lines, it was very difficult to handle.

[0005] According to some aspects of the present embodiment, it is possible to provide a processing system, a program, a processing method, and the like that are capable of associating audio and lines with a small workload or the like. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to a processing system including a voice data storage unit that stores voice data by a series of utterances by a speaker, a line data storage unit that stores a plurality of line data, a decomposition unit that decomposes the voice data into a plurality of speech voice data in a plurality of speech sections, and a correspondence unit that associates the decomposed speech voice data with the line data. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or an information storage medium that can be read by a computer and that stores the program.

[0007] According to one aspect of the present disclosure, voice data and line data from a series of utterances by a speaker are acquired, and the voice data is decomposed into a plurality of pieces of speech voice data in a plurality of speech sections. The decomposed speech voice data and the line data are then associated with each other. In this way, it is possible to associate data of each of a series of a plurality of lines uttered by the speaker with data of each of the plurality of speech voices in the voice data. Therefore, it is possible to reduce the burden on an operator who checks which of the plurality of lines each of the plurality of speech voices corresponds to.

[0008] In addition, in one aspect of the present disclosure, the matching unit may perform the matching by performing a matching process between spoken text data generated by voice recognition of the decomposed spoken voice data and dialogue text data, which is the dialogue data in a text data format.

[0009] In this way, it becomes possible to determine which line data out of the multiple line data each piece of uttered voice data in the voice data should correspond to, based on the result of the matching process between the utterance text data and the line text data.

[0010] In addition, in one aspect of the present disclosure, when the matching unit determines that the utterance text data and the dialogue text data do not match, the matching unit may perform a division process or a concatenation process of the utterance text data, and perform a matching process between the utterance text data after the division process or the concatenation process and the dialogue text data.

[0011] In this way, when a speech section longer or a speech period shorter than the speech period of a line is set, a division process or a concatenation process is performed on the speech text data, making it possible to match the speech text data after the division process or the concatenation process with the line text data.

[0012] In another aspect of the present disclosure, the association unit may perform a matching process between data obtained by converting the speech text data into syllables or phonemes and data obtained by converting the dialogue text data into syllables or phonemes.

[0013] In this way, it becomes possible to deal with cases in which speech recognition of spoken voice recognizes a term that is pronounced the same as a term in dialogue but is written differently.

[0014] In another aspect of the present disclosure, the decomposition unit may decompose the voice data into the speech voice data based on the loudness of the voice data, sound spectrum information, or speech probability.

[0015] In this way, it becomes possible to determine the volume, sound spectrum information, or utterance probability of voice data generated by a series of utterances by a speaker, and to decompose the voice data into utterance voice data of a plurality of utterance sections.

[0016] In another aspect of the present disclosure, the association unit may associate time information of the speech section of the speech voice data with the line text data associated with the speech voice data.

[0017] In this way, it becomes possible to identify the speech voice data of the speech section in the voice data by using the time information of the speech section associated with the line data.

[0018] In another aspect of the present disclosure, the association unit may label the speech voice data with identification information of the dialogue data.

[0019] In this way, when the voice data is decomposed into a plurality of pieces of speech voice data in a plurality of speech sections, the line data corresponding to the decomposed speech voice data can be identified by the identification information of the line data.

[0020] In another aspect of the present disclosure, the association unit may label the speech voice data with matching result information of the speech voice data and the line data.

[0021] In this way, it becomes possible to determine the degree to which the associated speech voice data and line data match, based on the matching result information labeled to the speech voice data.

[0022] In one aspect of the present disclosure, the association unit may label the speech voice data to which the line data has not been associated with information indicating that a matching line does not exist.

[0023] In this way, even if the audio data contains utterances other than dialogue, it is possible to identify such utterances other than dialogue by using information indicating the absence of matching dialogue.

[0024] In another aspect of the present disclosure, the association unit may label the speech voice data with information about the line data or information about the speaker.

[0025] In this way, it becomes possible to determine what type of line data the spoken voice data corresponds to, or what speaker the voice data is from, based on information about the line data or information about the speaker.

[0026] In addition, one aspect of the present disclosure may include a subtitle data generation unit that generates subtitle data to be displayed corresponding to the output of the audio data, based on the dialogue data associated with the spoken audio data.

[0027] In this way, it is possible to generate more appropriate subtitle data with fewer errors, compared to when subtitle data is generated based on voice recognition data of voice data.

[0028] In another aspect of the present disclosure, the subtitle data generation unit may perform a translation process on the dialogue data associated with the speech voice data, and generate the subtitle data based on the dialogue data after the translation process.

[0029] In this way, it is possible to generate subtitle data with fewer translation errors and with more accurate translation of lines, compared to a case where subtitle data is generated by translating speech recognition data of speech data.

[0030] Furthermore, one aspect of the present disclosure relates to a processing method in which a computer executes the following processes: storing voice data from a series of utterances by a speaker; storing a plurality of pieces of line data in text data format; decomposing the voice data into a plurality of pieces of spoken voice data in a plurality of speech sections; and associating the decomposed spoken voice data with the line data. [Brief description of the drawings]

[0031] [Figure 1] 2 shows an example of the configuration of a processing system according to the present embodiment. [Diagram 2] 13 shows an application example of the processing system of the present embodiment. [Diagram 3] 13 shows an application example of the processing system of the present embodiment. [Figure 4] FIG. [Diagram 5] 4 is a flowchart illustrating a processing example of the present embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a speech period detection technique. [Figure 7] FIG. 4 is an explanatory diagram of a matching process between utterance text data and dialogue text data. [Figure 8] FIG. 4 is an explanatory diagram of a matching process between utterance text data and dialogue text data. [Figure 9] FIG. 4 is an explanatory diagram of a process for dividing spoken text data. [Figure 10] FIG. 4 is an explanatory diagram of a concatenation process of spoken text data. [Figure 11] 11 is an explanatory diagram of a process of associating time information of an utterance section with spoken text data. [Figure 12] FIG. 4 is an explanatory diagram of identification information of script data and dialogue data. [Figure 13] FIG. 13 is an explanatory diagram of information to be labeled in speech voice data. [Figure 14] 10 is a flowchart illustrating another processing example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the contents of the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential configurations.

[0033] 1. Processing System An example of the configuration of the processing system of this embodiment is shown in Fig. 1. Note that the configuration of the processing system is not limited to that shown in Fig. 1, and various modifications are possible, such as omitting some of the components (parts) or adding other components.

[0034] 1 includes a processing unit 100 and a storage unit 170. The processing system may also include an operation unit 160, a display unit 190, a sound output unit 192, and a communication unit 196.

[0035] The processing unit 100 (processor) performs processing such as decomposing audio data, matching speech data with dialogue data, generating subtitle data, presenting information, or storing information, based on various information, programs, or operation information stored in the storage unit 170. Each processing (function) of the present embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each processing of the present embodiment can be realized by a processor that operates based on information such as a program, and a memory that stores information such as a program. The functions of each unit of the processor may be realized by individual hardware, or may be realized by integrated hardware. For example, the processor includes hardware, and the hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. For example, the processor may be configured with one or more circuit devices (e.g., ICs, etc.) mounted on a circuit board, or one or more circuit elements (e.g., resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to a CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may be a hardware circuit using an ASIC. The processor may include an amplifier circuit or a filter circuit for processing an analog signal. The memory (storage unit) may be a semiconductor memory such as an SRAM or a DRAM, or a register. Alternatively, the memory may be a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the instructions are executed by the processor to realize the processing (function) of each unit of the processing unit 100. The instructions here may be an instruction set constituting a program, or may be instructions that instruct the hardware circuit of the processor to operate.

[0036] The processing unit 100 includes a decomposing unit 102 and a matching unit 104. The processing unit 100 can also include a subtitle data generating unit 106 and a presenting unit 108. The decomposing unit 102, the matching unit 104, the subtitle data generating unit 106, and the presenting unit 108 will be described in detail below. Note that the configuration of the processing unit 100 is not limited to this, and various modifications are possible, such as omitting some of these components or adding other components.

[0037] The operation unit 160 allows the user to input various pieces of information such as operation information, and the function thereof can be realized by an input device such as a keyboard, a touch panel display, or an operation controller.

[0038] The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, etc., and its functions can be realized by a semiconductor memory, a HDD, an SSD, an optical disk device, etc. The storage unit 170 includes a voice data storage unit 172 and a dialogue data storage unit 174. The voice data storage unit 172 and the dialogue data storage unit 174 will be described in detail later.

[0039] The information storage medium 180 (a computer-readable medium) stores programs, data, etc., and its functions can be realized by an optical disk, a HDD, a semiconductor memory, etc. The processing unit 100 performs various processes of this embodiment based on the programs (data) stored in the information storage medium 180. This information storage medium 180 can store programs (programs for causing a computer to execute the processes of each unit) for causing a computer (a device equipped with an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of this embodiment.

[0040] The display unit 190 outputs an image generated by this embodiment, and its function can be realized by an LCD, an organic EL display, a CRT, an HMD, etc. The sound output unit 192 outputs a sound generated by this embodiment, and its function can be realized by a speaker, a headphone, etc. The communication unit 196 (communication interface) communicates with an external device such as a server system or a terminal device via a network, and its function can be realized by hardware such as a communication ASIC or a communication processor, communication firmware, etc.

[0041] As shown in FIG. 1, the processing system of this embodiment includes a decomposition unit 102, an association unit 104, an audio data storage unit 172, and a dialogue data storage unit 174.

[0042] The voice data storage unit 172 stores voice data of a speaker. For example, the voice data storage unit 172 stores voice data by a series of utterances by a speaker. The speaker is, for example, a voice actor, an actor, a public speaker, a lecturer, a presenter, or a video distributor. The series of utterances by the speaker is, for example, a plurality of utterances including a plurality of lines constituting a script or manuscript. For example, when a speaker makes a plurality of utterances including a plurality of lines, voice data is acquired by recording (recording, etc.) these plurality of utterances. Then, the voice data including the series of utterance voice data of the speaker is stored in the voice data storage unit 172. The voice data storage unit 172 may save the voice data or may temporarily store the voice data. For example, the voice data storage unit 172 may temporarily store voice data input in a stream format.

[0043] The line data storage unit 174 stores a plurality of line data. For example, the line data storage unit 174 stores a plurality of line data in a text data format or the like. For example, the line data storage unit 174 stores data of a plurality of lines constituting a script or manuscript or the like. The line data may be stored in a data format other than the text data format. For example, the line data may be stored in a format that allows the contents of the lines to be specified, and may be stored in a format that allows matching processing with, for example, spoken text data or spoken voice data. For example, the line data may be stored in a format other than the text data format, and may be converted to the text data format when matching processing with the spoken text data is performed. The text data may be something like hypertext data. For example, the line data storage unit 174 stores line data of a script or manuscript or the like that corresponds to voice data by a series of utterances by a speaker. The line data storage unit 174 may save the line data, or may temporarily store the line data.

[0044] The decomposition unit 102 performs a decomposition process of the voice data. For example, the decomposition unit 102 decomposes the voice data into a plurality of speech voice data in a plurality of speech sections. The decomposition unit 102 can also be called a division unit, a section unit, or a trimming unit, and the decomposition into a plurality of speech voice data can also be called division, sectioning, or trimming into a plurality of speech voice data. For example, the decomposition unit 102 performs a process of decomposing voice data by a series of utterances of a speaker stored in the voice data storage unit 172 into a plurality of speech voice data. For example, when a speaker makes a series of a plurality of utterances, the decomposition unit 102 identifies a plurality of speech sections corresponding to the series of a plurality of utterances, and decomposes (divides, sections, trims) the voice data into a plurality of speech voice data in the identified plurality of speech sections. For example, each of the plurality of speech voice data is the speech voice data in each of the plurality of speech sections. The decomposition of the audio data by the decomposition unit 102 does not have to actually divide (trim) the audio data, but may involve identifying multiple speech sections in the audio data based on time information of each speech section (information on the start time and end time of the section, etc.).

[0045] The associating unit 104 associates the speech voice data with the line data. For example, the associating unit 104 associates the speech voice data decomposed by the decomposing unit 102 with the line data stored in the line data storage unit 174. For example, the line data storage unit 174 stores a plurality of line data constituting a script or manuscript, and performs a process of associating each line data of the plurality of line data with each line data of the plurality of speech voice data decomposed by the decomposing unit 102. The association between the speech voice data and the line data can be realized by, for example, labeling specific information of the line data to the speech voice data, labeling specific information of the speech voice data to the line data, or a process of linking the speech voice data and the line data by given information or the like. Then, for example, the presenting unit 108 (notifying unit) performs a process of presenting (notifying) the result of associating the speech voice data with the line data to the user. For example, the presentation unit 108 performs processing such as presenting to the user line data associated with the spoken voice data, presenting to the user line data associated with the spoken voice data, or presenting to the user link information between the spoken voice data and the line data.

[0046] The matching unit 104 also performs a matching process between the speech text data corresponding to the speech voice data and the line text data, which is line data in text data format. For example, the matching unit 104 performs a matching process between the speech text data generated by speech recognition of the decomposed speech voice data and the line text data, which is line data in text data format, to perform the matching. For example, the matching unit 104 performs a speech recognition process of the speech voice data in the speech section to generate the speech text data. Then, the matching unit 104 performs a matching process to determine the degree to which the speech text data and the line text data match. Then, the matching unit 104 performs a matching process between the speech voice data (the speech text data) and the line data (the line text data) based on the result of the matching process. For example, when the matching process determines that the degree of matching is high, the speech voice data and the line data are associated with each other. For example, when the matching process calculates an error rate, the matching process determines that the speech voice data and the line data have an error rate lower than a given threshold value and associates them with each other.

[0047] The association unit 104 also performs a division process or a concatenation process of the spoken text data (speech voice data) during the matching process. For example, when the association unit 104 determines that the spoken text data and the dialogue text data do not match, the association unit 104 performs a division process or a concatenation process of the spoken text data. In this case, the time division at the time of division is specified by forced alignment or phoneme segmentation. The forced alignment is a process of estimating the appearance position of each phoneme in the voice from, for example, a pair of a phoneme notation string and a voice. The phoneme segmentation is a process of automatically recognizing the start and end times of phonemes (segmented sounds) included in the voice data. Then, the association unit 104 performs a matching process between the spoken text data and the dialogue text data after the division process or the concatenation process. For example, when the association unit 104 determines that the spoken text data and the dialogue text data do not match (for example, when the error rate of the matching process is higher than a threshold), the association unit 104 divides the spoken text data into first spoken text data and second spoken text data. Then, a matching process is performed between the first utterance text data and the dialogue text data, and a matching process is performed between the second utterance text data and the dialogue text data. Alternatively, when the association unit 104 determines that the utterance text data and the dialogue text data do not match, the association unit 104 connects the utterance text data with the utterance text data before and after the utterance text data to generate third utterance text data. Then, a matching process is performed between the third utterance text data and the dialogue text data.

[0048] The matching unit 104 also performs a matching process between data obtained by converting the speech text data into syllables or phonemes and data obtained by converting the dialogue text data into syllables or phonemes. Then, based on the result of the matching process, the speech voice data and the dialogue data are matched. A syllable is a unit of sound in a language. For example, a syllable is a combination of a vowel and a consonant, and is a sound sequence that is perceived as a unit, and is a type of segmental unit that separates consecutive linguistic sounds. In Japanese, for example, a syllable is Hiragana or Katakana. A phoneme is the smallest phonetic unit that distinguishes meanings, and is the smallest phonological unit. For example, a phoneme is a vowel, a consonant, or a semivowel. The matching unit 104 converts the speech text data and the dialogue text data into syllables or phonemes and performs a matching process between the two.

[0049] Furthermore, the association unit 104 associates time information of the speech voice data with the line text data. For example, the association unit 104 associates time information of a speech section of the speech voice data with the line text data associated with the speech voice data. The time information of the speech section is information for identifying the time of the speech section, such as the start time, end time, or time interval information of the speech section. By associating the time information in this manner, it becomes possible to identify the time information of the speech section corresponding to the speech text data among the voice data of a series of utterances by the speaker, and to extract the speech voice data in the speech section. Then, it becomes possible to associate the speech text data with the speech voice data in the extracted speech section.

[0050] Furthermore, the correspondence unit 104 labels the speech voice data with the identification information of the line data. By labeling the speech voice data with the identification information of the line data in this manner, the speech voice data and the line data are associated with each other. For example, when the speech voice data is identified, labeling is performed so that the corresponding line data can be identified from among the multiple line data. The identification information of the line data is information for identifying each of the multiple line data. The labeling of the identification information of the line data for the speech voice data can be realized, for example, by setting the line data as tag information, link information, or additional information of the speech voice data.

[0051] Furthermore, the association unit 104 labels the speech voice data with matching result information between the speech voice data and the line data. For example, a matching process is performed between the speech text data corresponding to the speech voice data and the line text data corresponding to the line data, and the result of this matching process is labeled to the speech voice data. For example, information on the result of the matching process is set as tag information, link information, additional information, or the like of the speech voice data. The result of the matching process is, for example, a matching rate between the speech text data and the line data, an error rate in the matching process, or information indicating whether or not matching has been achieved.

[0052] Furthermore, the matching unit 104 labels the speech voice data to which the line data is not associated with information indicating that there is no matching line. The information indicating that there is no matching line is, for example, information indicating that the speech voice data (speech text data) and the line data (line text data) do not match and are NG, information indicating that the matching rate is equal to or lower than a reference threshold, information indicating that the error rate of the matching process is equal to or higher than a reference threshold, or information indicating that the speech voice is an utterance other than a line.

[0053] Furthermore, the association unit 104 labels the speech voice data with information on the line data (e.g., identification information or type information of the line data) or information on the speaker (e.g., identification information or type information of the speaker). The information on the line data is, for example, identification information of a script or manuscript composed of lines, or information indicating the category or attribute of the line data. The information on the speaker is identification information of a speaker such as a voice actor, information on a game character corresponding to the speaker, or information indicating the gender, age, affiliation, etc. of the speaker.

[0054] Labeling of the first information for the speech voice data can be said to be, for example, to link data indicating the first information to the speech voice data and present it to make it easier to identify the content of the first information in the speech voice. For example, labeling is to give information for identification to various information. For example, by labeling the speech voice data with identification information of line data, it becomes easier to identify what lines are spoken as the speech voice. Furthermore, by labeling the speech voice data with matching result information, it becomes easier to identify the matching result between the speech voice and the lines. Furthermore, by labeling the speech voice data with information of line data or information of the speaker, it becomes easier to identify the content of the lines in the speech voice and what kind of speaker the speech voice is.

[0055] The decomposition unit 102 also decomposes the voice data into speech voice data based on the loudness, spectrum information, or speech probability of the voice data. For example, when the loudness (amplitude) of the voice data is equal to or greater than a given threshold, the decomposition unit 102 determines that the voice data in that section is speech voice data. Alternatively, based on spectrum information of the voice in the voice data, the decomposition unit 102 determines that the voice data in a section determined to be a frequency band of speech is speech voice data. Alternatively, based on a trained model, for example, a speech probability, which is the probability that the voice in the voice data is speech voice, is obtained, and the voice data in a section in which the speech probability is equal to or greater than a given threshold is determined to be speech voice data. The trained model is, for example, a model that is trained (deep learning) using training voice data and a correct answer label indicating that the sound is speech voice. When voice data is input to this trained model, a speech probability indicating that the voice of the voice data is speech voice is output, and the decomposition unit 102 decomposes the voice data into speech voice data based on this speech probability.

[0056] Further, the subtitle data generating unit 106 generates subtitle data for the audio data. For example, the subtitle data generating unit 106 generates subtitle data to be displayed corresponding to the output of the audio data, based on line data associated with the spoken audio data. For example, the subtitle data generating unit 106 generates subtitle data to be displayed when audio corresponding to the audio data is output by the sound output unit 192 or the like, based on line data associated with the spoken audio data. For example, the subtitle data generating unit 106 generates subtitle data corresponding to the audio in each of the multiple speech sections, based on line data associated with the spoken audio data in that speech section. The generated subtitle data is displayed, for example, by the display unit 190.

[0057] The subtitle data generating unit 106 also translates the dialogue data. For example, the subtitle data generating unit 106 translates the dialogue data associated with the spoken voice data, and generates subtitle data based on the dialogue data after the translation process. Instead of translating the spoken text data obtained by speech recognition of the spoken voice data, the subtitle data generating unit 106 translates the dialogue data, which is dialogue text data matching the spoken text data. In this case, the subtitle data generating unit 106 may translate the dialogue data into a language specified by the terminal device. For example, the subtitle data generating unit 106 performs a multilingual translation process, which is a translation process into a language selected from a plurality of languages. Then, subtitles corresponding to the dialogue are displayed on the screen of the display unit 190 based on the subtitle data generated based on the dialogue data after the translation process.

[0058] This embodiment may be realized as a program that causes a computer to function as the above-described voice data storage unit 172, the line data storage unit 174, the decomposition unit 102, and the association unit 104, or as an information storage medium that stores the program. This embodiment may also be realized by a processing method in which a computer executes a process of storing voice data based on a series of utterances by a speaker, a process of storing a plurality of line data, a process of decomposing the voice data into a plurality of speech voice data in a plurality of speech sections, and a process of associating the decomposed speech voice data with the line data.

[0059] The processing system of this embodiment may be realized by a terminal device such as a PC (personal computer), or may be realized by a server system connected to the terminal device via a network. Alternatively, the processing of the processing system of this embodiment may be realized by distributed processing between the terminal device and the server system.

[0060] 2. Method of the Present Invention Next, the method of this embodiment will be described in detail.

[0061] 2.1 Matching speech data and dialogue data When creating amusement content such as game content using the voice of a voice actor or video content by a video distributor, the voice is recorded, for example, in units of lines set in advance, and manually digitized. However, there is a problem of time cost for an operator who manually divides the voice, and it is difficult to deal with the case where there is a large number of lines. Also, at various events such as events, events, speeches, and concerts, the contents of the speaker's speech may be displayed as subtitles. In this case, the speaker's speech is recognized to generate text data for the subtitles. However, since it is difficult to perform accurate speech recognition, there is a problem that erroneous subtitles are generated and unnatural subtitles are displayed. In particular, when the text data obtained by speech recognition is translated and the translated text data is used to display subtitles, problems such as incomprehensible subtitles may occur.

[0062] Therefore, in this embodiment, a method is adopted in which, from input voice data, delimited data for each line that can be used for content, events, etc. Specifically, in this embodiment, speech voice data in which various lines are input as each voice of a series of voices is compared with the corresponding line data. In this way, a process is performed to link the speech voice data corresponding to the lines, and by labeling the data, data that is easy to use when creating content such as game content, or data suitable for displaying subtitles at events is created.

[0063] FIG. 2 shows an application example of the processing system of this embodiment. In FIG. 2, a speaker 300, such as a speaker or a moderator, is speaking into a microphone 302 at a venue. The voice data of the speaker 300 is input to a PA device 310 and output from a speaker 304 installed at the venue, so that the audience 330 can hear the contents of the speech. In this case, the speaker gives a speech or hosts a meeting according to a script prepared in advance. The voice data of the speaker 300 is speech-recognized by the processing system of this embodiment, which is realized by, for example, a cloud, to generate speech text data. For example, when the speaker 300 utters a series of lines constituting a script for a speech or hosting, the voice data of this series of utterances is input to the processing system and speech recognition is performed. In this case, in this embodiment, the voice data of the series of utterances of the speaker 300 is decomposed into a plurality of speech voice data, and speech recognition is performed on each of the decomposed speech voice data to generate speech text data. Further, data of the manuscript spoken by the speaker 300 is stored in the system, and a matching process is performed between the spoken text data and text data of a plurality of lines constituting the manuscript, thereby associating the spoken voice data (spoken text data) with the line data (line text data). Then, a translation process is performed on the line data associated with the spoken voice data, and subtitle data generated by the translation process is sent to the terminal device 332 of the audience member 330, and the subtitles are displayed on the display unit of the terminal device 332. In this case, since a multilingual translation process is performed in Fig. 2, for example, English subtitles are displayed on the terminal device 332 of the American audience member 330, and Chinese subtitles are displayed on the terminal device 332 of the Chinese audience member 330.

[0064] In this case, as shown in FIG. 2, the speech recognition of the speech data of the speaker 300 may erroneously recognize "lecture" as "park" or insert a term such as "ahh" that is not suitable for translation. Therefore, if the subtitles based on the results of this speech recognition are displayed as is on the terminal device 332 of the audience member 330, the subtitles will be displayed inappropriately and unnaturally. In particular, if the results of the speech recognition are translated into another language, there is a risk that even more inappropriate subtitles will be displayed. In this case, a method can be considered in which an operator who edits the subtitles corrects the erroneously recognized terminology or deletes unnecessary terminology in real time, and the edited subtitles are displayed on the terminal device 332. However, this method may cause problems such as an excessive workload for the operator who edits the subtitles or a very long time lag between when the speaker 300 speaks and when the subtitles are displayed.

[0065] In this embodiment, the speech data of the speaker 300 is decomposed into a plurality of speech data. Then, a matching process is performed between each speech text data generated by speech recognition of each speech data and each line text data of a plurality of line text data of the manuscript data. Then, the speech data determined to match is associated with the line data, and subtitles based on the line data associated with the speech data are displayed on the terminal device 332 of the audience member 330. When translation is performed, a translation process is performed on the line data associated with the speech data, and the subtitles after the translation process are displayed on the terminal device 332 of the audience member 330. In this way, the "park" that is erroneously recognized by speech recognition is replaced with the term "lecture" in the line data, and unnecessary terms such as "ahh" are deleted, and the subtitles are displayed. After the erroneously recognized term is replaced with the correct term and the unnecessary term is deleted, a translation process is performed, and the translated subtitles are displayed. Therefore, it is possible to display appropriate subtitles corresponding to the manuscript on the terminal device 332 of the audience member 330.

[0066] Fig. 3 shows another application example of the processing system of this embodiment. In Fig. 3, a speaker 340 such as a voice actor is recording game audio in a studio, and the speaker 340 is speaking lines from a game script into a microphone 342. For example, the speaker 340 speaks a series of lines that constitute the game script. Audio data of the speech of the speaker 340 is input to PA equipment 350 and output from a speaker 344 installed in the studio. A director 346 is present in the studio to supervise the recording of the game audio, and checks the content of the speech of the speaker 340, who is a voice actor, and issues instructions such as NG if the speech is inappropriate.

[0067] The voice data recorded in this manner is input to the processing system of this embodiment for producing game contents, and is voice-recognized to generate spoken text data. For example, when a speaker 340, who is a voice actor, speaks a series of lines constituting a game script, the voice data of this series of utterances is input to the processing system and voice recognition is performed. In this case, in this embodiment, the voice data of the series of utterances of the speaker 340 is decomposed into a plurality of spoken voice data, and voice recognition is performed on each of the decomposed spoken voice data, thereby generating spoken text data. In addition, data of the script spoken by the speaker 340 is stored in the system, and a matching process is performed between the spoken text data and the text data of the plurality of lines constituting the script, so that the spoken voice data input by the speaker 340 is associated with appropriate line data that matches the spoken voice data. In this way, game voice data constituting the game content is produced. The game voice data is, for example, data in which identification information of each of the plurality of lines of the line data is associated with each of the plurality of spoken voice data. When a game character played by a voice actor speaks a line, the speech voice data corresponding to the line data is read and played back based on the identification information of the line data, thereby realizing a game in which the character speaks in the voice of the voice actor.

[0068] In this case, as a method of comparison with the present embodiment, a method in which a voice actor records the voice for each of a plurality of lines is considered. However, with this method, when the number of lines is large, the work of recording the lines in a studio becomes very cumbersome and difficult. Another method is also considered in which an editor for game production manually divides the voice data recorded by the voice actor uttering a series of lines into spoken voice data corresponding to each line. However, with this method, the worker needs to work while comparing the contents of the lines and the contents of the voice data, so the time cost of the worker becomes very large. Furthermore, the series of utterances of the voice actor in the voice data includes conversations with the director and utterances of lines that have been rejected, so the worker needs to distinguish between these and work, which further increases the worker's workload.

[0069] In this embodiment, the decomposition unit 102 decomposes the voice data of a series of utterances by the speaker, and the decomposed voice data is associated with each of the multiple lines of line data in the association unit 104. This eliminates the need to record the voice of the speaker for each of the multiple lines of line data, simplifying and facilitating the voice recording work. In addition, it is no longer necessary for an operator to check and divide the voice data of a series of utterances, making it possible to significantly reduce the time cost of the operator.

[0070] FIG. 4 is a method for explaining a specific example of the method of this embodiment. As shown in FIG. 4, the voice data is voice data resulting from the utterance of a series of lines by a speaker. In this embodiment, this voice data is decomposed into a plurality of speech voice data in a plurality of speech sections SS1 to SS9. For example, as described later, based on the volume of the sound in the voice data, the spectrum information of the sound, or the speech probability, a break in a series of utterances is detected, and the plurality of speech sections SS1 to SS9 are specified and set, thereby decomposing the voice data into a plurality of speech voice data. Then, the decomposed speech voice data in the plurality of speech sections SS1 to SS9 is associated with each line data of a plurality of line data constituting the script data or manuscript data. For example, this association is realized by performing a matching process between each speech voice data and each line data.

[0071] In FIG. 4, there is no matching line data for the speech data in the speech sections SS1 and SS2, and the speech data in the speech sections SS1 and SS2 are determined to be response utterances, such as conversations between a voice actor and a director.

[0072] As for the speech voice data of the speech section SS3, since there was no matching line data, the speech text data of the speech voice data of the speech section SS3 is divided. The time division at the time of division is detected, for example, by using forced alignment or phoneme segmentation. Then, the speech text data of the speech section SS31 after the division process is determined to match with the data of line 1. On the other hand, as for the speech text data of the speech section SS32 after the division process, there is no matching line data, and it is determined to be a response utterance by conversation or the like. As a result, the speech voice data of the speech section SS31 is associated with the data of line 1.

[0073] As for the speech voice data in the speech section SS4, it is determined that it matches the data for line 2, but the matching error rate is high and the degree of matching is very low. For this reason, it is determined that the correspondence between the speech voice data in the speech section SS4 and the data for line 2 is NG.

[0074] As for the speech voice data in the speech section SS5, since there was no matching line data, a division process is performed on the speech text data of the speech voice data in the speech section SS5. Then, the speech text data in the speech section SS51 after the division process is determined to match the data for line 3, and the speech text data in the speech section SS52 after the division process is determined to match the data for line 2. Therefore, the speech voice data in the speech section SS51 is associated with the data for line 3, and the speech voice data in the speech section SS52 is associated with the data for line 2. This situation is considered to be a situation in which the speaker speaks line 2 in the speech section SS4, but it does not go well and is rejected by the director, etc., and the speaker speaks line 2 again in the speech section SS52.

[0075] Moreover, the speech voice data in the utterance section SS6 is determined to match with the data of line 4. Therefore, the speech voice data in the utterance section SS6 comes to correspond to the data of line 4.

[0076] As for the speech voice data in the speech sections SS7 and SS8, since there was no matching line data, a process of concatenating the speech text data of the speech voice data in the speech sections SS7 and SS8 is performed. After the concatenation process, the speech text data in the speech section SS78 is determined to match the data for line 5. As a result, the speech voice data in the speech section SS78 is associated with the data for line 5.

[0077] Moreover, the speech voice data in the utterance section SS9 is determined to match with the data of line 6. Therefore, the speech voice data in the utterance section SS9 is associated with the data of line 6.

[0078] In this way, in Fig. 4, the speech voice data of the speech sections SS31, SS52, SS51, SS6, SS78, and SS9 are associated with the data of line 1, line 2, line 3, line 4, line 5, and line 6, respectively. In the application example of Fig. 2, subtitles of line 1, line 2, line 3, line 4, line 5, and line 6 are displayed. For example, subtitles translated from line 1, line 2, line 3, line 4, line 5, and line 6 are displayed. In the application example of Fig. 3, the speech voice data of the speech sections SS31, SS52, SS51, SS6, SS78, and SS9 are used as the speech voice data of line 1, line 2, line 3, line 4, line 5, and line 6 of a game character, for example.

[0079] Fig. 5 is a flow chart for explaining a processing example of this embodiment. First, the processing system acquires voice data and line data from a series of utterances by a speaker (step S1). For example, in Fig. 2, voice data from a series of utterances by a speaker, lecturer, or presenter is stored in voice data storage unit 172 and is acquired by reading it out from voice data storage unit 172. Also, line data is acquired by reading it out from line data storage unit 174. Note that voice data may be acquired as stream data, in which case stream-format voice data is temporarily stored in voice data storage unit 172, for example.

[0080] Next, the processing system decomposes the acquired voice data into a plurality of speech voice data in a plurality of speech sections (step S2). For example, in Fig. 4, the voice data by a series of utterances by a speaker is decomposed into speech voice data in speech sections SS1 to SS9. For example, the voice data is divided into speech sections SS1 to SS9, and the speech voice data in the divided speech sections SS1 to SS9 is identified.

[0081] Next, the processing system performs a matching process between the speech text data and the dialogue text data (step S3). For example, a matching process is performed between the speech text data obtained by speech recognition of the speech voice data in the speech section and the dialogue text data, which is dialogue data in text data format. For example, a matching process is performed to obtain a matching error rate or a matching degree between the speech text data and the dialogue text data. Then, when the processing system determines that the speech text data and the dialogue text data do not match, a division process or a concatenation process is performed on the speech text data (steps S4 and S5). For example, as shown in FIG. 4, a process is performed to divide the speech text data in the speech section SS3 into speech text data in the speech sections SS31 and SS32, and a process is performed to divide the speech text data in the speech section SS5 into speech text data in the speech sections SS51 and SS52. A process is also performed to concatenate the speech text data in the speech sections SS7 and SS8 as the speech text data in the speech section S78. Then, a matching process is performed between the speech text data after the division process or concatenation process and the dialogue text data (step S3). 4, for example, it is determined that the speech text data in the speech section SS31 after the division process matches the text data of line 1. It is also determined that the speech text data in the speech sections SS51 and SS52 after the division process match the text data of lines 3 and 2. It is also determined that the speech text data in the speech section SS78 after the concatenation process matches the text data of line 5.

[0082] On the other hand, when the processing system determines that the utterance text data and the line text data match, it performs a process of associating the matched line data with the utterance voice data (step S6). For example, when the matching error rate between the first utterance text data and the first line text data is equal to or lower than a threshold, the processing system performs a process of associating the first line data, which is the first line text data, with the first utterance voice data corresponding to the first utterance text data.

[0083] As described above, according to the present embodiment, voice data and line data from a series of utterances by a speaker are acquired, and the voice data is decomposed into a plurality of pieces of spoken voice data in a plurality of speech sections. Then, the decomposed spoken voice data and the line data are associated with each other. In this way, it is possible to associate data of each line of a series of a plurality of lines uttered by the speaker with data of each of the plurality of spoken voices in the voice data. Therefore, it is possible to reduce the burden on an operator or the like who checks which line of the plurality of spoken voices corresponds to each of the plurality of spoken voices. In addition, since the voice data is voice data obtained by a speaker uttering a plurality of lines as a series of lines, it is also possible to reduce the burden on the speaker in recording the voice, etc. Therefore, it is possible to provide, for example, spoken voice data that is easy to use for content such as game content, and line data that is easy to use for generating subtitles and translating, etc.

[0084] In this embodiment, the above-mentioned correspondence is achieved by performing a matching process between the spoken text data generated by the speech recognition of the decomposed speech data and the dialogue text data, which is dialogue data in text data format. In this way, it becomes possible to determine which dialogue data in the multiple dialogue data each piece of speech data in the speech data should be associated with, based on the result of the matching process between the spoken text data and the dialogue text data. This makes it possible to associate appropriate dialogue data with the speech data, for example, even if there is an erroneous recognition in the speech recognition of the speech data or if unnecessary speech by the speaker is included in the speech data.

[0085] In this embodiment, when it is determined that the utterance text data and the dialogue text data do not match, a division process or a concatenation process is performed on the utterance text data. Then, a matching process is performed between the utterance text data after the division process or the concatenation process and the dialogue text data. In this way, when an utterance section longer than the utterance period of the dialogue is set, for example, a division process is performed on the utterance text data, so that the utterance text data after the division process and the dialogue text data can be matched. When an utterance period shorter than the utterance period of the dialogue is set, a concatenation process is performed on the utterance text data, so that the utterance text data after the concatenation process and the dialogue text data can be matched.

[0086] 2.2 Specific processing examples Next, a specific processing example of this embodiment will be described. In this embodiment, the voice data is decomposed into speech voice data based on the loudness, sound spectrum information, or speech probability in the voice data. That is, in this embodiment, as shown in FIG. 4, the voice data by a series of utterances by a speaker is decomposed into a plurality of speech voice data in a plurality of speech sections SS1 to SS9, and at this time, the voice data is decomposed into speech voice data based on the loudness, sound spectrum information, or speech probability in the voice data. In this way, when the loudness (amplitude) of the voice data by a series of utterances by the speaker becomes a loudness (amplitude) corresponding to the uttered voice, when the sound spectrum becomes a spectrum corresponding to the uttered voice, or when the speech probability is determined to be uttered, the voice data can be decomposed into speech voice data with the section as the speech section.

[0087] For example, FIG. 6 is an explanatory diagram of a process of decomposing voice data into speech voice data based on the speech probability. In FIG. 6, the speech probability at each timing of the voice data is obtained, and the section where the speech probability is equal to or greater than a given threshold value VT is set as the speech section, and the voice data is decomposed into a plurality of speech voice data in a plurality of speech sections SS1 to SS9. The speech probability can be obtained, for example, by inputting the voice data to a trained model. For example, a trained model that outputs the speech probability for a speech voice is generated by performing a learning process using teacher data in which the speech data for learning and a correct answer label indicating whether or not it is a speech voice are set. Then, this trained model is stored in the storage unit 170 of FIG. 1, and the decomposing unit 102 inputs the voice data by a series of utterances by the speaker to the trained model, whereby the speech probability can be obtained. Then, as shown in FIG. 6, the section where the speech probability is equal to or greater than a threshold value VT is set as the speech section, and the speech voice data is cut out. Alternatively, the loudness of the sound in the voice data may be calculated based on, for example, the amplitude of the sound, and a section where the loudness of the sound is equal to or greater than a threshold may be regarded as a speech section, and the speech data may be extracted from the voice data. Alternatively, the spectrum of the sound in the voice data may be calculated by, for example, frequency analysis of the sound, and a section where the spectrum of the sound is determined to correspond to the spectrum of human speech may be regarded as a speech section, and the speech data may be extracted.

[0088] In addition, in this embodiment, a matching process is performed between data obtained by converting spoken text data into syllables or phonemes and data obtained by converting dialogue text data into syllables or phonemes. For example, a matching process is performed between data obtained by converting spoken text data into syllables such as hiragana or katakana and data obtained by converting dialogue text data into syllables such as hiragana or katakana. Alternatively, a matching process is performed between data obtained by converting spoken text data into phonemes such as vowels or consonants and data obtained by converting dialogue text data into phonemes such as vowels or consonants. In this way, it becomes possible to deal with, for example, a case in which a term in a dialogue is recognized as a term that has the same pronunciation but a different spelling in speech recognition of a spoken voice.

[0089] For example, in FIG. 7, spoken text data is generated by speech recognition of spoken voice data. This spoken text data is converted into syllables (hiragana, katakana) or phonemes (vowels, consonants). The dialogue text data is also converted into syllables (hiragana, katakana) or phonemes (vowels, consonants). Then, a matching process is performed between the spoken text data converted into syllables or phonemes and the dialogue text data converted into syllables or phonemes. For example, in FIG. 7, a matching process is performed between the spoken text data TA1 to TA5 and the dialogue text data TB1, and a matching process is performed between the spoken text data TA1 to TA5 and the dialogue text data TB2. Similarly, a matching process is performed between the spoken text data TA1 to TA5 and the dialogue text data TB3, and a matching process is performed between the spoken text data TA1 to TA5 and the dialogue text data TB4. In this case, in FIG. 7, both the spoken text data and the dialogue data are converted into syllables or phonemes. Therefore, even if, in speech recognition of a spoken text, a kanji character is recognized that is different from the kanji character of a term in a line, for example, it is possible to realize an appropriate matching process.

[0090] FIG. 8 is a diagram for explaining a specific example of the matching process. In FIG. 8, the error rate in the matching process between the speech text data TA2 and the dialogue text data TB1 is 0.00, and it is shown that the speech text data TA2 and the dialogue text data TB1 match as shown in FIG. 7. In addition, the error rate between the speech text data TA3 and the dialogue text data TB2 is 0.12, which indicates that the error rate is slightly larger. For example, in FIG. 7, the speech text data TA3 contains the term "Ah, Ie" that does not exist in the dialogue text data TB2, so the error rate is increased. In addition, the error rates between the speech text data TA4 and TA5 and the dialogue text data TB3 and TB4 are 0.07 and 0.08, respectively, which indicates that the error rate is slightly larger. For example, in FIG. 7, the speech text data TA4 contains the term "Very" that does not exist in the dialogue text data TB3, so the error rate is slightly increased. In addition, in the speech text data TA5, the term "Fufu" that exists in the dialogue text data TB4 is not present, so the error rate is slightly increased. Therefore, by determining whether the error rate of the utterance text data and the dialogue text data exceeds a threshold value, it becomes possible to determine whether the utterance text data and the dialogue text data match.

[0091] The matching process between the speech text data and the dialogue text data can be realized by calculating the edit distance. The edit distance is a distance that indicates the closeness between a first string and a second string, and can be calculated by the minimum number of operations of substitution (S), insertion (I), and deletion (D). For example, the greater the minimum number of operations of substitution, insertion, and deletion required to match the first string and the second string, the longer the edit distance will be, and the higher the error rate in the matching process will be.

[0092] FIG. 9 is an explanatory diagram of the division process of the spoken text data. The spoken text data TA1 obtained by speech recognition of the spoken voice data in the speech section is "The building's keywords are traditional and authentic", and the spoken text data TA2 obtained by speech recognition of the spoken voice data in the next speech section is "And vanished". On the other hand, the dialogue text data TB1 is "The building's keywords are", and the next dialogue text data TB2 is "'traditional', 'authentic' and 'high quality'". In FIG. 9, a matching process is performed to compare the spoken text data TA1 converted into katakana, which is a syllable, with the dialogue text data TB1 converted into katakana, which is a syllable. In this matching process, the above-mentioned edit distance is obtained. Specifically, the first 11 characters of the spoken text data TA1 match the dialogue text data TB1 (H), but in the spoken text data TA1, 12 characters need to be inserted (I) into the dialogue text data TB1. For this reason, it is determined that the edit distance is large, the error rate of the matching process is greater than the threshold, and the spoken text data TA1 and the dialogue text data TB1 do not match. In this case, the spoken text data TA1 is divided into spoken text data "The building's keyword is" and spoken text data "traditional, authentic". Note that the time division between "The building's keyword is" and "traditional, authentic" during the division process is determined by forced alignment or phoneme segmentation. Then, a matching process is performed to compare the spoken text data "The building's keyword is" after the division process with the dialogue text data "The building's keyword is", and it is determined that the two match.

[0093] FIG. 10 is an explanatory diagram of the concatenation process of spoken text data. By the division process described in FIG. 9, the spoken text data TA1, "The building keywords are traditional and authentic," is divided into the spoken text data TA11, "The building keywords are," and the spoken text data TA12, "traditional and authentic." Then, a matching process is performed between the spoken text data TA11 and the dialogue text data TB1, and it is determined that the two match. On the other hand, the spoken text data TA12 and the dialogue text data TB2 are determined not to match by the matching process, so a concatenation process is performed between the spoken text data TA12, "traditional and authentic," and the spoken text data TA2, "and then vanished." Then, a matching process is performed between the spoken text data after the concatenation process and the dialogue text data TB2, "'traditional,' "'authentic,' and 'high-quality,'" and it is determined that they match because the error rate is very small. In Figure 10, the word "quality" in the dialogue is mistakenly recognized as "disappeared" in the speech recognition of the spoken voice, but by converting it into syllables such as katakana or phonemes, it becomes possible to properly match the two.

[0094] In this embodiment, time information of the speech section of the speech voice data is associated with the line text data associated with the speech voice data. For example, in FIG. 11, speech data by a series of utterances by a speaker is decomposed into speech voice data of a plurality of speech sections SS1, SS2, and SS3. In this case, a start time ST1 and an end time EN1 are associated with the line text data TB1 associated with the speech voice data of the speech section SS1 as the time information of the speech section SS1. Also, a start time ST2 and an end time EN2 are associated with the line text data TB2 associated with the speech voice data of the speech section SS2 as the time information of the speech section SS2, and a start time ST3 and an end time EN3 are associated with the line text data TB3 associated with the speech voice data of the speech section SS3 as the time information of the speech section SS3. In this way, it becomes possible to specify, for example, the speech voice data of the speech sections SS1, SS2, and SS3 in the voice data by using the time information (ST1, EN1, ST2, EN2, ST3, and EN3) of the speech sections SS1, SS2, and SS3 associated with the line text data TB1, TB2, and TB3. For example, it becomes possible to extract the speech voice data of the speech sections SS1, SS2, and SS3 corresponding to the line text data TB1, TB2, and TB3 from the voice data by a series of utterances of a speaker. This makes it possible to generate data in which the speech voice data of the speech sections SS1, SS2, and SS3 are associated with the line text data TB1, TB2, and TB3. This makes it possible to generate game data in which the speech voice data of the line data is associated with the line data of a character, taking a game system as an example. Note that in FIG. 11, the start time and end time of the speech time are used as the time information of the speech section, but this embodiment is not limited to this. For example, the time information of the utterance period may be information on the start time and time interval of the utterance period, information on the end time and time interval of the utterance period, information on the center time and time interval of the utterance period, or the like.

[0095] Furthermore, in this embodiment, speech voice data is labeled with the identification information of the line data. For example, in FIG. 12, IDS1 is the identification information of the script data (or manuscript data). IDB1, IDB2, IDB3, and IDB4 are the identification information of the line data constituting the script data specified by the identification information IDS1. For example, in FIG. 12, the identification information IDB1, IDB2, IDB3, and IDB4 of the line data are associated with the script data of IDS1, for example, by consecutive numbers. In this way, by specifying the identification information IDS1 of the script data and the identification information IDB1, IDB2, IDB3, and IDB4 of the line data, it becomes possible to identify the line data of IDB1, IDB2, IDB3, and IDB4 in the script data of IDS1.

[0096] In FIG. 13, the speech data is labeled with the identification information of the dialogue data. For example, the speech data DVS1, DVS2, DVS3, and DVS4 in FIG. 13 are a plurality of speech data in a plurality of speech sections described in FIG. 4. In FIG. 13, the speech data DVS2, DVS3, and DVS4 are labeled with the identification information IDB1, IDB2, and IDB3 of the dialogue data described in FIG. 12. In this way, when the speech data is decomposed into a plurality of speech data DVS2, DVS3, and DVS4 in a plurality of speech sections, the dialogue data corresponding to the decomposed speech data can be identified by the identification information IDB1, IDB2, and IDB3. As a result, for example, even when the speech data is cut out into the speech data DVS2, DVS3, and DVS4 in a plurality of speech sections, the dialogue data corresponding to the cut out speech data DVS2, DVS3, and DVS4 can be identified by the identification information IDB1, IDB2, and IDB3. Therefore, it becomes possible to use data that is a set of speech voice data and line data for various contents such as game contents.

[0097] In this embodiment, the speech data is labeled with matching result information of the speech data and the line data. For example, in FIG. 13, the speech data DVS1, DVS2, DVS3, and DVS4 obtained by decomposing the speech data are labeled with error rates ER2, ER3, and ER4 in the matching process as matching result information of the speech data and the line data. This error rate can be obtained, for example, by the edit distance between the speech text data and the line text data as described above. Note that the matching result information is not limited to such an error rate, and may be, for example, information on the degree of matching between the speech text data and the line text data, or information on whether the speech text data and the line text data match. In this way, it becomes possible to determine the degree to which the associated speech data and the line data match in the matching process based on the matching result information labeled on the speech data. As a result, it becomes possible to determine, for example, whether the speech data associated with the line data should be used as the speech data of the line data using the labeled matching result information.

[0098] In this embodiment, the speech voice data to which the line data is not associated is labeled with information indicating that there is no matching line. For example, in FIG. 13, it is determined that there is no line data matching the speech voice data DVS1 by the matching process between the speech text data and the line text data. Therefore, the speech voice data DVS1 is labeled with "NON", which is information indicating that there is no matching line. Taking FIG. 4 as an example, the speech voice data in the speech sections SS1, SS2, and SS32 are determined to be voice data by responses that are not lines, so information indicating that there is no matching line is labeled for these speech voice data. For example, when the error rate in the matching process is high, the information indicating that there is no matching line is labeled in this way. In this way, even if there is an utterance other than a line in the voice data by a series of utterances by the speaker, it is possible to identify and remove such an utterance other than a line by using information indicating that there is no matching line (NON). The information indicating that a matching line does not exist may take various forms. For example, the information indicating that a line does not exist may be information indicating that the spoken voice data is a response voice data as shown in Fig. 4, or information indicating that the error rate exceeds a predetermined value.

[0099] In this embodiment, the spoken voice data is labeled with information on the line data (identification information, type information) or information on the speaker (identification information, type information). For example, in FIG. 13, as the information on the line data, the identification information IDS1 of the script data (or manuscript data) constituting the line data is labeled to the spoken voice data. As the information on the speaker, the identification information IDAC1 of the voice actor (or the character played by the voice actor) who is the speaker is labeled to the spoken voice data. In this way, it becomes possible to determine what type of line data the spoken voice data corresponds to, or what kind of speaker the voice data is, from the information on the labeled line data or the information on the speaker. Taking FIG. 13 as an example, it becomes possible to determine the script or manuscript constituting the lines of the spoken voice data, the voice actor who is uttering the spoken voice data, and the character played by the voice actor, from the information on the line data or the information on the speaker.

[0100] The information on the line data may be attribute information such as the category, genre, or group of the line data, and the information on the speaker may be information such as the gender, age, or affiliation of the speaker. For example, information indicating two or more pieces of information among the identification information of the line data, the matching result information, the information on the line data, and the information on the speaker may be associated with the spoken voice data. For example, it may be possible to specify two or more pieces of information among the identification information of the line data, the matching result information, the information on the line data, and the information on the speaker by one piece of information associated with each spoken voice data.

[0101] FIG. 14 is a flowchart showing another processing example of this embodiment. FIG. 14 shows a processing example corresponding to the application example of FIG. 2. Steps S11 to S16 in FIG. 14 are similar to steps S1 to S6 in FIG. 5, so detailed explanations are omitted. In FIG. 14, when the matched dialogue data is associated with the spoken voice data in step S16, the processing system translates the dialogue data associated with the spoken voice data (step S17). Taking FIG. 9 and FIG. 10 as an example, the translation processing is performed on the dialogue text data TB1 and TB2, which are "The keywords of the building are," "Traditional," "Authentic," and "Quality." That is, in the spoken text data, the term "Quality" is erroneously recognized as the term "Disappearance," but in this embodiment, the translation processing of the dialogue data is performed, so that the correct term "Quality" is translated. Then, the processing system generates subtitle data based on the dialogue data after the translation processing (step S18). For example, in FIG. 2, the subtitles based on the subtitle data generated in this way are displayed on the terminal device 332 of the audience member 330. This allows the audience 330 to check the content of the speech or presentation given by the speaker 300, such as a speaker or a moderator, by looking at the subtitles. If translation processing is not required, subtitle data may be generated based on the dialogue data without translation processing of the dialogue data.

[0102] In this manner, in this embodiment, the subtitle data displayed in response to the output of the voice data is generated based on the dialogue data associated with the spoken voice data. In this manner, it is possible to generate more appropriate subtitle data with fewer errors than when the subtitle data is generated based on the voice recognition data of the voice data. Taking Figs. 9 and 10 as an example, if the term "disappeared" obtained by the voice recognition of the spoken voice data is displayed as a subtitle, a situation occurs in which an unnatural subtitle that does not match the context is displayed. In this regard, according to this embodiment, since the subtitle data is generated based on the dialogue data stored as accurate data without errors, it is possible to prevent a situation in which an unnatural subtitle that does not match the context is displayed.

[0103] In this embodiment, a translation process is performed on the dialogue data associated with the speech voice data, and subtitle data is generated based on the dialogue data after the translation process. For example, when multiple language translation is required, a process is performed to translate the dialogue data into a specified language to generate subtitle data. In this way, compared to a case where subtitle data is generated by translating speech recognition data of speech data, subtitle data can be generated with less translation errors and with lines translated more accurately. Taking Figs. 9 and 10 as an example, if the term "disappeared" obtained by speech recognition of the speech voice data is translated and displayed as a subtitle, a situation occurs where a subtitle with an unclear translation is displayed or a subtitle with an inappropriate translation that does not fit the context is displayed. In this regard, according to this embodiment, since subtitle data is generated by translating the dialogue data stored as accurate data without errors, it is possible to prevent a situation where a subtitle with an unclear translation is displayed or a subtitle with an inappropriate translation that does not fit the context is displayed.

[0104] Although the present embodiment has been described in detail as above, it will be easily understood by those skilled in the art that many modifications are possible without substantially departing from the novel matters and effects of the present disclosure. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader meaning or synonymy can be replaced with that different term anywhere in the specification or drawings. In addition, the storage process of voice data, the storage process of dialogue data, the process of decomposing into spoken voice data, the process of matching spoken voice data with dialogue data, etc. are not limited to those described in the present embodiment, and methods equivalent to these are also included in the scope of the present disclosure. [Explanation of symbols]

[0105] 100... processing unit, 102... decomposition unit, 104... association unit, 106... subtitle data generation unit, 108... presentation unit, 160... operation unit, 170... storage unit, 172... audio data storage unit, 174... dialogue data storage unit, 180... information storage medium, 190... display unit, 192... sound output unit, 196... communication unit, 300... speaker, 302... microphone, 304... speaker, 310... PA equipment, 330... audience, 332... terminal device, 340... speaker, 342... microphone, 344... speaker, 346... director, 350... PA equipment, DVS1 to DVS4... speech voice data, IDB1 to IDB4... identification information, SS1 to SS9... speech section, TA1 to TA5... speech text data, TB1 to TB5... dialogue text data

Claims

1. a voice data storage unit for storing voice data based on a series of utterances by a speaker; a line data storage unit that stores a plurality of line data; A decomposition unit that decomposes the voice data into a plurality of speech voice data in a plurality of speech sections; a correspondence unit that associates the decomposed speech voice data with the line data; A processing system comprising:

2. In claim 1, The association unit is A processing system characterized by performing a matching process between spoken text data generated by voice recognition of the decomposed spoken voice data and dialogue text data, which is the dialogue data in text data format, thereby performing the correspondence.

3. In claim 2, The association unit is A processing system characterized by performing a division process or a concatenation process on the utterance text data when it is determined that the utterance text data and the line text data do not match, and performing a matching process between the utterance text data after the division process or the concatenation process and the line text data.

4. In claim 2, The association unit is A processing system comprising: a processing for performing a matching process between data obtained by converting the speech text data into syllables or phonemes and data obtained by converting the dialogue text data into syllables or phonemes.

5. In any one of claims 1 to 4, The decomposition unit includes: A processing system comprising: a processor for processing said voice data; said processor for processing said voice data; and a processor for processing said voice data based on a volume of sound, spectrum information of sound, or speech probability in said voice data.

6. In any one of claims 1 to 4, The association unit is A processing system comprising: a processing unit for processing a speech data including a speech section of the speech data, the processing unit correlating time information of the speech section of the speech data with the line text data that is associated with the speech data.

7. In any one of claims 1 to 4, The association unit is A processing system comprising: a processing unit for processing said speech data, said processing unit labeling said speech data with identification information of said line data.

8. In any one of claims 1 to 4, The association unit is A processing system comprising: labeling the speech voice data with matching result information between the speech voice data and the line data.

9. In any one of claims 1 to 4, The association unit is The processing system is characterized in that, for the speech voice data to which the line data cannot be associated, information indicating that a matching line does not exist is labeled.

10. In any one of claims 1 to 4, The association unit is A processing system comprising: labeling information of the line data or information of the speaker for the speech voice data.

11. In any one of claims 1 to 4, A processing system comprising: a subtitle data generating unit that generates subtitle data to be displayed corresponding to an output of the audio data, based on the dialogue data associated with the spoken audio data.

12. In claim 11, The caption data generating unit a processing system for performing a translation process on the dialogue data associated with the speech voice data, and generating the subtitle data based on the dialogue data after the translation process.

13. a voice data storage unit for storing voice data based on a series of utterances by a speaker; a line data storage unit that stores a plurality of line data in a text data format; A decomposition unit that decomposes the voice data into a plurality of speech voice data in a plurality of speech sections; As an association unit that associates the decomposed speech voice data with the line data, A program that causes a computer to function.

14. storing speech data of a series of utterances by a speaker; storing a plurality of pieces of dialogue data in a text data format; A process of decomposing the voice data into a plurality of speech voice data in a plurality of speech sections; A process of associating the decomposed speech voice data with the line data; A processing method characterized by being executed by a computer.

Images