Language processing device, language processing method, and program

Abstract

Improve technology for converting voice data into written language. [Solution] The language processing device includes an acquisition unit that acquires voice data and a control unit. The control unit executes voice recognition processing, speaker separation processing, text segmentation processing, and text refinement processing. The voice recognition processing converts utterances included in the voice data into text. The speaker separation processing separates and identifies the utterances of multiple speakers included in the voice data by speaker. The text segmentation processing separates the text converted from the utterances by the voice recognition processing into segments for each speaker identified by the speaker separation processing. The text refinement processing uses a language model to refine the text segmented by speaker to improve readability.

Description

[Technical Field]

[0001] The present invention relates to a language processing device, a language processing method, and a program for converting voice data into text language. [Background technology]

[0002] In recent years, speech recognition systems have made it possible to accurately transcribe speech data, such as conference and interview audio, word for word. However, the resulting text often has low readability and requires revision. To utilize speech transcripts converted into text by speech recognition, complex processing is required, including the removal of word fragments typical of spoken language, standardization of spelling, correction of errors, and elimination of redundant expressions. This series of conversion processes is called transcription refinement. Various proposals have been made to improve the readability of speech transcripts.

[0003] For example, Non-Patent Document 1 proposes using machine learning to output highly readable written sentences directly from speech, without generating speech transcriptions converted from speech data using speech recognition. This method has the advantage of being able to correct speech recognition errors, etc., using acoustic clues in the input speech data.

[0004] Non-Patent Document 2 describes the creation of a corpus used for training to convert Japanese sentences from spoken to written language, with the aim of improving the readability of text converted from spoken language. This corpus takes into consideration seven conversion rules, such as the removal of fillers, hesitations, and redundant expressions, the addition of punctuation marks, the standardization of writing style, the restoration of omitted particles, and the standardization of kana-kanji spelling. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Masato Mimura, Tatsuya Kawahara, "End-to-end speech-to-written conversion for Diet proceedings," Natural Language Processing, Vol. 30 (2023), No. 1, pp. 88-124 [Non-patent document 2] Ai Iori, Eihiko Takashima, Ryo Masumura, "Spoken-to-Written Conversion Corpus for Japanese Texts," Proceedings of the 26th Annual Conference of the Association for Natural Language Processing (March 2020), pp. 157-160 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the method of Cited Document 1 is not necessarily effective in correcting errors based on meaning determined from the context and / or common sense. Furthermore, the corpus of Non-Patent Document 2 covers only a limited number of rules for converting spoken language to written language, leaving room for improvement before it can be used to refine transcripts. Furthermore, when the number of rules for converting spoken language to written language increases, uniform application of all rules may not necessarily result in high-quality text. For example, uniform application of rules may result in problems such as a change in the meaning of a sentence when a particle is corrected without a clear meaning before correcting a word or phrase that is clearly evident from the context.

[0007] Therefore, an object of the present disclosure, which has been made with attention to these points, is to improve the technology for converting voice data into written language. [Means for solving the problem]

[0008] In order to solve the above problem, a language processing device according to an embodiment of the present disclosure includes: an acquisition unit that acquires voice data; a control unit that performs a speech recognition process to convert utterances included in the voice data into text, a speaker separation process to separate and identify utterances of a plurality of speakers included in the voice data for each speaker, a text segmentation process to separate the text converted from the utterances by the speech recognition process for each speaker identified by the speaker separation process, and a text refinement process to improve readability of the text separated for each speaker by using a language model; Equipped with.

[0009] In order to solve the above problem, a language processing method according to an embodiment of the present disclosure includes: A language processing method executed by a control unit of a language processing device, Acquiring audio data; a speech recognition process for converting speech contained in the voice data into text; a speaker separation process for separating and identifying utterances of a plurality of speakers included in the voice data; a text segmentation process for segmenting the text converted from the speech by the speech recognition process for each speaker identified by the speaker separation process; a text refinement process for improving readability of the text divided for each speaker using a language model; Includes.

[0010] In order to solve the above problem, a program according to an embodiment of the present disclosure includes: Acquiring audio data; a speech recognition process for converting speech contained in the voice data into text; a speaker separation process for separating and identifying utterances of a plurality of speakers included in the voice data; a text segmentation process for segmenting the text converted from the speech by the speech recognition process for each speaker identified by the speaker separation process; a text refinement process for improving readability of the text divided for each speaker using a language model; to be executed by the computer. [Effects of the Invention]

[0011] According to an embodiment of the present disclosure, the technology for converting audio data into written language can be improved. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic configuration diagram illustrating an example of a language processing system including a language processing device according to an embodiment. [Figure 2] 2 is a functional block diagram showing an example of the configuration of a control unit in FIG. 1. FIG. [Figure 3] 2 is a flowchart showing the procedure of a voice data arranging process executed by the control unit of FIG. 1; [Figure 4] 4A and 4B are diagrams showing an example of text before and after the speech recognition post-processing of FIG. 3. [Figure 5] FIG. 4 is a diagram including a flowchart explaining the generation of a speech recognition post-processing model and the procedure of the speech recognition post-processing of FIG. [Figure 6] 4 is a flowchart illustrating the procedure of the speaker separation process of FIG. 3. [Figure 7] 10A and 10B are diagrams illustrating an example of detection of speaking times for each speaker of audio data. [Figure 8] FIG. 10 is a diagram illustrating an example of adding speaker information to a speech transcript. [Figure 9] 4 is a flowchart illustrating the procedure of the pre-processing of text adjustment in FIG. 3. [Figure 10] 4A and 4B are diagrams showing an example of text before and after the pre-processing of text refinement in FIG. 3. [Figure 11] FIG. 4 is a diagram illustrating an outline of the procedure of the text tidying process of FIG. 3. [Figure 12] FIG. 10 is a diagram showing an example of a prompt for step 1. [Figure 13] FIG. 10 is a diagram showing an example of text before and after step 1. [Figure 14] FIG. 10 is a diagram showing an example of a prompt for step 2. [Figure 15] FIG. 10 is a diagram showing an example of text before and after step 2. [Figure 16]FIG. 10 is a diagram showing an example of a prompt for step 3. [Figure 17] FIG. 10 is a diagram showing an example of text before and after step 3. [Figure 18] FIG. 10 is a diagram showing an example of a prompt for step 4. [Figure 19] FIG. 10 is a diagram showing an example of text before and after step 4. [Figure 20] 4 is a flowchart illustrating the procedure of the text arranging process of FIG. 3. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] (Overall composition) 1, a language processing system 10 including a language processing device 20 according to an embodiment of the present disclosure will be described. The language processing system 10 is included in an information system within an organization such as a company, a school, or a research institute. The language processing system 10 may be a system located in a specific location, or may be a system that is geographically distributed.

[0015] The language processing system 10 includes a language processing device 20. The language processing system 10 may further include a user terminal 30. The language processing device 20 is a computer that performs speech recognition processing and text editing processing on speech transcripts generated by the speech recognition. The language processing device 20 may be a general-purpose computer such as a workstation or a PC (Personal Computer), or a dedicated computer. The language processing device 20 may be constructed as a server on the cloud. The user terminal 30 may include, for example, a PC, a tablet terminal, a smartphone, and a portable information device. The user terminal 30 can send speech data to the language processing device 20 and receive the text edited from the transcript.

[0016] In this application, audio data refers to acoustic data containing human speech. Examples of audio data include data on meetings, discussions, speeches, lectures, conversations, and songs. Audio data may be in any format, including linear PCM (Linear Pulse Code Modulation), MP3 (MPEG-1 audio layer 3), WAV (Windows Media Audio), AAC (Advanced Audio Coding), and WMA (Windows Media Audio). Audio data can also be referred to as spoken language or spoken words.

[0017] In one embodiment, the language processing device 20 is further configured to be able to communicate with a speech recognition system 50, a speaker separation system 60, and a language model system 70 external to the language processing system 10 via a network 40. The speech recognition system 50, the speaker separation system 60, and the language model system 70 may include a server device that provides cloud services. In another embodiment, some or all of the speech recognition system 50, the speaker separation system 60, and the language model system 70 may be included in the language processing system 10. Alternatively, some or all of the functions of the speech recognition system 50, the speaker separation system 60, and the language model system 70 may be executed by a control unit 21 of the language processing device 20.

[0018] (Configuration of language processing device) The language processing device 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0019] The control unit 21 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (central processing unit) or a GPU (graphics processing unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit). The control unit 21 executes processes related to the operation of the language processing device 20 while controlling each part of the language processing device 20.

[0020] 2, the control unit 21 may include components such as a speech recognition unit 21a, a speech recognition post-processing unit 21b, a speaker separation unit 21c, a text editing pre-processing unit 21d, and a text editing processing unit 21e. Each component may be configured as a software module or a hardware module.

[0021] The speech recognition unit 21a converts speech data input to the language processing device 20 into a speech transcript, which is text information.

[0022] The speech recognition post-processing unit 21b inserts punctuation marks and tags redundant expressions and named entities into the speech transcription converted from the speech data by the speech recognition unit 21a. In one embodiment, the speech recognition post-processing unit 21b includes a speech recognition post-processing model 21f that performs the processing of the speech recognition post-processing unit 21b by machine learning.

[0023] The speaker separating unit 21c separates and identifies the speeches of multiple speakers included in the audio data by speaker.

[0024] The pre-processing unit 21d processes the tags assigned to the speech transcript processed by the speech recognition post-processing unit 21b, and divides the speech transcript into segments for each speaker identified by the speaker separation unit 21c and for each defined maximum number of characters.

[0025] The text processing unit 21e includes a text processing system. The text processing unit 21e applies text processing rules (described later) to the speech transcription text before text processing that has been processed by the text pre-processing unit 21d, thereby converting the text into highly readable written language.

[0026] The processing of each of the speech recognition unit 21a, speech recognition post-processing unit 21b, speaker separation unit 21c, text refinement pre-processing unit 21d, and text refinement processing unit 21e will be described in more detail below as processing executed by the control unit 21.

[0027] Returning to the description of the language processing device 20 in FIG. 1, the storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 22 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores programs and data used in the operation of the language processing device 20, and data obtained by the operation of the language processing device 20. The information stored in the storage unit 22 may be updatable with information obtained from the network 40 via the communication unit 23, for example.

[0028] The storage unit 22 may sequentially store the voice data and the character information being processed at each stage when the language processing device 20 acquires voice data, converts it into a voice transcription using voice recognition, and performs a text editing process. The storage unit 22 may store a learning model to be used when an inference process using machine learning is performed at each stage. The storage unit 22 may further store information required for the process performed by the control unit 21, such as information on idiomatic phrases and four-character idioms to be written in kanji, a name dictionary, etc.

[0029] The communication unit 23 includes a communication interface for communicating with the user terminal 30 in the language processing system 10 and for communicating with an external system via the network 40. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface includes, for example, a wired LAN (Local Area Network) interface and a USB (Universal Serial Bus). In the case of wireless communication, the communication interface may include, for example, an interface compatible with a mobile communication standard such as wireless LAN, LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation).

[0030] The communication unit 23 may receive speech data to be subjected to speech recognition and text processing from the user terminal 30. That is, the communication unit 23 functions as an acquisition unit that acquires speech data. To execute the processing of the language processing device 20, the communication unit 23 may send and receive information to and from external systems such as the speech recognition system 50, the speaker separation system 60, and the language model system 70.

[0031] The input unit 24 accepts input to the language processing device 20. In addition to acquiring voice data via the communication unit 23, the language processing device 20 may also acquire voice data from the input unit 24. That is, the input unit 24 functions as an acquisition unit that acquires voice data. For example, the input unit 24 may include an input port or a memory slot for receiving a file containing voice data. Furthermore, the input unit 24 may include a keyboard, a mouse, a touch panel, etc. that accept operations on the language processing device 20.

[0032] The output unit 25 outputs a well-formed transcription obtained as a result of the language processing device 20 processing the speech data. In addition to outputting the well-formed transcription via the communication unit 23, the language processing device 20 may output the well-formed transcription from the output unit 25. The output unit 25 may include a display such as an LCD (Liquid Crystal Display) and a printer. The output unit 25 may also include an output port for outputting the output result to an external storage medium.

[0033] The functions of the language processing device 20 are realized by executing a program relating to the language processing method of this embodiment in a processor corresponding to the control unit 21. That is, the functions of the language processing device 20 are realized by software. The program causes a computer to execute the operations of the language processing device 20, thereby causing the computer to function as the language processing device 20. That is, the computer functions as the language processing device 20 by executing the operations of the language processing device 20 in accordance with the program.

[0034] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes a non-transitory computer-readable medium. The computer-readable recording medium is, for example, a magnetic recording device, an optical disc, a magneto-optical recording medium, or a semiconductor memory. The program can be distributed, for example, by selling, transferring, or lending a portable recording medium such as a DVD (digital versatile disc) or a CD-ROM (compact disc read only memory) on which the program is recorded. The program can also be distributed by storing the program in the storage of an external server and transmitting the program from the external server to another computer. The program can also be provided as a program product.

[0035] (User terminal configuration) The user terminal 30 may include a control unit, a memory unit, a communication unit, an input unit, and an output unit, similar to the language processing device 20. The user terminal 30 functions as a client that performs speech recognition and text editing of speech data, using the language processing device 20 that functions as a server. A user may input speech data into the input unit of the user terminal 30, have it transmitted to the language processing device 20, and receive the edited transcription resulting from processing by the language processing device 20 at the user terminal 30. A user may also input speech data directly to the language processing device 20 and obtain the output result without using the user terminal 30.

[0036] (Processing performed by the language processing device) The language processing executed by the control unit 21 of the language processing device 20 will be described with reference to Fig. 3. The processing in Fig. 3 is executed by each component of the control unit 21.

[0037] First, the control unit 21 acquires voice data via the communication unit 23 or the input unit 24 (step S101).

[0038] [Speech recognition processing] Next, the processing of the control unit 21 branches to steps S102 and S104. In step S102, the control unit 21 (voice recognition unit 21a) performs voice recognition processing on the voice data (S102). From the input voice data, the control unit 21 generates a transcript that accurately reproduces the utterance word for word using a voice recognition model generated by machine learning. This transcript can also be called a "raw transcript."

[0039] In one embodiment, the control unit 21 may transmit speech data to an external speech recognition system 50 equipped with a speech recognition model via the network 40 to obtain a speech transcript. In another embodiment, the language processing device 20 may not use an external speech recognition system 50. The control unit 21 may generate a transcript by performing speech recognition processing using a speech recognition model installed inside the language processing system 10 or in the language processing device 20. Note that the speech recognition processing includes a process of tokenizing the speech data, as will be described later.

[0040] [Speech recognition post-processing] Next, the control unit 21 (speech-recognition post-processing unit 21b) performs speech-recognition post-processing on the speech transcript (step S103). A transcript that is an accurate transcription of spoken words, word for word, is not easy to read as is. For example, a transcript output as a result of speech recognition does not contain punctuation marks or question marks. Also, the transcript contains redundant expressions and named entities. Therefore, the speech transcript needs to be converted into a form that is easy for humans to read. The speech-recognition post-processing prepares for this conversion.

[0041] Post-recognition processing includes adding punctuation and question marks to the transcript and tagging (annotating) redundant expressions and named entities. Redundant expressions include "fillers," "repairs," and "backchannels." "Fillers" are utterances that fill gaps in spoken dialogue and include "um" and "er," etc. "Repairs" are repetitions of the same or similar words. "Repairs" include repetitions to correct slips of the tongue. "Backchannels" are words that show agreement and / or affirmation to the interlocutor and include "I see" and "yes." Named entities include, for example, "kanji numerals" and "personal names." In this embodiment, it is assumed that all numbers in the transcript are written in kanji numerals and all personal names are written in katakana.

[0042] Figure 4 shows an example of text before and after post-speech recognition processing of a transcript before post-speech recognition processing. Tags are represented, for example, by parentheses containing a letter indicating the tag type and the target string. Tag representation is not limited to this. In the example of Figure 4, the letters "F," "D," and "B" indicate the tag types of redundant expressions "filler," "repeat," and "backchannel," respectively. The letters "S" and "P" indicate the tag types of named entities "kanji numerals" and "person's name," respectively. The underlines in the post-speech recognition processing text in Figure 4 are added to indicate tagged sections; the actual post-speech processing text does not include underlines.

[0043] The control unit 21 can perform speech recognition post-processing using a speech recognition post-processing model 21f, which is a sequence labeling model that adds punctuation and detects and tags redundant and named entities. Here, the "sequence labeling model" refers to a machine learning model that labels sequential data such as characters, words, and sentences. As shown in FIG. 5, the speech recognition post-processing model 21f is generated by performing machine learning using training data consisting of numerous tagged speech transcripts, in which tags and punctuation marks are manually added to the transcripts. The speech recognition post-processing involves acquiring the transcript, which is the text after speech recognition (step S201), and applying the speech recognition post-processing model 21f to perform inference to add punctuation and question marks and perform tagging (step S202). This enables automation of speech recognition post-processing.

[0044] The inventors generated the speech recognition post-processing model 21f using 63,364 speech transcripts as training data and Tohoku University BERTv3 as a base model. The inventors then converted the generated speech recognition post-processing model 21f using ONNX (Open Neural Network eXchange) and further quantized the model. Furthermore, during inference, the Viterbi algorithm, an optimal path solution method, was used to improve inference performance. The speech recognition post-processing model 21f generated in this way was evaluated using 500 evaluation data sets, and high performance was obtained.

[0045] [Speaker separation processing] Returning to the flowchart of Fig. 3, the control unit 21 (speaker separation unit 21c) performs processing to separate the voice data into speakers in parallel with the processing of steps S102 and S103 (step S104). Details of the speaker separation processing will be described with reference to Fig. 6. First, the control unit 21 detects the speech start and end times for each speaker from the voice data (step S301). An example of detecting the speech time for each speaker is shown in Fig. 7.

[0046] Next, the control unit 21 acquires data of the speech transcription tokenized by the speech recognition process (step S302). After tokenization, the text of the speech transcription is divided into morpheme units by a morphological analyzer. The data divided into morpheme units includes information on the start and end times of the speech.

[0047] The control unit 21 compares the timestamps of the morphemes in the transcribed sentence obtained as a result of the speech recognition processing with the start and end times of each utterance detected in step S301 to identify the speaker for each morpheme (step S303). The control unit 21 assigns speaker information for each morpheme to the speech transcribed sentence (step S304). An example of assigning speaker information to a speech transcribed sentence is shown in FIG. 8. The transcribed sentence to which speaker information has been assigned may include, for example, each piece of information: {token ID, token, start time, end time, . . . , speaker ID}.

[0048] In one embodiment, the control unit 21 may cause a cloud-based speaker separation system 60 to execute the speaker separation process of FIG. 6 . The control unit 21 may transmit audio data to the speaker separation system 60 and obtain, from the speaker separation system 60, information on a transcript annotated with speaker information resulting from the speaker separation. In another embodiment, the control unit 21 may have a function for performing speaker separation and may be able to perform speaker separation without using an external speaker separation system 60. Speaker separation can be performed based on the characteristics of the speakers' voices. Publicly known techniques can be used for speaker separation. For speaker separation, "pyannote.audio," an open source framework for speaker separation using Python, can be used.

[0049] [Text preprocessing] Returning to Fig. 3, the control unit 21 (text pre-processing unit 21d) executes text pre-processing (step S105). The text pre-processing includes a number of processes for adjusting the speech transcription text so that it is suitable for the text pre-processing before proceeding to the next text pre-processing (step S106). The contents of each process included in the text pre-processing will be described with reference to the flowchart in Fig. 9.

[0050] The control unit 21 deletes redundant expressions that were detected by the speech recognition post-processing model 21f in step S202 and to which tags were assigned (step S401). For example, tags of the tag types "F (filler)," "D (repeat)," and "B (backchannel)" are deleted.

[0051] Next, the control unit 21 deletes tags of expressions that should remain as Chinese numerals from among the tags for "S (Kanji numerals)" and excludes them from the tagging targets (step S402). The speech recognition process of this embodiment converts all numbers contained in the speech data into Kanji numerals and outputs them. However, the numbers include a mixture of those that should be converted to Arabic numerals and those that should remain as Kanji numerals. The latter includes, for example, numbers contained in proverbs, idioms, and four-character idioms. The speech recognition post-processing model 21f can be trained to perform tagging, including distinguishing between Arabic numerals and Kanji numerals. However, in step S402, incorrect tagging is excluded from the tagging targets using a rule base. The rule base is constructed based on a collection of cases where Kanji numerals should remain as they are.

[0052] After step S402, the control unit 21 converts the numbers in the expression including the numbers tagged with "S (Kanji numerals)" into Arabic numerals (step S403). The control unit 21 leaves the number expressions whose tags have been removed in step S402 as Kanji numerals.

[0053] Next, the control unit 21 performs conversion based on a name dictionary for personal names included in the speech transcription to which the speech recognition post-processing model 21f has assigned a tag indicating a personal name, "P (personal name)" (step S404). The personal name dictionary stores the spelling of personal names in association with their pronunciation. The control unit 21 can replace the pronunciation of personal names written in katakana in the speech transcription with kanji by referring to the personal name dictionary. The personal name dictionary may be stored in the storage unit 22 or in a computer external to the language processing device 20.

[0054] Next, the control unit 21 divides the speech transcript of the text by speaker using the speaker separation result of step S104 (step S405). The speech transcript converted from the speech data may contain a mixture of utterances from multiple speakers. Therefore, if the transcript is processed as is, the utterances of multiple speakers may be combined into a single sentence, which may degrade the quality of the transcription and / or adversely affect processing for each speaker. To avoid this, the control unit 21 divides the speech transcript by speaker in pre-transcription processing.

[0055] Following step S405, the control unit 21 controls the length of the input sentence for the text refinement process (step S406). In one aspect, it is desirable to process utterances from the same speaker as long sentences as possible. For example, when multiple word candidates are possible due to homonyms, such as "rain" (rain) and "candy" (candy), it may be possible to identify which word to adopt from the context. Furthermore, a long sentence increases the likelihood of identifying the content referred to by a demonstrative pronoun. However, if the input sentence is too long, the language model may unintentionally summarize the sentence during the text refinement process described below. Taking this into consideration, the control unit 21 divides the speech transcript into units of input sentences so that summarization does not occur. For example, in one embodiment, the control unit 21 divides the speech transcript at periods, question marks, or speaker transitions. If a continuous sentence of 400 characters or more does not contain periods or speaker transitions, the unit divides the sentence at the token (morpheme) boundary closest to 400 characters. Furthermore, sentences divided by periods or question marks before and after the separated sentences are joined together, and if the length fits within 430 characters, it is adjusted accordingly. Note that the above number of characters is an example. The number of characters may vary depending on the language model used for the text refinement process.

[0056] Figure 10 shows an example of a speech transcript before and after preprocessing. The underlines are added to indicate the changes, and are not included in the actual text after preprocessing.

[0057] [Text Processing] The control unit 21 (text refining processing unit 21e) performs text refining processing (step S106) on the speech transcript that has been subjected to the text refining preprocessing as described above. The text refining processing includes multiple steps as shown in FIG. 11. The control unit 21 inputs the speech transcript that has been subjected to the text refining preprocessing and an input sentence that includes step setting information to the text refining system that performs the text refining processing. The text refining system is executed using a language model. The text refining system can perform text refining processing using a language model provided by an external language model system 70 provided by the cloud. The language processing device 20 may have an internal language model. The language model may be a unique language model constructed by training a commonly provided large-scale language model for this text refining processing.

[0058] In each step, the control unit 21 generates an instruction sentence called a prompt to be given to the language model based on the text resulting from the processing in the previous stage and the text editing rules for that step. The prompt includes the text to be processed in that step, the instruction sentence, and the text editing rules for that step. The prompt may further include example sentences before and after text editing based on the text editing rules for that step.

[0059] In one embodiment, the large-scale language model Claude can be used as the language model. In this case, the following prompting techniques can be added: Wrap each element in an instruction in XML tags: This allows the language model to accurately understand the instruction. · Command the output to be wrapped in XML tags: Makes the output easier to parse.

[0060] The control unit 21 performs the text refining process step by step until it obtains the processing result of the step set in the step setting information. In the example of Fig. 11, the control unit 21 performs the text refining process from step 1 to step 4. In this way, the control unit 21 sequentially gives prompts to the language model and performs the process step by step. Thereby, the process is repeated up to the step specified in the step setting information to obtain the final result.

[0061] Figure 12 shows an example of a prompt sentence for step 1. (1) to (4) are provided for explanatory purposes. (1) assigns a role to the language model to make it behave at the level of a professional editor with high linguistic ability. (2) specifies the task to be performed on the input sentence and the input / output format. (3) provides the editing rules for the corresponding step. (4) presents several examples of the intended output. The prompts for each step are basically structured in the same way. In Figures 14, 16, and 18 below, the same symbols (1) to (4) are used as in Figure 12.

[0062] In one embodiment, the text refining process includes up to four steps. Each step of the text refining process is described below.

[0063] <Step 1> Step 1 involves minor refining, primarily aimed at processing word fragments and standardizing spelling. Step 1 includes the following refining rules 1-1 to 1-7. 1-1. Change Chinese numerals that should be Arabic numerals to Arabic numerals (half-width). 1-2. Correctly spell the names of famous people in katakana. 1-3. Remove fillers, non-function words, interjections, and characters that clearly have no meaning in the context. 1-4. Delete any backchannels (however, leave any that would change the meaning of the sentence if deleted). 1-5. Make some adjustments to the parts where you have asked for feedback. 1-6. Regarding English words and units written in katakana, those that should generally be written in alphabets or English should be spelled correctly. Also, regarding symbols, use %, dB, or m. 1-7. Correct any missing or incorrect punctuation.

[0064] The prompt sentence for step 1, an example of which is shown in Figure 12, has the input sentence shown in Figure 13 and step setting information instructing the processing up to step 3 entered in the place of {input}. The input sentence for step 1 is the speech transcription sentence after pre-processing for text refinement. The language model processes the text refinement rules in order. All of these processes are carried out while adhering to the rules that " " should not be used, and that even if the end of the input sentence is cut off in the middle of the sentence, the end is left as it is without being corrected. The same applies to the following steps. The text refinement rules 1-1, 1-2, 1-3, 1-4, and 1-7 overlap in content with the processing before text refinement, but correct any unfinished tasks and mistakes. The processing result of step 1 is the " " at the end of the prompt. <output>and is passed to the next step. In Figure 13, the parts changed in step 1 are underlined. The actual input / output statements in step 1 are not underlined. The same applies to the examples in steps 2 to 4 below.

[0065] <Step 2> Step 2 involves more elaborate editing aimed primarily at improving grammar. Step 2 includes the following editing rules 2-1 to 2-7. 2-1. Remove phrases that don't make sense in context. 2-2. Correct simple or obvious mistakes or misreadings. 2-3. Correct incorrect use of words and particles (such as missing particles). 2-4. Correct the excessive use of meaningless sentence-final particles and interjections to a certain extent. 2-5. Correct the use of the same particle repeatedly to a certain extent. 2-6. Any rephrasing regarding corrections to the content will be revised. 2-7. Correct any words that are clearly a speech recognition error based on the context.

[0066] Figure 14 shows an example of a prompt sentence in step 2. The processing result of step 1 is substituted into the {step1} part of the prompt in step 2. The language model processes the refinement rules of step 2 in order. The processing result of step 2 is <output>15 shows an example of the processing result of step 1, which is the input sentence of step 2, and the processing result of step 2.

[0067] <Step 3> Step 3 involves more elaborate refinement aimed primarily at improving fluency. Step 3 includes the following refinement rules 3-1 to 3-8. 3-1. Correct broken expressions. 3-2. Delete one of the duplicated words. 3-3. When a sentence contains repeated similar phrases or redundant phrases, revise certain parts while also paying attention to the speaker's tone. 3-4. Correct the inconsistency between subject and predicate. 3-5. Correct any parts where the anaphoric relationship between words is inappropriate (context is out of whack). 3-6. If words are inverted and the meaning is difficult to grasp, there is a risk of misunderstanding, or the text is difficult to read, revise the text. 3-7. If words are missing or omitted and the meaning is unclear or difficult to grasp, add the words or delete the unclear words. 3-8. If a sudden change in speech disrupts the flow of the conversation, correct the wording or change the word order.

[0068] Figure 16 shows an example of a prompt sentence in step 3. The processing result of step 2 is substituted into the {step2} part of the prompt in step 3. The language model processes the refinement rules of step 3 in order. The processing result of step 3 is <output>17 shows an example of the processing result of step 2, which is the input sentence of step 3, and the processing result of step 3.

[0069] <Step 4> Step 4 involves more extensive refining, primarily for the purpose of improving conciseness. Step 4 includes the following refining rules 4-1 through 4-4. 4-1. Delete any parts that are not directly related to the agenda. 4-2. Delete any monologue that is out of context. 4-3. Organize the text for the explanation of characters (notation). 4-4. If the text contains repeated or redundant phrases, go further than step 3 of revising the text and revise everything.

[0070] Figure 18 shows an example of a prompt sentence in step 4. The processing result of step 3 is substituted into the {step3} part of the prompt in step 3. The language model processes the refinement rules in step 4 in order. The processing result of step 4 is <output>19 shows an example of the processing result of step 3, which is the input sentence of step 4, and the processing result of step 4.

[0071] Next, the procedure for the text arranging process in step S106 in FIG. 3 will be described with reference to the flowchart in FIG.

[0072] First, the control unit 21 acquires the text of the speech transcription sentence that has been subjected to the text-editing pre-processing in step S105 (step S501).

[0073] The control unit 21 receives a designation of the number of steps N to be processed as step setting information from the user, either by input from the input unit 24 or via the communication unit 23 (step S502). The number of processing steps N indicates up to which steps out of steps 1 to 4 the processing will be performed. The number of processing steps can be selected depending on the purpose and use of the text to be edited. For example, in cases where it is necessary to faithfully reproduce the content of statements, including slip-ups and the like, such as in the case of parliamentary minutes, step 1 or step 2 is appropriate. Furthermore, in cases where it is preferable to grasp the gist of statements rather than the details, such as in the case of interviews or minutes, step 3 or step 4, which eliminates redundancy, is appropriate.

[0074] The control unit 21 assigns 1 to a variable "k" that indicates the step to be processed at the current step (step S503).

[0075] Control unit 21 generates the prompt of step 1 based on the preprocessed text acquired in step S501 and the sentence organization rules of step 1 (step S504).

[0076] The control unit 21 inputs the prompt of step k (k=1 in this case) to the language model (step S505). When an external language model system 70 is used, the control unit 21 transmits the prompt to the language model system 70 via the communication unit 23.

[0077] The control unit 21 acquires the processing result of step k (in this case, k=1) from the language model (step S506). When an external language model system 70 is used, the control unit 21 receives the processing result from the language model system 70 via the communication unit 23.

[0078] Next, if the processed step k has not reached step N (step S507: No), the control unit 21 adds 1 to k (step S508) and proceeds to the processing of the next step k.

[0079] Control unit 21 generates a prompt for step k based on the processing result of the previous step (step k-1) and the shaping rules for the current step (step k) (step S509).

[0080] Thereafter, the control unit 21 executes the processes of steps S505 and S506 for the current step k. The control unit 21 executes the processes of each step in sequence, and repeats the processes of steps S508, S509, S505, and S506 until the process of step N is completed (step S507: No).

[0081] When the control unit 21 completes the processing of step N (step S507: Yes), it outputs the processing result of step N to the output unit 25 and / or causes the output result to be output to the user terminal 30 via the communication unit 23 (step S508).

[0082] As described above, the voice data acquired in step S101 of FIG. 3 is converted into text data, subjected to text refinement processing, and output as a sentence with improved readability.

[0083] As described above, according to this embodiment, the control unit 21 divided the speech transcript before text refinement by speaker, and performed text refinement processing using a language model on the text divided for each speaker. This prevents or reduces the occurrence of speech from multiple speakers being mixed in the text to be refined, allowing the language processing device 20 to perform text refinement processing with high accuracy.

[0084] Furthermore, the control unit 21 controls each unit of text input to the language model so that it is as long as possible within a predetermined number of characters converted from the speech of the same speaker. This allows the language processing device 20 to improve the accuracy of text refinement by having the language model take context into consideration, while preventing or reducing unintended summarization by the language model.

[0085] Furthermore, this embodiment includes a speech recognition post-processing model generated by machine learning that tags specific types of expressions contained in the speech transcript. This reduces the processing load for tagging the speech transcript, and by performing processing according to the type of tag in the subsequent pre-processing of the transcription, it is possible to generate a speech transcript that is suitable for the transcription processing.

[0086] Furthermore, in this embodiment, in the text refinement process, text refinement rules are defined for multiple stages and incorporated into the text refinement system of the language processing device 20. This makes it possible to automate the text refinement process and to refine text to a certain level of quality. Furthermore, in this embodiment, the text refinement process is divided into multiple steps, and refinement is performed up to the step required by the user. This makes it possible to control the degree of refinement according to the purpose or use.

[0087] Furthermore, the text refining process of this embodiment is performed in multiple steps, with small-scale, simple processing that has little impact on the context being performed in earlier steps, and more difficult, in-depth text refining such as changing word order being performed in later steps. This prevents twisted sentences, makes the text refining process smoother, and improves the quality of the text refining.

[0088] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each means or step can be rearranged so as not to be logically inconsistent, and multiple means or steps can be combined or divided into one. [Explanation of symbols]

[0089] 10 Language Processing Systems 20 Language Processing Device 21 Control section 21a Voice recognition unit 21b Speech recognition post-processing section 21c Speaker separation unit 21d Text Preprocessing 21e Text Processing 21f Speech recognition post-processing model 22 Memory section 23 Communication Department (Acquisition Department) 24 Input section (acquisition section) 25 Output section 30 User terminal 40 Network 50 Voice Recognition System 60 Speaker Separation System 70 Language Model System< / output> < / output> < / output> < / output>

Claims

1. an acquisition unit that acquires voice data; a control unit that performs a speech recognition process to convert utterances included in the voice data into text, a speaker separation process to separate and identify utterances of a plurality of speakers included in the voice data for each speaker, a text segmentation process to separate the text converted from the utterances by the speech recognition process for each speaker identified by the speaker separation process, and a text refinement process to improve readability of the text separated for each speaker by using a language model; Equipped with In the text segmentation process, when utterances by a plurality of speakers are mixed in one sentence of the text converted from the utterance by the speech recognition process, the control unit segments the text for each speaker so that one sentence of the text to be subjected to the text refinement process does not include utterances by the plurality of speakers. Language processor.

2. 2. The language processing device according to claim 1, wherein the control unit, in the speaker separation process, acquires or generates speakers and speech start and end times for each morpheme included in the speech data, and, in the text segmentation process, separates the text converted by the speech recognition by speaker using the results of the speaker separation process.

3. 2. The language processing device according to claim 1, wherein the control unit controls the language model so that each unit of text input at one time is text converted from an utterance of the same speaker and has a predetermined number of characters or less, the predetermined number of characters being a number of characters determined so that the language model does not summarize the text.

4. 2. The language processing device according to claim 1, wherein the control unit includes a speech recognition post-processing model that performs a process of assigning tags to specific types of expressions included in the text converted from the speech data by the speech recognition process, and the control unit performs specific processing on the text to which the tags have been assigned according to the types of the tags before the text refinement process.

5. The language processing device according to claim 4 , wherein the speech recognition post-processing model is generated by machine learning using a plurality of training data in which the tags are added to text converted from speech data in advance.

6. 5. The language processing device of claim 4, wherein the specific type of expression includes an expression containing Chinese numerals, and when the text to which the tag is assigned includes a tag assigned to an expression containing Chinese numerals, the control unit deletes the tag from the expression containing Chinese numerals that should remain as Chinese numerals according to a predetermined rule, and then converts the Chinese numerals included in the expression containing Chinese numerals to which the tag is not deleted into Arabic numerals.

7. the text refinement process includes processing based on a plurality of text refinement rules, the plurality of text refinement rules are classified into N stages, N is an integer of 2 or more, The control unit is configured to be able to sequentially execute the text processing from the first stage to the Nth stage, the control unit causes the language model to execute a process of applying the text to be processed as a processing target in the first stage of the text processing to the text processing rule classified into the first stage and outputting a processing result of the first stage; wherein the control unit causes the language model to execute a process of outputting the processing result of the k-th stage, applying a processing rule classified into the k-th stage to the processing result of the k-th stage, where k is an integer equal to or greater than 2 and equal to or less than N, in the k-th stage of text refinement processing. The language processing device according to claim 1 .

8. The language processing device according to claim 7 , wherein the control unit receives a designation of the stage indicating up to which stage the sentence rectification process is to be performed, and executes the sentence rectification process from the first stage to the designated stage.

9. The language processing device according to claim 7 , wherein the sentence refinement rules classified into the first stage include rules for processing word fragments and standardizing notation.

10. The language processing device according to claim 7 , wherein the refinement rules classified into the Nth stage include rules for modifying sentences and paragraphs to simplify them.

11. 8. The language processing device according to claim 7, wherein the control unit generates a prompt including the text refinement target, the text refinement rule, and predetermined example sentences before and after text refinement based on the text refinement rule at each stage of the text refinement process, and inputs the prompt to the language model to cause the language model to execute the text refinement rule.

12. A language processing method executed by a control unit of a language processing device, Acquiring audio data; a speech recognition process for converting speech contained in the voice data into text; a speaker separation process for separating and identifying utterances of a plurality of speakers included in the voice data; a text segmentation process for segmenting the text converted from the speech by the speech recognition process for each speaker identified by the speaker separation process; a text refinement process for improving readability of the text divided for each speaker using a language model; Including, In the text segmentation process, when utterances by a plurality of speakers are mixed in one sentence of the text converted from the utterance by the speech recognition process, the text is segmented for each speaker, so that one sentence of the text to be subjected to the text refinement process does not contain utterances by the plurality of speakers. Language processing methods.

13. Acquiring audio data; a speech recognition process for converting speech contained in the voice data into text; a speaker separation process for separating and identifying utterances of a plurality of speakers included in the voice data; a text segmentation process for segmenting the text converted from the speech by the speech recognition process for each speaker identified by the speaker separation process; a text refinement process for improving readability of the text divided for each speaker using a language model; A program for causing a computer to execute the above, In the text segmentation process, when a single sentence of the text converted from the speech by the speech recognition process contains a mixture of speeches from multiple speakers, the program segments the text by speaker, so that a single sentence of the text to be subjected to the text refinement process does not contain speeches from the multiple speakers.

Images