Language processing device, language processing method, and program

Abstract

We will improve the technology for converting audio data into text. [Solution] The language processing device includes an acquisition unit for acquiring audio data and a control unit. The control unit performs speech recognition processing, speaker separation processing, text segmentation processing, and text formatting processing. Speech recognition processing converts utterances contained in the audio data into text. Speaker separation processing separates and identifies the utterances of multiple speakers contained in the audio data, one speaker at a time. Text segmentation processing divides the text converted from the utterances by speech recognition processing into text for each speaker identified by the speaker separation processing. Text formatting processing performs text formatting on the texts divided by speaker, using a language model to improve readability.

Description

Technical Field

[0001] The present invention relates to a language processing apparatus, a language processing method, and a program for converting voice data into a character language.

Background Art

[0002] In recent years, voice data such as conference and interview sound sources can be accurately transcribed word for word by a voice recognition system. However, the text often has low readability and often needs to be corrected. In order to utilize the voice transcription text converted from voice recognition, complex processing such as removing word fragments peculiar to spoken language, unifying notations, correcting errors, and eliminating redundant expressions is required. This series of conversion processing is called sentence regularization of the voice transcription text. Conventionally, various proposals have been made to improve the readability of the voice transcription text.

[0003] For example, in Non-Patent Document 1, it is proposed to output a written language sentence with high readability directly from voice using machine learning without generating a voice transcription sentence converted from voice data by voice recognition. This method has the advantage that it can correct voice recognition errors and the like using acoustic clues in the input voice data.

[0004] In Non-Patent Document 2, it is described that a corpus for learning is created for the purpose of improving the readability of text converted from spoken language and converting Japanese sentences from spoken language to written language. This corpus takes into account seven conversion rules such as deletion of fillers, hesitations, and redundant expressions, punctuation mark addition, style unification, restoration of omitted auxiliary words, and unification of kana-kanji notation.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] However, the method in Reference 1 is not necessarily effective for correcting errors based on meaning determined from context and / or common sense. Furthermore, the corpus in Non-Patent Document 2 has limited rules for converting spoken language to written language, so there is room for improvement in its use for editing transcribed texts. Also, when the number of rules for converting spoken language to written language increases, applying all rules uniformly may not always result in high-quality edited texts. For example, if rules are applied uniformly, and the meaning of the sentence is unclear before correcting errors in words or phrases that are clearly evident from the context, then correcting particles or other elements may cause the meaning of the sentence to change, which is problematic.

[0007] Therefore, the purpose of this disclosure, which focuses on these points, is to improve the technology for converting speech data into written language. [Means for solving the problem]

[0008] To solve the above problems, a language processing device according to one embodiment of this disclosure is provided. An acquisition unit that acquires audio data, A control unit that performs speech recognition processing to convert utterances contained in the audio data into text, speaker separation processing to separate and identify utterances of multiple speakers contained in the audio data for each speaker, text splitting processing to divide the text converted from the utterances by the speech recognition processing into text for each speaker identified by the speaker separation processing, and text formatting processing to improve readability of the text divided for each speaker using a language model. It is equipped with.

[0009] To solve the above problems, a language processing method according to one embodiment of this disclosure is provided. A language processing method performed by the control unit of a language processing device, Acquiring audio data and A speech recognition process that converts the utterances contained in the aforementioned audio data into text, Speaker separation processing that separates and identifies the utterances of multiple speakers contained in the aforementioned audio data, A text splitting process that divides the text converted from the utterance by the speech recognition process into separate texts for each speaker identified by the speaker separation process, For the text divided by speaker, a language model is used to perform a text formatting process to improve readability. Includes.

[0010] To solve the above problems, a program according to one embodiment of this disclosure is provided. Acquiring audio data and A speech recognition process that converts the utterances contained in the aforementioned audio data into text, Speaker separation processing that separates and identifies the utterances of multiple speakers contained in the aforementioned audio data, A text splitting process that divides the text converted from the utterance by the speech recognition process into separate texts for each speaker identified by the speaker separation process, For the text divided by speaker, a language model is used to perform a text formatting process to improve readability. Have the computer execute it. [Effects of the Invention]

[0011] According to one embodiment of the present disclosure, the technology for converting voice data into character languages can be improved.

Brief Description of Drawings

[0012] [Figure 1] It is a schematic configuration diagram showing an example of a language processing system including a language processing device according to one embodiment. [Figure 2] It is a functional block diagram showing an example of the configuration of the control unit in FIG. 1. [Figure 3] It is a flowchart showing the procedure of sentence arrangement processing of voice data executed by the control unit in FIG. 1. [Figure 4] It is a diagram showing an example of text before and after post-processing of speech recognition in FIG. 3. [Figure 5] It is a diagram including the generation of a post-processing model for speech recognition and a flowchart explaining the procedure of the post-processing of speech recognition in FIG. 3 [Figure 6] It is a flowchart explaining the procedure of speaker separation processing in FIG. 3. [Figure 7] It is a diagram showing an example of detection of the speaking time of each speaker of voice data. [Figure 8] It is a diagram showing an example of assignment of speaker information to a speech-to-text sentence. [Figure 9] It is a flowchart explaining the procedure of pre-sentence arrangement processing in FIG. 3. [Figure 10] It is a diagram showing an example of text before and after pre-sentence arrangement processing in FIG. 3. [Figure 11] It is a diagram schematically showing the procedure of sentence arrangement processing in FIG. 3. [Figure 12] It is a diagram showing an example of the prompt in step 1. [Figure 13] It is a diagram showing an example of text before and after step 1. [Figure 14] It is a diagram showing an example of the prompt in step 2. [Figure 15] It is a diagram showing an example of text before and after step 2. [Figure 16]This figure shows an example of the prompt for Step 3. [Figure 17] This figure shows an example of the text before and after Step 3. [Figure 18] This figure shows an example of the prompts for Step 4. [Figure 19] This figure shows an example of the text before and after Step 4. [Figure 20] Figure 3 is a flowchart illustrating the procedure for text formatting. [Modes for carrying out the invention]

[0013] The embodiments of this disclosure will be described below with reference to the drawings.

[0014] (Overall structure) Referring to Figure 1, a language processing system 10 including a language processing device 20 according to one embodiment of this disclosure will be described. The language processing system 10 is included in an information system within an organization such as a company, school, or research institution. The language processing system 10 may be a system located in a specific location, or it may be a system distributed geographically.

[0015] The language processing system 10 includes a language processing device 20. The language processing system 10 may further include a user terminal 30. The language processing device 20 is a computer that performs speech recognition processing and text editing processing on the speech transcript generated by speech recognition. The language processing device 20 can be a general-purpose computer such as a workstation or a PC (Personal Computer), or a dedicated computer. The language processing device 20 may be built as a server on the cloud. The user terminal 30 includes, for example, a PC, a tablet terminal, a smartphone, and a portable information device. The user terminal 30 can send speech data to the language processing device 20 and receive the edited transcript.

[0016] In this application, audio data refers to sound data including human speech. Audio data includes, for example, data from meetings, discussions, lectures, speeches, conversations, and songs. Audio data may be in any data format, including Linear PCM (Linear Pulse Code Modulation), MP3 (MPEG-1 audio layer 3), WAV (Windows Media Audio), AAC (Advanced Audio Coding), and WMA (Windows Media Audio). Audio data can be rephrased as spoken language or spoken language.

[0017] In one embodiment, the language processing device 20 is further configured to communicate with a speech recognition system 50, a speaker separation system 60, and a language model system 70 outside the language processing system 10 via a network 40. The speech recognition system 50, the speaker separation system 60, and the language model system 70 may include a server device that provides cloud services. In another embodiment, some or all of the speech recognition system 50, the speaker separation system 60, and the language model system 70 may be included in the language processing system 10. Alternatively, some or all of the functions of the speech recognition system 50, the speaker separation system 60, and the language model system 70 may be performed by the control unit 21 of the language processing device 20.

[0018] (Configuration of a language processing device) The language processing device 20 is comprised of a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0019] The control unit 21 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The control unit 21 controls each part of the language processing device 20 and executes processes related to the operation of the language processing device 20.

[0020] The control unit 21 may include the following components, as shown in the functional block diagram of Figure 2: a speech recognition unit 21a, a speech recognition post-processing unit 21b, a speaker separation unit 21c, a pre-processing unit 21d, and a sentence editing unit 21e. Each component may be configured as a software module or as a hardware module.

[0021] The speech recognition unit 21a converts the speech data input to the language processing device 20 into speech transcript text, which is text information.

[0022] The speech recognition post-processing unit 21b inserts punctuation marks and tags redundant expressions and named entities into the speech transcript converted from the speech data by the speech recognition unit 21a. In one embodiment, the speech recognition post-processing unit 21b includes a speech recognition post-processing model 21f that performs processing on the speech recognition post-processing unit 21b using machine learning.

[0023] The speaker separation unit 21c separates and identifies the utterances of multiple speakers contained in the audio data, one speaker at a time.

[0024] The pre-processing unit 21d processes the tags assigned to the speech transcript processed by the post-processing unit 21b, and also divides the speech transcript into segments for each speaker identified by the speaker separation unit 21c and for each segment with a defined upper limit of characters.

[0025] The text formatting processing unit 21e includes a text formatting system. The text formatting processing unit 21e applies the text formatting rules described later to the pre-formatting audio transcript processed by the pre-formatting processing unit 21d to convert it into highly readable written language.

[0026] The processing of each part, the speech recognition unit 21a, the post-speech recognition processing unit 21b, the speaker separation unit 21c, the pre-sentence editing processing unit 21d, and the sentence editing processing unit 21e, will be described in more detail below as processing performed by the control unit 21.

[0027] Returning to the description of the language processing device 20 in Figure 1, the storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). The RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). The ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 22 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 22 stores programs and data used for the operation of the language processing device 20, and data obtained by the operation of the language processing device 20. The information stored in the storage unit 22 may be updateable with information obtained from the network 40 via the communication unit 23, for example.

[0028] The memory unit 22 may sequentially store the audio data and character information being processed at each stage in which the language processing device 20 acquires audio data, converts it into a speech-to-text transcript using speech recognition, and performs text editing. The memory unit 22 may also store the learning model used if machine learning inference processing is performed at each stage. Furthermore, the memory unit 22 may store information necessary for the processing performed by the control unit 21, such as information on idiomatic expressions and four-character idioms to be written as kanji, as well as a dictionary of personal names, etc.

[0029] The communication unit 23 includes a communication interface for communicating with a user terminal 30 within the language processing system 10 and for communicating with an external system via the network 40. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may include, for example, a wired LAN (Local Area Network) interface and a USB (Universal Serial Bus) interface. In the case of wireless communication, the communication interface may include, for example, an interface compatible with a mobile communication standard such as wireless LAN, LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation).

[0030] The communication unit 23 may receive voice data subject to speech recognition and text formatting processing from the user terminal 30. In other words, the communication unit 23 functions as an acquisition unit that acquires voice data. The communication unit 23 may send and receive information with external systems such as the speech recognition system 50, the speaker separation system 60, and the language model system 70 in order to execute processing of the language processing device 20.

[0031] The input unit 24 receives input to the language processing unit 20. In addition to acquiring voice data via the communication unit 23, the language processing unit 20 may also acquire voice data from the input unit 24. That is, the input unit 24 functions as an acquisition unit for acquiring voice data. For example, the input unit 24 may include an input port or memory slot for receiving a file containing voice data. Furthermore, the input unit 24 may include a keyboard, mouse, touch panel, etc., for receiving operations on the language processing unit 20.

[0032] The output unit 25 outputs the formatted transcript obtained as a result of the language processing unit 20 processing the audio data. In addition to outputting the formatted transcript via the communication unit 23, the language processing unit 20 may also output the formatted transcript from the output unit 25. The output unit 25 may include a display such as an LCD (Liquid Crystal Display) and a printer. The output unit 25 may also include an output port for outputting the output result to an external storage medium.

[0033] The functions of the language processing device 20 are realized by executing a program relating to the language processing method of this embodiment on a processor corresponding to the control unit 21. In other words, the functions of the language processing device 20 are realized by software. The program causes the computer to perform the operations of the language processing device 20, thereby causing the computer to function as the language processing device 20. That is, the computer functions as the language processing device 20 by performing the operations of the language processing device 20 according to the program.

[0034] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes non-temporary computer-readable media. Examples of computer-readable recording media include magnetic recording devices, optical discs, magneto-optical recording media, or semiconductor memory. The program can be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs (digital versatile discs) or CD-ROMs (compact disc read-only memory) on which the program is recorded. Alternatively, the program may be distributed by storing it in the storage of an external server and transmitting it from the external server to other computers. The program may also be provided as a program product.

[0035] (User terminal configuration) The user terminal 30 may include a control unit, a storage unit, a communication unit, an input unit, and an output unit, similar to the language processing unit 20. The user terminal 30 functions as a client that performs speech recognition and text editing processing of speech data using the language processing unit 20, which functions as a server. The user may input speech data into the input unit of the user terminal 30, have it sent to the language processing unit 20, and receive the edited transcript resulting from the processing by the language processing unit 20 on the user terminal 30. The user may also input speech data directly to the language processing unit 20 and obtain the output result without using the user terminal 30.

[0036] (Processing performed by the language processing unit) Referring to Figure 3, the language processing performed by the control unit 21 of the language processing device 20 will be described. The processing shown in Figure 3 is performed by each component of the control unit 21.

[0037] First, the control unit 21 acquires voice data via the communication unit 23 or the input unit 24 (step S101).

[0038] [Speech recognition processing] Next, the processing of the control unit 21 branches to steps S102 and S104. In step S102, the control unit 21 (speech recognition unit 21a) performs speech recognition processing on the speech data (S102). From the input speech data, the control unit 21 generates a transcript that accurately reproduces the utterance word for word using a speech recognition model generated by machine learning. This transcript can also be called the "raw transcript".

[0039] In one embodiment, the control unit 21 may transmit audio data to an external speech recognition system 50 equipped with a speech recognition model via the network 40 to obtain a speech transcript. In another embodiment, the language processing device 20 does not need to use the external speech recognition system 50. The control unit 21 may generate the transcript by performing speech recognition processing within the language processing system 10 or by a speech recognition model installed in the language processing device 20. The speech recognition processing includes the tokenization of audio data, as will be described later.

[0040] [Post-processing of speech recognition] Next, the control unit 21 (post-speech recognition processing unit 21b) performs post-speech recognition processing on the speech transcript (step S103). A transcript that accurately transcribes spoken language word for word is not easy to read as is. For example, the transcript output as a result of speech recognition does not contain punctuation or question marks. In addition, the transcript contains redundant expressions and named entities. Therefore, the speech transcript needs to be converted into a form that is easy for humans to read. Post-speech recognition processing prepares for this conversion.

[0041] The post-processing of speech recognition includes adding punctuation and question marks to the transcribed text, and tagging (annotating) redundant expressions and named entities. Redundant expressions include "fillers," "corrections," and "acknowledgments." "Fillers" are utterances that fill gaps in spoken language, such as "um" and "well." "Corrections" are the repetition of the same or similar words. "Corrections" include repetition to correct a slip of the tongue. "Acknowledgments" are words that show agreement and / or affirmation to the person being spoken to, such as "I see" and "yes." Named entities include, for example, "Chinese numerals" and "personal names." In this embodiment, it is assumed that in the transcribed text, all numbers are written in Chinese numerals, and all personal names are written in katakana.

[0042] Figure 4 shows an example of the text before and after post-processing of speech recognition (SAS) applied to the transcript before post-processing. Tags are indicated, for example, by including an alphabet letter indicating the type of tag and the target string in parentheses. The method of notating tags is not limited to this. In the example in Figure 4, the letters "F," "D," and "B" indicate that the tag types are redundancy expressions: "filler," "rephrasing," and "interjection," respectively. The letters "S" and "P" indicate that the tag types are proper nouns: "Chinese numeral" and "person's name," respectively. The underlines in the post-processed text in Figure 4 are added to indicate tagged sections, and the actual post-processed text does not contain underlines.

[0043] The control unit 21 can perform speech recognition post-processing using a speech recognition post-processing model 21f, which is a sequence labeling model that adds punctuation and detects and tags redundant and named entities. Here, "sequence labeling model" refers to a machine learning model that labels ordered data such as characters, words, and sentences. As shown in Figure 5, the speech recognition post-processing model 21f is generated by preparing a large number of tagged speech transcripts as training data, to which tags and punctuation have been manually added, and performing machine learning. Speech recognition post-processing involves obtaining the transcript, which is the text after speech recognition (step S201), and applying the speech recognition post-processing model 21f to it to perform inference, thereby adding punctuation and question marks and performing tagging (step S202). This makes it possible to automate speech recognition post-processing.

[0044] The inventors used 63,364 speech transcripts as training data and generated a speech recognition post-processing model 21f using Tohoku University's BERTv3 as the base model. Furthermore, the inventors transformed the generated speech recognition post-processing model 21f using ONNX (Open Neural Network eXchange) and then quantized the model. During inference, the Viterbi algorithm, an optimal path-solving method, was used to improve inference performance. When the generated speech recognition post-processing model 21f was evaluated using 500 evaluation data points, high performance was obtained.

[0045] [Speaker separation processing] Returning to the flowchart in Figure 3, the control unit 21 (speaker separation unit 21c) performs speaker separation processing of the audio data in parallel with the processing in steps S102 and S103 (step S104). The details of the speaker separation process will be explained with reference to Figure 6. First, the control unit 21 detects the start and end times of speech for each speaker from the audio data (step S301). An example of detecting the speech time for each speaker is shown in Figure 7.

[0046] Next, the control unit 21 acquires the speech transcript data that has been tokenized by the speech recognition process (step S302). Tokenization divides the speech transcript text into morphemes by a morphological analyzer. The data divided into morphemes includes information about the start and end times of speech.

[0047] The control unit 21 compares the timestamps of the morphemes in the transcribed text obtained as a result of the speech recognition process with the start and end times of each utterance detected in step S301 to identify the speaker on a morpheme basis (step S303). The control unit 21 adds speaker information to the transcribed text on a morpheme basis (step S304). An example of adding speaker information to a transcribed text is shown in Figure 8. The transcribed text to which speaker information has been added may include, for example, information such as {token ID, token, start time, end time, ..., speaker ID}.

[0048] In one embodiment, the control unit 21 may have the speaker separation process shown in Figure 6 executed by a speaker separation system 60 provided via the cloud. The control unit 21 may transmit the audio data to the speaker separation system 60 and obtain information on the transcript text with speaker information added as a result of the speaker separation from the speaker separation system 60. In another embodiment, the control unit 21 may have a function to perform speaker separation and may perform speaker separation without using an external speaker separation system 60. Speaker separation can be performed based on the characteristics of the speaker's voice. Known technologies can be used for speaker separation. For speaker separation, the open-source Python framework for speaker separation, "pyannote.audio", can be used.

[0049] [Preprocessing for text formatting] Returning to Figure 3, the control unit 21 (pre-editing processing unit 21d) performs pre-editing (step S105). Pre-editing includes several processes to prepare the speech transcript for the editing process before proceeding to the next editing process (step S106). The contents of each process included in pre-editing will be explained with reference to the flowchart in Figure 9.

[0050] The control unit 21 deletes the redundant expressions that were detected and tagged by the speech recognition post-processing model 21f in step S202 (step S401). For example, tags of type "F (filler)", "D (rephrasing)", and "B (acknowledgment)" are deleted.

[0051] Next, the control unit 21 deletes the tags of expressions that should remain as Chinese numerals from the "S (Chinese numeral)" tags and excludes them from the tagging target (step S402). The speech recognition processing of this embodiment converts all numbers contained in the speech data into Chinese numerals and outputs them. However, some numbers should be converted to Arabic numerals, while others should remain as Chinese numerals. The latter include, for example, numbers contained in proverbs, idioms, and four-character idioms. The speech recognition post-processing model 21f can be learned to tag including the identification of whether to use Arabic numerals or Chinese numerals. However, in step S402, incorrect tags are excluded from the tagging target using a rule base. The rule base is constructed by collecting examples of cases where Chinese numerals should remain as they are.

[0052] After step S402, the control unit 21 converts the numbers in the representation that have the "S (Chinese numeral)" tag to Arabic numerals (step S403). For numerical representations from which the tag was removed in step S402, the control unit 21 leaves them as Chinese numerals.

[0053] Next, the control unit 21 performs a conversion on the names included in the speech transcript that have been tagged with the name tag "P (person's name)" by the speech recognition post-processing model 21f, based on the name dictionary (step S404). The name dictionary stores the spelling and pronunciation of names in association. The control unit 21 can replace the pronunciation of names written in katakana in the speech transcript with kanji by referring to the name dictionary. The name dictionary may be stored in the storage unit 22, or it may be stored in a computer outside the language processing device 20.

[0054] Next, the control unit 21 divides the speech transcript from the text into individual speakers using the results of speaker separation in step S104 (step S405). The speech transcript converted from the audio data may contain a mixture of utterances from multiple speakers. Therefore, if the text is edited as is, the utterances of multiple people may be combined into a single sentence, which can degrade the quality of the edited text and / or negatively affect the speaker-specific processing. To avoid this, the control unit 21 divides the speech transcript into individual speakers in the pre-editing process.

[0055] Following step S405, the control unit 21 controls the length of the input sentence for the text editing process (step S406). In one aspect, it is desirable to process utterances from the same speaker in the longest possible sentences. For example, when there are multiple word candidates for homophones such as "rain" and "candy," it may be possible to determine which word to use from the surrounding context. Also, with longer sentences, the likelihood of identifying what demonstrative pronouns refer to increases. However, it has been found that if the input sentence becomes too long, the language model may perform unintended summarization during the text editing process described later. Based on these considerations, the control unit 21 divides the speech transcript into units of input sentences of a length that does not cause summarization. For example, in one embodiment, the control unit 21 divides the speech transcript at periods, question marks, or at changes in speakers. If there are no periods or changes in speakers in a continuous sentence of 400 characters or more, it divides at the token (morpheme) closest to 400 characters. Furthermore, the sentences, separated by periods or question marks before and after each delimited sentence, are combined alternately to adjust the length to 430 characters or less. Note that the above character count is an example; it may vary depending on the language model used to perform the text formatting.

[0056] Figure 10 shows an example of the audio transcript before and after pre-processing. The underlined parts are added to indicate changes and are not included in the actual pre-processed text.

[0057] [Text formatting processing] The control unit 21 (text editing processing unit 21e) performs text editing processing (step S106) on the speech transcript that has undergone the text editing preprocessing described above. The text editing processing includes multiple steps as shown in Figure 11. The control unit 21 uses the input text, which includes the speech transcript that has undergone text editing preprocessing and step setting information, as input to the text editing system that performs the text editing processing. The text editing system is executed using a language model. The text editing system can perform text editing processing using a language model provided by an external language model system 70 provided by the cloud. The language processing unit 20 may have a language model internally. The language model may be a proprietary language model constructed by training a generally available large-scale language model for this text editing processing.

[0058] In each step, the control unit 21 generates an instruction sentence called a prompt to give to the language model, based on the text of the processing result of the previous step and the text formatting rules for that step. The prompt includes the text to be processed in that step, the instruction sentence, and the text formatting rules for that step. The prompt may further include example sentences before and after the text formatting based on the text formatting rules for that step.

[0059] In one embodiment, the large-scale language model Claude can be used as the language model. In this case, the following techniques can be added to the prompting method. • Enclose each element in the instruction with an XML tag: This ensures the language model understands the instructions accurately. • Instructs the system to enclose the output in XML tags: This improves the parsing of the output.

[0060] The control unit 21 performs text formatting processing step by step until it obtains the processing result of the step set in the step setting information. In the example in Figure 11, the control unit 21 performs text formatting processing from step 1 to step 4. In this way, the control unit 21 sequentially prompts the language model and executes processing step by step. By doing so, it repeats the processing up to the step specified in the step setting information and obtains the final result.

[0061] Figure 12 shows an example of a prompt statement for Step 1. (1) through (4) are added for illustrative purposes. (1) assigns a role to the language model to make it behave like a highly linguistic professional editor. (2) instructs the model on the tasks to be performed on the input sentence and the format of the input and output. (3) provides the formatting rules for the corresponding step. (4) presents several examples of the intended output. The prompts for each step are basically structured similarly. In Figures 14, 16, and 18 below, the same codes (1) through (4) are used as in Figure 12.

[0062] In one embodiment, the text formatting process includes up to four steps. The text formatting process for each step is described below.

[0063] <Step 1> Step 1 involves minor text editing, primarily aimed at processing word fragments and standardizing spelling. Step 1 includes the following editing rules 1-1 through 1-7. 1-1. Change any Chinese numerals that should be Arabic numerals to Arabic numerals (half-width). 1-2. Correct the katakana spelling of famous people's names. 1-3. Remove fillers, non-functional words, incomplete sentences, and characters that are clearly meaningless in context. 1-4. Delete any interjections or acknowledgments. (However, leave any that would change the meaning of the text if deleted.) 1-5. Make certain revisions to the parts where the question is asked for clarification. 1-6. English words and units written in katakana should be written in the correct spelling using the alphabet or English characters. Symbols should be written as %, dB, or m. 1-7. Correct any missing or incorrect punctuation marks.

[0064] The prompt for Step 1, shown as an example in Figure 12, has the input sentence shown in Figure 13 and step setting information instructing the system to perform the processing up to Step 3, in the {input} section. The input sentence for Step 1 is the speech transcript after pre-processing. The language model processes the sentence editing rules in order. All of these processes adhere to the rules of not using quotation marks and leaving the end of the input sentence as is, even if it is cut off in the middle of a sentence. The same applies to the following steps. The sentence editing rules in 1-1, 1-2, 1-3, 1-4, and 1-7 overlap in content with the pre-processing, but correct any unfinished tasks or errors. The processing result of Step 1 is at the end of the prompt. <output>This is output as a continuation and passed on to the next step. Note that in Figure 13, the parts that were changed in Step 1 are underlined. The actual input / output statements in Step 1 are not underlined. The same applies to the following examples of Steps 2-4.

[0065] <Step 2> Step 2 involves slightly more refined sentence editing, primarily aimed at improving grammatical accuracy. Step 2 includes the following sentence editing rules, 2-1 through 2-7. 2-1. Remove verbal tics that are meaningless in context. 2-2. Correct simple or obvious slips of the tongue or mispronunciations. 2-3. Correct misuse of words and particles (such as missing particles). 2-4. Correct the overuse of meaningless sentence-ending and interjectional particles to a certain extent. 2-5. Repeats of the same particle should be corrected to a certain extent. 2-6. Any rephrasing regarding corrections to the content should be revised. 2-7. Correct any words that, in context, can be clearly inferred to be errors in speech recognition.

[0066] Figure 14 shows an example of a prompt statement for Step 2. The processing result of Step 1 is substituted into the {step1} part of the prompt for Step 2. The language model processes the sentence formatting rules of Step 2 in order. The processing result of Step 2 is substituted into the end of the prompt. <output>This is output as a continuation and passed on to the next step. Figure 15 shows an example of the processing result of Step 1, which is the input statement for Step 2, and the processing result of Step 2.

[0067] <Step 3> Step 3 involves refining the sentences, primarily to improve fluency. Step 3 includes the following sentence refining rules, 3-1 through 3-8. 3-1. Refine informal or grammatical expressions. 3-2. Delete one of the duplicate words. 3-3. If similar phrases are repeated or the phrasing is redundant within a single sentence, revise certain parts of the sentence while also paying attention to the speaker's tone. 3-4. Correct the subject-predicate mismatch in the sentence. 3-5. Correct any parts where word relationships are inappropriate (disruption of context). 3-6. If the wording is inverted, making it difficult to understand the meaning, potentially leading to misunderstandings, or if the text is difficult to read, revise the sentence. 3-7. If words are missing or omitted, making the meaning unclear or difficult to understand, either add the missing words or delete the unclear words. 3-8. If a sudden change in topic prevents the conversation from continuing, correct the wording or rearrange the word order.

[0068] Figure 16 shows an example of a prompt statement for step 3. The processing result of step 2 is substituted into the {step2} part of the prompt for step 3. The language model processes the sentence formatting rules of step 3 in order. The processing result of step 3 is substituted into the end of the prompt. <output>This is output as a continuation and passed on to the next step. Figure 17 shows an example of the processing result of Step 2, which is the input statement for Step 3, and the processing result of Step 3.

[0069] <Step 4> Step 4 involves more extensive rewriting, primarily aimed at improving conciseness. Step 4 includes the following rewriting rules 4-1 through 4-4. 4-1. Delete any parts that are not directly related to the agenda. 4-2. Delete any unrelated monologues. 4-3. Edit the section explaining the characters (notation). 4-4. In cases of repetition of similar phrases or redundant phrasing, go beyond the refining steps in Step 3, Section 3, and refine the entire sentence.

[0070] Figure 18 shows an example of a prompt statement for step 4. The processing result of step 3 is substituted into the {step3} part of the prompt for step 3. The language model processes the sentence formatting rules of step 4 in order. The processing result of step 4 is substituted into the end of the prompt. <output>This is output as a continuation of the previous output. Figure 19 shows an example of the processing result of Step 3, which is the input statement for Step 4, and the processing result of Step 4.

[0071] Next, the text formatting procedure in step S106 of Figure 3 will be explained using the flowchart in Figure 20.

[0072] First, the control unit 21 obtains the text of the speech transcript that has undergone pre-processing in step S105 (step S501).

[0073] The control unit 21 receives a specification of the number of steps N to be processed as step setting information from the user via input from the input unit 24 or via the communication unit 23 (step S502). The number of processing steps N indicates which of steps 1 to 4 will be processed. The number of processing steps can be selected depending on the purpose and use of the text to be edited. For example, if it is necessary to faithfully reproduce the content of the speech, including any slips of the tongue, such as in parliamentary minutes, step 1 or step 2 is suitable. Also, if it is preferable to grasp the main idea rather than the details of the speech, such as in interviews or minutes, step 3 or step 4, which eliminates redundancy, is suitable.

[0074] The control unit 21 assigns 1 to the variable "k", which indicates the step to be processed in the current step (step S503).

[0075] The control unit 21 generates the prompt for step 1 based on the preprocessed text obtained in step S501 and the formatting rules of step 1 (step S504).

[0076] The control unit 21 inputs the prompt for step k (in this case, k=1) to the language model (step S505). If an external language model system 70 is used, the control unit 21 transmits the prompt to the language model system 70 via the communication unit 23.

[0077] The control unit 21 obtains the processing result of step k (in this case, k=1) from the language model (step S506). If the control unit 21 uses an external language model system 70, it receives the processing result from the language model system 70 via the communication unit 23.

[0078] Next, if the processed step k is not step N (step S507: No), the control unit 21 adds 1 to k (step S508) and proceeds to the processing of the next step k.

[0079] The control unit 21 generates a prompt for step k (step S509) based on the processing result of the previous step (step k-1) and the formatting rules for the current step (step k).

[0080] The control unit 21 then executes the processes of steps S505 and S506 for the current step k. The control unit 21 sequentially executes the processes of each step, and repeats the processes of steps S508, S509, S505 and S506 until the process of step N is completed (step S507: No).

[0081] When the control unit 21 completes the processing in step N (step S507: Yes), it outputs the processing result of step N to the output unit 25 and / or causes the output result to be output to the user terminal 30 via the communication unit 23 (step S508).

[0082] As described above, the audio data acquired in step S101 of Figure 3 is converted into text data, subjected to formatting processing, and output as a text with improved readability.

[0083] As described above, according to this embodiment, the control unit 21 divides the speech transcript before editing into individual speakers and performs editing processing using a language model on the text divided for each speaker. This prevents or reduces the mixing of utterances from multiple speakers in the text to be edited, so that the language processing device 20 can perform editing processing with high accuracy.

[0084] Furthermore, the control unit 21 controls the text input to the language model so that each unit of text input at one time is as long as possible, while remaining below a predetermined number of characters converted from the utterance of the same speaker. This allows the language processing device 20 to improve the accuracy of text editing by having the language model consider the context, while preventing or reducing unintended summarization by the language model.

[0085] Furthermore, this embodiment includes a speech recognition post-processing model generated by machine learning that tags specific types of expressions contained in the speech transcript. This reduces the processing load of tagging the speech transcript, and by performing processing according to the type of tag in the subsequent pre-processing stage, it is possible to generate a speech transcript suitable for editing.

[0086] Furthermore, in this embodiment, the text editing process is divided into multiple stages, and the text editing rules are defined and incorporated into the text editing system of the language processing device 20. This automates the text editing process and enables text editing of a certain quality. In addition, in this embodiment, the text editing process is divided into multiple steps, and the text is edited up to the step required by the user. This makes it possible to control the degree of text editing according to the purpose or use.

[0087] Furthermore, the text editing process in this embodiment is performed in multiple steps, with small-scale and simple processing that has little impact on the context being performed in the earlier steps, and more difficult and in-depth text editing, such as changing word order, being performed in the later steps. This prevents sentence distortion, makes the text editing process smoother, and improves the quality of the edited text.

[0088] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art can make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means or step can be rearranged in a logically consistent manner, and multiple means or steps can be combined into one or divided. [Explanation of Symbols]

[0089] 10. Language Processing Systems 20 Language Processing Devices 21 Control Unit 21a Speech Recognition Unit 21b Post-processing unit for speech recognition 21c Speaker separation section 21d Text preprocessing 21e Text formatting 21f Speech Recognition Post-Processing Model 22 Memory section 23 Communications Department (Acquisition Department) 24 Input section (acquisition section) 25 Output section 30 User terminals 40 Networks 50 Voice Recognition Systems 60 Speaker Separation System 70 Language Model Systems< / output> < / output> < / output> < / output>

Claims

1. An acquisition unit that acquires audio data, A control unit that performs speech recognition processing to convert utterances contained in the audio data into text, speaker separation processing to separate and identify the utterances of multiple speakers contained in the audio data for each speaker, text splitting processing to divide the text converted from the utterances by the speech recognition processing into text for each speaker identified by the speaker separation processing, and text formatting processing to improve readability of the text divided for each speaker using a language model. A language processing device equipped with the following features.

2. The language processing apparatus according to claim 1, wherein the control unit controls the text to be input to the language model at one time so that the text is less than or equal to a predetermined number of characters converted from the utterance of the same speaker.

3. The language processing apparatus according to claim 1, wherein the control unit includes a speech recognition post-processing model that performs a process of assigning tags to specific types of expressions contained in the text converted from the speech data by the speech recognition process, and the control unit performs a specific process on the text to which the tags have been assigned before the text editing process, according to the type of tags.

4. The speech recognition post-processing model is generated by machine learning using a plurality of training data sets to which the tags have been previously attached to text converted from speech data, as described in claim 3.

5. The language processing device according to claim 3, wherein the aforementioned specific type of expression includes an expression containing Chinese numerals, and when the control unit finds that the text to which the tag has been attached includes a tag attached to an expression containing Chinese numerals, it removes the tag from the expression containing Chinese numerals that should remain as Chinese numerals according to a predetermined rule, and then converts the Chinese numerals included in the expression containing Chinese numerals that have not had the tag removed into Arabic numerals.

6. The aforementioned text editing process includes processing based on multiple text editing rules, the multiple text editing rules are classified into N stages, and N is an integer of 2 or more. The control unit is configured to sequentially execute text formatting processes from the first to the Nth stage. The control unit causes the language model to perform a process in the first stage of text formatting processing, which involves applying the text formatting rules classified as the first stage to the text to be formatted, and outputting the processing result of the first stage. When k is an integer between 2 and N, the control unit causes the language model to perform the following processing in the k-th stage of text editing: the result of the (k-1)th stage is the text to be edited, a text editing rule classified as the k-th stage is applied, and the result of the k-th stage is output. The language processing device according to claim 1.

7. The language processing device according to claim 6, wherein the control unit receives a designation of the stages to which the text formatting processing should be performed, and performs the text formatting processing from the first stage to the designated stage.

8. The language processing device according to claim 6, wherein the sentence editing rules classified as the first stage include rules for processing word fragments and unifying notation.

9. The language processing device according to claim 6, wherein the sentence editing rules classified into the Nth stage include rules for making modifications that simplify sentences and texts.

10. The language processing apparatus according to claim 6, wherein the control unit generates a prompt in each of the steps of the text editing process that includes the text to be edited, the text editing rules, and example sentences before and after text editing based on the text editing rules, and causes the language model to execute the text editing rules by inputting the prompt to the language model.

11. A language processing method performed by the control unit of a language processing device, Acquiring audio data and, A speech recognition process that converts the utterances contained in the aforementioned audio data into text, Speaker separation processing that separates and identifies the utterances of multiple speakers contained in the aforementioned audio data, A text splitting process that divides the text converted from the utterance by the speech recognition process into separate texts for each speaker identified by the speaker separation process, For the text divided by speaker, a language model is used to perform a text formatting process to improve readability. A language processing method that includes this.

12. Acquiring audio data and, A speech recognition process that converts the utterances contained in the aforementioned audio data into text, Speaker separation processing that separates and identifies the utterances of multiple speakers contained in the aforementioned audio data, A text splitting process that divides the text converted from the utterance by the speech recognition process into separate texts for each speaker identified by the speaker separation process, For the text divided by speaker, a language model is used to perform a text formatting process to improve readability. A program that causes a computer to execute something.

Images