Information processing device, information processing method, and computer program

Abstract

An information processing device, method, and program for transcribing voice data into text data with high accuracy are provided. [Solution] In the information processing device 100, a processing unit inputs voice data into a voice recognition model that outputs transcription data in response to input of voice data, obtains transcription data of the voice data that is output in response to the input, inputs an input sentence including instructions for outputting proofreading information that matches pre-proofreading text containing transcription errors and the proofread text for the obtained transcription data into a large-scale language model, obtains a first proofreading proposal including proofreading information that matches pre-proofreading text and the proofread text that are output in response to the input, proofreads the transcription data based on the obtained first proofreading proposal, generates proofread transcription data, and outputs the generated proofread transcription data.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer program that perform processing related to conversion of audio data into text data. [Background technology]

[0002] In recent years, transcription tools that convert the contents of audio data into text have been attracting attention, and are being used in a variety of fields, such as creating minutes at meetings and recording telephone calls at call centers.

[0003] Conventionally, speech recognition engines that transcribe speech data have been developed by combining an acoustic model that converts speech features into a phoneme sequence, a pronunciation model that converts the phoneme sequence into word candidates, and a language model that predicts the sequence of word candidates and generates text. However, such conventional speech recognition engines focus on the problem of transcribing each utterance independently, and are unable to transcribe while taking into account the context of multiple utterances contained in the speech data. As a result, they are unable to transcribe naturally while taking into account the context of the speech data, and the accuracy of transcription has been lacking.

[0004] In response to this, so-called end-to-end transcription has been attracting attention in recent years, in which a speech recognition engine that uses a transformer-based model to generate text from speech features in a single model (for example, Patent Document 1). An end-to-end speech recognition engine that uses a transformer-based model enables more natural transcription that takes into account long contexts. Known examples of speech recognition engines that use a transformer-based speech recognition model include "Whisper" by OpenAI and "Wav2Vec" by Meta.

[0005] Although the accuracy of transcription is significantly improved by an end-to-end speech recognition engine using a transformer-based model compared to conventional speech recognition engines, as far as the inventors have verified, transcription using currently available end-to-end speech recognition models still contains many transcription errors. However, in order to improve the transcription accuracy of an end-to-end speech recognition model, a huge amount of high-quality text and speech data set is required, and it is not easy to improve the transcription accuracy of an end-to-end speech recognition model (for example, Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2022-010410 [Patent Document 2] JP 2024-010464 A Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention has been made in consideration of the problems with the conventional technology as described above, and in one aspect, it is an object of the present invention to provide an information processing device that enables more accurate transcription of voice data into text data. [Means for solving the problem]

[0008] In the course of their extensive research efforts to solve the above problems, the inventors came up with the idea that, as an attempt distinct from attempts to improve the speech recognition accuracy of a speech recognition model built into a speech recognition engine, it might be possible to correct transcription errors contained in the transcription data output by the speech recognition model with a large-scale language model by processing the transcription data output by the speech recognition model with a high degree of accuracy.

[0009] However, when a large-scale language model is instructed to proofread the transcription data and the proofread transcription data is directly output, the text data that is output contains many obvious proofreading errors, and the results are not necessarily satisfactory in terms of improving the accuracy of the transcription. This is thought to be because the large-scale language model is not good at continuing to apply the tasks that were initially instructed to it to the long input data without forgetting them.

[0010] Therefore, in the course of further research efforts, the inventors discovered that by feeding transcription data into a large-scale language model and outputting proofread data directly, rather than outputting proofread information that matches the proofreading points contained in the transcription data, or more preferably, the proofreading points containing transcription errors contained in the transcription data, with proofreading suggestions for the proofreading points, it is possible to reduce proofreading errors caused by the large-scale language model and obtain highly accurate transcription data.

[0011] That is, in one aspect, the present invention provides A means for inputting voice data into a voice recognition model that outputs transcription data in response to input of voice data, and acquiring transcription data of the voice data that is output in response to the input; a means for inputting an input sentence including an instruction for outputting pre-proofreading text including transcription errors and proofreading information associating the proofreading text with the obtained transcription data into a large-scale language model, and acquiring a first proofreading proposal including proofreading information associating the pre-proofreading text and the proofreading text that are output in response to the input; A means for proofreading the transcription data based on the obtained first proofreading proposal to generate proofread transcription data; and A means for outputting the generated proofread transcription data; The above-mentioned problems are solved by providing an information processing device comprising:

[0012] In a preferred embodiment, the input sentence input to the large-scale language model may further include an instruction to output the unproofread text and the proofread text as proofreading information only if the unproofread text and the proofread text have the same or similar pronunciation. Such an instruction may be, for example, an instruction to output the unproofread text and the proofread text as proofreading information only if the unproofread text and the proofread text have the same or similar pronunciation.

[0013] The similarity of the readings of the pre-proofreading text and the proofread text may be determined by the large-scale language model on its own, or a definition or condition may be given. For example, a definition or condition may be set that the readings of the pre-proofreading text and the proofread text are similar when the number of sounds with different readings is 2 or less, more preferably 1 or less, or the number of sounds with different readings is 33% or less, more preferably 25% or less, and even more preferably 20% or less of the number of sounds constituting the readings of the pre-proofreading text or the proofread text. When transcription data is provided to the large-scale language model and a pre-proofreading text and a proofread text containing a transcription error are output, a proofread text with a reading significantly different from that of the pre-proofreading text may be proposed by prioritizing the meaning of the sentence. If the readings of the pre-proofreading text and the proofread text are significantly different, this may be beyond the scope of proofreading the transcription error, which may be undesirable. When the input sentence input to the large-scale language model includes the above instruction, proofreading information including a more appropriate proofread text may be output as a proofreading proposal for proofreading the transcription result.

[0014] In a preferred embodiment, when executing the means for obtaining the first proofreading proposal, a word list that associates words with their pronunciations may be referenced. That is, in a preferred embodiment, the input sentence input to the large-scale language model may further include an instruction to output the proofreading information by referring to a word list that associates words with their pronunciations. Such an instruction may be, for example, an instruction to output the proofreading information by referring to a word list that associates words with their pronunciations.

[0015] The large-scale language model can execute a task according to an instruction to output the proofreading information by referring to a word list that associates a word with a reading of the word, but more preferably, a more specific instruction is given. For example, the instruction to output proofreading information by referring to a word list may be an instruction to output proofreading information including the text and the word included in the transcription data as a pre-proofreading text and a post-proofreading text, respectively, only when the reading of the text and the reading of the word included in the word list are the same or similar to each other. More preferably, the instruction to output proofreading information including the text and the word, respectively, associated with the pre-proofreading text and the post-proofreading text, may be an instruction to output proofreading information including the text and the word included in the transcription data as a pre-proofreading text and a post-proofreading text, respectively, only when the reading of the text and the reading of the word included in the word list are the same or similar to each other, and the meaning of the sentence obtained by replacing the text with the word in the sentence including the text included in the transcription data is not unnatural. In addition, in a sentence including the text included in the transcription data, an instruction to consider whether the meaning of the sentence obtained by replacing the text with the word is not unnatural may be explicitly stated or implicit in the input sentence. In addition, when the meaning of the sentence is not unnatural, the meaning of the sentence is natural.

[0016] Technical terms, new words, personal names, homonyms, and the like used in a specific field or a specific group are difficult for a speech recognition engine or a large-scale language model to understand, and it is not easy to transcribe them correctly. On the other hand, when an information processing device according to one aspect of the present invention has the above-mentioned configuration using a word list, by referring to a pre-recorded word list, it becomes possible to more accurately correct the reading and writing of technical terms, new words, personal names, homonyms, and the like that have not been learned by a large-scale language model or whose contents are difficult for a large-scale language model to understand or output, or to proofread the transcription data for specific terms as desired by the user.

[0017] In addition to the words and their pronunciations, the word list may contain other information, such as the part of speech of the word, the context in which the word is used, the field in which the word is used, etc. If the word list contains such other information, it goes without saying that the input sentence input to the large-scale language model may contain an instruction to output the proofreading information by referring to the other information.

[0018] In a preferred embodiment, the information processing device according to one aspect of the present invention may further include a means for acquiring a second proofreading proposal by selecting proofreading information in which the unproofread text and the proofread text have the same or similar pronunciation from the proofreading information included in the first proofreading proposal. In this case, the information processing device according to one aspect of the present invention proofreads the transcription data based on the acquired second proofreading proposal, and generates proofread transcription data.

[0019] When transcription data is provided to a large-scale language model and an uncorrected text and its corrected text containing transcription errors are output, a corrected text with a reading significantly different from that of the uncorrected text is proposed, which may exceed the scope of transcription errors. In an input sentence input to a large-scale language model, an instruction is given to output the uncorrected text and the corrected text as proofreading information only when the uncorrected text and the corrected text have the same or similar readings in the input sentence, and the output of the proofread text with a reading significantly different from that of the uncorrected text is suppressed, but it is not easy to completely eliminate such output. In response to this, an information processing device according to one aspect of the present invention further includes a means for acquiring a second proofreading proposal obtained by selecting proofreading information included in the first proofreading proposal that has the same or similar readings as the uncorrected text and the corrected text, and when the proofreading proposal (first proofreading proposal) output by the large-scale language model is inspected, proofreading information with the same or similar readings as the uncorrected text and the corrected text can be selected, thereby realizing more appropriate transcription.

[0020] There are no particular limitations on the specific method of implementing the above-mentioned means for obtaining a second proofreading proposal by selecting proofreading information from the first proofreading proposal in which the reading of the pre-proofreading text and the proofread text are identical or similar, and the means may be implemented using a rule-based algorithm, a model such as a large-scale language model or AI, or a combination thereof.

[0021] When using a rule-based algorithm for selection, for example, if the number of sounds whose readings differ between the pre-proofreading text and the proofreading text is two or less, more preferably one or less, or if the number of sounds whose readings differ is 33% or less of the total number of sounds that make up the reading of the pre-proofreading text or the proofreading text, more preferably 25% or less, and even more preferably 20% or less, it can be determined that the readings of the pre-proofreading text and the proofreading text are similar and extracted.

[0022] On the other hand, when the selection is performed using a large-scale language model, for example, an input sentence including an instruction to select and output only the proofreading information, of the proofreading information included in the first proofreading proposal, whose readings are the same or similar between the unproofreaded text and the proofread text, may be input to the large-scale language model, and the output may be obtained as the second proofreading proposal. Here, in the input sentence input to the large-scale language model, a definition or condition of the same or similar readings may be given in more detail, and for example, a definition or condition may be set that the readings of both are similar when the number of sounds whose readings differ between the reading of the unproofreaded text and the reading of the proofread text is two sounds or less, more preferably one sound or less, or when the number of sounds whose readings differ is 33% or less, more preferably 25% or less, and even more preferably 20% or less of the number of sounds constituting the reading of the unproofreaded text or the proofread text.

[0023] The uncorrected text and the proofread text may be output in any unit. For example, the uncorrected text and the proofread text may be output in sentence or speech units, or in word or phrase units. Furthermore, the proofreading information may further include other elements in addition to the uncorrected text and the proofread text. Other elements included in the proofreading information include, for example, information that can identify the proofreading point, i.e., the position of the uncorrected text, in the transcription data. Such information may be any information as long as it can identify the position of the uncorrected text in the transcription data. For example, a sentence or speech containing the uncorrected text, and if necessary, one or more speeches or sentences before and after it may be output. Furthermore, the position of the uncorrected text in the transcription data may be identified by the number of lines, the number of characters, the number of bytes, the paragraph number, the order number or number of the sentence or speech in the entire transcription data, etc.

[0024] In a preferred embodiment, an information processing device according to an aspect of the present invention executes the means for dividing the transcription data into predetermined units and acquiring a first proofreading proposal including proofreading information that associates a pre-proofreading text with a proofread text for each of the divided predetermined units. That is, in a preferred embodiment, the means for acquiring a first proofreading proposal divides the acquired transcription data into predetermined units and inputs an input sentence including an instruction for outputting a first proofreading proposal including a pre-proofreading text including a transcription error and proofreading information that associates the proofread text for each of the transcription data divided into the predetermined units into a large-scale language model, thereby acquiring a first proofreading proposal including proofreading information that associates a pre-proofreading text output in response to the input with the proofread text.

[0025] There are no particular limitations on the specified unit, but for example, it is preferable that the unit is a unit containing at least two utterances or sentences, more preferably a unit containing 2 to 1000 utterances or sentences, preferably a unit containing 5 to 500 utterances or sentences, and more preferably a unit containing 10 to 300 utterances or sentences.

[0026] A sentence means a unit separated by a period. On the other hand, an utterance means a speech section included in the voice data. The voice data is composed of a speech section and a silent section (or a non-speech section), and each speech section is separated by a silent section. Those skilled in the art can easily recognize the units of speech included in the voice data. In addition, a voice recognition model that can distinguish and output a speech section and a silent section of the voice data, such as "Whisper" by OpenAI, may be used. In addition, it is optional to detect a speech section included in the voice data by using a voice activity detection technique. In this case, the voice data may be processed by speech activity detection before being transcribed by the voice recognition model.

[0027] According to the findings of the present inventors, if the input transcription data is too short, the context cannot be understood, and the oversight of the proofreading part, i.e., the proofreading omission of the part including the transcription error becomes noticeable. On the other hand, many large-scale language models are not good at understanding long tasks and at continuing to apply the task that was initially instructed to long input data without forgetting it, and if the input transcription data becomes too long, the proofreading omission becomes noticeable. In response to this, by dividing the transcription data into predetermined units, inputting the transcription data for each of the divided predetermined units, and outputting a first proofreading proposal including proofreading information that associates the pre-proofreading text with the proofreading text, it becomes possible to proofread the transcription data more thoroughly.

[0028] In another aspect, the present invention also provides, for example, an information processing method realized by the information processing device according to one aspect of the present invention, and a computer program for causing a computer to function as the information processing device. Effect of the Invention

[0029] According to the present invention, it is possible to transcribe voice data into text data with high accuracy. [Brief description of the drawings]

[0030] [Figure 1] 1 is a diagram showing an example of a configuration of an information processing system including an information processing device according to an aspect of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an example of a configuration of an embodiment of an information processing device according to an aspect of the present invention. [Diagram 3] 1 is a diagram showing an example of a flow of information processing by an information processing system including an information processing device according to an aspect of the present invention. [Figure 4] A figure showing an example of transcription data input to a large-scale language model, and proofreading information that associates pre-proofreading text with proofread text and is output by the large-scale language model in accordance with the transcription data. [Diagram 5] A figure showing an example of transcription data input to a large-scale language model, and proofreading information that associates pre-proofreading text with proofread text and is output by the large-scale language model in accordance with the transcription data. [Figure 6] 1 is a diagram showing an example of a flow of information processing by an information processing system including an information processing device according to an aspect of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. However, the following embodiments are merely examples of the embodiments of the present invention, and are not intended to limit the technical scope of the present invention.

[0032] [1. Embodiment 1] [1.1. Configuration of the First Embodiment] First, the configuration of the information processing device 100 according to the first embodiment will be described.

[0033] The information processing device 100 is an information processing device that transcribes voice data and outputs the transcription data.

[0034] "Audio data" is data including audio. The audio data may be basically any type as long as it can be input to a voice recognition model that outputs transcription data in response to input of audio data described later, but may be provided, for example, as a file in an audio data format. The audio data format file may be, for example, an MP3, M4A, or WAV format file, but is not limited thereto. The audio data may also be provided as a file in a file format that includes data other than audio, such as video. The file format that includes data other than audio together with audio data may be, for example, an MP4 or WebM format file, but is not limited thereto.

[0035] On the other hand, the "transcription data" is data including text obtained by transcribing (converting) the voice included in the voice data. The format of the transcription data output by the information processing device 100 may basically be any format, but as an example, it may be, for example, a file in TXT or CSV format.

[0036] As shown in FIG. 1, an information processing device 100 is communicably connected via a network N to an external information processing device 200, an external information processing device 300, an external storage 400, and a user terminal 500 which is a terminal device used by a user.

[0037] The information processing device 200 is an information processing device equipped with a voice recognition model, for example, a cloud server. When the information processing device 200 receives a request to transcribe voice data from the information processing device 100, the information processing device 200 uses the voice recognition model to execute transcription of the voice data and converts the voice data into text data (transcription data).

[0038] The information processing device 300 is an information processing device equipped with a large-scale language model, for example, a cloud server. When the information processing device 300 receives a proofreading request for the transcription data from the information processing device 100, the information processing device 300 inputs an input sentence including an instruction for causing the large-scale language model to output a proofreading proposal corresponding to the transcription data, and causes the large-scale language model to generate a proofreading proposal corresponding to the transcription data.

[0039] The external storage 400 is a storage device in which various data such as voice data provided from the user terminal 500, transcription data generated in the information processing device 200, proofreading proposals generated in the information processing device 300, and proofread transcription data are stored as needed, and is, for example, an online storage.

[0040] In Fig. 1, the information processing devices 100, 200, and 300 are depicted as if they are one physical information processing device, but the information processing device 100 does not necessarily have to be one physical information processing device. For example, the information processing device 100 may be a distributed information processing device, for example, a distributed server. This also applies to Fig. 3 described later.

[0041] In addition, in Fig. 1, the external storage 400 is also depicted as being one physical storage device, but it does not necessarily have to be one physical storage device. For example, it may be a distributed storage device, for example, a distributed type storage. This is also true for Fig. 3 described later.

[0042] Next, a more detailed configuration of the information processing device 100 will be described.

[0043] As shown in FIG. 2, the information processing device 100 conceptually includes a communication unit 110, a storage unit 120, and a processing unit .

[0044] <Communication Unit 110> The communication unit 110 is a means for transmitting and / or receiving various types of information to and from various devices via the network N. The communication unit 110 may be configured to include, for example, a network interface card (NIC).

[0045] <Storage section 120> The storage unit 120 is a means for storing various types of information, and may be configured to include, for example, memories including RAM (Random Access Memory) such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory), flash memories such as NOR type flash memory and NAND type flash memory, and / or a hard disk.

[0046] <Processing section 130> The processing unit 130 is a means for executing various processes, and may include a processor such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU).

[0047] The processing unit 130 is conceptually configured to include, in terms of function, a transcription request acquisition unit 131, a transcription data acquisition unit 132, a proofreading proposal acquisition unit 133, a transcription data proofreading unit 134, and a proofread transcription data output unit 135. Each unit will be described below.

[0048] <Transcription request acquisition unit 131> The transcription request acquisition unit 131 is a means for acquiring a transcription request for audio data.

[0049] The transcription request acquisition unit 131 receives a transcription request for audio data from the user terminal 500 via the communication unit 110, for example, to acquire the transcription request for audio data.

[0050] The transcription request is accompanied by information for identifying the audio data to be transcribed. The information for identifying the audio data may be, for example, a file path of the audio data. For example, the user terminal 500 may upload the audio data to the external storage 400, and provide the file path of the audio data uploaded to the external storage 400 to the transcription request acquisition unit 131 of the information processing device 100 as information for identifying the audio data.

[0051] <Transcription data acquisition section 132> The transcription data acquisition unit 132 is a means for acquiring transcription data of the voice data specified in the transcription request. For example, the transcription data acquisition unit 132 directly or indirectly inputs the voice data specified in the transcription request to a voice recognition model that outputs transcription data in response to input of voice data, and acquires the transcription data that is output in response to the input.

[0052] In this example, a voice recognition model that outputs transcription data in response to input of voice data is provided in an information processing device 200, which is an external information processing device separate from the information processing device 100. The transcription data acquisition unit 132 transmits, for example, a transcription request for voice data specified in the transcription request acquired by the transcription request acquisition unit 131 to the information processing device 200. In the information processing device 200 that receives the transcription request from the transcription data acquisition unit 132, the voice data specified in the transcription request is input to a voice recognition model that outputs transcription data in response to input of voice data, and text data output in response to the input is acquired as transcription data, which is then transmitted to the information processing device 100. The transcription data acquisition unit 132 of the information processing device 100 receives the transcription data via the communication unit 110, thereby acquiring the transcription data of the voice data.

[0053] In one embodiment, the information processing device 200 may be configured to store the transcription data generated by the voice recognition model in the external storage 400. In this case, the transcription data acquisition unit 132 of the information processing device 100 may be configured to acquire the transcription data by reading the transcription data stored in the external storage 400. In any case, it is sufficient that the information processing device 100 is capable of acquiring the transcription data generated by the voice recognition model.

[0054] As described above, the information processing device 200 having the voice recognition model that outputs transcription data in response to input of voice data may be a single physical information processing device, or may be a distributed information processing device, for example, a distributed server. In this example, the voice recognition model is provided in the information processing device 200, which is an external information processing device, but in another aspect, the information processing device 100 may be configured to have the voice recognition model.

[0055] The voice recognition model that outputs transcription data in response to input of voice data may be basically any type as long as it is a trained model that can transcribe the voice contained in the input voice data and generate text data in accordance with the content of the voice data, and may be, for example, a transformer-based voice recognition model. Examples of such voice recognition models include, but are not limited to, "Whisper" provided by OpenAI, Inc. and "Speech-to-Text" provided by Alphabet, Inc.

[0056] Furthermore, a speech recognition model that outputs transcription data in response to input of speech data may be fine-tuned to generate transcription data, etc. For example, the model may be fine-tuned to understand technical terms according to a field, to understand terms used in a specific organization such as a company, to understand a specific language, and / or to understand the habits of speakers in a specific organization such as a company.

[0057] The voice data may be pre-processed by an appropriate method before being transcribed by the voice recognition model. For example, the file format of the voice data may be converted into another file format, or noise contained in the voice data may be removed. The pre-processing of the voice data may be performed by the information processing device 100 or by another information processing device, for example, the information processing device 200.

[0058] <Proofreading proposal acquisition unit 133> The proofreading proposal acquisition unit 133 is a means for acquiring a proofreading proposal (first proofreading proposal) for proofreading the transcription data. For example, the proofreading proposal acquisition unit 133 inputs an input sentence including an instruction for outputting a proofreading proposal corresponding to the transcription data acquired by the transcription data acquisition unit 132 directly or indirectly into a large-scale language model, and acquires a proofreading proposal (first proofreading proposal) for the transcription data to be output in response to the input.

[0059] In this example, the large-scale language model is provided in an information processing device 300, which is an external information processing device separate from the information processing device 100. That is, the proofreading proposal acquisition unit 133 transmits an input sentence including an instruction for generating a proofreading proposal (first proofreading proposal) corresponding to the transcription data acquired by the transcription data acquisition unit 132 to the information processing device 300 via the communication unit 110, thereby inputting an input sentence including an instruction for generating a proofreading proposal (first proofreading proposal) for proofreading the transcription data acquired by the transcription data acquisition unit 132 to the large-scale language model provided in the information processing device 300, and causing the large-scale language model to generate a proofreading proposal. The information processing device 300 transmits the proofreading proposal (first proofreading proposal) generated by the large-scale language model to the information processing device 100, and the proofreading proposal acquisition unit 133 of the information processing device 100 receives the proofreading proposal via the communication unit 110.

[0060] In one embodiment, the information processing device 300 may be configured to store the proofreading proposal (first proofreading proposal) generated by the large-scale language model in the external storage 400. In this case, the external storage 400 may be configured to store, for example, the transcription data and the proofreading proposal (first proofreading proposal) generated by the large-scale language model in association with each other. In this way, the proofreading proposal acquisition unit 133 of the information processing device 100 can acquire a proofreading proposal corresponding to the transcription data by reading out the proofreading proposal stored in the external storage 400 in association with the transcription data. In any case, it is sufficient that the information processing device 100 can acquire the proofreading proposal (first proofreading proposal) generated by the large-scale language model and can identify the correspondence between each proofreading proposal and the transcription data.

[0061] As described above, the information processing device 300 including the large-scale language model that outputs a proofreading proposal (first proofreading proposal) according to the transcription data may be a single physical information processing device, or may be a distributed information processing device, for example, a distributed server. In this example, the large-scale language model is provided in the information processing device 300, which is an external information processing device, but in another aspect, the information processing device 100 may be configured to include the large-scale language model.

[0062] The large-scale language model that generates a proofreading proposal (first proofreading proposal) according to the transcription data may be, for example, a Transformer-based large-scale language model. The Transformer-based large-scale language model may be, for example, "GPT" provided by OpenAI, Inc., "Gemini" provided by Google DeepMind, Inc., "Claude" provided by Anthropic, Inc., "tsuzumi" provided by Nippon Telegraph and Telephone Corporation, etc., but is not limited thereto.

[0063] Additionally, the large-scale language model may be fine-tuned as appropriate, for example, to understand technical terms according to a field, to understand terms used within a given organization such as a company, to understand a particular language, and / or to understand the habits of speakers within a given organization such as a company.

[0064] The input sentence including the instruction for outputting the proofreading proposal (first proofreading proposal) according to the transcription data is also referred to as a prompt including the instruction for outputting the (first proofreading proposal) according to the transcription data. When the large-scale language model receives the input of the input sentence or the prompt including the predetermined instruction, the large-scale language model executes the task instructed by the instruction included in the input sentence or the prompt.

[0065] In a preferred embodiment, the proofreading proposal (first proofreading proposal) based on the transcription data may include proofreading information that associates pre-proofreading text, including a transcription error, included in the transcription data with proofread text of the pre-proofreading text. The proofreading proposal (first proofreading proposal) may include one or more pieces of proofreading information. For example, if there are two proofreading points, a total of two pieces of proofreading information corresponding to the respective proofreading points may be included.

[0066] The pre-proofreading text is text that includes a transcription error included in the transcription data, and may be, for example, text in which the meaning of a sentence or utterance containing the text is unclear or unnatural as it is. The transcription data output by the speech recognition model may contain a large amount of text whose meaning is unclear or unnatural, for example, because the characters representing the sound are incorrect even if the sound is correct. The definition of the pre-proofreading text may be explicitly given to the input sentence to be input to the large-scale language model.

[0067] On the other hand, the proofread text is a text in which the transcription errors contained in the pre-proofread text have been proofread. The proofread text may be, for example, a text in which the transcription errors contained in the pre-proofread text have been corrected so that the pronunciation of the pre-proofread text and the proofread text is the same or similar, and further, in a sentence or utterance including the pre-proofread text included in the transcription data, the meaning of the sentence or utterance in which the pre-proofread text is replaced with the proofread text is not unnatural. In this example, the proofread text may have a similar pronunciation to the pre-proofread text, but in a preferred embodiment, the proofread text may only have the same pronunciation as the pre-proofread text. The definition of the proofread text may be explicitly given to the input sentence input to the large-scale language model. The same or similar pronunciation is as described above, so a description will be omitted.

[0068] According to the findings of the present inventors, when transcription data is input to a large-scale language model and the transcription data is directly proofread, that is, when the large-scale language model is directly made to output the proofread transcription data, many transcription errors remain unproofread. In contrast, by giving the large-scale language model a more specific task of outputting a proofreading proposal (first proofreading proposal) including proofreading information that associates pre-proofreading text including transcription errors with the proofread text, it becomes possible to perform proofreading with fewer omissions.

[0069] The input sentence for generating a proofreading proposal according to the transcription data may further include other information. Examples of other information that may be included in the input sentence may include various information such as the situation or context in which the voice data was recorded, examples of input and output, points to note when performing a task, etc. Examples of the situation or context in which the voice data was recorded may include, but are not limited to, information on the field to which the content of the voice data is related, such as law, medicine, economics, sports, entertainment, etc., the type of speaker, such as elementary school student, high school student, university student, or working adult, and the language of the voice data, such as Japanese, English, or Chinese.

[0070] An input sentence including an instruction for generating proofreading information corresponding to the transcription data may be generated, for example, by using a predetermined prompt template. Specifically, the transcription data to be proofread may be written in a predetermined prompt template, and / or a file path specifying the transcription data may be written in a predetermined prompt template, and this may be used as the input sentence. The generation of the input sentence including an instruction for generating a proofreading proposal corresponding to the transcription data may be configured to be performed by the information processing device 100, or may be configured to be performed by an external information processing device, for example, the information processing device 300. An AI agent may also be used to generate the input sentence. There is no particular restriction on the storage location of the prompt template, and it may be stored in the storage unit 120 of the information processing device 100, or in an external storage device, for example, the external storage 400.

[0071] <Transcription Data Proofreading Department 134> The transcription data proofreading unit 134 proofreads the transcription data based on the first proofreading proposal acquired by the proofreading proposal acquisition unit 133.

[0072] For example, if the first proofreading proposal acquired by the proofreading proposal acquisition unit 133 includes proofreading information that associates pre-proofreading text, which is included in the transcription data and includes a transcription error, with proofread text of the pre-proofreading text, the transcription data proofreading unit 134 executes a process of replacing each of the pre-proofreading texts included in the transcription data with the proofread text corresponding to the pre-proofreading text, thereby generating proofread transcription data.

[0073] <Proofread Transcription Data Output Section 135> The proofread transcription data output unit 135 is a means for outputting the proofread transcription data generated by the transcription data proofreading unit 134.

[0074] For example, the proofread transcription data output unit 135 may transmit the proofread transcription data to the user terminal 500 via the communication unit 110. Alternatively, the proofread transcription data may be stored in the external storage 400 in association with the original voice data.

[0075] [1.2. An example of information processing according to embodiment 1] Next, a description will be given of an example of information processing by the information processing device 100. A flow of an example of information processing by the information processing device 100 is shown in FIG.

[0076] A user operates the user terminal 500 to transmit voice data and a request for transcription of the voice data to the information processing device 100 (step S1). For example, a predetermined display screen provided by a voice recognition system is displayed on the screen of the user terminal 500, and the voice data to be transcribed is uploaded to the external storage 400 through the display screen, and a transcription request for the voice data is transmitted to the information processing device 100 by pressing a transcription execution button displayed on the display screen.

[0077] When the information processing device 100 receives a transcription request for the voice data from the user terminal 500, the information processing device 100 transmits the transcription request for the voice data to the information processing device 200 including a voice recognition model (step S2). The transcription request may be a transcription request that specifies the voice data to be transcribed, and may be, for example, a request that specifies a file path of the voice data to be transcribed.

[0078] When the information processing device 200 including the voice recognition model receives a transcription request for the voice data from the information processing device 100, the information processing device 200 inputs the voice data to the voice recognition model. The voice recognition model transcribes the voice included in the voice data to generate text data. This generates transcription data (step S3).

[0079] The information processing device 200 transmits the generated transcription data (text data) to the information processing device 100 (step S4).

[0080] In this example, the voice recognition model provided in the information processing device 200 is generated based on the voice data specified by the information processing device 100 as follows: sesame The Democratic Republic of the Congo, Central Africa year is. sesame teeth MiragonThe city was destroyed by volcanic lava, and most of the streets, especially the center of town, were buried.' The information processing device 200 generates transcription data including the text and transmits it to the information processing device 100, as described below (FIG. 4).

[0081] When the information processing device 100 receives transcription data from the information processing device 200 having a speech recognition model, the information processing device 100 transmits the transcription data and a request for proofreading the transcription data to the information processing device 300 having a large-scale language model in order to proofread the transcription data (step S5).

[0082] The proofreading request for the transcription data transmitted to the information processing device 300 includes an input sentence including an instruction to output proofreading information corresponding to the transcription data. In this example, the input sentence including an instruction to output a proofreading proposal corresponding to the transcription data is a prompt including an instruction to output a first proofreading proposal including proofreading information that associates pre-proofreading text included in the transcription data with proofread text of the pre-proofreading text, specific examples of input and output, and points to note when performing a task.

[0083] In this example, an input sentence including an instruction for generating a proofreading proposal corresponding to the transcription data is generated by using a prompt template stored in the storage unit 120 of the information processing device 100. Specifically, the information processing device 100 writes a file path of the transcription data to be proofread in the prompt template stored in the storage unit 120, and generates an input sentence. Note that the generation of an input sentence for generating a proofreading proposal corresponding to the transcription data does not necessarily have to be performed by the information processing device 100, and may be performed by, for example, the information processing device 300. In this case, the proofreading request transmitted to the information processing device 300 only needs to be accompanied by information identifying the transcription data, for example, a file path of the transcription data.

[0084] When the information processing device 300 equipped with a large-scale language model receives from the information processing device 100 a proofreading request for the transcription data including an input sentence including an instruction to output a proofreading suggestion based on the transcription data, the information processing device 300 inputs the input sentence into the large-scale language model and causes the large-scale language model to generate a proofreading suggestion based on the transcription data (step S6).

[0085] In this example, the output of the large-scale language model is shown in Figure 4, and the " sesame teeth Miragon Volcano The city was destroyed by lava, and most of the streets, especially the center of the city, were buried. sesame teeth Miragon Volcano For the sentence "The city was destroyed by lava and most of the streets, especially the center of town, were buried," proofreading information is output that includes a combination of "Miragon volcano" as the pre-proofreading text and "Niragongo volcano" as the post-proofreading text. In this way, the output of the large-scale language model includes proofreading feedback in which the pre-proofreading text and the post-proofreading text are associated with each other. In this example, a sentence including the pre-proofreading text is output in association with the pre-proofreading text and the post-proofreading text as information identifying the proofreading part.

[0086] The information processing device 300 transmits the proofreading information generated by the large-scale language model to the information processing device 100 as a first proofreading proposal (step S7).

[0087] When the information processing device 100 receives the first proofreading proposal from the information processing device 300 equipped with the large-scale language model, the information processing device 100 proofreads the transcription data based on the received first proofreading proposal (step S8).

[0088] For example, in this example, the first proofreading proposal includes a pre-proofreading text, its proofread text, and a sentence including the pre-proofreading text as information for identifying the proofreading portion, in association with each other. Therefore, the information processing device 100 generates proofread transcription data by replacing the pre-proofreading text with the proofread text in the sentence included in the transcription data. Specifically, the proofread transcription data includes the following: Sesame teeth Nyiragongo The city was destroyed by volcanic lava, burying most of the streets, especially the center of town."

[0089] The information processing device 100 outputs the generated proofread transcription data to the user terminal 500 (step S9). This allows the user to obtain the transcription data of the uploaded voice data.

[0090] [1.3. An example of information processing according to embodiment 1] Next, an example of information processing will be described in which the information processing device 100 uses a word list in which words are associated with the pronunciations of the words.

[0091] The steps are as described above, namely, a step in which the user terminal 500 transmits a transcription request for voice data to the information processing device 100 (step S1); a step in which the information processing device 100, having received the transcription request for voice data, transmits the transcription request for the voice data to the information processing device 200 having a voice recognition model (step S2); a step in which the information processing device 200 having the voice recognition model transcribes the voice contained in the voice data and generates transcription data (step S3); and a step in which the information processing device 200 transmits the generated transcription data to the information processing device 100 (step S4), and therefore will not be described here.

[0092] On the other hand, in this example, the voice recognition model recognizes FindingsThe Special Investigation Unit is an investigative agency that investigates political corruption, tax evasion, economic crimes, etc. Through this investigation, it was discovered that Politician A frequently meets with Politician B. Findings The following explanation assumes that transcription data including the text "I got the text." is generated, and that a word list is stored in external storage 400, and that the word "Chiken" and its pronunciation "Chiken" are registered in association with each other in the word list (FIG. 5).

[0093] When the information processing device 100 receives transcription data from the information processing device 200 having a speech recognition model, the information processing device 100 transmits a request for proofreading the transcription data to the information processing device 300 having a large-scale language model in order to proofread the transcription data (step S5).

[0094] The proofreading request for the transcription data transmitted to the information processing device 300 includes an input sentence including an instruction for outputting a proofreading proposal corresponding to the transcription data.

[0095] In this example, the input sentence including an instruction to output a proofreading suggestion corresponding to the transcription data is a prompt including an instruction to output a proofreading suggestion corresponding to the transcription data by referring to the word list stored in external storage 400. In addition to an instruction to output a file path of the transcription data and proofreading information that associates pre-proofreading text included in the transcription data with the proofread text, the prompt further includes a file path of the word list stored in external storage 400 and an instruction to output the proofreading information by referring to the word list.

[0096] In this example, the instruction to output the proofreading information by referring to the word list is an instruction to output proofreading information in which the text and the word correspond to pre-proofreading text and the proofreading text, respectively, when the reading of the text included in the transcription data is identical or similar to the reading of a word included in the word list, the meaning of a sentence including the text included in the transcription data is unnatural, and the meaning of the sentence including the text included in the transcription data is not unnatural when the text is replaced with the word.

[0097] In this example, the voice recognition model outputs the transcription data containing the phrase "Tokyo Findings The Special Investigation Department is an investigative agency that investigates political corruption, tax evasion, economic crimes, etc.," contains the text "Kenken," which is pronounced the same as the word "Chiken," which is pronounced the same as the word "Chiken" in the word list. Findings The sentence "The Special Investigation Department is an investigative agency that investigates political corruption, tax evasion, economic crimes, etc." is unnatural in its original form, and the sentence "Tokyo District Prosecutor's Office The Special Investigation Unit is an investigative agency that investigates political corruption, tax evasion, economic crimes, etc.," is not an unnatural meaning.

[0098] On the other hand, the transcription data output by the speech recognition model included the line, "This investigation has revealed that politician A frequently meets with politician B." Findings The sentence "I got the text 'Knowledge', which has the same reading as the word "Chiken" in the word list. However, the sentence "This investigation has revealed that politician A frequently meets with B. Findings The meaning of the sentence "I obtained it." is not unnatural.

[0099] Therefore, in this example, the large-scale language model is used to generate FindingsThe Special Investigation Unit is an investigative agency that investigates political corruption, tax evasion, economic crimes, etc.," the character string "knowledge" is included in the pre-proofreading text as follows: District Prosecutor's Office " will be output as proofread text. On the other hand, "This investigation has revealed that politician A frequently meets with B. Findings For the sentence "I got the proof.", the pre-proofreading text and the post-proofreading text are not output.

[0100] By giving a predetermined instruction as described above, the large-scale language model functions as a means for determining whether a sentence included in the transcription data includes a text (character string) whose reading is the same as or similar to the reading of a word included in the word list; a means for determining whether the meaning of a sentence including the text (character string) is unnatural or not and whether the meaning of a sentence obtained by replacing the text (character string) included in the sentence with the word is unnatural or not when the means determines that the sentence included in the transcription data includes a text (character string) whose reading is the same as or similar to the reading of a word included in the word list; and a means for outputting the text (character string) as pre-proofreading text, the word as proofreading text, and, if necessary, the sentence including the text as information identifying the proofreading portion when the means determines that the above means can be achieved. A trained model other than the large-scale language model may be used.

[0101] The steps that are subsequently executed are: in the information processing device 300, transmitting a first proofreading proposal generated by the large-scale language model to the information processing device 100 (step S7); in the information processing device 100, receiving the first proofreading proposal from the information processing device 300 equipped with the large-scale language model, and proofreading the transcription data based on the received first proofreading proposal (step S8); and in the information processing device 100, outputting the proofread transcription data to the user terminal 500 (step S9). These steps are as described above, and therefore will not be described here.

[0102] [2. Embodiment 2] As another embodiment of the information processing device according to one aspect of the present invention, an embodiment will be described in which a proofreading proposal proofreading unit 136 is further added to the information processing device 100. Fig. 6 shows an example of the flow of information processing by an information processing system configured to include the information processing device 100 according to this embodiment.

[0103] The proofreading proposal proofreading unit 136 is a means for acquiring a second proofreading proposal obtained by proofreading the first proofreading proposal acquired by the proofreading proposal acquisition unit 133, which may be included in the processing unit 130. In this specification, a proofreading proposal obtained by proofreading the first proofreading proposal is referred to as a second proofreading proposal. When there is no part to be proofread, the contents of the first proofreading proposal and the second proofreading proposal may be the same.

[0104] In this example, the second proofreading proposal is generated in an external information processing device 600 separate from the information processing device 100. That is, the information processing device 100, for example, transmits to the information processing device 600 a proofreading request for the first proofreading proposal acquired by the proofreading proposal acquisition unit 133, and, as necessary, a file path for the first proofreading proposal and / or the first proofreading proposal. Upon receiving the proofreading request for the first proofreading proposal, the information processing device 600 proofreads the first proofreading proposal and generates a second proofreading proposal. Then, it transmits the generated second proofreading proposal to the information processing device 100. The proofreading proposal proofreading unit 136 of the information processing device 100 receives the second proofreading proposal via the communication unit 110, thereby acquiring the second proofreading proposal.

[0105] Furthermore, in this example, the transcription data proofreading unit 134 proofreads the transcription data based on the second proofreading proposal acquired by the proofreading proposal proofreading unit 136.

[0106] In this example, the second proofreading plan is generated by the information processing device 600, but may be generated by the information processing device 100. Also, it may be generated by an external information processing device (e.g., information processing device 200 or 300) other than the information processing device 600. Also, in FIG. 6, the information processing device 600 is depicted as being physically one information processing device, but the information processing device 600 does not necessarily have to be physically one information processing device. For example, it may be a distributed information processing device, for example, a distributed server.

[0107] The second proofreading proposal may be, for example, proofreading information selected from the proofreading information included in the first proofreading proposal acquired by the proofreading proposal acquisition unit 133, in which the readings of the pre-proofreading text and the proofread text are the same or similar. The first proofreading proposal acquired by the proofreading proposal acquisition unit 133 is output by a large-scale language model, and the first proofreading proposal output by the large-scale language model may include proofreading information in which the readings of the pre-proofreading text and the proofread text are neither the same nor similar. By selecting proofreading information in which the readings of the pre-proofreading text and the proofread text are the same or similar from the first proofreading proposal output by the large-scale language model, it is possible to obtain an appropriate proofreading result without changing the reading of the transcription result as much as possible.

[0108] The second proofreading proposal may be a selection of proofreading information included in the first proofreading proposal acquired by the proofreading proposal acquisition unit 133, in which the pronunciation of the pre-proofreading text and the proofread text are the same or similar, and further, in a sentence including the pre-proofreading text included in the transcription data, the meaning of the sentence obtained by replacing the pre-proofreading text with the proofread text is not unnatural and / or natural. The first proofreading proposal acquired by the proofreading proposal acquisition unit 133 is output by a large-scale language model, which is not good at performing long tasks. In particular, when transcription data with a relatively large capacity is provided, the proofreading proposal output by the large-scale language model may include a combination in which the meaning of the sentence becomes unnatural or unnatural when the pre-proofreading text is replaced with the proofread text. By extracting only proofreading information that does not make the meaning of the sentence unnatural when the pre-proofreading text is replaced with the proofread text from the proofreading information included in the proofreading proposal output by the large-scale language model, it is possible to more appropriately proofread the transcription result.

[0109] There is no particular limitation on the specific method for generating the second proofreading proposal, and a person skilled in the art may adopt an appropriate method. For example, the method may be realized by using a rule-based algorithm, a model such as a large-scale language model, or an AI, or a combination of these.

[0110] When a large-scale language model is used to generate the second proofreading proposal, the large-scale language model may be the same as or different from the large-scale language model used to generate the first proofreading proposal. For example, the large-scale language model used to generate the second proofreading proposal may be a Transformer-based large-scale language model, similar to the large-scale language model used to generate the first proofreading proposal. Examples of the Transformer-based large-scale language model include, but are not limited to, "GPT" provided by OpenAI, Inc., "Gemini" provided by Google DeepMind, Inc., "Claude" provided by Anthropic, Inc., and "tsuzumi" provided by Nippon Telegraph and Telephone Corporation. It goes without saying that the large-scale language model used to generate the second proofreading proposal may also be fine-tuned as appropriate.

[0111] Although the embodiments of the present invention have been described in detail above, these embodiments are merely examples, and the present invention can be embodied in other forms with various modifications and improvements based on the knowledge of a person having ordinary skill in the art. Furthermore, the above-mentioned "means" and "unit" can be appropriately interpreted. For example, the acquisition unit can be interpreted as an acquisition means. [Industrial Applicability]

[0112] The present invention makes it possible to accurately transcribe voice data into text data, thereby preventing forgetting to record, typos, and mishearing during daily meetings, business negotiations, interviews, and the like, and making it possible to record and share information with a high degree of accuracy. [Explanation of symbols]

[0113] 100 Information processing device 110 Communications Department 120 Storage section 130 Processing section 131 Transcription Request Acquisition Unit 132 Transcription Data Acquisition Department 133 Proofreading proposal acquisition department 134 Transcription Data Proofreading Department 135 Proofread Transcription Data Output Unit 200 Information processing device (information processing device equipped with a voice recognition model) 300 Information processing device (information processing device equipped with a large-scale language model) 400 External Storage 500 Terminal Equipment 600 Information processing device N Network

Claims

1. A means for inputting voice data into a voice recognition model that outputs transcription data in response to input of the voice data; means for acquiring transcription data of the speech data output from the speech recognition model in response to the input; a means for generating an input sentence including the two instructions for the acquired transcription data based on a prompt template including an instruction for outputting proofreading information that associates pre-proofreading text including transcription errors with proofread text obtained by proofreading the pre-proofreading text for the transcription errors, and an instruction for outputting the proofreading information by referring to a predetermined word list that associates words with the pronunciation of the words; A means for inputting the generated input sentence into a large-scale language model; a means for acquiring a first proofreading proposal including proofreading information that associates pre-proofreading text and proofread text output from the large-scale language model in response to the input; A means for proofreading the transcription data based on the obtained first proofreading proposal and generating proofread transcription data; and A means for outputting the generated proofread transcription data; An information processing device comprising:

2. The input sentence and the prompt template further include: An instruction to output the uncorrected text and the corrected text as proofreading information only if the pronunciation of the uncorrected text and the corrected text is the same or similar, 2. The information processing apparatus according to claim 1,

3. The instruction for outputting the proofreading information by referring to a word list includes: and when the reading of the text included in the transcription data is the same as or similar to the reading of the word included in the word list, and the meaning of the sentence obtained by replacing the text with the word in a sentence including the text included in the transcription data is not unnatural, the instruction includes instructions to output the text and the word as a pre-proofreading text and a proofreading text, respectively.

3. The information processing apparatus according to claim 2.

4. a means for acquiring a second proofreading proposal by selecting proofreading information in which the reading of the pre-proofreading text and the proofreading text are the same or similar from the proofreading information included in the first proofreading proposal; The transcription data is proofread based on the obtained second proofreading proposal to generate proofread transcription data. The information processing device according to claim 1 .

5. The means for obtaining a second proofreading proposal utilizes a large-scale language model to obtain the second proofreading proposal.

5. The information processing apparatus according to claim 4.

6. The means for obtaining a second revision proposal comprises: An input sentence including an instruction for selecting and outputting only the proofreading information in which the reading of the pre-proofreading text and the proofreading text are the same or similar from among the proofreading information included in the first proofreading proposal is input to a large-scale language model, and the output is obtained as a second proofreading proposal.

6. The information processing apparatus according to claim 5,

7. Dividing the acquired transcription data into predetermined units, and acquiring a first proofreading proposal including proofreading information that associates pre-proofreading text with proofread text for each of the transcription data divided into the predetermined units.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. The information processing apparatus according to claim 7 , wherein the predetermined unit includes at least two utterances or sentences.

9. 1. A computer-implemented information processing method, comprising: inputting speech data into a speech recognition model that outputs transcription data in response to the input of the speech data; obtaining transcription data of the speech data output from the speech recognition model in response to the input; generating an input sentence including the two instructions for the acquired transcription data based on a prompt template including an instruction for outputting proofreading information that associates pre-proofreading text including transcription errors with proofread text obtained by proofreading the pre-proofreading text for the transcription errors, and an instruction for outputting the proofreading information by referring to a predetermined word list that associates words with the pronunciation of the words; inputting the generated input sentence into a large-scale language model; obtaining a first proofreading proposal including proofreading information that associates pre-proofreading text and proofread text output from the large-scale language model in response to the input; Proofreading the transcription data based on the obtained first proofreading proposal to generate proofread transcription data; and outputting the generated proofread transcription data; An information processing method comprising:

10. inputting speech data into a speech recognition model that outputs transcription data in response to the input of the speech data; obtaining transcription data of the speech data output from the speech recognition model in response to the input; generating an input sentence including the two instructions for the acquired transcription data based on a prompt template including an instruction for outputting proofreading information that associates pre-proofreading text including transcription errors with proofread text obtained by proofreading the pre-proofreading text for the transcription errors, and an instruction for outputting the proofreading information by referring to a predetermined word list that associates words with the pronunciation of the words; inputting the generated input sentence into a large-scale language model; obtaining a first proofreading proposal including proofreading information that associates pre-proofreading text and proofread text output from the large-scale language model in response to the input; Proofreading the transcription data based on the obtained first proofreading proposal to generate proofread transcription data; and outputting the generated proofread transcription data; A computer program that causes a computer to execute the following:

Images