Information processing system, speech recognition system, information processing method, and program

Abstract

To reduce time and cost required for improving speech recognition accuracy.SOLUTION: The present invention is an information processing system communicable with terminal devices via a network, which comprises: a speech recognition result acquisition unit that acquires speech recognition results of speech data uttered by one or more users; a correction unit that transmits the speech recognition results and a unique vocabulary associated with the one or more users, to a large-scale language model that is a natural language processing model; a correction unit that acquires corrected speech recognition results that the large-scale language model has corrected the speech recognition results based on the unique vocabulary from the large-scale language model; and a communication unit that transmits the acquired corrected speech recognition results to the terminal devices.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a voice recognition system, an information processing method, and a program. [Background technology]

[0002] End-to-end models are known as speech recognition models with excellent accuracy and robustness. Unlike conventional DNN-HMM or GMM-HMM models, end-to-end models do not separate functions into an acoustic model, language model, and pronunciation dictionary, but instead directly convert speech data into text data using a single network, which has the advantage of not requiring advanced personal skills.

[0003] Since the speech recognition results are not necessarily converted exactly as spoken, there are known techniques for correcting the speech recognition results (see, for example, Patent Document 1). Patent Document 1 discloses a technique for correcting the results of recognizing speech data spoken by a user based on a category dictionary corresponding to the category into which the speech content has been classified. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, it takes time and costs money to improve the accuracy of speech recognition.

[0005] In view of the above problems, an object of the present invention is to reduce the time and cost required to improve the accuracy of speech recognition. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an information processing system capable of communicating with a terminal device via a network, the information processing system comprising: a speech recognition result acquisition unit that acquires speech recognition results for speech data spoken by one or more users; a correction unit that transmits the speech recognition results and unique vocabulary associated with the one or more users to a large-scale language model that is a model of natural language processing, and acquires from the large-scale language model corrected speech recognition results obtained by the large-scale language model correcting the speech recognition results based on the unique vocabulary; and a communication unit that transmits the acquired corrected speech recognition results to the terminal device. [Effects of the Invention]

[0007] By transmitting the unique vocabulary to a large-scale language model to obtain a corrected speech recognition result, the time and cost involved in improving the accuracy of speech recognition can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a DNN-HMM model and an End-to-End model. [Figure 2] FIG. 10 is a diagram illustrating the need to make the end-to-end model correspond to specific vocabulary. [Figure 3] FIG. 1 is a diagram for schematically explaining the re-learning of the End-to-End model. [Figure 4] FIG. 1 is a diagram illustrating how a large-scale language model modifies speech recognition results from an end-to-end model with a user's vocabulary. [Figure 5] This diagram compares and explains two methods for addressing specific vocabulary: prompt programming and fine tuning. [Figure 6] FIG. 1 is a diagram illustrating a system configuration of an example of a voice recognition system. [Figure 7] FIG. 2 is a diagram illustrating a hardware configuration of an example of an information processing system or a user terminal. [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing system and a user terminal. [Figure 9]10 is a conceptual diagram illustrating an example of a unique vocabulary management table stored in a unique vocabulary storage unit. FIG. [Figure 10] FIG. 2 is a diagram illustrating a large-scale language model in detail. [Figure 11] 1A and 1B are diagrams illustrating examples of recognition errors that tend to occur in speech recognition and examples of corrections using a large-scale language model. [Figure 12] FIG. 10 is a diagram illustrating a comparison between real-time recognition and batch recognition. [Figure 13] FIG. 10 is a sequence diagram illustrating a process in which the language processing device 40 corrects the speech recognition result (character data) obtained by the speech recognition system (batch processing). [Figure 14] 10 is an example of a flowchart illustrating a process in which a language processing device corrects a speech recognition result (character data) obtained by speech recognition performed by a speech recognition system. [Figure 15] 23 is a diagram showing an example of unique vocabulary registered by a user on the unique vocabulary input screen of FIG. 22. FIG. [Figure 16] FIG. 2 is a diagram showing an example of a voice recognition result (character data) generated by the information processing system through voice recognition. [Figure 17] FIG. 10 is a diagram illustrating an example of a “check_prompt” function. [Figure 18] FIG. 10 is a diagram illustrating an example of a “correction_prompt” function. [Figure 19] FIG. 10 is a diagram showing an example of a check_prompt sent to the language processing device. [Figure 20] FIG. 10 is a diagram illustrating an example of a correction_prompt prompt generated by the “correction_prompt” function. [Figure 21] FIG. 10 is a diagram showing an example of a response message returned by the language processing device in response to a correction_prompt. [Figure 22] FIG. 10 is a diagram illustrating an example of a unique vocabulary input screen displayed on a user terminal. [Figure 23] FIG. 10 is a diagram showing an example of a voice recognition screen on which a user terminal displays a voice recognition result (character data). [Figure 24] 10 is an example of a flowchart illustrating a flow of voice recognition and correction of a voice recognition result (character data). [Figure 25] 1 is an example of a sequence diagram illustrating a process in which a language processing device corrects a speech recognition result (character data) obtained by speech recognition performed by a speech recognition system (real-time recognition). [Figure 26] FIG. 10 is a diagram showing an example of a voice recognition screen on which a user terminal displays a voice recognition result (character data). DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A voice recognition system and an information processing method performed by the voice recognition system will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings.

[0010] <Speech recognition model> First, the speech recognition model will be described with reference to Fig. 1. Fig. 1(a) shows the flow of speech recognition in speech recognition model 206 that employs a DNN-HMM model, and Fig. 1(b) shows the recognition flow in speech recognition model 207 that employs an end-to-end model. Note that the processes indicated by the step numbers (S1, S2, S3, S4) shown below correspond to the step numbers in Fig. 1(a) and (b), respectively.

[0011] S1: First, audio is picked up by a microphone or the like, and the analog audio data is converted into digital audio data by PCM conversion.

[0012] S2: In acoustic analysis, audio data is cut into fixed short time intervals (audio data separated by short time intervals is called a frame), and a logarithmic amplitude spectrum is obtained by Fourier transform, and a Mel frequency spectrum is generated by discrete cosine transform of the Mel frequency spectrum to generate a cepstrum. The acoustically analyzed audio data is called an acoustic feature.

[0013] S3: In the conversion process, speech recognition is performed on the acoustic features, and the acoustic features are converted into speech recognition results (character data) in Japanese, English, etc.

[0014] First, the acoustic model 201 of the speech recognition model 206 is a DNN-HMM model. In the case of a DNN-HMM model, the DNN estimates the HMM state of a phoneme from acoustic features. First, the HMM models each individual phoneme. When the HMM is modeled using triphones, each phoneme is preceded and followed by a phoneme whose pronunciation is taken into account, so the HMM is defined by three states and the transition probability between states (including cases where no transition occurs). The HMM state indicates which phoneme has been estimated. Since speech is a continuous sequence of phonemes, the DNN estimates HMM states one after the other. The DNN-HMM model recognizes speech that passes through a path that maximizes the probability of transitioning between multiple HMMs.

[0015] The language model 202 is a model that expresses the likelihood of words being connected in terms of probability, and probabilistically evaluates the output of the DNN-HMM model based on the connections between words. The pronunciation dictionary 203 is a dictionary that lists how a word is pronounced for each written form, and converts the pronunciation into an appropriate word.

[0016] The GMM-HMM model has slightly lower recognition accuracy than the DNN-HMM model. In the GMM-HMM model, a GMM (Gaussian Mixture Model) corresponds to each of the three states of the HMM, and outputs the probability of outputting a voice (≒phoneme) for each acoustic feature.

[0017] The end-to-end model 204 does not require the acoustic model 201, language model 202, or pronunciation dictionary 203. The end-to-end model 204 outputs characters, words, etc. directly from acoustic features using a neural network. End-to-end models 204 include CTC (Connectionist Temporal Classification), RNN-Transducer, and attention-enabled encoder-decoder models. These or other end-to-end models 204 may be used in this embodiment. CTC assumes conditional independence of each output and predicts the next output without reference to past outputs. CTC uses LSTM to output phonemes or characters. RNN-Transducer is a model that eliminates the "conditional independence" of CTC and adds the character of the previous output to the input. Attention-enabled encoder-decoder models are currently considered to have the highest recognition accuracy. Attention-enabled encoder models input acoustic features into the LSTM of the encoder and output distributed representations (numerical vectors) instead of phonemes or words. The decoder input is the value obtained by multiplying the attention weight by the encoder output, and the decoder outputs phonemes or characters.

[0018] S4: Output processing provides words or characters to the user by, for example, displaying the speech recognition results (character data) on a display or outputting them from a speaker.

[0019] In this way, unlike the DNN-HMM model (or GMM-HMM) model, the End-to-End model 204 is not functionally divided into the acoustic model 201, language model 202, and pronunciation dictionary 203, but has a simple configuration in which speech is directly converted into text data using a single neural network, which has the advantage of not requiring advanced personal skills.

[0020] However, it is known that customization (support for specific vocabulary) for each user is difficult for the End-to-End model 204. That is, in the DNN-HMM model, customization for each user is relatively easy by using a pronunciation dictionary 203 specific to each user. However, in the End-to-End model 204, the acoustic model 201, the language model 202, and the pronunciation dictionary 203 are not functionally separated, making it difficult to use a pronunciation dictionary specific to each user.

[0021] Figure 2 is a diagram explaining the need to make the End-to-End model 204 correspond to the specific vocabulary. The speech uttered in Figure 2 is "A person who tested positive for infectious disease A." However, the speech recognition result is "A person who has 'requested' testing for infectious disease A," and the character that should be recognized as "positive" is erroneously recognized as "request."

[0022] In order to reduce such misrecognition, a method of training (also called customization) has been devised for the End-to-End model 204. Figure 3 is a diagram that schematically explains the training of the End-to-End model 204. In Figure 3, the developer trained the End-to-End model on a large number of news articles, web pages, etc., so that it can now correctly recognize "people who have tested positive for infectious disease A."

[0023] However, with the refresher training shown in Figure 3, it is difficult to deal with unique vocabulary that is often used by users but not generally used frequently. In refresher training of unique vocabulary, the user is asked to provide audio and text data containing the unique vocabulary, and the training unit uses this data to train the end-to-end model 204. However, refresher training must be carried out after annotation (associating audio with text, setting audio timestamps, etc.), which requires time (which can take several weeks to several months) and costs, and the learning work has been a significant burden.

[0024] [First embodiment] <Outline of the speech recognition model of this embodiment> In this embodiment, the information processing system 20 utilizes a large-scale language model, thereby easily achieving customization of the end-to-end model without requiring additional learning of the end-to-end model.

[0025] FIG. 4 is a diagram illustrating how a large-scale language model modifies the speech recognition results of an end-to-end model with the user's vocabulary. (1) A user 211 registers his / her own unique vocabulary 212 in the information processing system 20. Since the unique vocabulary is not learned by the speech recognition engine, the end-to-end model 204 is unlikely to correctly recognize the unique vocabulary. (2) The user 211 sends voice data to the information processing system 20. The information processing system 20 converts the voice data into a voice recognition result (text data) using the end-to-end model 204. The information processing system 20 sends the unique vocabulary and the voice recognition result (text data) to the large-scale language model 213. (3) The large-scale language model 213 matches the speech recognition results (character data) with the unique vocabulary. (4) The large-scale language model 213 corrects any misrecognized character strings found as a result of matching.

[0026] Figure 5 is a diagram comparing two methods for dealing with unique vocabulary (prompt programming and fine tuning). Prompt programming refers to constructing a natural language appropriately and instructing a large-scale language model to process it. In this embodiment, this prompt programming is used to correct the speech recognition results (character data). Prompt programming teaches desired transformations to a large-scale language model at low cost, and is expected to achieve a fairly high level of accuracy. Fine tuning is a type of incremental learning that is expected to achieve higher accuracy, but requires training costs.

[0027] In this way, in this embodiment, the large-scale language model 213 corrects the speech recognition result (character data) using the unique vocabulary of the user 211, eliminating the need for time-consuming and costly follow-up training. This reduces the drawback of the end-to-end model, which has high recognition accuracy but is not easy to customize. Furthermore, flexible customization for each user 211 becomes possible.

[0028] <Terminology> The unique vocabulary is a vocabulary that has not been learned by the large-scale language model. The vocabulary that has not been learned by the large-scale language model includes vocabulary that the large-scale language model cannot recognize or clearly original vocabulary. The user prepares the unique vocabulary by checking or predicting vocabulary that has not been learned by the large-scale language model.

[0029] Voice recognition refers to the process of converting voice data uttered by a person into character data (text data). In this embodiment, conversion into Japanese character data will be described, but the type of language is not important.

[0030] <System configuration example> The system configuration of the speech recognition system 100 according to this embodiment will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the system configuration of the speech recognition system 100. The speech recognition system 100 shown in FIG. 6 includes an information processing system 20, a user terminal 10, and a language processing device 40. The information processing system 20 and the user terminal 10 are communicably connected via networks N1 and N2. The information processing system 20 can communicate with the language processing device 40 via the network N1. The user terminal 10 can communicate with the language processing device 40 via the networks N1 and N2. The user terminal 10 does not need to be connected to the network N2 at all times; it is sufficient that the user terminal 10 is connected to the network N2 when using the information processing system 20 or the language processing device 40.

[0031] The user terminal 10 is located in a facility such as a company or a home, and is connected to a network N2. The network N2 may be a LAN, Wi-Fi (registered trademark), wide area Ethernet (registered trademark), a mobile phone network such as 4G, 5G, or 6G, etc. The network N1 is a wide area network such as the Internet or a WAN.

[0032] The language processing device 40 provides a service that allows users to converse with artificial intelligence (AI). Systems using large-scale language models (LLMs) are known as such language processing devices 40. That is, the language processing device 40 functions as a device having a large-scale language model. A large-scale language model is a natural language processing model trained using a large amount of text data. The language processing device 40 acquires a large amount of text data in advance and learns knowledge from it using deep learning, reinforcement learning, and the like. Using this knowledge, the language processing device 40 provides a response message to a chat message (which may be entered via a keyboard or generated by voice recognition). The interface through which a user inputs a chat message to the language processing device 40 is called a prompt. A prompt may include chat messages (natural language) such as instructions and questions, as well as sample data such as image data or files. The chat message may be a conversational sentence or phrase composed of natural language spoken by a human. A language processing device 40 that generates sentences from data based on chat messages is sometimes called a generative AI. In this embodiment, the language processing device 40 corrects the speech recognition result (character data) obtained by speech recognition of speech data by a user by the information processing system 20 using the specific vocabulary.

[0033] Systems using large-scale language models such as GPT-3, GPT-4, Transformer, BERT, PaLM, Llama, and Dolly (all of which are registered trademarks) are known as the language processing device 40. As an example, ChatGPT using GPT-3 or GPT-4 can be used in the speech recognition system 100 of this embodiment, but systems using other large-scale language models may also be used.

[0034] The information processing system 20 performs speech recognition on the speech data transmitted from the user terminal 10 and generates a speech recognition result (text data). An existing end-to-end model is assumed to be used for the speech recognition. The information processing system 20 also accepts registration of unique vocabulary from each user. The information processing system 20 transmits the speech recognition result (text data) and unique vocabulary to the language processing device 40 and receives corrected speech recognition results (text data). The information processing system 20 transmits the corrected speech recognition result (text data) to the user terminal 10.

[0035] The information processing system 20 has one or more information processing devices. From the perspective of the user terminal 10, the information processing system 20 also functions as a server that accepts requests. A server is a computer, software, or the like that provides information, processing results, etc. in response to requests from clients. The information processing system 20 provides a user interface to the user terminal 10 and receives voice data from the user terminal 10. The information processing system 20 generates a request message from the voice recognition result (character data) and the unique vocabulary, and transmits the generated request message to the language processing device 40. In response to the request message, the information processing system 20 receives a response message from the language processing device 40 that includes the corrected voice recognition result (character data). Because the information processing system 20 mediates communication between the user terminal 10 and the language processing device 40, the user does not have to go through the trouble of connecting the user terminal 10 to the information processing system 20 and then switching the connection to the language processing device 40.

[0036] The information processing system 20 may be realized by cloud computing or by a single information processing device. Cloud computing refers to a form in which resources on a network are used without being aware of specific hardware resources. The information processing system 20 may exist on the Internet or on-premise.

[0037] Furthermore, the functions of the information processing system 20 may be distributed among multiple information processing devices, or multiple information processing devices with the same functions may exist, and the number of information processing devices performing voice recognition may be increased or decreased depending on the processing load.

[0038] Furthermore, in FIG. 6, the information processing system 20 and the language processing device 40 are depicted as separate devices, but the information processing system 20 and the language processing device 40 may be integrated.

[0039] The user terminal 10 is, for example, a terminal device such as a PC (Personal Computer), smartphone, or tablet terminal used by a user or developer. A web browser or a native application runs on the user terminal 10. The user operates the user terminal 10 to input voice. The user operates the user terminal 10 to use various services provided by the information processing system 20 and the language processing device 40.

[0040] The user terminal 10 may be any information processing device. Examples of such information processing devices include, in addition to a PC, an output device such as an electronic whiteboard or digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, a medical device, a network home appliance, a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, and a wearable PC.

[0041] <Hardware configuration example> The hardware configuration of the information processing system 20 and the user terminal 10 included in the speech recognition system 100 will be described with reference to Fig. 7. The language processing device 40 has the same hardware configuration as that shown in Fig. 7 or is an information processing device compatible with cloud computing.

[0042] <<Information processing system or user terminal>> Fig. 7 is a diagram showing an example of the hardware configuration of the information processing system 20 or the user terminal 10 according to this embodiment. As shown in Fig. 7, the information processing system 20 or the user terminal 10 is constructed by a computer 500, and includes a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, a media I / F 516, and a microphone 517.

[0043] Of these, the CPU 501 controls the overall operation of the computer 500. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components, such as the CPU 501, shown in FIG. 7.

[0044] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. The DVD-RW drive 514 is not limited to a DVD-RW, and may be a DVD-R or the like. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory. The microphone 517 collects surrounding sounds and converts them into electrical signals.

[0045] <About the function> Next, the functional configuration of the speech recognition system 100 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the functional configuration of the information processing system 20 and the user terminal 10.

[0046] <<User device>> The user terminal 10 has a communication unit 11, a display control unit 12, and an operation reception unit 13. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 7 executing instructions contained in one or more programs installed in the user terminal 10. For example, the communication unit 11, the display control unit 12, and the operation reception unit 13 may be realized by at least one of a web browser and JavaScript (registered trademark, omitted below). When the user terminal 10 executes a native app, these may be realized by the native app.

[0047] The communication unit 11 transmits and receives various types of information to and from the information processing system 20. In this embodiment, the communication unit 11 receives screen information for voice input from the information processing system 20, and transmits voice data and unique vocabulary to the information processing system 20. The communication unit 11 also receives voice recognition results (character data) from the information processing system 20, which are the results of voice recognition of voice data. The communication unit 11 can also directly communicate with the language processing device 40.

[0048] The display control unit 12 interprets screen information of various screens and displays it on the display 506. The operation reception unit 13 receives various operations by the user on the various screens displayed on the display 506. The operation reception unit 13 also receives input of audio data collected by a microphone or the like.

[0049] <<Information Processing System>> Information processing system 20 includes a communication unit 21, a speech recognition result acquisition unit 22, an acquisition / generation unit 23, a correction unit 24, a speech recognition model 25, a unique vocabulary storage unit 28, a prompt function information storage unit 29, and a voice file storage unit 30. Each functional unit included in information processing system 20 is a function or means realized by CPU 501 shown in FIG. 7 executing instructions included in one or more programs installed in information processing system 20. Furthermore, unique vocabulary storage unit 28, prompt function information storage unit 29, and voice file storage unit 30 are constructed in HD 504, RAM 503, etc. shown in FIG. 7. Unique vocabulary storage unit 28, prompt function information storage unit 29, and voice file storage unit 30 do not necessarily have to be included in information processing system 20, but may be located on a network accessible by information processing system 20.

[0050] The communication unit 21 transmits and receives various information to and from the user terminal 10. In this embodiment, the communication unit 21 transmits screen information for voice input to the user terminal 10 and receives voice data and unique vocabulary from the user terminal 10. The unique vocabulary is preferably stored in the unique vocabulary storage unit 28 in association with a user name or user identification information. That is, the communication unit 21 receives unique vocabulary transmitted by the user terminal 10 and input by a specific user to the user terminal 10. The information processing system 20 then associates a specific user out of one or more users with the unique vocabulary input by the specific user to the user terminal 10 and registers the associated unique vocabulary in the unique vocabulary storage unit 28 (the "unique vocabulary management table" described below). The unique vocabulary thus associated with a specific user name or user identification information is an example of unique vocabulary associated with a user. The user identification information includes a user ID, an email address, etc. The communication unit 21 also transmits a voice recognition result (character data) resulting from voice recognition of the voice data to the user terminal 10.

[0051] It is assumed that an existing method is used to identify users when multiple users are participating in a conference. For example, the communication unit 21 associates the user identification information that a user enters to log in when participating in a conference with the IP address of the user terminal 10 that sent the user identification information. When the user terminal 10 sends voice data to the information processing system 20, the communication unit 21 can identify the user identification information based on the IP address of the sender of the voice data. The user terminal 10 may always send the voice data and user identification information as a set to the information processing system 20. Alternatively, each user may register their voiceprint information and user identification information in the information processing system 20 before the conference begins. During the conference, the information processing system 20 analyzes the voice data to acquire the voiceprint information and identify the user identification information.

[0052] The speech recognition result acquisition unit 22 uses the speech recognition model 25 to perform speech recognition on the speech files stored in the speech file storage unit 30 or the speech data transmitted from the user terminal 10, and acquires the speech recognition result (text data). Specifically, the speech recognition result acquisition unit 22 may implement the following method to acquire the speech recognition result for speech data uttered by one or more users. For example, the speech recognition result acquisition unit 22 uses commonly known speaker diarization to estimate "who spoke and when" from the received speech file or speech data, and associates the user who spoke (the speaker) with the speech content. As a result, the speech recognition result acquisition unit 22 assigns information (speaker label) such as "Mr. A" or "Mr. B" to the speech recognition result. Note that speaker diarization may be performed in parallel (simultaneously) with the speech recognition process. The speech recognition result acquisition unit 22 may further recognize the speaker down to the individual level using commonly known speaker recognition (authentication) and assign a user name (e.g., Mr. A = Mr. Sato, Mr. B = Mr. Suzuki) associated with the speaker label assigned by speaker diarization. The speech recognition model 25 is assumed to be an end-to-end model, but may also be a DNN-HMM model or a GMM-HMM model. The language processing device 40 can modify the speech recognition result (text data) regardless of the speech recognition model. The speech recognition model 25 may not be included in the information processing system 20. In this case, the speech recognition result acquisition unit 22 transmits speech data to an external speech recognition model 25 via a network and receives the speech recognition result (text data) from the speech recognition model 25. The received speech recognition result (text data) may be in any language corresponding to the speech spoken by the user, and may be Japanese, English, Chinese, or any other language without limitations on country, region, etc. Furthermore, when a translation function is used in addition to this embodiment, the voice recognition result (character data) may be received in a language of a country, region, etc. different from the voice spoken by the user.

[0053] The acquisition / generation unit 23 generates a prompt using a prompt function stored in advance in the prompt function information storage unit 29. A prompt is an interface with the language processing device 40, and in this embodiment, it is an instruction sentence given to the language processing device 40 via the interface. The language processing device 40 returns a response message in response to the prompt, so in order for the information processing system 20 to acquire an appropriate response message from the language processing device 40, it is preferable that a prompt corresponding to the processing by the language processing device 40 is prepared in advance. The prompt function is a template (format) for generating this prompt. Details of the prompt function will be described with reference to FIGS. 17 and 18.

[0054] The correction unit 24 sends a request message to the language processing device 40 by calling an API 41 published by the language processing device 40. The request message includes a prompt. As a result of calling the API 41, the correction unit 24 receives a response message for the request message from the language processing device 40. The response message includes a response message to the prompt, etc. In this embodiment, the response message is the corrected speech recognition result (text data). The correction unit 24 corrects the speech recognition result (text data) in this manner. The correction unit 24 also sends the speech recognition result (text data) recognized by the speech recognition model 25 and unique vocabulary associated with one or more users to the large-scale language model, and acquires from the large-scale language model the corrected speech recognition result (text data) obtained by the large-scale language model correcting the speech recognition result (text data) based on the unique vocabulary. The correction unit 24 further corrects the speech recognition result based on the unique vocabulary when the large-scale language model determines that correction is necessary, and does not correct the speech recognition result based on the unique vocabulary when the large-scale language model does not determine that correction is necessary. The API 41 is one or more Web-APIs that the language processing device 40 makes public.

[0055] As described above, the language processing device 40 may be included in whole or in part in the information processing system 20 .

[0056] ● Unique Vocabulary Management Table 2001 FIG. 9 is a conceptual diagram illustrating an example of a unique vocabulary management table stored in the unique vocabulary storage unit 28. As shown in FIG. 9, in the unique vocabulary management table 2001, information for identifying a user, including a user name and an email address, and unique vocabulary are associated and managed for each unique vocabulary ID. Among these, the unique vocabulary ID is a management number for managing each unique vocabulary. User information for identifying a user, such as a user name (user ID) and an email address, may be information that is associated with each user and managed in advance by the information processing system 20. The unique vocabulary ID and the unique vocabulary corresponding to the unique vocabulary ID are, for example, text information registered in the unique vocabulary management table 2001 when a user inputs (registers) the unique vocabulary into the user terminal 10. Note that each piece of information managed in the unique vocabulary management table 2001 may be registered by the information processing system 20 acquiring the information via a network from an external server or the like that manages the information, instead of or in addition to the user inputting (registering) the information into the user terminal 10. In this way, the information processing system 20 has the above-mentioned specific vocabulary management table 2001 that registers a specific user among one or more users in association with the specific vocabulary input by the specific user to the user terminal 10. In this embodiment, the specific vocabulary management table 2001 functions as an example of the specific vocabulary storage unit 28.

[0057] <Additional information on large-scale language models> FIG. 10 is a diagram explaining the large-scale language model 213 in detail. The large-scale language model 213 is a type of deep learning model used in the field of natural language processing, and its main purpose is to learn from a huge amount of character data and achieve natural language generation and understanding like that of a human. The large-scale language model is a complex model that uses deep learning technology, and its foundation is a computational model called a neural network that mimics the function of neurons in the human brain. This neural network receives input data in a multi-layer structure, learns patterns, and performs predictions, classifications, etc. In particular, the large-scale language model learns language structures by inputting huge amounts of character data and adjusting a large number of parameters.

[0058] PaLM, ChatGPT, Llama, and other models can automatically generate text data with fluency and logic nearly equivalent to that of human writing. These models are large neural networks consisting of billions of parameters and trained on a vast amount of text data. In this embodiment, the information processing system 20 uses such a large-scale language model 213 to correct recognition errors that occur during speech recognition.

[0059] FIG. 11 is a diagram illustrating an example of a recognition error that is likely to occur in speech recognition and an example of correction using a large-scale language model.

[0060] Example 1 The correct utterance is "Let me first judge whether there are any problems." The voice recognition result was "Let me check if there are any problems first." A large-scale language model can correct the speech recognition result to "We will first judge whether there are any problems."

[0061] Example 2 The correct utterance is "Is it portrait rights and copyright?" The voice recognition result was "Was it copyright?" Large-scale language models can correct speech recognition results to "What about portrait rights and copyright?"

[0062] Example 3 The correct utterance is, "In the old days, we used to do things like introducing ourselves as an icebreaker." The voice recognition result was, "I didn't introduce myself during the ice preview." A large-scale language model can correct the speech recognition result to "In the old days, there was something called an icebreaker called introductions."

[0063] In examples 1 to 3, the characters that are intended to be recognized (judge, portrait rights, old times, icebreaker) are recognized as different characters (incorrect characters). In example 3, the characters that are intended to be recognized are not output. This is called a recognition failure (the "a" in "mukashi nanka saa" is not output, and the "a" in "toka iu no ataru" is not output). Also, in example 3, unnecessary characters that are not in the utterance are included ("yanaka" in "toka yanakana").

[0064] In this embodiment, for example, a user can register "judge, portrait rights, icebreaker" and the like as unique vocabulary in the information processing system 20, which makes it easier for the information processing system 20 to correctly convert voice data. Furthermore, even if the information processing system 20 is not directly related to the unique vocabulary, it can use a large-scale language model to insert correct characters if there are any recognition errors, and delete unnecessary characters if they are included.

[0065] <Real-time recognition and batch recognition> Speech recognition is broadly divided into real-time recognition and batch recognition, and the speech recognition method of this embodiment can be suitably applied to batch recognition and real-time recognition.

[0066] FIG. 12 is a diagram illustrating the comparison between real-time recognition and batch recognition. Examples of batch recognition include transcribing a meeting or creating minutes. In the case of batch recognition, the user terminal 10 and the information processing system 20 communicate using asynchronous HTTP. Asynchronous HTTP is a communication method in which the user terminal 10 and the information processing system 20 communicate without synchronization. The user terminal 10 can accept operations from the user even while the information processing system 20 is performing voice recognition, etc.

[0067] In the case of batch recognition, the user terminal 10 uploads a long voice file 271 to the information processing system 20. When the information processing system 20 finishes recognizing the entire voice file 271, it changes the status to "completed." After the status is changed to "completed," the user terminal 10 obtains all voice recognition results 272.

[0068] Examples of real-time recognition include voice input, voice operation, subtitle display, and interaction with a virtual human. In this case, the user terminal 10 and the information processing system 20 communicate via synchronous HTTP or Websocket. Synchronous HTTP is a communication method in which the user terminal 10 and the information processing system 20 communicate in synchronization. While the information processing system 20 is performing voice recognition, the user terminal 10 is in a standby state and cannot accept operations from the user.

[0069] In the case of communication using synchronous HTTP, the user terminal 10 uploads short voice data 273 of about several tens of milliseconds to the information processing system 20. Each time the information processing system 20 recognizes this short voice data 273, it transmits a voice recognition result 274 to the user terminal 10. The user terminal 10 obtains the voice recognition results 274 for the amount uploaded.

[0070] Websocket is a communication standard that enables two-way communication by establishing a session between the user terminal 10 and the information processing system 20. In communication using Websocket, the user terminal 10 transmits voice data 275 and 277 to the information processing system 20 one after another without waiting for a response from the information processing system 20. The information processing system 20 transmits to the user terminal 10 one after another a voice recognition result 276 in which recognition has been completed for the voice data 275 and a voice recognition result 278 in which recognition has been completed for the voice data 277.

[0071] Real-time recognition enables voice input, voice operation, etc. Furthermore, during a meeting, the user terminal 10 can display subtitles of participants' speech in real time. Users can also converse with AI in real time. One form of real-time conversation is a voice conversation with a virtual human. A virtual human is an artificial being with a human-like appearance created using computer graphics, artificial intelligence technology, etc. This virtual human itself also converses using a large-scale language model.

[0072] <Processing flow> Next, the flow of the speech recognition and correction process by the speech recognition system 100 will be described with reference to Figs. 13 to 21. Fig. 13 is a sequence diagram illustrating the process in which the language processing device 40 corrects the speech recognition result (character data) obtained by speech recognition by the speech recognition system 100. Speech recognition can be performed in real time or all at once. Fig. 13 explains the case of all at once recognition. Details of each step, including the process in which the language processing device 40 corrects the speech recognition result (character data), will be explained in Fig. 14, and the overall flow will be explained in Fig. 13.

[0073] S11: The user inputs vocabulary into the user terminal 10. The vocabulary may be input in real time via keyboard or voice, or may be prepared in advance as a file. There are two main cases in which the user inputs (registers) vocabulary in S11. The first is when the user confirms the speech recognition results shown in Figure 4 and determines that input (registration) of vocabulary is necessary. Specifically, suppose a user utters "This road is one-way" and the speech recognition result obtained by a large-scale language model (LLM) is "This road is one-way." In this case, the user determines that this is a misrecognition and inputs (registers) "One-way" as a new vocabulary. The other is when the user determines that the words or phrases contained in the utterance contain uniqueness or rarity. For example, if the user wants to utter "Today's blue sky is blue, indigo, and blue-and-blue," the user would input (register) "Blue, indigo, and blue-and-blue," a combination of "blue, indigo" and "blue-and-blue" that is rarely used and considered unusual, as a vocabulary. Note that other specific examples of words that may be input (registered) as unique words may be, for example, the words shown in FIG. 15, which will be described later. An example of a unique vocabulary input screen 290 is shown in FIG. 22. The communication unit 21 of the information processing system 20 receives unique vocabulary and user identification information such as a user ID or an email address from the user terminal 10. The communication unit 21 of the information processing system 20 may register the unique vocabulary in the unique vocabulary storage unit 28 in association with the user identification information such as the user ID or email address. The user identification information is input into the user terminal 10 when the user logs in to the information processing system 20, or has been input into the user terminal 10 in advance.

[0074] S12: The communication unit 11 of the user terminal 10 transmits the unique vocabulary to the information processing system 20, and the communication unit 21 of the information processing system 20 receives the unique vocabulary. The communication unit 21 may store the unique vocabulary in the unique vocabulary storage unit 28 in association with the user identification information.

[0075] S13: In the case of batch recognition, the user operates the user terminal 10 to specify an audio file collected during a conference, etc. The audio file contains a large amount of audio data recorded during the conference.

[0076] S14: The operation reception unit 13 of the user terminal 10 receives the designation of an audio file. The communication unit 11 of the user terminal 10 transmits the audio file to the information processing system 20, and the communication unit 21 of the information processing system 20 receives the audio file. The communication unit 21 stores the audio file in the audio file storage unit 30. The audio file may be stored on a network.

[0077] When the time comes for the information processing system 20 to perform batch processing, it starts voice recognition of the voice file.

[0078] S15: The voice recognition result acquisition unit 22 of the information processing system 20 performs voice recognition on the voice data included in the voice file using the voice recognition model 25 to acquire a voice recognition result (character data).

[0079] S16: Next, the acquisition and generation unit 23 of the information processing system 20 generates a check_prompt to be sent to the language processing device 40. The check_prompt is a prompt for inquiring of the language processing device 40 as to whether or not the speech recognition result (character data) needs to be corrected. The acquisition and generation unit 23 may include all of the speech recognition results (character data) in one check_prompt, or may generate check_prompts including divided speech recognition results (character data) and request the language processing device 40 to correct the results in multiple batches.

[0080] S17: The correction unit 24 of the information processing system 20 sends check_prompt to the language processing device 40. At this time, check_prompt includes the unique vocabulary that was received from the user terminal 10 in S12 and then stored in the unique vocabulary storage unit 28 in association with the user identification information.

[0081] S18: The correction unit 24 of the information processing system 20 receives the information indicating whether or not the voice recognition result (character data) needs to be corrected (yes, no).

[0082] S19: If the voice recognition result (character data) needs to be corrected, the acquisition / generation unit 23 generates a correction_prompt for requesting the language processing device 40 to correct the voice recognition result (character data).

[0083] S20: The correction unit 24 sends the correction_prompt to the language processing device 40.

[0084] S21: The correction unit 24 of the information processing system 20 receives the speech recognition result (character data) corrected by the language processing device 40.

[0085] S22: The communication unit 21 of the information processing system 20 transmits the voice recognition result (character data) to the user terminal 10. The transmission may be performed automatically or upon a request from the user. For example, the communication unit 21 may transmit the voice recognition result (character data) to the user's email address. Alternatively, the information processing system 20 may save the voice recognition result (character data) in a specific folder, URL, etc., and the user may download it from the folder, URL, etc. The information processing system 20 may transmit this URL to the user's email address. Note that the voice recognition result (character data) may or may not be corrected.

[0086] S23: The communication unit 11 of the user terminal 10 receives the voice recognition result (character data), and the display control unit 12 displays the voice recognition result (character data) on the display 506 of the user terminal 10. An example of the display of the voice recognition result (character data) is shown in FIG.

[0087] Next, Fig. 14 is a flowchart illustrating the process in which the language processing device 40 corrects the speech recognition result (character data) obtained by the speech recognition system 100. The process in Fig. 14 is started when the information processing system 20 performs batch processing. The timing when the information processing system 20 performs batch processing may be a fixed time, a user operation, or when a certain amount of speech files has accumulated. Each step in Fig. 14 will be explained below.

[0088] S110 The acquisition / generation unit 23 of the information processing system 20 acquires unique vocabulary from the unique vocabulary storage unit 28. In the case of conference audio, the audio file contains audio data of one or more users. At the batch processing stage, the users who attended the conference may not be known. The unique vocabulary that the acquisition / generation unit 23 acquires from the unique vocabulary storage unit 28 will be described in step S140.

[0089] Fig. 15 shows an example of unique vocabulary registered by a user on the unique vocabulary input screen 290 of Fig. 22. In Fig. 15, three unique vocabulary words 291, namely, "Brainstorm," "My Number," and "Good Boy Blog," are registered. These unique vocabulary words are registered in the unique vocabulary storage unit 28 in association with user identification information such as a user name (user ID) and an email address. Any number of unique vocabulary words 291 may be registered.

[0090] S120 The voice recognition result acquisition unit 22 of the information processing system 20 acquires a voice file from the voice file storage unit 30. The voice recognition result acquisition unit 22 acquires a voice recognition result (character data) by voice-recognizing all voice data included in the voice file using the voice recognition model 25. The voice recognition result acquisition unit 22 may recognize each block of voice data at a time, rather than recognizing the entire voice data all at once. A block of voice data may be voice data separated by a silent section of a certain length or more, or may be voice data separated by a certain time.

[0091] FIG. 16 shows the speech recognition result (character data) generated by the information processing system 20 through speech recognition. As shown in FIG. 16, the information processing system 20 recognized "Well, how about proposing a service using product X and audio / images for the Grest system?" The part recognized as "Grest system" is incorrect, and the part recognized as "Brainstorm system" is correct. In this case, the user does not need to check whether there is a recognition error.

[0092] S130 Next, the acquisition and generation unit 23 of the information processing system 20 prepares two types of prompts to send to the language processing device 40. The large-scale language model accepts instructions from the user in natural language. In this embodiment, the correction unit 24 uses prompt programming to request the large-scale language model to correct the speech recognition result (character data).

[0093] One of the prepared prompts is a prompt that the language processing device 40 uses to determine whether correction is necessary in step S140. The other is a prompt that the language processing device 40 uses to correct the speech recognition result (character data) in step S170. Templates for each prompt are prepared in the form of functions. The function for generating a prompt that determines whether correction is necessary is called the "check_prompt" function. The function for generating a prompt that corrects the speech recognition result (character data) is called the "correction_prompt" function.

[0094] Figure 17 shows the "check_prompt" function. The "check_prompt" function has two arguments: dictionary 221 and recognition_results 222. Dictionary 221 is the user's unique vocabulary, and recognition_results 222 is the speech recognition results (character data). When generating a prompt, the user's unique vocabulary and speech recognition results (character data) are obtained using these arguments, and the prompt is generated. That is, the contents of dictionary 221 are set under a display called Dictionary 224, and the contents of recognition_results 222 are set under a display called Speech Recognition Results 225.

[0095] The "check_prompt" function has a message 223 that reads, "Please refer to the dictionary below and answer whether or not there are any parts of the Japanese that are incorrect in the speech recognition results." The "check_prompt" function also has a message 226 that reads, "Are there any parts of the Japanese that are incorrect (answer with either "yes" or "no")?" These messages 223 and 226 are sent to the language processing device 40 as part of the prompt. The language processing device 40 generates a response message (yes or no) based on the prompt generated by the "check_prompt" function and returns it to the information processing system 20.

[0096] Figure 18 shows the "correction_prompt" function. The "correction_prompt" function has two arguments: dictionary 231 and recognition_results 232. Dictionary 231 is the user's unique vocabulary, and recognition_results 232 is the speech recognition results (character data). When generating a prompt, these arguments are used to obtain the user's unique vocabulary and speech recognition results (character data), and the prompt is generated. In other words, the contents of dictionary are set under a display called dictionary 234, and the contents of recognition_results are set under a display called speech recognition results 235.

[0097] The "correction_prompt" function has a message 233 that reads, "Please refer to the dictionary below and correct any incorrect Japanese in the speech recognition result. However, please keep the corrections to a minimum." The "correction_prompt" function also has a message 236 that reads, "Correction." These messages 233 and 236 are sent to the language processing device 40 as part of the prompt. The language processing device 40 generates a response message (corrected speech recognition result (character data)) based on the prompt generated by the "correction_prompt" function, sets it under the message 236, and returns it to the information processing system 20.

[0098] S140 The acquisition and generation unit 23 of the information processing system 20 inputs the unique vocabulary of step S110 and the speech recognition result (character data) of step S120 into the "check_prompt" function prepared in step S130, and generates check_prompt for the language processing device 40 to determine whether correction is necessary. The correction unit 24 of the information processing system 20 transmits the generated check_prompt to the language processing device 40.

[0099] Fig. 19 is an example of check_prompt sent to the language processing device 40. As shown in Fig. 17, check_prompt has a message 241 saying "Please refer to the dictionary below and answer whether or not there are any parts of the Japanese that are incorrect in the speech recognition results." Also, check_prompt has dictionary contents 242 and recognition_results contents 243. Also, check_prompt has a message 244 saying "Are there any parts of the Japanese that are incorrect (answer with either "Yes" or "No")?"

[0100] In the case of batch recognition, the following types of speech recognition results are possible: (a) The user who made the utterance is not identified in any way. (b) The user (participant) who spoke is identified. For example, the participants of the meeting are attached to an audio file, or the information processing system 20 can acquire the participants of the meeting from a conference room reservation system. For example, a conference is assumed in which multiple people actually gather in a conference room. (c) The user who spoke is identified for each piece of text data. For example, when a user logs in from each user terminal 10 during a conference, the user identification information of the user who sent each piece of voice data is stored in the voice file for each piece of voice data. For example, an online conference is assumed.

[0101] In the case of (a), the contents 242 of the dictionary are all the unique vocabulary words stored in the unique vocabulary storage unit 28. In the case of (b), the contents 242 of the dictionary are the unique vocabulary words stored in the unique vocabulary storage unit 28 that are associated with the participants of the conference (user identification information). In the case of (c), check_prompt may be generated to include the speech recognition result (character data) for each speaker. The acquisition and generation unit 23 generates check_prompt, for example, every time the speaker changes. In this case, the contents 242 of the dictionary are the unique vocabulary associated with the user who is the speaker, among the unique vocabulary stored in the unique vocabulary storage unit 28. However, since generating check_prompt every time the speaker changes imposes a high processing load, the acquisition and generation unit 23 may generate check_prompt in the same manner as in (b).

[0102] The specific vocabulary set in correction_prompt can be considered in the same way as (a) to (c).

[0103] S150 The correction unit 24 of the information processing system 20 transmits the prompt generated in step S140 to the language processing device 40, thereby obtaining a response message of "yes" or "no" from the language processing device 40. If the response message is "yes," the information processing system 20 can determine that correction is necessary. On the other hand, if the response message is "no," the information processing system 20 can determine that correction is not necessary.

[0104] S160 The correction unit 24 of the information processing system 20 determines whether the response message in step S150 is "present" or "absent." If "present," the process proceeds to step S170, and if "absent," the process proceeds to step S200.

[0105] S170 If the response message is "yes," the speech recognition result (character data) needs to be corrected, so the acquisition / generation unit 23 of the information processing system 20 performs a process to make the correction. The acquisition / generation unit 23 of the information processing system 20 inputs the unique vocabulary from step S110 and the speech recognition result (character data) from step S120 into the "correction_prompt" function prepared in step S130, and generates a correction_prompt for correcting the speech recognition result (character data). For example, the following prompt is generated.

[0106] Figure 20 is an example of a correction_prompt generated by the "correction_prompt" function. As shown in Figure 18, correction_prompt has a message 251 that reads, "Please refer to the dictionary below and correct any incorrect Japanese in the speech recognition results. However, please keep corrections to a minimum." Also, correction_prompt has dictionary content 252 and recognition_results content 253. Also, correction_prompt has a message 254 that reads "correct."

[0107] S180 The correction unit 24 of the information processing system 20 transmits the correction_prompt (correction request) generated in step S170 to the language processing device 40, thereby obtaining a response message from the language processing device 40. The response message to the correction_prompt may include the corrected speech recognition result (character data). Note that if the language processing device 40 is a generation AI, it may not always correctly correct the speech recognition result (character data).

[0108] Fig. 21 shows an example of a response message returned by the language processing device 40 in response to the correction_prompt. The response message in Fig. 21 is "Well, how about proposing a service using product X and speech recognition for brainstorming?". It can be confirmed that the correction has correctly corrected the part that the speech recognition engine had mistakenly recognized as "brainstorming."

[0109] S190 In response to a request from the user terminal 10, the communication unit 21 of the information processing system 20 transmits the corrected speech recognition result (character data) obtained in step S180 to the user terminal 10. The user terminal 10 receives the corrected speech recognition result (character data) and outputs (displays) the received speech recognition result (character data) on the display 506. Therefore, the user can obtain the speech recognition result (character data) corrected with their own unique vocabulary without being aware of the correction made by the large-scale language model. Note that the communication unit 21 may transmit the result by email, push notification, etc. regardless of a request from the user terminal 10.

[0110] S200 In step S160, if the correction unit of the information processing system 20 receives a response message of "No" from the language processing device 40 (a case in which correction of the speech recognition result (character data) is not necessary), it means that correction of the speech recognition result (character data) acquired in step S120 was not necessary. In this case, the correction unit 24 does not perform correction processing. In response to a request from the user terminal 10, the communication unit 21 of the information processing system 20 transmits the speech recognition result (character data) acquired in step S120 to the user terminal 10. The user terminal 10 receives the corrected speech recognition result (character data) and outputs (displays) the received speech recognition result (character data) on the display 506. Note that, regardless of a request from the user terminal 10, the communication unit 21 of the information processing system 20 may transmit the speech recognition result (character data) to the user terminal 10 by email, push notification, or the like.

[0111] In the flowchart of FIG. 14, if the response message from the language processing device 40 is "yes" in step S160, the correction unit 24 generates a correction_prompt for correcting the speech recognition result (character data). However, the correction unit 24 may send a correction_prompt to the language processing device 40 without sending a check_prompt to the language processing device 40. If the correction unit 24 sends a correction_prompt to the language processing device 40 without sending a check_prompt, a correct speech recognition result (character data) may also be corrected. Therefore, the correction unit 24 sends a check_prompt to the language processing device 40, and if the correction is "yes," sends a correction_prompt to the language processing device 40, thereby improving the correction accuracy (recognition accuracy). There are many possible methods for correcting vocabulary using prompt programming, but in this embodiment, high accuracy can be expected by breaking down the prompts for the correction process into small parts and proceeding with the correction step by step.

[0112] <Screen example> Fig. 22 shows an example of a unique vocabulary input screen 290 displayed by the user terminal 10. In Fig. 22, three unique vocabulary words 291, "Brainstorm," "My Number," and "Oriko Blog," have been entered on the unique vocabulary input screen 290. The user may enter the unique vocabulary words using a keyboard or by voice input. The user may also enter the kanji characters by associating them with their readings.

[0113] 23 shows an example of a voice recognition screen 260 on which the user terminal 10 displays the voice recognition result (character data). The voice recognition screen 260 has a recognition result display section 261 and a recognition status display section 262. The recognition status display section 262 indicates, by an icon 263, whether or not the voice is currently being recognized. The recognition result display section 261 displays the voice recognition result (character data) transmitted from the information processing system 20. The recognition result display section 261 displays various applications used for character input.

[0114] The recognition status display unit 262 and the recognition result display unit 261 may be provided as separate applications. The user can input speech to any of the recognition result display units 261 by starting the application for the recognition status display unit 262 and then starting the application for the recognition result display unit 261. If the recognition status display unit 262 and the recognition result display unit 261 are the same application, the user can input speech to one application by starting that application. The speech recognition screen 260 may have the following display variations. Specifically, the speech recognition result corrected by a large-scale language model may be displayed in a different display format (bold, underlined, blinking, etc.) from the other uncorrected portions. Furthermore, the speech recognition result before correction and the speech recognition result corrected by a large-scale language model (or only the corrected portions) may be displayed on the same screen.

[0115] <For real-time recognition> Next, with reference to FIGS. 24 and 25, the processing flow for real-time recognition will be described. FIG. 24 is a flowchart illustrating the flow of speech recognition and correction of speech recognition results (text data). A possible scenario in which real-time recognition is used is when a conversation (utterance) turn (a continuous unit in a conversation from the start of speech by one speaker to the end) is relatively long. Furthermore, when the intervals between speakers are relatively long, for example, when a game of shogi or go is being commented on by multiple commentators (speakers), and the commentary is displayed with subtitles for broadcasting or distribution. A possible processing method for these cases is to first display the speech recognition result (text data) containing erroneous text, and then replace it with the correct speech recognition result (text data) once the LLM is generated. In other words, real-time recognition may be applied when the content of a speaker's utterance can be sufficiently corrected even if it is erroneously converted.

[0116] In the case of real-time recognition, the communication unit 21 of the information processing system 20 receives voice data from the user terminal 10 (S201).

[0117] The voice recognition result acquisition unit 22 acquires a voice recognition result (character data) by performing voice recognition on the voice data transmitted from the user terminal 10 using the voice recognition model 25 (S202).

[0118] The communication unit 21 transmits the speech recognition result (character data) to the user terminal 10. The user terminal 10 displays the speech recognition result (character data) before correction on the display 506 of the user terminal 10 (S203). Note that an end-to-end model may be used as the speech recognition model 25.

[0119] Next, the acquisition / generation unit 23 acquires the speech recognition result (character data) (S204). Then, the correction unit 24 corrects the speech recognition result (character data) using the unique vocabulary (S205). The process of this step S205 may be the same as that in FIG. 14.

[0120] The communication unit 21 transmits the corrected speech recognition result (character data) to the user terminal 10. As a result, the user terminal 10 outputs (displays) the corrected speech recognition result (character data) on the display 506 (S206).

[0121] Fig. 25 is a sequence diagram illustrating the process in which the language processing device 40 corrects the speech recognition result (character data) obtained by the speech recognition system 100. Fig. 25 illustrates the case of real-time recognition. The explanation of Fig. 25 will mainly focus on the differences from Fig. 13.

[0122] First, the process in steps S11 and S12 in which the user terminal 10 transmits the unique vocabulary to the information processing system 20 may be the same as that in Fig. 13. However, the user terminal 10 does not need to transmit the unique vocabulary immediately before real-time recognition, but may transmit it before speech recognition.

[0123] S31: The user speaks to the user terminal 10.

[0124] S32: The operation reception unit 13 of the user terminal 10 acquires the voice data. The communication unit 11 of the user terminal 10 transmits the voice data to the information processing system 20, and the communication unit 21 of the information processing system 20 receives the voice data.

[0125] S15: The voice recognition result acquisition unit 22 of the information processing system 20 performs voice recognition on the voice data using the voice recognition model 25 to acquire a voice recognition result (character data).

[0126] S33: In the case of real-time recognition, the communication unit 21 of the information processing system 20 transmits the speech recognition result (character data) before correction and the recognition ID to the user terminal 10. That is, even if there is a recognition error in the speech recognition result (character data), the information processing system 20 transmits the speech recognition result (character data) to the information processing system 20 in order to maintain real-time performance. Then, if there is a recognition error, the information processing system 20 transmits the speech recognition result (character data) corrected in subsequent processing to the user terminal 10. The recognition ID is identification information for the speech recognition process, and is used to identify the speech recognition result (character data) to be corrected by the user terminal 10.

[0127] S34: The communication unit 11 of the user terminal 10 receives the speech recognition result (character data), and the display control unit 12 displays the speech recognition result (character data) before correction on the display 506 of the user terminal 10. An example of the display of the speech recognition result (character data) is shown in Fig. 26. In this way, the user terminal 10 can display the speech recognition result in real time.

[0128] After step S15, the information processing system 20 executes steps S16 to S21, similar to Fig. 13. The detailed processing of steps S16 to S21 may be similar to that of Fig. 14. In real-time processing, if the user who made the utterance is identified by logging in or the like, the acquisition and generation unit 23 sets the unique vocabulary of that user as the prompt. In real-time processing, if the user who made the utterance is not identified, the acquisition and generation unit 23 sets all the unique vocabulary stored in the unique vocabulary storage unit 28 or the unique vocabulary of all participants in the conference as the prompt, similar to (a) and (b) above.

[0129] S35: If the language processing device 40 determines that correction is "necessary," the communication unit 21 transmits the speech recognition result (character data) corrected by the language processing device 40 and the recognition ID to the user terminal 10.

[0130] S36: The communication unit 11 of the user terminal 10 receives the corrected speech recognition result (character data) and the recognition ID. If the received recognition ID matches the recognition ID of the uncorrected speech recognition result (character data) that is already displayed, the display control unit 12 replaces (corrects) the uncorrected speech recognition result (character data) with the received speech recognition result (character data). This allows correction of any recognition errors that may be included in the displayed speech recognition result (character data).

[0131] It takes several seconds for the information processing system 20 to execute the processes of steps S16 to S21 between the information processing system 20 and the language processing device 40. However, according to the process of FIG. 25, the speech recognition result (character data) can be displayed in real time, and if an error is included, the incorrectly recognized speech recognition result (character data) can be corrected with a minimum time delay.

[0132] Figure 26 shows an example of a voice recognition screen 260 on which the user terminal 10 displays the voice recognition result (character data). The explanation of Figure 26 will mainly focus on the differences from Figure 23. The voice recognition screen 260 in Figure 26(a) includes the incorrectly recognized character 281, "Grest system."

[0133] Thereafter, the conversation progresses as shown in FIG. 26(b), and character data 283 saying "That's good" is displayed. Also, in FIG. 26(b), the user terminal 10 receives the corrected speech recognition result, and the characters 281 that were erroneously recognized as "Grest-related" are corrected to characters 282 saying "Breest-related." The display control unit 12 may highlight the corrected characters 282 in red, for example. Also, when the user hovers the mouse over or clicks on the characters 282, the display control unit 12 may display the erroneously recognized characters 281.

[0134] Furthermore, if the language processing device 40 is unable to completely correct the character, the user may specify the incorrectly recognized character 281 on the speech recognition screen 260 of FIG. 26(a). The user terminal 10 may transmit the incorrectly recognized character 281 or a speech recognition result (character data) including the character 281 to the information processing system 20, and the information processing system 20 may transmit a correction_prompt to the language processing device 40, which may result in the character being correctly corrected. The user may also input a character into which the incorrectly recognized character 281 has been correctly converted. The user terminal 10 transmits the correctly converted character to the information processing system 20. The information processing system 20 stores this character as a unique vocabulary in the unique vocabulary storage unit 28, so that the character will be correctly converted from the next time onwards. Note that FIG. 26 may also have variations of the display as shown in FIG. 23.

[0135] <Major Effects> In the speech recognition system 100 of this embodiment, the information processing system 20 transmits the user's unique vocabulary and the speech recognition result to a large-scale language model to correct the speech recognition result, eliminating the need for time-consuming and costly retraining. Even if an utterance contains the user's unique vocabulary, it is easy to convert it into a correct utterance. This reduces the drawback of end-to-end models, which have high recognition accuracy but are difficult to customize. In addition, flexible customization for each user 211 is possible.

[0136] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0137] For example, in this embodiment, speech recognition is described as being performed using an end-to-end model, a DNN-HMM model, or a GMM-HMM model, but the speech recognition results corrected by the large-scale language model may be those recognized using any speech recognition model.

[0138] Furthermore, in this embodiment, the information processing system 20 includes the speech recognition result and the unique vocabulary in the prompt and requests the language processing device 40 to make corrections. However, it is also possible to have the large-scale language model learn the unique vocabulary in advance, and send only the speech recognition result to the large-scale language model to request corrections.

[0139] In addition, the configuration examples in Figure 8 and the like are divided according to main functions to make it easier to understand the processing by the user terminal 10, the information processing system 20, and the language processing device 40. The method of dividing the processing units, the names, etc. do not limit the present invention. The processing by the user terminal 10, the information processing system 20, and the language processing device 40 can be divided into even more processing units depending on the processing content. Also, it is possible to divide one processing unit so that it includes even more processes.

[0140] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a conventional circuit module, and other devices designed to perform each of the above-described functions.

[0141] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, information processing system 20 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0142] Furthermore, the information processing system 20 may be integrated into one server device, or may be divided into multiple devices.

[0143] <Aspect> [Aspect 1] An information processing system capable of communicating with a terminal device via a network, a speech recognition result acquisition unit that acquires speech recognition results for speech data uttered by one or more users; a correction unit that transmits the speech recognition result and the unique vocabulary associated with the one or more users to a large-scale language model that is a model for natural language processing, and receives a corrected speech recognition result from the large-scale language model that has corrected the speech recognition result; a communication unit that transmits the corrected speech recognition result to the terminal device; An information processing system having the above. [Aspect 2] 2. The information processing system of claim 1, wherein the correction unit uses prompt programming that notifies the large-scale language model of processing content in natural language to request correction of the speech recognition result acquired by the speech recognition result acquisition unit. [Aspect 3] the speech recognition result acquisition unit acquires speech recognition results for all speech data included in the speech file, 2. The information processing system according to aspect 1, wherein the correction unit requests the large-scale language model to correct all of the speech recognition results obtained by speech recognition of all of the speech data in one or more separate rounds. [Aspect 4] the voice recognition result acquisition unit acquires the voice recognition result obtained by performing voice recognition in real time on the voice data transmitted from the terminal device; the communication unit transmits the acquired speech recognition result to the terminal device; the correction unit receives, after the communication unit transmits the speech recognition result to the terminal device, a corrected speech recognition result obtained by the large-scale language model correcting the speech recognition result using the unique vocabulary associated with the one or more users, from the large-scale language model; 4. The information processing system according to any one of aspects 1 to 3, wherein the communication unit transmits the corrected speech recognition result acquired by the correction unit to the terminal device. [Aspect 5] the correction unit queries the large-scale language model about whether or not the speech recognition result needs to be corrected, along with the unique vocabulary associated with the user and the speech recognition result; if a response is received from the large-scale language model that correction is required; The information processing system according to any one of aspects 1 to 4, wherein the correction unit receives from the large-scale language model a corrected speech recognition result in which the large-scale language model corrects the speech recognition result using unique vocabulary associated with the one or more users. [Aspect 6] 6. The information processing system according to claim 5, wherein the correction unit corrects the speech recognition result based on the unique vocabulary when the large-scale language model determines that correction is necessary, and does not correct the speech recognition result based on the unique vocabulary when the large-scale language model does not determine that correction is necessary. [Aspect 7] The information processing system according to any one of aspects 1 to 6 further includes a unique vocabulary storage unit that registers a specific user among the one or more users in association with the unique vocabulary input by the specific user to the terminal device. [Aspect 8] 8. The information processing system according to any one of aspects 1 to 7, wherein the speech recognition result acquisition unit acquires the speech recognition result obtained by speech recognition using an end-to-end model. [Explanation of symbols]

[0144] 10 User Devices 20 Information Processing Systems 40 Language Processing Device [Prior art documents] [Patent documents]

[0145] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-40904

Claims

1. An information processing system capable of communicating with a terminal device via a network, a speech recognition result acquisition unit that acquires a speech recognition result of speech data uttered by one or more users; a correction unit that transmits the speech recognition result and unique vocabulary associated with the one or more users to a large-scale language model that is a model for natural language processing, and obtains from the large-scale language model a corrected speech recognition result in which the large-scale language model corrects the speech recognition result based on the unique vocabulary; a communication unit that transmits the corrected speech recognition result to the terminal device; An information processing system having the above.

2. 2. The information processing system according to claim 1, wherein the correction unit requests correction of the speech recognition result acquired by the speech recognition result acquisition unit using prompt programming that notifies the large-scale language model of processing content in natural language.

3. the speech recognition result acquisition unit acquires speech recognition results for all speech data included in the speech file, The information processing system according to claim 1 , wherein the correction unit requests the large-scale language model to correct all of the speech recognition results obtained by speech recognition of all of the speech data in one or more separate requests.

4. the voice recognition result acquisition unit acquires the voice recognition result obtained by performing voice recognition in real time on the voice data transmitted from the terminal device; the communication unit transmits the acquired speech recognition result to the terminal device; the correction unit receives, after the communication unit transmits the speech recognition result to the terminal device, a corrected speech recognition result from the large-scale language model in which the large-scale language model corrects the speech recognition result using the unique vocabulary associated with the one or more users; The information processing system according to claim 1 , wherein the communication unit transmits the corrected speech recognition result acquired by the correction unit to the terminal device.

5. the correction unit queries the large-scale language model about whether or not the speech recognition result needs to be corrected, along with the unique vocabulary associated with the user and the speech recognition result; If a response is received from the large-scale language model that correction is required, 2. The information processing system according to claim 1, wherein the correction unit receives, from the large-scale language model, a corrected speech recognition result in which the large-scale language model corrects the speech recognition result using unique vocabulary associated with the one or more users.

6. 6. The information processing system according to claim 5, wherein the correction unit corrects the speech recognition result based on the unique vocabulary when the large-scale language model determines that correction is necessary, and does not correct the speech recognition result based on the unique vocabulary when the large-scale language model does not determine that correction is necessary.

7. 2. The information processing system according to claim 1, further comprising a unique vocabulary storage unit that registers a specific user among the one or more users in association with the unique vocabulary input by the specific user to the terminal device.

8. The information processing system according to claim 1 , wherein the speech recognition result acquisition unit acquires the speech recognition result obtained by speech recognition using an end-to-end model.

9. A speech recognition system comprising a terminal device, an information processing system, and a language processing device having a large-scale language model that is a model of natural language processing, the terminal device and the information processing system being communicably connected via a network, The information processing system includes: a speech recognition result acquisition unit that acquires a speech recognition result of speech data uttered by one or more users; a correction unit that transmits the speech recognition result and a unique vocabulary associated with the one or more users to the language processing device, and obtains from the language processing device a corrected speech recognition result obtained by the large-scale language model of the language processing device correcting the speech recognition result based on the unique vocabulary; a communication unit that transmits the corrected speech recognition result to the terminal device, The terminal device a display control unit that displays the corrected speech recognition result received from the information processing system on a display unit;

10. The speech recognition system according to claim 9 , wherein the large-scale language model modifies one or more characters included in the speech recognition result acquired by the speech recognition result acquisition unit with a character string included in the unique vocabulary.

11. 10. The speech recognition system according to claim 9, wherein the large-scale language model corrects a recognition error that is not included in the speech recognition result acquired by the speech recognition result acquisition unit, or an insertion of an unnecessary character that is included in the speech recognition result.

12. An information processing method performed by an information processing system capable of communicating with a terminal device via a network, comprising: obtaining speech recognition results for speech data spoken by one or more users; a process of transmitting the speech recognition result and a unique vocabulary associated with the one or more users to a large-scale language model that is a model for natural language processing, and acquiring a modified speech recognition result from the large-scale language model, the modified speech recognition result being obtained by the large-scale language model modifying the speech recognition result based on the unique vocabulary; and transmitting the corrected speech recognition result to the terminal device.

13. An information processing system capable of communicating with a terminal device via a network, obtaining speech recognition results for speech data spoken by one or more users; a process of transmitting the speech recognition result and a unique vocabulary associated with the one or more users to a large-scale language model that is a model for natural language processing, and acquiring a modified speech recognition result from the large-scale language model, the modified speech recognition result being obtained by the large-scale language model modifying the speech recognition result based on the unique vocabulary; a process of transmitting the corrected speech recognition result to the terminal device; A program to execute.

Images