Information processing system, speech recognition system, information processing method, and recording medium

Abstract

An information processing system is communicable with a terminal apparatus through a network. The information processing system includes: a speech recognition result acquisition unit to acquire a speech recognition result of audio data of speech spoken by one or more users; a correction unit to transmit the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing and to acquire, from the large language model, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and a communication unit to transmit the acquired corrected speech recognition result to the terminal apparatus.

Description

[DESCRIPTION][Title of Invention]INFORMATION PROCESSING SYSTEM, SPEECH RECOGNITION SYSTEM, INFORMATION PROCESSING METHOD, AND RECORDING MEDIUM [Technical Field]

[0001] The present disclosure relates to an information processing system, a speech recognition system, an information processing method, and a recording medium.[Background Art]

[0002] The end-to-end model is known as a highly accurate and robust speech recognition model. The end-to-end model does not divide functions into an acoustic model, a language model, and a pronunciation dictionary like the known Deep Neural Network-Hidden Markov Model (DNN-HMM) model or Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) model and directly converts audio data into character data using a single network. The end- to-end model has the advantage of not requiring advanced personal skills.Speech is not always recognized and converted exactly as spoken. For this reason, PTL 1 discloses a technology of correcting a speech recognition result. Specifically, PTL 1 discloses correcting a result obtained by recognizing data of speech spoken by a user on the basis of a category dictionary corresponding to a category into which the content of the speech is classified.The technology according to the background art takes time and cost to enhance the accuracy of speech recognition.[Citation List][Patent Literature]

[0003] [PTL 1]Japanese Unexamined Patent Application Publication No. 2018-40904[Summary of Invention][Technical Problem]

[0004] An object of the present disclosure is to reduce the time and cost to enhance the accuracy of speech recognition.[Solution to Problem]

[0005] The present disclosure described herein provides an information processing system communicable with a terminal apparatus through a network. The information processing system includes: a speech recognition result acquisition unit to acquire a speech recognition result of audio data of speech spoken by one or more users; a correction unit to transmit thespeech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing and to acquire, from the large language model, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and a communication unit to transmit the acquired corrected speech recognition result to the terminal apparatus.The present disclosure described herein provides a speech recognition system including a terminal apparatus, an information processing system, and a language processing apparatus having a large language model that is a model for natural language processing. The terminal apparatus, the information processing system, and the language processing apparatus are communicably connected to each other through a network. The information processing system includes: a speech recognition result acquisition unit to acquire a speech recognition result of audio data of speech spoken by one or more users; a correction unit to transmit the speech recognition result and unique vocabulary associated with the one or more users to the language processing apparatus and to acquire, from the language processing apparatus, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model of the language processing apparatus based on the unique vocabulary; and a communication unit to transmit the acquired corrected speech recognition result to the terminal apparatus. The terminal apparatus includes a display control unit to display the corrected speech recognition result received from the information processing system on a display.The present disclosure described herein provides an information processing method performed by an information processing system communicable with a terminal apparatus through a network. The information processing method includes: acquiring a speech recognition result of audio data of speech spoken by one or more users; transmitting the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing; acquiring, from the large language model, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and transmitting the acquired corrected speech recognition result to the terminal apparatus. The present disclosure described herein provides a recording medium storing a plurality of program codes which, when executed by one or more processors of an information processing system communicable with a terminal apparatus through a network, causes the one or more processors to perform a method. The method includes: acquiring a speech recognition result of audio data of speech spoken by one or more users; transmitting the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing; acquiring, from the large language model, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and transmitting the acquired correctedspeech recognition result to the terminal apparatus.[Advantageous Effects of Invention]

[0006] According to an embodiment of the present disclosure, unique vocabulary is sent to a large language model to obtain a corrected speech recognition result. This reduces the time and cost to enhance the accuracy of speech recognition.[Brief Description of Drawings]

[0007] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings.[FIG. 1A]FIG. 1A is a diagram for explaining a Deep Neural Network-Hidden Markov Model (DNN- HMM model).[FIG. IB]FIG. IB is a diagram for explaining an end-to-end model.[FIG. 2]FIG. 2 is a diagram for explaining the necessity to adopt an end-to-end model to a unique vocabulary.[FIG. 3]FIG. 3 is a diagram schematically illustrating incremental learning of the end-to-end model ofFIG. 2.[FIG. 4]FIG. 4 is a diagram illustrating how a large language model corrects a result of speech recognition by an end-to-end model with a user’s vocabulary.[FIG. 5]FIG. 5 is a diagram illustrating comparison between two methods for dealing with the unique vocabulary: prompt programming and fine tuning.[FIG. 6]FIG. 6 is a diagram illustrating the configuration of a speech recognition system.[FIG. 7]FIG. 7 is a block diagram illustrating the hardware configuration of an information processing system or a user terminal.[FIG. 8]FIG. 8 is a diagram illustrating the functional configuration of the information processing system and the user terminal of FIG. 7.[FIG. 9]FIG. 9 is an illustration of a unique vocabulary management table stored in a unique vocabulary storage unit.[FIG. 10]FIG. 10 is a diagram illustrating details of a large language model.[FIG. 11]FIG. 11 is a diagram illustrating examples of recognition errors that are likely to occur in speech recognition and an examples of corrections by the large language model.[FIG. 12]FIG. 12 is a diagram illustrating real-time recognition and batch recognition in comparison.[FIG. 13]FIG. 13 is a sequence diagram of a process in which the language processing apparatus corrects a speech recognition result (character data) obtained by speech recognition performed by the speech recognition system in the case of batch processing.[FIG. 14]FIG. 14 is a flowchart of a process in which the language processing apparatus corrects a speech recognition result (character data) obtained by speech recognition performed by the speech recognition system.[FIG. 15]FIG. 15 is a diagram illustrating an example of the unique vocabulary registered by a user on a unique vocabulary input screen of FIG. 22.[FIG. 16]FIG. 16 is a diagram illustrating a speech recognition result (character data) generated by the information processing system performing speech recognition.[FIG. 17]FIG. 17 is a diagram illustrating an example of a “ check _prompt” function.[FIG. 18]FIG. 18 is a diagram illustrating an example of a “correct! on prompt” function.[FIG. 19]FIG. 19 is a diagram illustrating an example of a check_prompt transmitted to the language processing apparatus.[FIG. 20]FIG. 20 is a diagram illustrating an example of a correction_prompt generated by a“ correct! on_prompt” function.[FIG. 21]FIG. 21 is a diagram illustrating an example of a reply message transmitted by the language processing apparatus in response to a correction_prompt.[FIG. 22]FIG. 22 is a diagram illustrating an example of a unique vocabulary input screen displayed by the user terminal.[FIG. 23]FIG. 23 is a diagram illustrating an example of a speech recognition screen on which the user terminal displays a speech recognition result (character data). [FIG. 24]FIG. 24 is a flowchart of the flow of speech recognition and correction of the speech recognition result (character data).[FIG. 25]FIG. 25 is a sequence diagram of a process in which the language processing apparatus corrects a speech recognition result (character data) obtained by speech recognition performed by the speech recognition system in the case of real-time recognition.[FIG. 26A]FIG. 26A is a diagram illustrating an example of a speech recognition screen on which the user terminal displays a speech recognition result (character data).[FIG. 26B]FIG. 26B is a diagram illustrating an example of a speech recognition screen on which the user terminal displays a speech recognition result (character data).The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views. [Description of Embodiments]

[0008] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0009] A speech recognition system and an information processing method performed by the speech recognition system are described below as an example of an embodiment of the present disclosure with reference to the drawings.

[0010] Speech Recognition ModelFirst, a speech recognition model is described below with reference to FIG. 1. FIG. 1A illustrates a flow of speech recognition in a speech recognition model 206 that employs the Deep Neural Network- Hidden Markov Model (DNN-HMM model). FIG. IB illustrates a flow of recognition in a speech recognition model 207 that employs the end-to-end model.Processes indicated by the step numbers SI, S2, S3 and S4 described below are processes corresponding to the step numbers in FIG. 1A and FIG. IB.

[0011] First, in step SI, voice is picked up by, for example, a microphone, and analog audio data is converted into digital audio data by pulse code modulation (PCM) conversion.

[0012] In step S2, acoustic analysis is performed. In the acoustic analysis, the audio data is segmented in fixed short time intervals and subjected to Fourier transform to obtain a logarithmic magnitude spectrum. The audio data obtained by the segmentation in short time intervals is called a “frame.” A Mel frequency spectrum is then generated, and the Mel frequency spectrum is subjected to discrete cosine transform to generate a cepstrum. The audio data on which the acoustic analysis has been performed is referred to as “acoustic features.”

[0013] In step S3, conversion processing is performed. In the conversion processing, speech recognition is performed on the acoustic features, and thus the acoustic features are converted into a speech recognition result (character data) in, for example, Japanese or English.

[0014] An acoustic model 201 of the speech recognition model 206 is a DNN-HMM model. In the DNN-HMM model, the DNN estimates the HMM state of a phoneme from acoustic features. The HMM models each phoneme individually. When the HMM is modeled using triphones, a phoneme is preceded and followed by phonemes whose pronunciations are considered. Accordingly, the HMM is defined by three states and the transition probabilities between the states (including a case where no transition occurs). The HMM state indicates which phoneme has been estimated. Since speech is a continuous sequence of phonemes, the DNN estimates the HMM state one after the other. The DNN-HMM model recognizes speech that passes through a path where the probability of transition between multiple HMMs is maximized.

[0015] A language model 202 is a model that represents the degree of connection between words in terms of probability. The language model 202 probabilistically evaluates the output of the DNN-HMM model based on the connections between words. A pronunciation dictionary 203 is a dictionary that records how a certain word is pronounced for its written form. The pronunciation dictionary 203 converts pronunciations into appropriate words.

[0016] A Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) model is a model having a slightly lower recognition accuracy than the DNN-HMM model.In the GMM-HMM model, a Gaussian mixture model (GMM) corresponds to each of the three states of the HMM, and outputs the probability of outputting speech (approximately phoneme) for acoustic features.

[0017] The end-to-end model 204 does not need the acoustic model 201, the language model 202, and the pronunciation dictionary 203. The end-to-end model 204 outputs characters, words, etc. directory from acoustic features using a neural network. There are various types of end- to-end model 204, such as a Connectionist Temporal Classification (CTC), a Recurrent Neural Network (RNN)-Transducer, or an encoder-decoder model with an attention mechanism. In the present embodiment, these types or other types of the end-to-end model 204 may be used. The CTC assumes conditional independence of each output, and predicts the next output without reference to past outputs. The CTC uses Long Short Term Memory (LSTM) to output phonemes or characters. The RNN-Transducer is a model that eliminates the “conditional independence” in the CTC and adds the character of the previous output to the input. The encoder-decoder model with an attention mechanism is currently considered to have the highest recognition accuracy. The encoder model with an attention mechanism inputs acoustic features to the LSTM of the encoder and outputs distributed representations (numerical vectors) instead of a phoneme or a word. A value obtained by multiplying the attention weight by the output of the encoder is used as an input to the decoder, and the decoder outputs a phoneme or a character.

[0018] In step S4, output processing is performed. In the output processing, for example, the speech recognition result (character data) is displayed on a display or outputted from a speaker to provide words or characters to a user.

[0019] As described above, the end-to-end model 204 is not functionally divided into the acoustic model 201, the language model 202, and the pronunciation dictionary 203, as in the DNN- HMM model (or GMM-HMM) model. In other words, the end-to-end model 204 has a simple configuration that directly converts speech into character data using a single neural network. Accordingly, the end-to-end model 204 has an advantage of not requiring advanced person skills.

[0020] However, it is known that it is difficult to customize the end-to-end model 204 according to a user. The customization is, for example, the adaptation of unique vocabulary. In other words, since the DNN-HMM model uses the pronunciation dictionary 203 that is unique to a user, customization according to a user is performed in a relatively simple manner. However, since the end-to-end model 204 does not separate functions of the acoustic model 201, the language model 202, and the pronunciation dictionary 203, and has a difficulty in using a pronunciation dictionary unique to a user.

[0021] FIG. 2 is a diagram for explaining the necessity to adopt the end-to-end model 204 to a unique vocabulary. In the example illustrated in FIG. 2, the speech uttered is “a person who tested positive for infectious disease A”. However, the result of speech recognition is "a person requesting testing for infectious disease A.” In this case, the result of speech recognition includes an error “requesting” (in Japanese characters “Hail”, pronounced as “Yousei”), which should be correctly recognized as “positive” (in Japanese character “1^'14”, pronounced as “Yousei”).

[0022] In order to reduce such erroneous recognition, a method of performing incremental learning has been devised for the end-to-end model 204. Such incremental learning may be referred to as “customization.” FIG. 3 is a diagram schematically illustrating the incremental learning of the end-to-end model 204. In the example illustrated in FIG. 3, the developer has trained the end-to-end model using, for example, a large number of news articles and web pages. As a result, the end-to-end model correctly recognizes “a person who tested positive for infectious disease A.”

[0023] However, in the incremental learning as described with reference to FIG. 3, it is difficult to deal with unique vocabulary that is frequently used by a particular user but is not generally used frequently. In the incremental learning of specific vocabulary, a user provides, for example, voice and character data including the unique vocabulary, and the learning unit causes the end-to-end model 204 to learn the unique vocabulary using the provided data. However, the incremental learning, which may take time, sometimes, several weeks to several months, and is costly, has to be performed after annotation such as association between voice and characters and setting of a time stamp of voice. This makes the learning process a significant burden.

[0024] First EmbodimentOverview of Speech Recognition Model According to Present EmbodimentIn the present embodiment, an information processing system 20 uses a large language model to implement the customization of the end-to-end model in a simple manner without performing the incremental learning on the end-to-end model.

[0025] FIG. 4 is a diagram illustrating how a large language model corrects a result of speech recognition by the end-to-end model with a user’s vocabulary.(1) A user 211 registers a unique vocabulary 212 that is unique to the user himself / herself in the information processing system 20. Since the speech recognition engine does not learn the unique vocabulary, the end-to-end model 204 is unlikely to correctly recognize the unique vocabulary.(2) The user 211 transmits audio data to the information processing system 20. The information processing system 20 converts the audio data into a speech recognition result (character data) using the end-to-end model 204. The information processing system 20 transmits the unique vocabulary and the speech recognition result (character data) to a large language model 213.(3) The large language model 213 matches the speech recognition result (character data) with the unique vocabulary.(4) The large language model 213 corrects an erroneously recognized character string found as the result of the matching.

[0026] FIG. 5 is a diagram illustrating comparison between two methods for dealing with the unique vocabulary. The process of constructing natural language appropriately and instructing a large language model to perform processing is called prompt programming. In the present embodiment, the speech recognition result (character data) is corrected by using the prompt programming. Prompt programming allows for the training of desired transformation into a large language model at low cost with reasonably high accuracy. Fine-tuning is incremental learning, which can be expected to achieve higher accuracy, but it incurs a training cost.

[0027] As described above, in the present embodiment, the large language model 213 corrects the speech recognition result (character data) using the unique vocabulary of the user 211, eliminating the need for time-consuming and costly incremental learning. This reduces the drawback of the end-to-end model that has high recognition accuracy but is not easy to customize. This further implements flexible customization for each user 211.

[0028] TermsThe term “unique vocabulary” refers to vocabulary that the large language model has not learned. Examples of the vocabulary that the large language model has not learned include vocabulary that the large language model cannot recognize and vocabulary that is apparently original. The user checks or predicts vocabulary that the large language model has not learned, and prepares unique vocabulary.

[0029] The term “speech recognition” refers to a process of converting data of speech spoken by a human into character data (text data). In the present embodiment, conversion into character data in English is described. However, the speech data can be converted into character data in any language.

[0030] Example of System ConfigurationA system configuration of the speech recognition systemlOO according to the present embodiment is described below with reference to FIG. 6. FIG. 6 is a diagram illustrating anexample of the configuration of the speech recognition system 100. The speech recognition system 100 illustrated in FIG. 6 includes an information processing system 20, a user terminal 10, and a language processing apparatus 40. The information processing system 20 and the user terminal 10 are communicably connected with each other through a network Nl and a network N2. The information processing system 20 is communicable with the language processing apparatus 40 through the network Nl. Further, the user terminal 10 is communicable with the language processing apparatus 40 through the network N 1 and the network N2. The user terminal 10 does not have to be always connected to the network N2. It suffices that the user terminal 10 connects to the network N2 when the user terminal 10 uses the information processing system 20 or the language processing apparatus 40.

[0031] The user terminal 10 is provided in a facility such as a company and or home, and is connected to the network N2. Examples of the network N2 include a local area network (LAN), Wireless Fidelity (Wi-Fi®), wide-area Ethernet®, and a fourth generation (4G), fifth generation (5G), or sixth generation (6G) mobile network. The network Nl is a wide area network (WAN) such as the Internet.

[0032] The language processing apparatus 40 provides a service for a user to converse with an artificial intelligence (Al) in a conversational manner. As a system such as the language processing apparatus 40, a system using a large language model (LLM) has been proposed. In other words, the language processing apparatus 40 functions as an apparatus having a large language model. The large language model is a model for natural language processing that have been trained using a large amount of character data. The language processing apparatus 40 captures a huge amount of character data in advance and learns knowledge from the character data using, for example, deep learning or reinforcement learning. The language processing apparatus 40 uses the learned knowledge to provide a reply message to a chat message. The chat message may be input from a keyboard or may be text obtained by speech recognition. An interface with which the user inputs a chat message to the language processing apparatus 40 is called a prompt. The prompt may be input in a form of a chat message (natural language) such as an instruction or a question, or in a form of image data or sample data such as a file. The chat message may be a conversational sentence or words of a natural language spoken by humans. The language processing apparatus 40 that generates a sentence for data based on a chat message may be referred to as a “generation artificial intelligence (Al).” In the present embodiment, the language processing apparatus 40 corrects a speech recognition result (character data) obtained by the information processing system 20 performing speech recognition on data of speech by a user, using unique vocabulary.

[0033] As the language processing apparatus 40, a system using a large language model such as GPT-3®, GPT-4®, Transformer®, BERT®, PaLM®, Llama®, or Dolly® has been proposed.The speech recognition systemlOO according to the present embodiment may use, for example, a ChatGPT® using GPT-3® or GPT-4®. Alternatively, the speech recognition system 100 may use a system using any other large language model.

[0034] The information processing system 20 performs speech recognition on audio data transmitted from the user terminal 10, and generates a speech recognition result (character data). It is assumed that an existing end-to-end model is used for the speech recognition. Further, the information processing system 20 receives registration of unique vocabulary that is unique to each of users from the corresponding user. The information processing system 20 transmits the speech recognition result (character data) and the unique vocabulary to the language processing apparatus 40, and receives the corrected speech recognition result (character data). The information processing system 20 transmits the corrected speech recognition result (character data) to the user terminal 10.

[0035] The information processing system 20 includes one or more information processing apparatuses. The information processing system 20 serves as a server that receives a request from the user terminal 10. A server is a computer or software that functions to provide information and a processing result in response to a request from a client. The information processing system 20 provides the user terminal 10 with a user interface to receive audio data from the user terminal 10. The information processing system 20 generates a request message on the basis of the speech recognition result (character data) and the unique vocabulary, and transmits the generated request message to the language processing apparatus 40. The information processing system 20 receives a response message including the corrected speech recognition result (character data) from the language processing apparatus 40 as a response to the request message. Since the information processing system 20 mediates communication between the user terminal 10 and the language processing apparatus 40, the user does not have to switch the connection to the language processing apparatus 40 after connecting the user terminal 10 to the information processing system 20.

[0036] The information processing system 20 may be implemented by cloud computing. Alternatively, the information processing system 20 may be implemented by a single information processing apparatus. The term “cloud computing” refers to Internet-based computing where resources on a network are used or accessed without identifying specific hardware resources. The information processing system 20 may reside on the Internet. Alternatively, the information processing system 20 may reside in an on-premises environment.

[0037] The functions of the information processing system 20 may be allocated among multiple information processing apparatuses. Alternatively, the information processing system 20 mayinclude multiple information processing apparatuses having the same functions, and the number of information processing systems that perform speech recognition may be increased or decreased depending on the processing load.

[0038] Although an example in which the information processing system 20 and the language processing apparatus 40 are separate apparatuses is described with reference to FIG. 6, the information processing system 20 and the language processing apparatus 40 may be configured as a single entity.

[0039] The user terminal 10 is, for example, a terminal apparatus such as a personal computer (PC), a smartphone, or a tablet terminal used by the user or the developer. In the user terminal 10, a web browser or a native application operates. The user operates the user terminal 10 to input speech. The user operates the user terminal 10 to use various services provided by the information processing system 20 and the language processing apparatus 40.

[0040] The user terminal 10 may be any information processing apparatus. Examples of the information processing apparatus include, in addition to a PC, an electronic whiteboard, an output device such as digital signage, a head-up display (HUD), an industrial machine, an imaging device, a sound collecting device, a medical device, a networked home appliance, a mobile phone, a smartphone, a tablet communication terminal, a game console, a personal digital assistant (PDA), a digital camera, and a wearable PC.

[0041] Example of Hardware ConfigurationThe hardware configurations of the information processing system 20 and the user terminal 10 of the speech recognition system 100 are described below with reference to FIG. 7. The language processing apparatus 40 has the same or substantially the same hardware configuration as illustrated in FIG. 7 or a hardware configuration of an information processing apparatus that supports cloud computing.

[0042] Information Processing System or User TerminalFIG. 7 is a block diagram illustrating a hardware configuration of the information processing system 20 or the user terminal 10 according to the present embodiment. As illustrated in FIG. 7, the information processing system 20 and the user terminal 10 each are implemented by a computer 500. The computer 500 includes a central processing unit (CPU) 501, a read-only memory (ROM) 502, a random-access memory (RAM) 503, a hard disk (HD) 504, a hard disk drive (HDD) controller 505, a display 506, an external device connection interface (PF) 508, a network PF 509, a bus line 510, a keyboard 511, a pointing device 512, a digital versatile disc rewritable (DVD-RW) drive 514, a medium PF 516, and a microphone 517.

[0043] The CPU 501 controls the overall operation of the computer 500. The ROM 502 stores programs such as an initial program loader (IPL) to boot the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as a program. The HDD controller 505 controls reading or writing of various data from or to the HD 504 under control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or image. The external device connection PF 508 is an interface for connecting the computer 500 to various external devices. Examples of the external devices include a universal serial bus (USB) memory and a printer.The network PF 509 is an interface for performing data communication using the network N2. The bus line 510 is, for example, an address bus or a data bus, which electrically connects the components illustrated in FIG. 7, such as the CPU 501.

[0044] The keyboard 511 is an example of an input device including multiple keys to be used for inputting characters, numerical values, or various instructions. The pointing device 512 is an example of an input device that allows a user to select or execute various instructions, select a target for processing, or move a cursor being displayed. The DVD-RW drive 514 controls reading or writing of various data from or to a DVD-RW 513, which is an example of a removable recording medium. The DVD-RW is merely one example of the removable storage medium. In another example, the DVD-RW drive 514 controls reading or writing of data with respect to a digital versatile disc recordable (DVD-R). The medium PF 516 controls reading and writing (storing) of data from and to a recording medium 515 such as a flash memory. The microphone 517 collects surrounding sounds and converts the sounds into electrical signals.

[0045] FunctionsThe functional configuration of the speech recognition system 100 is described below with reference to FIG. 8. FIG. 8 is a diagram illustrating an example of the functional configuration of the information processing system 20 and the user terminal 10.

[0046] User TerminalThe user terminal 10 includes a communication unit 11, a display control unit 12, and an operation reception unit 13.These functional units are functions or means that are implemented by the CPU 501 illustrated in FIG. 7 executing instructions included in one or more programs installed on the user terminal 10. For example, the communication unit 11, the display control unit 12, and the operation reception unit 13 may be implemented by at least one of web browser and JavaScript®. Alternatively, the communication unit 11, the display control unit 12, and the operation reception unit 13 may be implemented by a native application when the user terminal 10 executes the native application.

[0047] The communication unit 11 transmits and receives various kinds of information to and from the information processing system 20. In the present embodiment, the communication unit 11 receives screen information for voice input from the information processing system 20 and transmits audio data and unique vocabulary to the information processing system 20. The communication unit 11 receives a speech recognition result (character data), which is a result of speech recognition performed on the audio data, from the information processing system 20. The communication unit 11 is also communicable directly with the language processing apparatus 40.

[0048] The display control unit 12 interprets screen information of various screens to display the screens on the display 506.The operation reception unit 13 receives various operations by the user on the screens displayed on the display 506. The operation reception unit 13 further receives input of audio data collected by, for example, a microphone.

[0049] Information Processing SystemThe information processing system 20 includes a communication unit 21, a speech recognition result acquisition unit 22, an acquisition and generation unit 23, a correction unit 24, a speech recognition model 25, a unique vocabulary storage unit 28, a prompt function information storage unit 29, and an audio file storage unit 30. The functional units of the information processing system 20 are functions or means that are implemented by the CPU 501 illustrated in FIG. 7 executing instructions included in one or more programs installed on the information processing system 20. The unique vocabulary storage unit 28, the prompt function information storage unit 29, and the audio file storage unit 30 are implemented in the HD 504 or the RAM 503 illustrated in FIG. 7. The unique vocabulary storage unit 28, the prompt function information storage unit 29, and the audio file storage unit 30 may not be included in the information processing system 20 but may reside on a network accessible by the information processing system 20.

[0050] The communication unit 21 transmits and receives various kinds of information to and from the user terminal 10. In the present embodiment, the communication unit 21 transmits the screen information for voice input to the user terminal 10 and receives the audio data and the unique vocabulary from the user terminal 10. The unique vocabulary is preferably stored in the unique vocabulary storage unit 28 in association with a user name or user identification information. In other words, the communication unit 21 receives the unique vocabulary transmitted by the user terminal 10 and input to the user terminal 10 by a particular user. Then, the information processing system 20 registers a particular user among one or more users in association with unique vocabulary input to the user terminal 10 by the particular userin the unique vocabulary storage unit 28 (i.e., “unique vocabulary management table” described later). The unique vocabulary associated with the user name or the user identification information of the particular user as described above is an example of unique vocabulary associated with a user. The user identification information includes a user identifier (ID) and an email address. The communication unit 21 transmits the speech recognition result (character data), which is a result of speech recognition performed on the audio data, to the user terminal 10.

[0051] In a case where multiple users participate in a meeting, existing methods are used to identify the users. For example, the communication unit 21 associates user identification information that a user inputs for logging in when participating in a meeting with an internet protocol (IP) address of the user terminal 10 that has transmitted the user identification information. When the user terminal 10 transmits audio data to the information processing system 20, the communication unit 21 can identify the user identification information based on the IP address of the sender of the audio data. The user terminal 10 may transmit the audio data in association with the user identification information to the information processing system 20 each time the user terminal 10 transmits the audio data. Alternatively, a method can be used in which each of the users registers his / her voiceprint information and the user identification information in the information processing system 20 before the start of the meeting. During the meeting, the information processing system 20 analyzes the audio data to acquire voiceprint information and identifies user identification information based on the acquired voiceprint information.

[0052] The speech recognition result acquisition unit 22 performs speech recognition on an audio file stored in the audio file storage unit 30 or audio data transmitted from the user terminal 10 using the speech recognition model 25, and thus acquires a speech recognition result (character data). Specifically, the speech recognition result acquisition unit 22 may implement the following method in order to acquire the speech recognition result for audio data spoken by one or more users. For example, the speech recognition result acquisition unit 22 uses commonly known speaker diarization to predict “who spoke and when” from the received audio file or audio data and associates the user (speaker) who spoke with the content of speech using the prediction result. Thus, the speech recognition result acquisition unit 22 assigns information (speaker label) such as “Mr. A” or “Mr. B” to the speech recognition result. The speaker diarization may be performed in parallel (concurrently) with speech recognition processing. The speech recognition result acquisition unit 22 may recognize the speaker down to the individual level using commonly known speaker recognition (authentication) and assign a user name (e.g., Mr. A=Mr. Sato, Mr. B=Mr. Suzuki) associated with the speaker label assigned by the execution of the speaker diarization. The speech recognition model 25 is assumed to be the end-to-end model, but may be DNN-HMM modelor GMM-HMM model. The language processing apparatus 40 can correct the speech recognition result (character data) regardless of the speech recognition model. The information processing system 20 does not necessarily include the speech recognition model 25. In this case, the speech recognition result acquisition unit 22 transmits audio data to the speech recognition model 25 that is external to the information processing system 20 through a network, and receives a speech recognition result (character data) from the speech recognition model 25 that is external to the information processing system 20. The received speech recognition result (character data) may be in any language of any country or region corresponding to speech spoken by the user, such as Japanese, English, or Chinese. Further, when a translation function is used in conjunction with the present embodiment, the speech recognition result (character data) may be received in a language of a country or a region different from the language of the speech spoken by the user.

[0053] The acquisition and generation unit 23 generates a prompt by using a prompt function that is pre-stored in the prompt function information storage unit 29. A prompt is an interface with the language processing apparatus 40. In the present embodiment, a prompt is an instruction sentence given to the language processing apparatus 40 by an interface. Since the language processing apparatus 40 returns a reply message corresponding to a prompt, it is preferable that a prompt corresponding to processing by the language processing apparatus 40 is preliminarily prepared so that the information processing system 20 acquires an appropriate reply message from the language processing apparatus 40. The prompt function is a template (format) for generating this prompt. The details of the prompt function are described later with reference to FIG. 17 and FIG. 18.

[0054] The correction unit 24 calls an application interface (API) 41 that is made open to the public by the language processing apparatus 40, and transmits a request message to the language processing apparatus 40. The request message includes the prompt. As a result of calling API41, the correction unit 24 receives a response message to the request message from the language processing apparatus 40. The response message includes a reply message to the prompt. In the present embodiment, this reply message is a corrected speech recognition result (character data). Thus, the correction unit 24 corrects a speech recognition result (character data). Further, the correction unit 24 transmits the speech recognition result (character data) obtained by the speech recognition by the speech recognition model 25 and the unique vocabulary associated with one or more users to a large language model, and acquires a corrected speech recognition result (character data) obtained by the large language model correcting the speech recognition result (character data) based on the unique vocabulary from the large language model. The correction unit 24 further corrects the speech recognition result based on the unique vocabulary when the large language model determines that correction is necessary, and does not correct the speech recognition result based on theunique vocabulary when the large language model determines that correction is not necessary. The API 41 is one or more web- APIs that is made open to the public by the language processing apparatus 40.

[0055] As described above, the entire or a part of the language processing apparatus 40 may be included in the information processing system 20.

[0056] Unique Vocabulary Management Table 2001FIG. 9 is an illustration of a unique vocabulary management table 2001 stored in the unique vocabulary storage unit 28. As illustrated in FIG. 9, in the unique vocabulary management table 2001, information for identifying a user including a user name and an email address is associated with unique vocabulary for each of unique vocabulary IDs. The unique vocabulary ID is a management number for managing each unique vocabulary. The user information for identifying a user such as the user name (user ID) and the email address may be information that is preliminarily associated with each other and managed by the information processing system 20. The unique vocabulary ID and the unique vocabulary corresponding to the unique vocabulary ID are text information registered in the unique vocabulary management table 2001, for example, when the user inputs (registers) the unique vocabulary to the user terminal 10. The information items managed in the unique vocabulary management table 2001 may be registered by being acquired by the information processing system 20 from, for example, an external server that manages the information items through a network, instead of or in addition to being input (registered) by the user to the user terminal 10. As described above, the information processing system 20 includes the unique vocabulary management table 2001 that registers a particular user among one or more users in association with unique vocabulary input to the user terminal 10 by the particular user. In the present embodiment, the unique vocabulary management table 2001 functions as an example of the unique vocabulary storage unit 28.

[0057] Notes on Large Language ModelFIG. 10 is a diagram illustrating details of the large language model 213. The large language model 213 is a kind of deep learning model used in the field of natural language processing. The main purpose of the large language model 213 is to learn huge amounts of character data and implement human-like natural language generation and understanding. The large language model is a complex model using a deep learning technology and is based on a computational model called a neural network that imitates the operation of neurons of the human brain. The neural network has a multi-layer structure to receive input data, learns patterns, and performs predictions and classification. In particular, a large language model learns a language structure by adjusting a large number of parameters with huge character data as an input.

[0058] PaLM®, ChatGPT®, Llama®, etc. can automatically generate character data with a level of fluency and logic almost equal to that of human writing. These models are huge neural networks with billions of parameters and are trained on a huge amount of character data. In the present embodiment, the information processing system 20 uses the large language model 213 as described above to correct recognition errors that occur during speech recognition.

[0059] FIG. 11 is a diagram illustrating examples of recognition errors that are likely to occur in speech recognition and an examples of corrections by the large language model.

[0060] Example 1The correct speech is “First, we judge if there are any problems.”The speech recognition result is "First, we jig if there are any problems.”The large language model can correct the speech recognition result to “First, we jig if there are any problems.”

[0061] Example 2The correct speech is “Is that portrait right and copyright?”The speech recognition result is “Is that portrait written and copyright?”The large language model can correct the speech recognition result to “Is that portrait right and copyright?”

[0062] Example 3The correct speech is “Um, in the past, there was a thing called “introduction of others” for ice-breaking.”The speech recognition result is “In the paste, there was thing called as “introduction of others” for ice brew.”The large language model can correct the speech recognition result to “Um, in the past, there was a thing called “introduction of others” for ice-breaking.”

[0063] In Examples 1 to 3, the characters to be recognized (i.e., judge, portrait right, past, and icebreaking) are recognized as different characters. In other words, the characters to be recognized are recognized incorrectly. In example 3, the character to be recognized is not output. This is called recognition failure. Specifically, “Um” of the “Um, in the past” is not output, and “a” of “was a thing” is not output. Further, in Example 3, an unnecessary character that is not included in the utterance is included. Specifically, “as” of “called as” is output.

[0064] In the present embodiment, for example, the user registers “judge, portrait right, icebreaking” as unique vocabulary in the information processing system 20. This makes it easier for the information processing system 20 to correctly convert audio data. Further, the information processing system 20 can insert a correct character when there is a recognition failure and delete an unnecessary character when the unnecessary character is included by using the large language model, even when the insertion or the deletion does not relate to the unique vocabulary.

[0065] Real-Time Recognition and Batch RecognitionSpeech recognition is roughly classified into real-time recognition and batch recognition. The speech recognition method according to the present embodiment can be suitably applied to batch recognition and real-time recognition.

[0066] FIG. 12 is a diagram illustrating real-time recognition and batch recognition in comparison. Examples of the batch recognition include transcribing a meeting and creating minutes. In the case of batch recognition, the user terminal 10 and the information processing system 20 communicate with each other over asynchronous Hypertext Transfer Protocol (HTTP). The asynchronous HTTP is a communication method in which the user terminal 10 and the information processing system 20 communicate with each other without synchronization. The user terminal 10 can receive an operation from the user even while the information processing system 20 is performing speech recognition.

[0067] In the batch recognition, the user terminal 10 uploads an audio file 271 having a long recording time to the information processing system 20. When the information processing system 20 finishes recognizing the entirety of the audio file 271, the information processing system 20 changes the status to “completed.” The user terminal 10 obtains a speech recognition result 272 of all speech after the status is changed to “completed.”

[0068] Examples of the real-time recognition include voice input, voice operation, subtitle display, and interaction with a virtual human. In this case, the user terminal 10 and the information processing system 20 communicate with each other using synchronous HTTP or WebSocket. The synchronous HTTP is a communication method in which the user terminal 10 and the information processing system 20 communicate with each other while synchronizing with each other. While the information processing system 20 is performing speech recognition, the user terminal 10 is in a standby state and cannot receive an operation from the user.

[0069] In the communication over synchronous HTTP, the user terminal 10 uploads an audio data 273 of a short time of about tens of milliseconds to the information processing system 20. Each time the information processing system 20 recognizes the audio data 273 of a short time,the information processing system 20 transmits a speech recognition result 274 to the user terminal 10.The user terminal 10 obtains the speech recognition result 274 corresponding to the uploaded audio data.

[0070] WebSocket is a communication standard that enables two-way communication in which a session is established between the user terminal 10 and the information processing system 20. In communication using WebSocket, the user terminal 10 transmits audio data 275 and audio data 277 one after another to the information processing system 20 without waiting for a response from the information processing system 20. The information processing system 20 successively transmits a speech recognition result 276 obtained by performing speech recognition on the audio data 275 and a speech recognition result 278 obtained by performing speech recognition on the audio data 277 to the user terminal 10.

[0071] The real-time recognition enables voice input and voice operation. Further, the user terminal 10 can display the subtitle of the speech of a participant in real time during, for example, a meeting. Furthermore, the real-time recognition allows a user to have real-time conversation with an Al. One aspect of the real-time conversation is a voice conversation with a virtual human. A virtual human is an artificial being with a human-like appearance, created using computer graphics and artificial intelligence technology. The virtual human itself also converses using a large language model.

[0072] Process FlowA flow of process of speech recognition and correction by the speech recognition system 100 is described below with reference to FIG. 13 to FIG. 21. FIG. 13 is a sequence diagram of a process in which the language processing apparatus 40 corrects a speech recognition result (character data) obtained by speech recognition performed by the speech recognition system 100. The speech recognition may be performed in real time or in batches. With reference FIG. 13, a case where the batch recognition is performed is described. The details of steps including processing of correcting the speech recognition result (character data) by the language processing apparatus 40 is described with reference to FIG. 14, and the overview of the flow is described with reference to FIG. 13.

[0073] In step S 11, a user inputs a unique vocabulary to the user terminal 10. The unique vocabulary may be input in real time from a keyboard or by voice. Alternatively, the unique vocabulary may be pre-prepared as a file. In step Si l, there are two cases where the user inputs (registers) the unique vocabulary. One is a case in which the user checks the speech recognition result illustrated in FIG. 4 and determines that the unique vocabulary is to be input (registered). Specifically, it is assumed that when a user says, “This road is one-way,”the speech recognition result obtained by the large language model (LLM) is “This road is one weigh.” In this case, the user determines that the recognition is erroneous, and newly inputs (registers) “one-way” as unique vocabulary. The other is a case in which it is determined that the speech includes a unique or rare word. For example, when the user wants to say, “Today’s blue sky is blue indigo azure,” the user preliminarily inputs (registers) “indigo azure” as unique vocabulary, as “indigo” and “azure” are typically used individually rather than combining them into a single phrase. Other specific examples of a word or phrase input (registered) as unique vocabulary” includes those illustrated in FIG. 15 described later. FIG. 22 illustrates an example of a unique vocabulary input screen 290. The communication unit 21 of the information processing system 20 receives the unique vocabulary and user identification information such as a user ID or an email address from the user terminal 10.The communication unit 21 of the information processing system 20 may register the unique vocabulary in association with user identification information such as the user ID and the email address in the unique vocabulary storage unit 28. The user identification information is input to the user terminal 10 when the user logs in to the information processing system 20, or is preliminarily input to the user terminal 10.

[0074] In step S12, the communication unit 11 of the user terminal 10 transmits the unique vocabulary to the information processing system 20, and the communication unit 21 of the information processing system 20 receives the unique vocabulary. The communication unit 21 may store the unique vocabulary in association with the user identification information in the unique vocabulary storage unit 28.

[0075] In step S 13, in the case of batch recognition, the user operates the user terminal 10 to designate an audio file of audio collected in, for example, a meeting. The audio file includes a large amount of audio data recorded during the meeting.

[0076] In step S14, the operation reception unit 13 of the user terminal 10 receives the designation of the audio file. The communication unit 11 of the user terminal 10 transmits the designated audio file to the information processing system 20, and the communication unit 21 of the information processing system 20 receives the audio file. The communication unit 21 stores the audio file in the audio file storage unit 30. The audio file may be stored on a network.

[0077] When the time comes for the information processing system 20 to perform batch processing, the information processing system 20 starts speech recognition of the audio file.

[0078] In step S15, the speech recognition result acquisition unit 22 of the information processing system 20 acquires a speech recognition result (character data) by performing speechrecognition on the audio data included in the audio file using the speech recognition model 25.

[0079] In step S16, the acquisition and generation unit 23 of the information processing system 20 generates a check_prompt to be transmitted to the language processing apparatus 40. The check_prompt is a prompt for inquiring of the language processing apparatus 40 whether the speech recognition result (character data) needs to be corrected. The acquisition and generation unit 23 may include the entirety of the speech recognition result (character data) in one check_prompt. Alternatively, the acquisition and generation unit 23 may generate multiple check_prompts each including a part of the speech recognition result (character data) obtained by dividing the speech recognition result and request the language processing apparatus 40 to perform correction in multiple times.

[0080] In step S17, the correction unit 24 of the information processing system 20 transmits check_prompt to the language processing apparatus 40. The transmitted check_prompt includes the unique vocabulary that received from the user terminal 10 in step S12 and then stored in the unique vocabulary storage unit 28 in association with the user identification information.

[0081] In step S18, the correction unit 24 of the information processing system 20 receives information indicating whether the speech recognition result (character data) needs to be corrected, i.e., whether correction necessity of the speech recognition result is “yes” or “no.”

[0082] When the correction necessity of the speech recognition result (character data) is “yes,” in step S19, the acquisition and generation unit 23 generates a correction_prompt for requesting the language processing apparatus 40 to correct the speech recognition result (character data).

[0083] In step S20, the correction unit 24 transmits the correction_prompt to the language processing apparatus 40.

[0084] In step S21, the correction unit 24 of the information processing system 20 receives the speech recognition result (character data) corrected by the language processing apparatus 40.

[0085] In step S22, the communication unit 21 of the information processing system 20 transmits the speech recognition result (character data) to the user terminal 10. The transmission may be performed automatically or may be performed in response to a request from the user. For example, the communication unit 21 may transmit the speech recognition result (character data) to the email address of the user. Alternatively, the information processing system 20 may store the speech recognition result (character data) in, for example, a predeterminedfolder or a uniform resource locator (URL), and the user may download the speech recognition result from the folder or the URL. The information processing system 20 may transmit the URL to the email address of the user. The speech recognition result (character data) is either the speech recognition result to which no correction is made or the corrected speech result.

[0086] The communication unit 11 of the user terminal 10 receives the speech recognition result (character data). In step S23, the display control unit 12 displays the speech recognition result (character data) on the display 506 of the user terminal 10. FIG. 23 is a diagram illustrating a display example of the speech recognition result (character data).

[0087] FIG. 14 is a flowchart of a process in which the language processing apparatus 40 corrects the speech recognition result (character data) obtained by speech recognition performed by the speech recognition system 100. The process of FIG. 14 is started at a timing when the information processing system 20 performs the batch processing. The timing at which the information processing system 20 performs the batch processing may be a fixed time, in response to a user’s operation, or when a certain amount of audio files are accumulated. Each step of the process of FIG. 14 is described below.

[0088] In step SI 10, the acquisition and generation unit 23 of the information processing system 20 acquires the unique vocabulary from the unique vocabulary storage unit 28.When audio is collected during a meeting, the audio file includes one or more pieces of audio data of one or more users. When the batch processing is performed, a user who attended the meeting may not be identified. The unique vocabulary that the acquisition and generation unit 23 acquires from the unique vocabulary storage unit 28 is described later with reference to step S140.

[0089] FIG. 15 is a diagram illustrating an example of the unique vocabulary registered by the user on the unique vocabulary input screen 290 illustrated in FIG. 22. In FIG. 15, three unique vocabulary words 291, i.e., “brainstorming,” “My Number,” and “Clever Blog,” is registered. The “My Number” in individual number under the Japan’s Social Security and Tax Number System. These unique vocabulary words are registered in the unique vocabulary storage unit 28 in association with user identification information such as a user name (user ID) and an email address. Any number of the unique vocabulary words 291 may be registered.

[0090] In step S120, the speech recognition result acquisition unit 22 of the information processing system 20 acquires the audio file from the audio file storage unit 30. The speech recognition result acquisition unit 22 acquires a speech recognition result (character data) by performing speech recognition on all audio data included in the audio file using the speech recognitionmodel 25. The speech recognition result acquisition unit 22 may recognize each chunk of audio data individually rather than recognizing the entire audio data at a time. The chunk of audio data may be audio data separated by a silent section of a certain length or more oar audio data separated by a certain time period.

[0091] FIG. 16 is a diagram illustrating a speech recognition result (character data) generated by the information processing system 20 performing speech recognition. As illustrated in FIG. 16, the information processing system 20 recognizes that “Well, then, for the grain storming, how about proposing a service that uses X product and audio / visuals?” The part that is recognized as “grain” is incorrect, and it should be recognized “brain.”At this time, the user does not have to check whether or not there is a recognition error.

[0092] In step S130, the acquisition and generation unit 23 of the information processing system 20 prepares two types of prompts to be transmitted to the language processing apparatus 40. The large language model accepts an instruction from the user in natural language. In the present embodiment, the correction unit 24 uses prompt programming to request the large language model to correct the speech recognition result (character data).

[0093] One of the prepared prompts is a prompt for the language processing apparatus 40 to determine whether the speech recognition result has to be corrected in step S 140. The other is a prompt for the language processing apparatus 40 to correct the speech recognition result (character data) in step S170. A template for each of the prompts is provided in functional form. A function for generating the prompt for determining whether the speech recognition result has to be corrected is called a “check_prompt” function. A function for generating the prompt for correcting the speech recognition result (character data) is called a “correct! on prompt” function.

[0094] FIG. 17 is a diagram illustrating the “check_prompt” function. The " check _prompt” function has two arguments: dictionary 221 and recognition_results 222. The dictionary 221 is a unique vocabulary, and recognition_results 222 is a speech recognition result (character data). When generating a prompt, the user’s unique vocabulary and the speech recognition result (character data) are acquired by these arguments, and the prompt is generated. In other words, the content of the dictionary 221 is set under a display 224 of Dictionary, and the content of the recognition_results 222 is set under a display 225 of Speech recognition result.

[0095] The “check_prompt” function includes a message 223 that reads “Please refer to the dictionary below and answer whether there are any strange parts of English in the speech recognition result.” The " check _prompt" function further includes a message 226 “Strange parts of English (Answer with either Yes or No).” The message 223 and the message 226 aresent to the language processing apparatus 40 as part of the prompt. The language processing apparatus 40 generates a reply message (i.e., Yes or No) based on the prompt generated by the “check_prompt” function and transmits the reply message to the information processing system 20.

[0096] FIG. 18 is a diagram illustrating the “correct! on prompt” function. The “correction prompt” function has two arguments: dictionary 231 and recognition_results 232. The dictionary 231 is a unique vocabulary, and recognition_results 232 is a speech recognition result (character data). When generating a prompt, the user's unique vocabulary and the speech recognition result (character data) are acquired by these arguments, and the prompt is generated. In other words, the content of the dictionary is set under a display 234 of Dictionary, and the content of the recognition_results is set under a display 235 of Speech recognition result.

[0097] The “ correct! on prompt” function includes a message 233 that reads “Please refer to the dictionary below and correct any strange parts of English in the speech recognition result. Please keep corrections to a minimum.” The “ correct! on_prompt” function further includes a message 236 “Correct.” The message 233 and the message 236 are sent to the language processing apparatus 40 as part of the prompt. The language processing apparatus 40 generates a reply message (corrected speech recognition result (character data)) based on the prompt generated by the “correction jorompt” function, sets the reply message under the message 236, and transmits the reply message to the information processing system 20.

[0098] In step S140, the acquisition and generation unit 23 of the information processing system 20 inputs the unique vocabulary acquired in step SI 10 and the speech recognition result (character data) acquired in step SI 20 to the “ check _prompt” function prepared in step SI 30, and generates check_prompt for the language processing apparatus 40 to determine whether correction is needed. The correction unit 24 of the information processing system 20 transmits the generated check_prompt to the language processing apparatus 40.

[0099] FIG. 19 is a diagram illustrating an example of the check_prompt transmitted to the language processing apparatus 40. As illustrated in FIG. 17, the check_prompt has a message 241 that reads “Please refer to the dictionary below and answer whether there are any strange parts of English in the speech recognition result.” The check_prompt further includes a content 242 of the dictionary and a content 243 of the recognition results. The “check prompt” further includes a message 244 “Strange parts of English (Answer with either Yes or No) ”

[0100] In the case of the batch recognition, the speech recognition result is classified into, for example, the following types (a) to (c):(a) A user who spoke is not identified in any way.(b) A user (participant) who spoke is identified. The type (b) corresponds a case in which user identification information of participants in a meeting is attached to an audio file or a case in which the information processing system 20 can acquire the user identification information of participants in a meeting from a meeting room reservation system. For example, a meeting in which multiple persons actually gather in a meeting room is assumed.(c) A user who has spoken is identified for each character data. The type (c) corresponds a case in which users log in from respective ones of the user terminals 10 during a meeting, and user identification information of a user who has transmitted each of pieces of audio data is stored in an audio file for each of the pieces of audio data. For example, an online meeting is assumed.

[0101] In the case of (a), the content 242 of the dictionary is all the unique vocabulary words stored in the unique vocabulary storage unit 28.In the case of (b), the content 242 of the dictionary is one or more unique vocabulary words associated with the participant (user identification information) of the meeting among the unique vocabulary words stored in the unique vocabulary storage unit 28.In the case of (c), check_prompt is preferably generated to include the speech recognition result (character data) for each speaker. The acquisition and generation unit 23 generates check_prompt, for example, each time the speaker is switched. In this case, the content 242 of the dictionary is one or more unique vocabulary words associated with a user who is a speaker among the unique vocabulary words stored in the unique vocabulary storage unit 28. However, generating check_prompt every time the speaker is switched imposes a high processing load, the acquisition and generation unit 23 may generate check_prompt in the same or substantially the same manner as in (b).

[0102] The above types (a) to (c) are also applied to the unique vocabulary to be set in correction_prompt.

[0103] In step S150, the correction unit 24 of the information processing system 20 transmits the prompt generated in step S140 to the language processing apparatus 40 and acquires a reply message of “yes” or “no” from the language processing apparatus 40. When the reply message is “yes", the information processing system 20 determines that the speech recognition has to be corrected. By contrast, when the reply message is “no,” the information processing system 20 determines that there is no need to correct the speech recognition.

[0104] In step S160, the correction unit 24 of the information processing system 20 determines whether the reply message acquired in step SI 50 is “yes" or “no.” When the reply message is “yes,” the process proceeds to step SI 70. When the reply message is “no,” the process proceeds to step S200.

[0105] When the reply message is “yes,” the speech recognition result (character data) has to be corrected. Accordingly, in step S170, the acquisition and generation unit 23 of the information processing system 20 performs processing for correcting the speech recognition result. The acquisition and generation unit 23 of the information processing system 20 inputs the unique vocabulary acquired in step S 110 and the speech recognition result (character data) acquired in step SI 20 to the “correction_prompt” function prepared in step S130, and thereby generates correction_prompt for correcting the speech recognition result (character data). For example, the prompt as described below is generated.

[0106] FIG. 20 is a diagram illustrating an example of correction_prompt generated by the “correction_prompt” function. As illustrated in FIG. 20, “correctionjorompt” includes a message 251 that reads “Please refer to the dictionary below and correct any strange parts of English in the speech recognition result. Please keep corrections to a minimum.” The correction_prompt further includes a content 252 of the dictionary and a content 253 of the recognition results. The “correction_prompt” further includes a message 254 “Correct.”

[0107] In step SI 80, the correction unit 24 of the information processing system 20 transmits the correction_prompt (correction request) generated in step S170 to the language processing apparatus 40 and acquires a reply message from the language processing apparatus 40. The reply message to the correction_prompt may include a corrected speech recognition result (character data). When the language processing apparatus 40 is the generative Al, the speech recognition result (character data) is not always corrected correctly.

[0108] FIG. 21 is a diagram illustrating an example of the reply message transmitted by the language processing apparatus 40 in response to the correction_prompt. The reply message illustrated in FIG. 21 is “Well, then, for the brainstorming, how about proposing a service that uses X product and audio / visuals?” The reply message indicates that the part that the voice recognition engine erroneously recognized as “grain” by the speech recognition engine is correctly corrected as “brain” by the correction processing.

[0109] In step S190, the communication unit 21 of the information processing system 20 transmits the corrected speech recognition result (character data) obtained in step SI 80 to the user terminal 10 in response to a request from the user terminal 10. The user terminal 10 receives the corrected speech recognition result (character data), and outputs (displays) the received speech recognition result (character data) to the display 506. Thus, the user can obtain the speech recognition result (character data) corrected based on the user’s own unique vocabulary without being conscious of the correction made by the large language model. Alternatively, the communication unit 21 may transmit the corrected speech recognition resultby, for example, an email or push notification, regardless of the request from the user terminal 10.

[0110] When the correction unit 24 of the information processing system 20 receives “no” as the reply message from the language processing apparatus 40 in step S160 (i.e., in a case where there is no need to correct the speech recognition result (character data)), no correction of the speech recognition result (character data) acquired in step S 120 is needed.In this case, the correction unit 24 does not perform the correction processing. The communication unit 21 of the information processing system 20 transmits the speech recognition result (character data) acquired in step S120 to the user terminal 10 in response to a request from the user terminal 10. The user terminal 10 receives the speech recognition result (character data) on which no correction processing is performed. In step S200, the user terminal 10 outputs (displays) the received speech recognition result (character data) to the display 506. Alternatively, the communication unit 21 of the information processing system 20 may transmit the speech recognition result (character data) to the user terminal 10 by, for example, an email or push notification, regardless of the request from the user terminal 10.

[0111] In the flowchart of FIG. 14, when the reply message from the language processing apparatus 40 is “yes” in step SI 60, the correction unit 24 generates the correction_prompt for correcting the speech recognition result (character data). Alternatively, the correction unit 24 may transmit the correction_prompt to the language processing apparatus 40 without transmitting the check_prompt to the language processing apparatus 40. When the correction unit 24 transmits the correction_prompt to the language processing apparatus 40 without transmitting the check_prompt, there may be a case where the speech recognition result (character data) is modified even when the speech recognition result is correct. In view of such a case, the correction unit 24 transmits the check_prompt to the language processing apparatus 40 and transmits correction_prompt to the language processing apparatus 40 when the need for correction is “yes.” This enhances the correction accuracy (recognition accuracy). Although there are many possible ways of correcting vocabulary by prompt programming, in the present embodiment, prompts for correction processing are broken down by multiple levels, and correction is performed step by step. Thus, correction with high accuracy can be expected.

[0112] Examples of ScreensFIG. 22 is a diagram illustrating an example of the unique vocabulary input screen 290 displayed by the user terminal 10. In FIG. 22, the three unique vocabulary words 291, “brainstorming,” “My Number,” and “Clever Blog” are input on the unique vocabulary input screen 290. The user may input the unique vocabulary with a keyboard or by voice input. Incase of the Japanese language, the user may input Kanji characters and their readings in association with each other.

[0113] FIG. 23 is a diagram illustrating an example of a speech recognition screen 260 on which the user terminal 10 displays a speech recognition result (character data). The speech recognition screen 260 includes a recognition result display section 261 and a recognition state display section 262. The recognition state display section 262 includes an icon 263 that indicates whether speech is currently being recognized. The recognition result display section 261 displays a speech recognition result (character data) transmitted from the information processing system 20. The recognition result display section 261 is any kind of application used for character input.

[0114] The recognition state display section 262 and the recognition result display section 261 may be provided by different applications. The user can input speech to the recognition result display section 261 implemented by a desired application used for character input, by starting up an application for the recognition state display section 262 and then starting up the desired application for the recognition result display section 261. When the recognition state display section 262 and the recognition result display section 261 are implemented by the same application, the user can input speech to one application by starting up the application. The speech recognition screen 260 may have display variations as follows. Specifically, the speech recognition result corrected by the large language model may be displayed in a different display form from other parts that are not corrected. Examples of the different display form include bold, underlined, and blinking. The speech recognition result before correction and the speech recognition result (or only the corrected parts) corrected by the large language model may be displayed on the same screen.

[0115] In the Case of Real-Time RecognitionA flow of process performed in the case of the real-time recognition is described below with reference to FIG. 24 and FIG. 25. FIG. 24 is a flowchart of the flow of speech recognition and correction of the speech recognition result (character data). The real-time recognition is used, for example, in a situation where a conversation turn (speech turn), which is a continuous unit from the start to the end of speech by one speaker during the conversation, is relatively long. Further, the real-time recognition is used, for example in a situation where the speech intervals between speakers is relatively long, for example, in a case where multiple commentators (speakers) comment on a game such as Japanese chess or Go, and the commentary contents are displayed with subtitles for broadcasting or distribution. Examples of a process to be performed in these cases include first displaying a speech recognition result (character data) including an erroneously converted text, and then replacing the erroneously converted text with a correct speech recognition result (character data) once the FEMgenerates the correct speech recognition result. In other words, the real-time recognition may be applied to a case where even when the speech content of a speaker is erroneously converted, the content can be sufficiently corrected.

[0116] In the case of the real-time recognition, in step S201 the communication unit 21 of the information processing system 20 receives audio data from the user terminal 10.

[0117] In step S202, the speech recognition result acquisition unit 22 acquires a speech recognition result (character data) by performing speech recognition on the audio data transmitted from the user terminal 10 using the speech recognition model 25.

[0118] The communication unit 21 transmits the speech recognition result (character data) to the user terminal 10. In step S203, the user terminal 10 displays the speech recognition result (character data) on which correction is not yet performed on the display 506 of the user terminal 10. The end-to-end model may be used as the speech recognition model 25.

[0119] In step S204, the acquisition and generation unit 23 acquires the speech recognition result (character data). In step S205, the correction unit 24 corrects the speech recognition result (character data) using the unique vocabulary. The processing in step S205 may be performed in the same or substantially the same manner as described with reference to FIG. 14.

[0120] The communication unit 21 transmits the corrected speech recognition result (character data) to the user terminal 10. In step S206, the user terminal 10 outputs (displays) the corrected speech recognition result (character data) on the display 506.

[0121] FIG. 25 is a sequence diagram of a process in which the language processing apparatus 40 corrects a speech recognition result (character data) obtained by speech recognition performed by the speech recognition system 100. With reference FIG. 25, a case where the real-time recognition is performed is described. In the following description with reference to FIG. 25, the differences from FIG. 13 are described.

[0122] First, the processing of transmitting the unique vocabulary to the information processing system 20 by the user terminals 10 in steps Si l and S 12 are performed in the same or substantially the same manner as described with reference to FIG. 13. However, the user terminal 10 does not have to transmit the unique vocabulary immediately before the real-time recognition and may transmit the unique vocabulary before the speech recognition.

[0123] In step S31, the user inputs speech to the user terminal 10.

[0124] In step S32, the operation reception unit 13 of the user terminal 10 acquires audio data. The communication unit 11 of the user terminal 10 transmits the audio data to the information processing system 20, and the communication unit 21 of the information processing system 20 receives the audio data.

[0125] In step S15, the speech recognition result acquisition unit 22 of the information processing system 20 acquires a speech recognition result (character data) by performing speech recognition on the audio data using the speech recognition model 25.

[0126] In the case of the real-time recognition, in step S33, the communication unit 21 of the information processing system 20 transmits the speech recognition result (character data) on which correction is not performed yet and a recognition ID to the user terminal 10. In other words, even where the speech recognition result (character data) includes a recognition error, the information processing system 20 transmits the speech recognition result (character data) to the information processing system 20 in order to ensure the real-time property. When the speech recognition result includes a recognition error, the information processing system 20 transmits the speech recognition result (character data) corrected in a later step to the user terminal 10. The recognition ID is identification information of the speech recognition processing, and is used by the user terminal 10 to identify a speech recognition result (character data) to be corrected.

[0127] The communication unit 11 of the user terminal 10 receives the speech recognition result (character data). In step S34, the display control unit 12 displays the speech recognition result (character data) on which correction is not performed yet on the display 506 of the user terminal 10. FIG. 26A and FIG. 26B are diagrams illustrating display examples of the speech recognition result (character data). Thus, the user terminal 10 can display the speech recognition result in real time.

[0128] After step S15, the information processing system 20 performs steps S16 to S21 in the same or substantially the same manner as described with reference to FIG. 13. The detailed processing in steps S16 to S21 may be performed in the same or substantially the same manner as described with reference to FIG. 14. In the real-time processing, when a user who has spoken is identified by, for example, logging in, the acquisition and generation unit 23 sets the unique vocabulary words of the identified user in the prompt. In the real-time processing, when a user who has spoken is not identified, the acquisition and generation unit 23 sets all of the unique vocabulary stored in the unique vocabulary storage unit 28 or the unique vocabulary words of all participants in a meeting in the prompt, as in the abovedescribed (a) and (b).

[0129] When the language processing apparatus 40 determines that the speech recognition result needs to be corrected, in step S35, the communication unit 21 transmits the speech recognition result (character data) corrected by the language processing apparatus 40 and the recognition ID to the user terminals 10.

[0130] The communication unit 11 of the user terminal 10 receives the corrected speech recognition result (character data) and the recognition ID. In step S36, when the received recognition ID matches the recognition ID of the speech recognition result (character data) on which correction is not performed yet and that has already been displayed, the display control unit 12 replaces (corrects) the speech recognition result (character data) on which correction is not performed yet with the received speech recognition result (character data). Thus, even when the displayed speech recognition result (character data) includes a recognition error, the recognition error is corrected.

[0131] It takes about several seconds for the information processing system 20 to perform steps S16 to S21 between the information processing system 20 and the language processing apparatus 40. However, according to the process of FIG. 25, the speech recognition result (character data) is displayed in real time, and when the speech recognition result includes a recognition error, the erroneously recognized speech recognition result (character data) can be corrected with a minimum time delay.

[0132] FIG. 26A and FIG. 26B are diagrams illustrating examples of the speech recognition screen 260 on which the user terminal 10 displays a speech recognition result (character data). In the following description with reference to FIG. 26A and FIG. 26B, the differences from FIG. 23 are described. The speech recognition screen 260 illustrated in FIG. 26A includes a character string 281 “Grain” which is erroneously recognized.

[0133] After the screen of FIG. 26A is displayed, the conversation progresses as illustrated in FIG. 26B, and character data 283 that reads “That’s nice.” is displayed. Further, in FIG. 26B, the user terminal 10 receives the corrected speech recognition result, and thus the character string 281 “Grain,” which is erroneously recognized, is corrected to a character string 282 “Brain.” The display control unit 12 may highlight the corrected character string 282 in, for example, red. Further, when the user places the mouse over or clicks on the performs mouseover or clicks on the character string 282, the display control unit 12 may display the character string 281 which is erroneously recognized.

[0134] Furthermore, when the language processing apparatus 40 fails to correct an erroneously recognized character or character string, the user may designate the character string 281 on the speech recognition screen 260 illustrated in FIG. 26A. The user terminal 10 transmits thecharacter string 281 which is erroneously recognized or the speech recognition result (character data) including the character string 281 to the information processing system 20, and the information processing system 20 transmits correction_prompt to the language processing apparatus 40. Accordingly, the speech recognition result may be corrected correctly. Alternatively, the user may input a character string that is the correct conversion of the character string 281 which is erroneously recognized. The user terminal 10 transmits the correctly converted character string to the information processing system 20. The information processing system 20 stores the received character string in the unique vocabulary storage unit 28 as a unique vocabulary, and therefore, the character string is correctly converted from the next time. The speech recognition screen 260 illustrated in each of FIG. 26A and FIG. 26B may have display variations as described with reference to FIG. 23.

[0135] In the speech recognition system 100 according to the present embodiment, the information processing system 20 transmits the unique vocabulary of a user and a speech recognition result to a large language model to correct the speech recognition result. This eliminates the need for time-consuming and costly incremental learning. Speech including the unique vocabulary of a user is converted to correct speech in a simple manner. This reduces the drawback of the end-to-end model that has high recognition accuracy but is not easily customized. This further implements flexible customization for each user 211.

[0136] Other Application ExamplesThe above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may be combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

[0137] For example, although in the present embodiment, the speech recognition is performed using the end-to-end model, the DNN-HMM model, or the GMM-HMM model, the speech recognition result corrected by the large language model may be recognized by any speech recognition model.

[0138] In the present embodiment, the information processing system 20 includes the speech recognition result and the unique vocabulary in the prompt to request the language processing apparatus 40 to correct the speech recognition result. Alternatively, the large language model may be caused to learn the unique vocabulary in advance, and only the speech recognition result may be transmitted to the large language model to request the large language model to correct the speech recognition result.

[0139] In the block diagram such as FIG. 8, the functional units are divided into blocks in accordance with main functions of the user terminal 10, the information processing system 20, and the language processing apparatus 40, in order to facilitate understanding of the processes performed by the user terminal 10, the information processing system 20, and the language processing apparatus 40. The scope of the present disclosure is not limited by how the process units are divided or by the names of the process units. The processing of the user terminal 10, the information processing system 20, and the language processing apparatus 40 may be divided into more processing units according to the processing content. Also, one processing unit can be divided so as to include more processing units.

[0140] The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application- specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality. There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of an FPGA or ASIC.

[0141] The apparatuses or devices described in one or more embodiments are just one example of plural computing environments that implement the one or more embodiments disclosed herein. In some embodiments, the information processing system 20 includes multiple computing devices, such as a server cluster. The multiple computing devices communicate with one another through any type of communication link including, for example, a network and a shared memory, and perform the processes disclosed in the present disclosure.

[0142] The elements of the information processing system 20 may be implemented by one server apparatus or are allocated to multiple apparatuses.

[0143] The present disclosure includes the following aspects.Aspect 1According to Aspect 1, an information processing system is communicable with a terminal apparatus through a network.The information processing system includes a speech recognition result acquisition unit that acquires a speech recognition result of audio data spoken by one or more users.The information processing system includes a correction unit that transmits the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing and receives a corrected speech recognition result obtained by correcting the speech recognition result by the large language model from the large language model.The information processing system includes a communication unit that transmits the acquired corrected speech recognition result to the terminal apparatus.Aspect 2According to Aspect 2, in the information processing system of Aspect 1, the correction unit requests correction of the speech recognition result acquired by the speech recognition result acquisition unit using prompt programming that notifies the large language model of processing content in natural language.Aspect 3According to Aspect 3, in the information processing system of Aspect 1, the speech recognition result acquisition unit acquires speech recognition results for all of pieces of audio data included in an audio file.The correction unit requests the large language model to correct all of the speech recognition results obtained by performing speech recognition on the all pieces of audio data one time or a plurality of times.Aspect 4According to Aspect 4, in the information processing system of any one of Aspects 1 to 3, the speech recognition result acquisition unit acquires the speech recognition result obtained by performing speech recognition in real time on the audio data transmitted from the terminal apparatus.The communication unit transmits the speech recognition result acquired by the speech recognition result acquisition unit to the terminal apparatus.The correction unit receives, from the large language model, the corrected speech recognition result obtained by correcting the speech recognition result by the large language model using the unique vocabulary associated with the one or more users, after the communication unit transmits the speech recognition result to the terminal apparatus.The communication unit transmits the corrected speech recognition result acquired by the correction unit to the terminal apparatus.Aspect 5According to Aspect 5, in the information processing system of any one of Aspects 1 to 4, the correction unit transmits an inquiry inquiring of the large language model whether the speechrecognition result needs to be corrected, together with the unique vocabulary associated with the user and the speech recognition result.When a response from the large language model to the inquiry indicates that the speech recognition result needs to be corrected, the correction unit receives, from the large language model, the corrected speech recognition result obtained by correcting the speech recognition result by the large language model using the unique vocabulary associated with the one or more users.Aspect 6According to Aspect 6, in the information processing system of Aspect 5, the correction unit corrects the speech recognition result based on the unique vocabulary when the large language model determines that the speech recognition result needs to be corrected, and does not correct the speech recognition result based on the unique vocabulary when the large language model determines that the speech recognition result does not need to be corrected.Aspect 7According to Aspect 7, the information processing system of any one of Aspects 1 to 6 further includes a unique vocabulary storage unit that registers a particular user among the one or more users in association with the unique vocabulary input by the particular user to the terminal apparatus.Aspect 8 According to Aspect 6, in the information processing system of any one of Aspects 1 to 7, the speech recognition result acquisition unit acquires the speech recognition result obtained by speech recognition using an end-to-end model.

[0144] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Intemet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD- ROM), a magnetic tape device, or a solid state memory device.

[0145] This patent application is based on and claims priority to Japanese Patent Application No. 2024-032655, filed on March 5, 2024, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.[Reference Sings List]

[0146] 10 User terminal20 Information processing system40 Language processing apparatus

Claims

[CLAIMS]1. An information processing system communicable with a terminal apparatus through a network, the information processing system comprising: a speech recognition result acquisition unit configured to acquire a speech recognition result of audio data of speech spoken by one or more users; a correction unit configured to transmit the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing and to acquire, from the large language model, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and a communication unit configured to transmit the acquired corrected speech recognition result to the terminal apparatus.

2. The information processing system according to claim 1, wherein the correction unit requests correction of the speech recognition result acquired by the speech recognition result acquisition unit using prompt programming, the prompt programming for notifying the large language model of processing content in natural language.

3. The information processing system according to claim 1, wherein the speech recognition result acquisition unit acquires speech recognition results for all pieces of audio data included in an audio file, and the correction unit requests the large language model to correct all of the speech recognition results obtained by performing speech recognition on the all pieces of audio data one time or a plurality of times.

4. The information processing system according to any one of claims 1 to 3, wherein the speech recognition result acquisition unit acquires the speech recognition result obtained by performing speech recognition in real time on the audio data transmitted from the terminal apparatus, the communication unit transmits the speech recognition result acquired by the speech recognition result acquisition unit to the terminal apparatus, the correction unit receives, from the large language model, the corrected speech recognition result obtained by correcting the speech recognition result by the large language model using the unique vocabulary associated with the one or more users, after the communication unit transmits the speech recognition result to the terminal apparatus, and the communication unit transmits the corrected speech recognition result acquired by the correction unit to the terminal apparatus.

5. The information processing system according to any one of claims 1 to 4, whereinthe correction unit transmits an inquiry inquiring of the large language model whether the speech recognition result needs to be corrected, together with the unique vocabulary associated with the user and the speech recognition result, and when a response from the large language model to the inquiry indicates that the speech recognition result needs to be corrected, the correction unit receives, from the large language model, the corrected speech recognition result obtained by correcting the speech recognition result by the large language model using the unique vocabulary associated with the one or more users.

6. The information processing system according to claim 5, wherein the correction unit corrects the speech recognition result based on the unique vocabulary when the large language model determines that the speech recognition result needs to be corrected, and does not correct the speech recognition result based on the unique vocabulary when the large language model determines that the speech recognition result does not need to be corrected.

7. The information processing system according to any one of claims 1 to 6, further comprising a unique vocabulary storage unit that registers a particular user among the one or more users in association with the unique vocabulary input by the particular user to the terminal apparatus.

8. The information processing system according to any one of claims 1 to 7, wherein the speech recognition result acquisition unit acquires the speech recognition result obtained by speech recognition using an end-to-end model.

9. A speech recognition system comprising: a terminal apparatus; an information processing system; and a language processing apparatus having a large language model that is a model for natural language processing, the terminal apparatus, the information processing system, and the language processing apparatus being communicably connected to each other through a network, wherein the information processing system includes: a speech recognition result acquisition unit configured to acquire a speech recognition result of audio data of speech spoken by one or more users; a correction unit configured to transmit the speech recognition result and unique vocabulary associated with the one or more users to the language processing apparatus and to acquire, from the language processing apparatus, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model of thelanguage processing apparatus based on the unique vocabulary; and a communication unit configured to transmit the acquired corrected speech recognition result to the terminal apparatus, and the terminal apparatus includes a display control unit configured to display the corrected speech recognition result received from the information processing system on a display.

10. The speech recognition system according to claim 9, wherein the large language model corrects one or more characters included in the speech recognition result acquired by the speech recognition result acquisition unit with a character string included in the unique vocabulary.

11. The speech recognition system according to claim 9, wherein the large language model corrects the speech recognition result by adding one or more character that are missing in the speech recognition result acquired by the speech recognition result acquisition unit or deleting one or more characters that are erroneously included in the speech recognition result.

12. An information processing method performed by an information processing system communicable with a terminal apparatus through a network, the information processing method comprising: acquiring a speech recognition result of audio data of speech spoken by one or more users; transmitting the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing; acquiring, from the large language model, a corrected speech recognition result obtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and transmitting the acquired corrected speech recognition result to the terminal apparatus.

13. A recording medium storing a plurality of program codes which, when executed by one or more processors of an information processing system communicable with a terminal apparatus through a network, causes the one or more processors to perform a method comprising: acquiring a speech recognition result of audio data of speech spoken by one or more users; transmitting the speech recognition result and unique vocabulary associated with the one or more users to a large language model that is a model for natural language processing; acquiring, from the large language model, a corrected speech recognition resultobtained by correcting the speech recognition result by the large language model based on the unique vocabulary; and transmitting the acquired corrected speech recognition result to the terminal apparatus.

Images