Information processing device, learning device, information processing method, learning method, and recording medium

Abstract

The processor executes an increase process using a bias module to increase the likelihood of a candidate symbol including a registered symbol among at least one candidate symbol. The processor determines a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increase process has been executed. If the provisional output symbol includes the registered symbol, the processor executes a conversion process by referring to a combination table to convert the registered symbol into a specific symbol corresponding to the registered symbol, and outputs the provisional output symbol after the conversion process as an output symbol.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a learning device, an information processing method, a learning method, and a recording medium. [Background technology]

[0002] In recent years, information processing devices that perform transformation processing to convert input data into output symbols have been utilized in various fields. In particular, so-called end-to-end architectures that perform the transformation processing using only a single neural network are well known.

[0003] For example, an information processing device employing a speech recognition engine accepts input of speech data as input data. Then, the information processing device outputs text of the speech as output symbols. The symbols (output symbols) are, for example, words or sentences.

[0004] There is a priority output function that allows a speech recognition engine to preferentially output specific symbols as output symbols. In this priority output function, a user registers specific symbols representing proper nouns, such as people's names or place names, in the speech recognition engine. The speech recognition engine then preferentially outputs the specific symbols during speech recognition. This priority output function is also called a vocabulary registration function.

[0005] In conventional speech recognition engines, the vocabulary registration function was independent, making it easy to implement. However, in speech recognition engines with an end-to-end structure, the conversion from input data to an output symbol sequence is realized using a single neural network. Therefore, with speech recognition engines with an end-to-end structure, it is difficult to add and register vocabulary through partial changes.

[0006] To address this issue, JP 2022-531615 A ​​(Patent Document 1) discloses a speech recognition engine that alleviates the problem. This speech recognition engine links an end-to-end speech recognition model with a weighted finite-state transducer (WFST). Through this linkage, this speech recognition engine makes it easier for specific vocabulary to appear in speech recognition results, thereby alleviating the problem. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2022-531615 Summary of the Invention The problem the invention is trying to solve

[0008] When a configuration that makes it easier to output specific symbols by combining an end-to-end model and a WFST, as in Patent Document 1, is applied to a speech recognition engine, symbols that are not included in the training data of the end-to-end model may not be output as output symbols. In the technology described in Patent Document 1, a substitution process is performed when training a speech recognition model. The substitution process randomly replaces any symbol in the training data with a symbol that has the same pronunciation as the given symbol but is different from the given symbol. This substitution process generates a new symbol as new training data. In this way, the technology described in Patent Document 1 discloses a method for alleviating the above problem by diversifying the training data.

[0009] However, in the above-described replacement process, it is difficult for a user to comprehensively prepare all words that will be generated in the future. Therefore, even if a specific symbol is registered, the above-described replacement process often does not generate the specific symbol registration. In this case, a problem may arise in that it is difficult to output the specific symbol as an output symbol. In other words, in conventional information processing devices, a problem may arise in that it is difficult to preferentially output a specific symbol input by a user as an output symbol.

[0010] The object of the present disclosure is to solve the above-described problems, and to provide a technology that enables a specific symbol input by a user to be preferentially output as an output symbol. [Means for solving the problem]

[0011] An information processing device disclosed herein converts input data into output symbols and outputs the output symbols. The information processing device includes an interface that accepts user input of input data, a memory that stores a trained model, and a processor. The processor extracts features from the input data. The processor estimates at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol by applying the features to the trained model. The memory stores a combination table and a bias module. The combination table is a table that shows combinations of specific symbols designated by a user and registered symbols corresponding to the specific symbols. The bias module is a module that increases the likelihood of candidate symbols including the registered symbol. The processor further performs an increase process that uses the bias module to increase the likelihood of a candidate symbol that includes the registered symbol among the at least one candidate symbol. The processor determines a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increase process has been performed. Then, when the provisional output symbols include registered symbols, the processor refers to a combination table to execute a conversion process for converting the registered symbols into specific symbols corresponding to the registered symbols, and outputs the provisional output symbols that have undergone the conversion process as output symbols.

[0012] The learning device of the present disclosure is a learning device for updating a model. The learning device includes an interface that acquires learning data, which is a combination of learning input data and first learning output symbols, and a calculation device. The calculation device extracts features from the learning input data. The calculation device acquires output symbols by applying the features to a model. The calculation device generates second learning output symbols by performing preprocessing on the first learning output symbols. The calculation device updates the model to reduce the error between the output symbols and the second learning output symbols. The preprocessing is a process of generating a first symbol included in the first learning output symbols and a second symbol in which the first symbol is represented by an expression specified by a user.

[0013] The information processing method disclosed herein converts input data into output symbols and outputs the converted output symbols. The bias module is a module that increases the likelihood of candidate symbols including a registered symbol. The combination table is a table that indicates combinations of specific symbols designated by a user and registered symbols corresponding to the specific symbols. The information processing method includes extracting features from the input data. The information processing method includes estimating at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol by applying the features to a trained model. The information processing method includes performing an increase process using the bias module to increase the likelihood of a candidate symbol that includes a registered symbol corresponding to the specific symbol designated by the user, among the at least one candidate symbol. The information processing method includes performing an increase process using the bias module to increase the likelihood of a candidate symbol that includes a registered symbol, among the at least one candidate symbol. The information processing method includes determining a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increase process has been performed. The information processing method includes, when a provisional output symbol includes a registered symbol, performing a conversion process for converting the registered symbol into a specific symbol corresponding to the registered symbol by referring to a combination table, and outputting the provisional output symbol after the conversion process as an output symbol.

[0014] The training method disclosed herein is a training method for updating a model. The training method includes acquiring training data that is a combination of training input data and first training output symbols. The training method includes extracting features from the training input data. The training method includes acquiring output symbols by applying the features to a model. The training method includes generating second training output symbols by performing preprocessing on the first training output symbols. The training method includes updating the model to reduce an error between the output symbols and the second training output symbols. The preprocessing is a process of generating a first symbol included in the first training output symbols and a second symbol in which the first symbol is represented by an expression specified by a user.

[0015] The recording medium of the present disclosure is a non-transitory recording medium storing a program for causing a computer to convert input data into output symbols and output the output symbols. The bias module is a module that increases the likelihood of candidate symbols including a registered symbol. The combination table is a table that shows combinations of specific symbols designated by a user and registered symbols corresponding to the specific symbols. The program causes the computer to extract features from input data. The program causes the computer to estimate at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol by applying the features to a trained model. The program causes the computer to perform an increasing process using the bias module to increase the likelihood of a candidate symbol that includes a registered symbol corresponding to a specific symbol designated by a user, among the at least one candidate symbol. The program causes the computer to perform an increasing process using the bias module to increase the likelihood of a candidate symbol that includes a registered symbol, among the at least one candidate symbol. The program causes the computer to determine a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increasing process has been performed. When a provisional output symbol includes a registered symbol, the program executes a conversion process for converting the registered symbol into a specific symbol corresponding to the registered symbol by referring to a combination table, and outputs the provisional output symbol for which the conversion process has been executed as an output symbol.

[0016] A recording medium according to the present disclosure is a non-transitory recording medium storing a program for updating a model in a computer. The program causes the computer to acquire training data that is a combination of training input data and first training output symbols. The program causes the computer to extract features from the training input data. The program causes the computer to acquire output symbols by applying the features to a model. The program causes the computer to generate second training output symbols by performing preprocessing on the first training output symbols. The program causes the computer to update the model so as to reduce an error between the output symbols and the second training output symbols. The preprocessing is a process of generating a first symbol included in the first training output symbols and a second symbol in which the first symbol is represented by an expression specified by a user. [Effects of the Invention]

[0017] According to the present disclosure, a technique is provided that makes it possible to preferentially output a specific symbol input by a user as an output symbol. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram illustrating an overview of an information processing device according to the present disclosure. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the information processing device. [Figure 3] FIG. 2 is a functional block diagram of the information processing system. [Figure 4] FIG. 2 is a functional block diagram of the information processing device. [Figure 5] FIG. 10 is a diagram illustrating an example of a specific symbol table. [Figure 6] FIG. 10 is a diagram illustrating an example of a combination table. [Figure 7] FIG. 10 is a diagram illustrating an example of a bias table. [Figure 8] 10 is a flowchart showing main processing of the information processing device. [Figure 9] FIG. 2 is a functional block diagram of a learning device. [Figure 10] FIG. 10 is a diagram illustrating an example of a word changing process. [Figure 11] FIG. 10 is a diagram illustrating an example of preprocessing. [Figure 12] 10 is a flowchart of a learning device. [Figure 13] 10 is an example of a specific symbol table expressed in Japanese. [Figure 14] 10 is an example of a combination table expressed in Japanese. [Figure 15] This is an example of a bias table expressed in Japanese. [Figure 16] FIG. 10 is a diagram illustrating another example of application of input data. [Figure 17] FIG. 10 is a diagram illustrating an example of probability symbols. [Figure 18] 10 is a flowchart of a registration symbol determination method. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Although several embodiments will be described below, it is intended from the beginning of the application that the configurations described in each embodiment may be appropriately combined. Note that the same or corresponding parts in the drawings are designated by the same reference numerals, and their description will not be repeated.

[0020] Embodiment 1 [Summary of this disclosure] Hereinafter, the term "symbol" will be used. A symbol is a unit of information output from the information processing device 100. A symbol represents, for example, at least one character. A symbol is a character, such as hiragana, katakana, kanji, or the alphabet. For example, a symbol may be a single character (e.g., "A") or multiple characters, such as "America." When a symbol is defined as a single character, multiple characters are also referred to as a "symbol sequence." A symbol may also be information indicating an event, such as the start of the symbol or the end of the symbol.

[0021] In the following disclosure, information representing a single character and information consisting of multiple characters are collectively referred to as a "symbol." That is, both "A" and "America" ​​are "symbols."

[0022] Fig. 1 is a diagram for explaining an overview of an information processing device 100 of the present disclosure. In the example of Fig. 1, the information processing device 100, a microphone 2 (voice input device), and a display 3 (display device) are shown. The microphone 2 and the display 3 are connected to the information processing device 100.

[0023] In the example of FIG. 1, user A utters the words "i triple e." This "i triple e" means "IEEE." Note that "IEEE" is a symbol (word) that indicates the name of an organization called "Institute of Electrical and Electronics Engineers," and is an example of a proper noun. Information processing device 100 accepts input of speech data of the speech "i triple e," performs speech recognition processing, and outputs "IEEE" as text data with the output symbol. Display 3 displays the text data, "IEEE."

[0024] Furthermore, user A has registered "IEEE" as a specific symbol in information processing device 100, and desires that when user A utters the words "i triple e," "IEEE" be preferentially output as the output symbol.

[0025] 1, the output symbol contains "IEEE," which the user desires to output, but a conventional speech recognition device may display the text "i triple e," which is the speech itself, on the display 3. The information processing device 100 of this embodiment also employs an end-to-end structure.

[0026] [Hardware configuration of information processing device] 2 is a block diagram showing a hardware configuration of the information processing device 100 according to Embodiment 1. The information processing device 100 may be realized by, for example, a general-purpose computer or a dedicated computer.

[0027] As shown in FIG. 2, the information processing device 100 includes, as main hardware elements, a calculation unit 11, a memory unit 12, an audio interface 13, a communication unit 14, a display interface 15, an input device interface 16, and a reading unit 17.

[0028] The calculation unit 11 is a calculation entity (calculation device) that executes various processes by executing various programs, and is an example of a computer. The calculation unit 11 corresponds to the "processor" of the present disclosure. The calculation unit 11 is configured, for example, with a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), and a GPU (Graphics Processing Unit). The calculation unit 11 may be configured with at least one of the CPU, FPGA, and GPU, or may be configured with a CPU and an FPGA, an FPGA and a GPU, a CPU and a GPU, or all of the CPU, FPGA, and GPU. The calculation unit 11 is also referred to as "at least one processor." The calculation unit 11 may be configured with a processing circuitry. The calculation unit 11 may be configured with one chip or multiple chips. Furthermore, all or some of the functions of the calculation unit 11 may be provided in a server device (e.g., a cloud-based server device) not shown.

[0029] The storage unit 12 includes a volatile storage area (e.g., a working area) that temporarily stores program code, work memory, and the like when the calculation unit 11 executes any program. For example, the storage unit 12 is configured with a volatile memory device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). Furthermore, the storage unit 12 includes a non-volatile storage area. For example, the storage unit 12 is configured with a non-volatile memory device such as a ROM (Read Only Memory), a hard disk, or an SSD (Solid State Drive).

[0030] In the present embodiment, an example has been shown in which a volatile storage area and a nonvolatile storage area are included in the same storage unit 12, but the volatile storage area and the nonvolatile storage area may be included in different storage units. For example, the calculation unit 11 may include a volatile storage area, and the storage unit 12 may include a nonvolatile storage area. The information processing device 100 may include a microcomputer including the calculation unit 11 and the storage unit 12.

[0031] The storage unit 12 stores a processing program 301, a trained model D3, a decision model 305, a combination table D7, and a bias table D8. The processing program 301 includes a generation program and a learning program. The information processing program describes information processing in which the calculation unit 11 generates output text based on voice data (input data) acquired by the microphone 2 and the trained model D3. The learning program describes learning processing for updating (training) the trained model D3.

[0032] The trained model D3 includes a neural network 303 and parameters 304 used by the neural network 303. The trained model D3 is subjected to machine learning (update processing) using a plurality of training data, which are combinations of training input data and training output symbols.

[0033] The trained model D3 has, for example, a known deep neural network (DNN), convolutional neural network (CNN), long short-term memory (LSTM), Transformer, Conformer structure, etc. The decision model 305, combination table D7, and bias table D8 will be described later.

[0034] The audio interface 13 is an interface for connecting the microphone 2. The communication unit 14 communicates with a predetermined external device. The display interface 15 is an interface for connecting the display 3, and realizes input and output of data between the information processing device 100 and the display 3.

[0035] The input device interface 16 is an interface for connecting an input device 4 (for example, a keyboard, a mouse, etc.), and realizes input and output of data between the information processing device 100 and the input device.

[0036] The reading unit 17 reads various data stored in a removable disk 20, which is a storage medium. The removable disk 20 stores a processing program 301 (at least one of an information processing program and a learning program) according to the present disclosure. The removable disk 20 storing the processing program 301 is then sold or otherwise implemented. The reading unit 17 may also acquire the processing program 301 from the removable disk 20. The removable disk 20 corresponds to an example of a "non-transitory recording medium" according to the present disclosure.

[0037] [Functional block diagram of information processing system] 3 is a functional block diagram of an information processing system 30. The information processing system 30 includes an information processing device 100, a learning device 200, a learning data storage device 300, and a trained model storage device 400. The information processing device 100 outputs an output symbol D4 using a trained model D3 based on input data D1 and a specific symbol table D2.

[0038] The learning device 200 learns a trained model D3 (updates the parameters 304) to be used in the information processing device 100. The learning device 200 learns the model using the training data E stored in the training data storage device 300. The learning device 200 outputs a trained model E3, which is a trained model, to the trained model storage device 400.

[0039] [Functional block diagram of information processing device] 4 is a functional block diagram of the information processing device 100. The information processing device 100 includes an extraction unit 110, a determination unit 120, a generation unit 130, an estimation unit 140, and a conversion unit 150.

[0040] The information processing device 100 of this embodiment functions as a voice recognition device. Input data D1 and a specific symbol table D2 are input to the information processing device 100. In this embodiment, the input data D1 is voice data. The input data D1 is, for example, a digital signal obtained by converting a voice input to a microphone 2 through A / D (analog / digital) conversion or the like of the microphone 2. The output symbols D4 are text data of the voice represented by the voice data. For example, before the input data D1 is input to the information processing device 100, the specific symbol table D2 is input to the information processing device 100 from the input device 4 by a user.

[0041] Input data D1 is input to the extraction unit 110. The extraction unit 110 extracts features D5 from the input data D1 in a format suitable for input to a trained model D3. The method for extracting the features D5 includes, for example, dividing the audio signal, which is the input data D1, into segments that overlap at regular intervals. The extraction method further includes extracting a known logarithmic Mel filter bank or Mel frequency cepstrum from the divided signal. Other methods for extracting features may also be used.

[0042] The feature D5 is output to the estimation unit 140. The estimation unit 140 estimates at least one candidate symbol and the likelihood of each of the at least one candidate symbol by applying the feature D5 to the trained model D3. The likelihood is a parameter indicating the likelihood of a candidate symbol, and is also referred to as a score. For example, negative log-likelihood (NLL) is used as the score. When the score is negative log-likelihood, the smaller the score, the higher the likelihood. In this embodiment, an example is applied in which the smaller the score, the higher the likelihood. Note that, as a modified example, the configuration may be such that the larger the score, the higher the likelihood.

[0043] Furthermore, the estimation unit 140 performs an increasing process to increase the likelihood (a process to decrease the score) using a bias table D8 described later, and outputs the candidate bias with the maximum likelihood after the increasing process as a provisional output symbol D9.

[0044] Next, the specific symbol table D2 will be described. The specific symbol table D2 is a table storing at least one specific symbol. The specific symbol is a symbol that tends not to be included in the output symbols D4 when the information processing device 100 performs speech recognition processing on the input text without using the bias table D8 described below. Given this tendency, the specific symbol is typically a symbol that the user desires to be preferentially included in the output symbols D4. The specific symbol is a proper noun, a character with a special meaning, or the like. The specific symbol table D2 is information that is input in advance to the information processing device 100 by a user A or the like before inputting the input data D1 to the information processing device 100.

[0045] A registered symbol, which will be described later, is a symbol corresponding to a specific symbol. A registered symbol is, for example, a symbol that has the same meaning as a specific symbol but is expressed in a different way (expression type) than the specific symbol. A registered symbol may include a symbol that has the same pronunciation as the specific symbol, which is a vocabulary, but is read in a different way, such as hiragana, katakana, roman letters, or spelling. A registered symbol may also include a symbol in which the specific symbol is expressed using the International Phonetic Alphabet or the like. In this way, a registered symbol may have the same reading as the specific symbol but be expressed using a different symbol. Note that the registered symbol and the specific symbol may be the same.

[0046] 5 is an example of the specific symbol table D2. The specific symbol is, for example, a proper noun such as a person's name or a place name, or an organization name such as "IEEE" as mentioned above. In the specific symbol table, a specific symbol (IEEE) is associated with a phoneme symbol of the specific symbol as an alternative symbol that can be used in place of the specific symbol. The alternative symbol may not be present.

[0047] The determination unit 120 determines a registered symbol from the input specific symbol table D2. A registered symbol is a symbol that is preferentially output by the estimation unit 140 instead of a specific symbol. Note that "a symbol is preferentially output" means that the symbol, which is a candidate symbol (described later), has a high likelihood. Thus, in this embodiment, it is preferable that the likelihood of a registered symbol is high.

[0048] Next, the determination unit 120 extracts a specific symbol from the specific symbol table, and then converts the specific symbol into a registered symbol.

[0049] The conversion method from a specific symbol to a registered symbol may employ a method of determining a registered symbol based on supplemental information (not shown) included in the specific symbol table D2. The conversion method may be realized by a predetermined calculation. The conversion method may also be realized by, for example, specifying the representation type of the registered symbol by a user. Alternatively, a substitute symbol may be determined as the registered symbol.

[0050] The determining unit 120 then generates a combination table D7 by combining the specific symbols with the registered symbols. The combination table D7 is a table showing combinations of the specific symbols input by the user and the registered symbols corresponding to the specific symbols.

[0051] 6 is a diagram showing an example of the combination table D7. The combination table D7 is a table in which the specific symbol "IEEE" is associated with the phoneme symbol (registered symbol) of the specific symbol. The combination table D7 is stored in the storage unit 12 (FIG. 2). The combination table D7 is then output to the conversion unit 150, which will be described later.

[0052] 6. Furthermore, the determining unit 120 extracts a registered symbol D6 from the combination table D7 and outputs the registered symbol D6 to the generating unit 130. The registered symbol D6 is the symbol enclosed in a dashed line frame in FIG.

[0053] The generating unit 130 generates a bias table D8 by determining a bias value corresponding to the registered symbol D6 included in the combination table D7. The bias table D8 corresponds to an example of a "bias module" in the present disclosure. The bias table D8 is a table in which registered symbols are associated with bias values ​​that increase likelihood. The bias table D8 generated by the generating unit 130 is stored in the storage unit 12.

[0054] Fig. 7 shows an example of a bias table. In the example of Fig. 7, the bias value "-3.5" is associated with the IEEE phoneme symbol (registered symbol D6). Bias values ​​are also associated with the other registered symbols.

[0055] The learned model D3 may be trained such that, when the amount of registered symbols contained in the training output symbols included in the training data E of the learned model D3 is small, the likelihood of a symbol containing the registered symbol is smaller than the likelihood of a symbol containing another symbol different from the registered symbol.

[0056] Here, "i triple e" is divided into the word "i", the word "triple", and the word "e", and these words are contained in large numbers in the training data E. Therefore, "i triple e" is contained more often in the training data E than the IEEE phoneme symbol. Therefore, as shown in Figure 1, when "i triple e" is the input data D1, there is a tendency (premise) that "i triple e" will have a smaller score than the IEEE phoneme symbol ("i triple e" has a higher likelihood than the phoneme symbol). Incidentally, "i triple e" is also referred to as a "general word".

[0057] Under these assumptions, the bias value is set so that the score estimated by the estimation unit 140 is smaller for the phoneme symbol than for "i triple e." In other words, as a guideline for determining the bias value, the generation unit 130 determines a bias value that significantly reduces the score (increases the likelihood) for symbols that are included in small amounts in the training data E.

[0058] For example, the generation unit 130 determines the bias value using a determination model 305 (see FIG. 2). The determination model 305 is, for example, a model created using training symbols DT2 and includes a language model. For example, an N-gram language model is adopted as the language model. The language model outputs a large negative log-likelihood for symbols contained in the training symbols DT2 in small amounts, and outputs a small negative log-likelihood for symbols contained in the training symbols DT2 in large amounts. The determination model 305 can be expressed by equation (1).

[0059] Bias value = ρ - λ × (negative log likelihood from language model) (1) where ρ is a constant and λ is a positive constant. In this way, by using the decision model 305 according to equation (1), the generation unit 130 outputs a bias value that increases the likelihood of a candidate symbol including a registered symbol as the number of input registered symbols included in the registered symbols for training decreases. For example, when the number of input registered symbols included in the registered symbols for training is small, the bias value is −5, and when the number of input registered symbols included in the registered symbols for training is large, the bias value is −1. The input registered symbols are registered symbols input to the decision model 305.

[0060] The bias table may be generated in the WFST format described above. Also, a machine-learned model may be used instead of a language model.

[0061] Next, the processing of the estimation unit 140 using the bias table D8 will be described. As described above, the estimation unit 140 estimates at least one candidate symbol and the likelihood of each of the at least one candidate symbol by applying the feature D5 to the trained model D3. In the absence of the bias table D8, the estimation unit 140 identifies the candidate symbol with the smallest score (highest likelihood) from the at least one candidate symbol and outputs the candidate symbol as the provisional output symbol D9.

[0062] On the other hand, if the bias table D8 is present, the estimation unit 140 adds the bias value corresponding to the registered symbol in the bias table D8 to the score of the candidate symbol including the registered symbol.

[0063] For example, a case will be described in which the estimation unit 140 identifies the following first and second candidate symbols as candidate symbols. The first candidate symbol is "i triple e," and the score of the first candidate symbol is "0.0." The second candidate symbol is "IEEE phoneme symbol," and the score of the second candidate symbol is "1.0." In this example, the score of the first candidate symbol is smaller (higher likelihood) than the second candidate symbol.

[0064] In this example, if there is no bias table D8, the estimation unit determines the first candidate symbol as the tentative output symbol.

[0065] 7 exists, the second candidate symbol is registered as a registered symbol. Therefore, the estimation unit 140 performs an increase process to increase the likelihood of the candidate symbol (second candidate symbol) based on the bias value (-3.5) corresponding to the registered symbol. In this example, this increase process is a process of calculating a score of -2.5 by adding the bias value to the score.

[0066] In this way, the estimation unit 140 performs an increasing process of increasing the likelihood of a candidate symbol that includes a registered symbol among at least one candidate symbol, based on the bias value corresponding to the registered symbol.

[0067] Then, a provisional output symbol D9 is determined based on the likelihood of each of at least one candidate symbol after the increase process has been performed. In the above example, after the increase process, the score of the first candidate symbol is "0.0" and the score of the second candidate symbol is "-2.5". Therefore, the estimation unit 140 determines the second candidate symbol as the provisional output symbol D9. This provisional output symbol D9 is input to the conversion unit 150.

[0068] When the provisional output symbol D9 includes a registered symbol, the conversion unit 150 refers to the combination table D7 (FIG. 6). Then, the conversion unit 150 executes a conversion process to convert the registered symbol into a specific symbol corresponding to the registered symbol. Then, the conversion unit 150 outputs the provisional output symbol D9 that has been subjected to the conversion process as the output symbol D4.

[0069] In the above example, the provisional output symbol D9 includes the "IEEE phoneme symbol." Therefore, the conversion unit 150 converts the "IEEE phoneme symbol" (registered symbol) into "IEEE" (specific symbol) that corresponds to the "IEEE phoneme symbol" in the combination table D7 (FIG. 6). Then, the converted symbol, "IEEE," is output as the output symbol D4.

[0070] Next, the significance of the determination unit 120 will be explained. It is possible to generate the bias table D8 by regarding the specific symbol in the specific symbol table D2 as the registered symbol D6 without providing the determination unit 120. That is, in the above example, the generation unit 130 can generate the bias table D8 by using the specific symbol "IEEE" instead of the registered symbol "IEEE phoneme symbol." In the example of FIG. 7, this bias table D8 associates a bias value (-3.5) with "IEEE."

[0071] Here, since "IEEE" is a proper noun, the amount of data containing "IEEE" in the training data E used to train the trained model D3 is often zero or extremely small. In this case, the estimation unit 140 determines that the score of the candidate symbol "IEEE" is significantly higher than the score of "i triple e" and the score of "IEEE phoneme symbol."

[0072] In this case, it is assumed that even if the estimation unit 140 adds a bias value to the score of the candidate symbol "IEEE" using the bias table D8, the score will not be smaller than the score of "i triple e" and the score of "IEEE." In this case, "IEEE," which the user desired, will not be output as the output symbol D4.

[0073] The registered symbol "IEEE phoneme symbol" is expressed differently from the specific symbol "IEEE", but many individual phonemes of the phoneme symbol are included in the training data. Therefore, the score of the candidate symbol "IEEE phoneme symbol" is smaller than the score of the specific symbol "IEEE".

[0074] As described above, the registration of a specific symbol functions more effectively when the bias table D8 is generated using the registered symbol “IEEE phoneme symbol” than when the bias table D8 is generated using the specific symbol “IEEE.” In other words, the determining unit 120 has the significance of converting the specific symbol into a symbol (in this example, “IEEE phoneme symbol”) that has a higher likelihood than the specific symbol.

[0075] [flowchart] 8 is a flowchart showing the main processing of the information processing device 100. First, in step S12, the information processing device 100 extracts a feature D5 from the input data D1. Next, in step S14, the information processing device 100 estimates at least one candidate symbol and the likelihood of each of the at least one candidate symbol.

[0076] Next, in step S16, the information processing device 100 executes an increasing process to increase the likelihood of candidate symbols including the registration symbol D6 based on the bias value. Next, in step S18, the information processing device 100 determines a provisional output symbol D9 based on the likelihood after the increasing process. Next, in step S20, the information processing device 100 converts the registration symbol included in the provisional output symbol D9 into a specific symbol corresponding to the registration symbol in the combination table D7, and outputs it as the output symbol D4.

[0077] [Learning device] Next, learning device 200 (see FIG. 3) will be described. Learning device 200 learns model E5 when a predetermined start condition is met. Model E5 corresponds to trained model D3 before the start condition is met. The start condition is met, for example, when a user of learning device 200 performs a predetermined start operation.

[0078] 9 is a functional block diagram of the learning device 200. The learning device 200 includes an acquisition unit 210, an extraction unit 220, a processing unit 230, a model storage unit 235, a preprocessing unit 240, an update unit 250, and an output unit 260.

[0079] A plurality of pieces of training data E are stored in the training data storage device 300. The acquisition unit 210 acquires the plurality of pieces of training data E from the training data storage device 300. The training data E (teacher data) is composed of training input data E1 to be used for training the model E5 and first training output symbols E2 corresponding to the training input data E1. For example, the training input data E1 is audio data, and the first training output symbols E2 are text data obtained by transcribing the audio represented by the audio data. The training input data E1 is output to the extraction unit 220, and the first training output symbols E2 are output to the preprocessing unit 240. Furthermore, in the trained model D3, the more symbols included in the training output symbols, the higher the likelihood of a candidate symbol including those symbols.

[0080] The extraction unit 220 extracts, from the learning input data E1, feature values ​​E4 in a format suitable for input to a model. The specific processing method for extracting feature values ​​E4 from the learning input data E1 is the same as the processing for extracting feature values ​​D5 from input data D1 in the extraction unit 110 (FIG. 4) of the information processing device 100.

[0081] The model storage unit 235 stores the model E5. After reading out the model E5 from the model storage unit 235, the processing unit 230 applies the model E5 to the feature E4 extracted by the extraction unit 220 to obtain the output symbol E6. For example, the model E5 outputs a score (likelihood) indicating which symbol is likely to be output in each frame of the feature E4. Alternatively, when the feature E4, information on symbols that have already been output, and information on candidates for the symbol to be output next are given, the model E5 calculates a score for outputting the symbol candidate.

[0082] The preprocessing unit 240 applies predetermined preprocessing to the first training output symbols E2 to generate second training output symbols E7. The preprocessing includes, for example, applying morphological analysis to the transcribed text to divide it into words, and then performing modification processing to randomly modify some words (target words) to words that are expressed differently from the target words.

[0083] For example, the modification process includes a process of modifying a target word that is not written in hiragana to a word that is written in hiragana. The modification process also includes a process of modifying a target word that is not written in katakana to a word that is written in katakana. The modification process also includes a process of modifying a target word that is not written in romaji to a word that is written in romaji. The modification process also includes a process of modifying a target word to a word that is written with a lengthened vowel. FIG. 10 shows an example of a process of modifying a target word to a word that is written with a lengthened vowel.

[0084] The preprocessing of the preprocessing unit 240 will be further described. As described above, it is preferable that the likelihood of a registered symbol is high in the information processing device 100. Here, as described above, a registered symbol is a symbol that has the same meaning as a specific symbol and is expressed in a different way from the specific symbol (see FIG. 6).

[0085] Moreover, the first training output symbol E2 is, for example, an English sentence using the alphabet. The English sentence that is the first training output symbol E2 is, for example, a sentence that includes "i triple e", such as "I attend i triple e". In the following, "i triple e" and the like included in the English sentence are, as described above, "general words".

[0086] In other words, when the learning device 200 learns the model E5 using the first learning output symbol E2 as is, the model E5 is trained so that a high likelihood is assigned to common words such as "i triple e" and a low likelihood is assigned to phoneme symbols.

[0087] When model E5 is trained in this way, the score of the output symbol for a general word (first score) may become excessively smaller than the score of the output symbol for a phoneme symbol (second score). In this case, if the phoneme symbol (e.g., IEEE phoneme symbol) is the registered symbol, even if the score of the registered symbol (second score) is reduced by the bias value, the first score cannot become lower than the second score, and the output symbol for a general word may still be output. Furthermore, if λ in the formula for the bias value is set to a large value to compensate for the large difference in scores, the second problem may occur, in which the registered symbol may be erroneously output in an unrelated situation.

[0088] Therefore, assuming the first and second problems, the preprocessing by the preprocessing unit 240 is a process of applying morphological analysis to the text, dividing it into words, and then selecting some words and converting them to different notations. Morphological analysis is an example of the "predetermined processing" of the present disclosure.

[0089] Fig. 11 is a diagram for explaining preprocessing. In the example of Fig. 11, the text is "I attend i triple e". Then, the preprocessing unit 240 applies morphological analysis to this text to divide it into "I", "attend", and "i triple e". Note that the symbols after division are also referred to as unit symbols or simply words in the present disclosure.

[0090] The preprocessing unit 240 then selects "i triple e" as the first symbol. The preprocessing unit 240 then converts "i triple e" into the phoneme symbol for "IEEE," which is a different representation. The phoneme symbol for "IEEE" is an example of a second symbol. This allows the learning device 200 to increase the phoneme symbol training data (second symbol). Therefore, the learning device 200 can increase the likelihood of a candidate symbol containing the phoneme symbol training data (second symbol). In other words, the score (second score) of the output symbol based on the phoneme symbol can be appropriately reduced, the difference between the first score and the second score becomes smaller, and the assignment of a bias value to the registered symbol becomes more effective. The first training output symbol E2 that has undergone preprocessing by the preprocessing unit 240 is referred to as the "second training output symbol E7."

[0091] The representation type of the second symbol is specified by the user. The second training output symbol E7 includes a first symbol and a second symbol corresponding to the first symbol. The second symbol may be expressed as corresponding to a registered symbol. Typically, it is preferable that the second symbol is identical to the registered symbol. The second symbol may be included in the registered symbol, or the registered symbol may be included in the second symbol.

[0092] The update unit 250 calculates the error between the output symbol E6 (output symbol) output from the processing unit 230 and the second learning output symbol E7 output from the pre-processing unit 240. Next, the update unit 250 updates the model E5 so as to reduce the error. The update unit 250 outputs the obtained updated model E5 to the model storage unit 235. Here, the function for calculating the error between the output symbol E6 and the second learning output symbol E7 is, for example, a cross-entropy error function or a CTC error function (CTC: connectionist temporal classification). Furthermore, in the model update, the gradient of each parameter (parameter 304) of the model is calculated using a known backpropagation method, and then the model E5 is updated using an optimization method such as the SGD method (Stochastic Gradient Descent) or the Adam method.

[0093] After the learning of model E5 is completed, output unit 260 outputs model E5 stored in model storage unit 235 to learned model storage device 400 as learned model E3.

[0094] [Flowchart of the learning device] 12 is a flowchart of the learning device 200. In step S52, learning data E, which is a combination of learning input data E1 and a first learning output symbol E2, is acquired. Next, in step S54, the learning device 200 extracts a feature E4 from the learning input data E1.

[0095] Next, in step S56, the learning device 200 applies the feature E4 to the model E5 to obtain an output symbol E6. Next, in step S58, the learning device 200 performs preprocessing on the first training output symbol E2 to generate a second training output symbol E7 (see FIG. 11).

[0096] Next, in step 60, the learning device 200 updates the model E5 so as to reduce the error between the output symbol E6 and the first learning output symbol E2.

[0097] Next, learning device 200 determines whether a termination condition is met. The termination condition includes, for example, a condition that a predetermined amount of learning data E has been processed. If the termination condition is met (YES in step S62), that is, that learning device 200 has processed a predetermined amount of learning data E, the process proceeds to step S64. On the other hand, if the termination condition is not met (NO in step S62), the process returns to step S52. In step S64, learning device 200 outputs model E5 updated in step S60 as trained model E3.

[0098] At least one of the information processing device 100 and the learning device 200 may be implemented by an on-premise computer or may be located in the cloud. The information processing device 100 and the learning device 200 may be located on the same computer. Alternatively, the information processing device 100 and the learning device 200 may be located on different computers. When such a configuration is adopted, the learning device 200 may output the trained model to the trained model storage device 400 via a network.

[0099] [Summary] (1) For example, a user may want an information processing device to preferentially output a specific symbol desired by the user as an output symbol. In this case, an information processing device employing an end-to-end model has a problem in that it is difficult to customize the device by partially modifying it to additionally register vocabulary, because the device converts input data into an output symbol sequence using a single neural network.

[0100] Furthermore, in the technology disclosed in JP2022-531615A, a random permutation process is performed, and therefore, a specific symbol is often not generated in the permutation process. In this case, a problem may arise in that it is difficult to output the specific symbol as an output symbol. In other words, in conventional information processing devices, a problem may arise in that it is difficult to preferentially output a specific symbol input by a user as an output symbol.

[0101] Therefore, in the present disclosure, the information processing device 100 estimates at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol from input data. Next, the information processing device 100 performs an increase process to increase the likelihood of a candidate symbol including a registered symbol based on a bias module. This allows the registered symbol (or a candidate symbol including the registered symbol) to be preferentially determined as a provisional output symbol. Then, when the provisional output symbols include a registered symbol, the information processing device 100 performs a conversion process to convert the registered symbol into a specific symbol corresponding to the registered symbol by referring to a combination table, and outputs the provisional output symbol resulting from the conversion process as an output symbol.

[0102] Therefore, the user can easily customize the above by inputting a specific symbol. Also, by executing an increase process for preferentially outputting registered symbols and a replacement process for changing registered symbols to specific symbols, the specific symbols can be preferentially output as output symbols.

[0103] If it is desired to preferentially output specific symbols as output symbols, a configuration can be considered in which the trained model D3 is updated using training data that includes many specific symbols. However, if such a configuration is adopted, there is a risk that the information processing device 100 will preferentially output specific symbols that are not often used, such as proper nouns. In the present embodiment, such a configuration is not adopted, and the information processing device 100 can be made to preferentially output specific symbols desired by the user as output symbols.

[0104] (2) Furthermore, as preprocessing, the learning device 200 performs preprocessing to generate a first symbol included in the first learning output symbol and a second symbol in which the first symbol is represented by an expression specified by the user (FIG. 11). Therefore, the learning device 200 can increase the amount of learning data including the second symbol. Therefore, the learning device 200 updates the model E3 so that the likelihood of the second symbol (the symbol corresponding to the registered symbol) increases. Therefore, the learning device 200 can update the model so that the likelihood of the second symbol in which the first symbol is represented by an expression type specified by the user increases.

[0105] Embodiment 2 In the second embodiment, various other embodiments of the above configuration will be described.

[0106] [symbol] In the above embodiment 1, the symbols are mainly in English. However, the symbols may be expressed in other languages, such as Japanese.

[0107] Fig. 13 shows an example of the specific symbol table D2 expressed in Japanese. In the example of Fig. 13, proper nouns written in kanji are shown as specific symbols. Also, as alternative symbols, a first symbol expressed in katakana and a second symbol expressed in hiragana are shown.

[0108] Fig. 14 is an example of a combination table D7 expressed in Japanese. In the example of Fig. 14, proper nouns written in kanji are shown as specific symbols. Also, symbols written in katakana are shown as registered symbols.

[0109] Fig. 15 is an example of the bias table D8 expressed in Japanese. In the example of Fig. 15, a bias value is defined for each of three registered symbols.

[0110] The flow of processing by the information processing device 100 when such a configuration is adopted will be described. It is assumed that the specific symbol input by the user is "Choshi," a proper noun that is a place name. In the following, a configuration in which the speech is "I went to Choshi" will be described. In this configuration, it is preferable that the symbol "I went to Choshi" be output as the output symbol.

[0111] Assume that the estimation unit 140 has identified the following first and second candidate symbols as candidate symbols. The first candidate symbol is "I went into a good mood," and the score of the first candidate symbol is "0.0." "Morning" is a general noun that means the state of a verbal expression.

[0112] Also, the second candidate symbol is "I went to Choshi," and the score of the second candidate symbol is 1.0. In this example, the first candidate symbol has a smaller score (higher likelihood) than the second candidate symbol.

[0113] In this example, if the bias table D8 in FIG. 15 does not exist, the estimation unit determines the first candidate symbol as the tentative output symbol.

[0114] On the other hand, in this example, when the bias table D8 in Fig. 15 is present, the second candidate symbol is registered as a registered symbol as shown in Fig. 14. Therefore, the estimation unit 140 executes an increase process to increase the likelihood of the candidate symbol (second candidate symbol) based on the bias value (-3.5) corresponding to the registered symbol. In this example, this increase process is a process of calculating a score of -2.5 by executing a process of adding the bias value to the score.

[0115] In this way, the estimation unit 140 performs an increasing process of increasing the likelihood of a candidate symbol that includes a registered symbol among at least one candidate symbol, based on the bias value corresponding to the registered symbol.

[0116] Then, a provisional output symbol D9 is determined based on the likelihood of each of at least one candidate symbol after the increase process has been performed. In the above example, after the increase process, the score of the first candidate symbol is "0.0" and the score of the second candidate symbol is "-2.5". Therefore, the estimation unit 140 determines the second candidate symbol as the provisional output symbol D9. This provisional output symbol D9 is input to the conversion unit 150.

[0117] When the provisional output symbol D9 includes a registered symbol, the conversion unit 150 refers to the combination table D7 (FIG. 6). Then, the conversion unit 150 executes a conversion process to convert the registered symbol into a specific symbol corresponding to the registered symbol. Then, the conversion unit 150 outputs the provisional output symbol D9 that has been subjected to the conversion process as the output symbol D4.

[0118] In the above example, the provisional output symbol D9 includes "I went to Choshi." Therefore, the conversion unit 150 refers to the combination table D7 (FIG. 6) and converts "I went to Choshi" to "I went to Choshi," which includes the specific symbol. Then, the converted symbol "I went to Choshi" is output as the output symbol D4. As described above, even if the symbols are in another language, the information processing device 100 can appropriately output the specific symbol as the output symbol with priority.

[0119] Input Data and Output Symbology In the above embodiment, an example has been described in which the information processing device 100 functions as a speech recognition device. That is, an example has been described in which the input data is speech data and the output symbols are text of speech represented by the speech data. However, the input data and output symbols may be other examples. Figure 16 is a diagram showing such other examples.

[0120] In the example of Fig. 16, when the input data is speech data, the output symbols may be summary data. The summary data is a text summarizing the speech text represented by the speech data. In such a configuration, the information processing device functions as a speech summarizing device.

[0121] Furthermore, the input data may be any of sound data (e.g., sound effect data), still image data, and video data. In this case, the output symbol is explanatory text. The explanatory text is text (e.g., caption) that describes the object. The object is represented by sound using sound data, a still image using still image data, and a video using video data. In such a configuration, the information processing device functions as a captioning device.

[0122] As shown in FIG. 16, the information processing device 100 of the present disclosure can accept a variety of input data and output symbols.

[0123] [Bias value determination method] In the above embodiment, the generator 130 determines the bias value using an N-gram language model trained using training output symbols. However, the bias value may be determined by other methods.

[0124] For example, all bias values ​​of a plurality of registered symbols may be the same, as shown in parentheses in the bias value column of Fig. 15. In the example shown in parentheses in Fig. 15, all bias values ​​are -2.0. With this configuration, the bias values ​​used in the above-described increase process are all the same, which simplifies the increase process.

[0125] Furthermore, the generating unit 130 may determine a bias value that increases the likelihood to a greater extent as the number of characters in the registered symbol increases. In the above embodiment, the bias value that increases the likelihood to a greater extent is a negative number with a large absolute value.

[0126] Furthermore, the decision model 305 may be a machine-learned model that receives an input of a registered symbol and outputs a bias value.

[0127] A configuration in which the user determines the bias value may also be employed. The information processing device 100 employing such a configuration may be configured to input the bias value using the input device 4 (see FIG. 2). When such a configuration is employed, the bias value may be input (set) for a registered symbol already stored in the information processing device 100. The user may also input information associating the registered symbol with the bias value using the input device 4 (see FIG. 2).

[0128] [Preprocessing] In FIG. 11, a configuration has been described which is a pre-processing in which a first symbol is randomly selected from a plurality of unit symbols determined by a predetermined calculation (such as morphological analysis).

[0129] However, the preprocessing may be any other process as long as it increases the likelihood of candidate symbols including the registered symbol. For example, the preprocessing may be such that, from among the divided unit symbols, a unit symbol determined to be a noun or a proper noun is selected as the first symbol. The reason for this is that, in general, it is expected that many of the unit symbols (specific symbols) that users wish to register are difficult-to-read nouns or proper nouns.

[0130] In the example of FIG. 11, the unit symbol "i triple e" (proper noun only) is selected as the first symbol from among the first training output symbols, and the proper noun is converted into the second training output symbol.

[0131] Furthermore, the preprocessing identifies the number contained in all of the first learning output symbols E2 for each of the multiple unit symbols. Here, the first learning output symbols E2 are at least one of the symbols after they have been used in the learning device 200 and the symbols before they are used in the learning device 200. The reason for this is that, in general, the unit symbols (specific symbols) that a user wants to register are often unit symbols that are contained in a small number of the first learning output symbols E2.

[0132] In the example of Figure 11, the number of unit symbols "i triple e" contained in the first learning output symbol E2 is less than (smaller than) a predetermined value, so this unit symbol is selected as the first symbol, and the first symbol is converted into the second learning output symbol.

[0133] [Information attached to the symbol] The determining unit 120 may assign a predetermined information symbol to the symbol (registered symbol). The information symbol is a symbol having a predetermined meaning. The information symbol includes a start symbol indicating the start of the registered symbols and an end symbol indicating the start of the registered symbols. Start symbol' <bob>' is an abbreviation for beginning of biasing, and means the start of the registered symbol. It also means the end symbol. <eob>' is an abbreviation for end of biasing, and indicates the end of the registered symbols. In this way, the determining unit 120 can clearly identify the registered symbols by assigning the start symbol and the end symbol to the registered symbols.

[0134] Furthermore, the pre-processing unit 240 may assign a probability symbol to the first symbol. The probability symbol is a symbol indicating the probability (proportion) that the first symbol will be replaced with the second symbol.

[0135] 17 is a diagram showing an example of probability symbols 80. In FIG. 17, for the first symbol, <r20>An example is shown in which the probability symbol 80 is assigned. <r20>indicates that the first symbol is replaced with the second symbol with a probability of 20%. That is, in the example of Fig. 17, the first symbol is replaced with the second symbol with a probability of 1 in 5. Note that the probability symbol 80 may be specified by the user, or may be determined by the learning device 200 through a predetermined calculation.

[0136] [Method for determining registration symbols] (1) There may also be multiple specific symbols (for example, the case of FIG. 13). In this case, the determination unit 120 may use a different method for determining the registered symbol for each specific symbol, taking into consideration the performance of the registration process for the registered symbols. For example, with respect to a first specific symbol among the multiple specific symbols, the determination unit 120 may determine the first specific symbol itself as the registered symbol. Furthermore, with respect to a second specific symbol among the multiple specific symbols, the determination unit 120 may determine a symbol different from the second specific symbol as the registered symbol.

[0137] (2) In the case of Japanese speech recognition, in addition to "Choshi," there are other words pronounced as "choushi," such as "Choushi." In this situation, if the determination unit 120 determines "Choushi" as the registered symbol of "Choshi," the estimation unit 140 will output all symbols that should be output as "Choushi" as provisional output symbols D9 as "Choushi." In this case, after the conversion process is performed by the conversion unit 150, many of the places where "Choushi" should be output will end up as "Choushi." Even if "Choshi" is registered as the registered symbol, if the estimation unit 140 can output "Choshi" preferentially, it may determine "Choshi" as the registered symbol instead of "Choushi." The registered symbol may be determined in a way that reduces the number of cases where "Choushi" is mistakenly output as "Choushi."

[0138] (3) The method of determining the registered symbol D6 by the determination unit 120 (information processing device 100) is not limited to the above embodiment, and other methods may be used. FIG. 18 is an example of a flowchart of a first determination method of the registered symbol D6. In the example of FIG. 18, a plurality of training output symbols (e.g., first training output symbol E2) are used. The training output symbol is at least one of a symbol after it has been used in the learning device 200 and a symbol before it is used in the learning device 200.

[0139] In step S102, the determination unit 120 obtains a plurality of training output symbols. Next, in step S104, the determination unit 120 determines the number of specific symbols included in the plurality of training output symbols, and determines whether the number of specific symbols is equal to or greater than a predetermined value. If the number of specific symbols is equal to or greater than the predetermined value, the process proceeds to step S106. If the number of specific symbols is less than the predetermined value, the process proceeds to step S108.

[0140] In step S106, the determination unit 120 determines the specific symbol as the registered symbol. For example, in the example of FIG. 5, the specific symbol "IEEE" itself is stored as the registered symbol D6. Then, the processing of FIG. 18 ends. On the other hand, in step S106, the determination unit 120 determines a symbol expressed in a different way from the specific symbol as the registered symbol. For example, in the example of FIG. 5, the "IEEE phoneme symbol" corresponding to the specific symbol "IEEE" is stored as the registered symbol D6.

[0141] Next, the significance of the flowchart in Fig. 18 will be explained. When a specific symbol included in a plurality of learning output symbols is equal to or greater than a predetermined value (YES in step S104 in Fig. 18), the likelihood of candidate symbols including the specific symbol tends to be high. In such a case, the specific symbol can be determined as a registered symbol, so that the specific symbol can be preferentially output as a provisional output symbol. Note that in this case, since the specific symbol has been output as a provisional output symbol, the above-described replacement process is not executed.

[0142] On the other hand, when the specific symbol included in the plurality of learning output symbols is less than the predetermined value (NO in step S104 in FIG. 18), the likelihood of the candidate symbols including the specific symbol tends to be low. In such a case, by determining a symbol expressed differently from the specific symbol as the registered symbol, the registered symbol can be preferentially output as the provisional output symbol.

[0143] (4) Furthermore, as shown in parentheses in Fig. 18, the determination unit 120 may use a plurality of sample output symbols instead of a plurality (N: N is an integer equal to or greater than 2) of training output symbols. For example, the user inputs N pieces of input data (sample input data) to the information processing device 100, and causes the information processing device 100 to output N sample output symbols as output symbols. In the configuration shown in parentheses in Fig. 18, the determination unit 120 uses the N sample output symbols.

[0144] Next, the significance of the configuration in parentheses in Fig. 18 will be explained. When a specific symbol included in a plurality of sample output symbols is equal to or greater than a predetermined value (YES in step S104 in Fig. 18), the likelihood of candidate symbols including the specific symbol tends to be high. In such a case, the specific symbol can be determined as a registered symbol, so that the specific symbol can be preferentially output as a provisional output symbol. Note that in this case, since the specific symbol has been output as a provisional output symbol, the above-described replacement process is not executed.

[0145] On the other hand, if the specific symbol included in the plurality of sample output symbols is less than the predetermined value (NO in step S104 in FIG. 18), the likelihood of the candidate symbols including the specific symbol tends to be low. In such a case, by determining a symbol expressed differently from the specific symbol as the registered symbol, the registered symbol can be preferentially output as the provisional output symbol.

[0146] 18, it is possible to determine the registered symbol D6 so that the frequency with which it is output as the provisional output symbol D9 is increased as much as possible for each of the specific symbols included in the specific symbol table D2. Also, it is possible to reduce the number of specific symbols that are erroneously output as the output symbol for each of the specific symbols included in the specific symbol table D2.

[0147] [About output symbols] For example, as shown in Fig. 18, when the method for determining the registered symbol D6 is changed, the output symbol D4 may be changed in accordance with the change. Therefore, when the method for determining the registered symbol is changed, the information processing device 100 may determine a plurality of output symbols D4, select the best output symbol from the plurality of output symbols D4, and output the best output symbol.

[0148] [About combination tables and bias tables] In the above embodiment, an example has been described in which the information processing device 100 itself generates the combination table and the bias table. However, at least one of the combination table and the bias table may be generated by another device.

[0149] [Note] The bracketed text below is merely an example for reference purposes and is not intended to be limiting.

[0150] (1) An information processing device disclosed herein converts input data (input data D1) into output symbols (output symbols D4) and outputs the converted data. The information processing device includes an interface that accepts user input of input data, a memory that stores a trained model, and a processor. The processor extracts features (features E5) from the input data. The processor applies the features to the trained model to estimate at least one candidate symbol ("i triple e" and "IEEE phoneme symbol") and a likelihood representing the likelihood of each of the at least one candidate symbol. The memory stores a combination table (combination table D7 in FIG. 6) and a bias module (bias table D8 in FIG. 7). The combination table is a table that indicates combinations of specific symbols designated by a user and their corresponding registered symbols. The bias module is a module that increases the likelihood of candidate symbols that include the registered symbol. The processor further performs an increase process that uses the bias module to increase the likelihood of candidate symbols that include the registered symbol among the at least one candidate symbol. The processor determines a provisional output symbol (provisional output symbol D9) based on the likelihood of each of at least one candidate symbol after the increase process has been performed. Then, when the provisional output symbols include a registered symbol, the processor refers to the combination table and performs a conversion process to convert the registered symbol into a specific symbol corresponding to the registered symbol, and outputs the provisional output symbol that has been subjected to the conversion process as an output symbol (output symbol D4).

[0151] For example, there may be a case where a user preferentially wants a symbol (the specific symbol) to be output from an information processing device as an output symbol. In this case, an information processing device employing an end-to-end model has a problem in that it is difficult to customize the device by adding and registering vocabulary through partial changes, because the conversion from input data to an output symbol sequence is realized by a single neural network.

[0152] Furthermore, in the technology disclosed in JP2022-531615A, the specific symbol registration is often not generated by the above-described replacement process. In such cases, a problem may arise in that it is difficult to output the specific symbol as an output symbol. In other words, in conventional information processing devices, a problem may arise in that it is difficult to preferentially output the specific symbol input by a user as an output symbol.

[0153] Therefore, in the present disclosure, an information processing device estimates at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol from input data. Next, the information processing device performs an increase process to increase the likelihood of a candidate symbol including a registered symbol based on a bias module. This allows the registered symbol (or a symbol including the registered symbol) to be preferentially determined as a provisional output symbol. Then, if the provisional output symbols include a registered symbol, the information processing device performs a conversion process to convert the registered symbol into a specific symbol corresponding to the registered symbol by referring to a combination table, and outputs the provisional output symbol resulting from the conversion process as an output symbol.

[0154] Therefore, by inputting the specific symbol, the above-mentioned customization can be easily performed. Also, by executing an increase process for giving priority to outputting registered symbols and executing a replacement process for changing registered symbols to specific symbols, the specific symbols can be given priority to outputting as output symbols.

[0155] (2) Preferably, the bias module is a bias table (FIG. 7) in which registered symbols are associated with bias values ​​that increase the likelihood of candidate symbols including the registered symbols.

[0156] According to this configuration, the bias value corresponding to the registered symbol is reflected in the likelihood, thereby making it possible to perform the increase process.

[0157] (3) Preferably, the learned model is trained based on a plurality of learning data, each of which is a combination of a learning feature and a learning registered symbol. The memory stores a decision model (decision model 305) that outputs a bias value in response to an input registered symbol. The decision model outputs a bias value that increases the likelihood of a candidate symbol including a learning registered symbol as the number of input registered symbols included in the learning registered symbol decreases. The processor generates a bias table by associating the bias value output from the decision model with a registered symbol identical to the learning registered symbol.

[0158] According to this configuration, the smaller the number of registered symbols for training, the greater the likelihood of candidate symbols including the registered symbols is.The processor then generates a bias table by associating the bias values ​​output from the determination model with registered symbols identical to the registered symbols for training.Therefore, even if the likelihood of a candidate symbol including a registered symbol is low, the likelihood can be increased by the increasing process.

[0159] (4) Preferably, the processor generates the bias table based on user input (input device 4).

[0160] With this configuration, it is possible to generate a bias table in which bias values ​​according to the user's desire are defined.

[0161] (5) Preferably, the bias table defines the same bias value for each of a plurality of registered symbols (see the parentheses in FIG. 15).

[0162] With this configuration, the bias values ​​to be applied are all the same, so the increase process can be simplified.

[0163] (6) Preferably, the processor generates a bias module (processing of the generator 130 in FIG. 4).

[0164] With this configuration, the information processing device itself can generate the bias module. (7) Preferably, the interface accepts a user input of a specific symbol, and the processor generates a combination table indicating the specific symbol input to the interface and a combination of the specific symbol and a registered symbol corresponding to the specific symbol (processing of the determination unit 120).

[0165] According to this configuration, the information processing device itself can generate the combination table.

[0166] (8) Preferably, the interface acquires a plurality of sample output symbols output by applying a plurality of sample input data to the information processing device (see parentheses in FIG. 18). If the number of specific symbols included in the plurality of sample output symbols is equal to or greater than a predetermined value (YES in step S104), the processor determines the specific symbols as registered symbols. If the number of specific symbols included in the plurality of sample output symbols is less than a predetermined value (NO in step S104), the processor determines a symbol represented by a different expression from the specific symbols as a registered symbol.

[0167] According to this configuration, when the specific symbol included in the plurality of sample output symbols is equal to or greater than a predetermined value, the likelihood of the candidate symbols including the specific symbol tends to be high. In such a case, the specific symbol can be determined as a registered symbol, so that the specific symbol can be preferentially output as a provisional output symbol. Note that in this case, since the specific symbol has been output as a provisional output symbol, the above-described replacement process is not executed.

[0168] On the other hand, when the number of specific symbols included in the plurality of sample output symbols is less than a predetermined value, the likelihood of candidate symbols including the specific symbol tends to be low. In such a case, by determining a symbol expressed differently from the specific symbol as the registered symbol, the registered symbol can be preferentially output as the provisional output symbol.

[0169] (9) Preferably, the trained model is a model trained based on a plurality of training data, which are combinations of training input data and training output symbols. The interface acquires a plurality of training output symbols (FIG. 18). If the number of specific symbols included in the plurality of training output symbols is equal to or greater than a predetermined value (YES in step S104), the processor determines the specific symbols as registered symbols. If the number of specific symbols included in the plurality of training output symbols is less than a predetermined value (NO in step S104), the processor determines a symbol represented by a different expression from the specific symbol as a registered symbol.

[0170] According to this configuration, when a specific symbol included in a plurality of learning output symbols has a value equal to or greater than a predetermined value, the likelihood of a candidate symbol including the specific symbol tends to be high. In such a case, the specific symbol can be determined as a registered symbol, so that the specific symbol can be preferentially output as a provisional output symbol. Note that in this case, since the specific symbol has been output as a provisional output symbol, the above-described replacement process is not executed.

[0171] On the other hand, when the number of specific symbols included in the plurality of learning output symbols is less than a predetermined value, the likelihood of candidate symbols including the specific symbol tends to be low. In such a case, by determining a symbol expressed differently from the specific symbol as the registered symbol, the registered symbol can be preferentially output as the provisional output symbol.

[0172] (10) Preferably, the increasing process is a process that does not increase the likelihood of a candidate symbol that does not include a registered symbol among at least one candidate symbol (see the "Other" column in FIG. 15).

[0173] According to this configuration, it is possible to prevent an increase in the likelihood of candidate symbols that do not include a registered symbol, and as a result, it is possible to preferentially determine candidate symbols that include a registered symbol as provisional output symbols.

[0174] (11) Preferably, the registered symbol is either identical to the specific symbol, has a different expression from the specific symbol, or has the same pronunciation as the specific symbol but a different expression (see Figures 6 and 14).

[0175] According to this configuration, it is possible to define a registered symbol that is one of the above.

[0176] (12) Preferably, the specific symbol is expressed in one of Hiragana, Katakana, Roman letters, English, and International Phonetic Alphabet, and the registered symbol is expressed in the other one (see Figures 6 and 14).

[0177] With this configuration, the specific symbols and registered symbols can be expressed in hiragana, katakana, Roman letters, English, and the International Phonetic Alphabet.

[0178] (13) Preferably, the input data is audio data, and the output symbols are text of the audio represented by the audio data or text summarizing the content of the audio represented by the audio data (see FIG. 16).

[0179] According to this configuration, it is possible to output a text of the speech or a text summarizing the content of the speech.

[0180] (14) Preferably, the input data is any one of sound data, still image data, and video data, and the output symbol is text that indicates a description of the object represented by the input data (see FIG. 16).

[0181] According to this configuration, it is possible to output text that shows an explanation of the object indicated by the input data.

[0182] (15) The learning device of the present disclosure is a learning device for updating a model. The learning device includes an interface that acquires learning data, which is a combination of learning input data (learning input data E1) and first learning output symbols (first learning output symbols E2), and a calculation device. The calculation device extracts features (feature E4) from the learning input data. The calculation device acquires output symbols (output symbols E6) by applying the features to a model. The calculation device generates second learning output symbols by performing preprocessing (FIG. 11) on the first learning output symbols. The calculation device updates the model to reduce the error between the output symbols and the second learning output symbols. The preprocessing is a process of generating a first symbol included in the first learning output symbols and a second symbol in which the first symbol is represented by an expression specified by a user (FIG. 11).

[0183] According to this configuration, as preprocessing, a process is executed to generate a first symbol included in the first training output symbols and a second symbol in which the first symbol is represented by an expression specified by a user. Furthermore, the calculation device updates the model to increase the likelihood of the second symbol, and therefore, can update the model to increase the likelihood of the second symbol in which the first symbol is represented by an expression specified by a user.

[0184] (16) Preferably, the second symbol is either identical to the first symbol, or is expressed differently from the specific symbol, or is pronounced the same as the specific symbol but is expressed differently (FIG. 11).

[0185] According to this configuration, it is possible to define the second symbol as one of the above.

[0186] (17) Preferably, the first training output symbols are text. The preprocessing involves dividing the text into unit symbols by performing a predetermined process on the text, and selecting the unit symbols as the first symbols (FIG. 11).

[0187] According to this configuration, when the first training output symbols are text, words included in the text can be selected as the first symbols.

[0188] (18) Preferably, the unit symbol is a noun or a proper noun (FIG. 11). With this configuration, the model can be updated to increase the likelihood of the second symbol corresponding to the first symbol being a noun or a proper noun.

[0189] (19) Preferably, in the pre-processing, if the number of unit symbols included in the first learning output symbol is less than a predetermined value, the unit symbol is selected as the first symbol (FIG. 11).

[0190] According to this configuration, even if the number of symbols included in the first learning output symbols is small, the symbols can be selected as the first symbols, and the likelihood of the second symbols corresponding to the first symbols can be increased.

[0191] (20) Preferably, the pre-processing adds information to the first symbol indicating the probability that the first symbol is transformed into the second symbol (see FIG. 17).

[0192] With this configuration, it is possible to control the probability that the first symbol is converted into the second symbol.

[0193] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the description of the above embodiments, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0194] 2 microphone, 3 display, 4 input device, 11 calculation unit, 12 memory unit, 13 voice interface, 14 communication unit, 15 display interface, 16 input device interface, 20 removable disk, 30 information processing system, 80 probability symbol, 100 information processing device, 110, 220 extraction unit, 120 determination unit, 130 generation unit, 140 estimation unit, 150 conversion unit, 200 learning device, 210 acquisition unit, 230 processing unit, 235 model storage unit, 240 preprocessing unit, 250 update unit, 260 output unit, 300 learning data storage device, 301 processing program, 303 neural network, 304 parameters, 305 decision model, 400 learned model storage device. < / eob> < / bob>

Claims

1. An information processing device that converts input data into output symbols and outputs the symbols, an interface for accepting user input of said input data; a memory for storing the trained model; a processor; The processor: extracting features from the input data; applying the feature quantity to the trained model to estimate at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol; The memory stores a combination table and a bias module; the combination table is a table showing combinations of specific symbols designated by a user and registered symbols corresponding to the specific symbols; The bias module is a module that increases the likelihood of a candidate symbol including the registered symbol; The processor further comprises: performing an increasing process using the bias module to increase the likelihood of a candidate symbol including the registered symbol among the at least one candidate symbol; determining a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increasing process has been performed; When the provisional output symbols include the registered symbols, the information processing device refers to the combination table, performs a conversion process to convert the registered symbols into the specific symbols corresponding to the registered symbols, and outputs the provisional output symbols after the conversion process as the output symbols.

2. 2. The information processing apparatus according to claim 1, wherein the bias module is a bias table in which bias values ​​that increase the likelihood of candidate symbols including the registered symbols are defined in association with the registered symbols.

3. The learned model is trained based on a plurality of learning data, which are combinations of learning features and learning registered symbols; the memory stores a decision model that outputs a bias value when the enrollment symbol is input; the decision model outputs a bias value that increases the likelihood of a candidate symbol including the registered symbol as the number of the registered symbols included in the training symbols decreases; The information processing apparatus according to claim 2 , wherein the processor generates the bias table by associating the bias values ​​output from the decision model with the registered symbols that are the same as the registered symbols for learning.

4. The information processing apparatus according to claim 2 , wherein the processor generates the bias table based on a user input.

5. The information processing device according to claim 2 , wherein the bias table defines the same bias value for each of the plurality of registered symbols.

6. The information processing device according to claim 1 , wherein the processor generates the bias module.

7. the interface accepts user input of the specific symbol; The information processing apparatus according to claim 1 , wherein the processor generates, as the combination table, a table showing combinations of the specific symbols input to the interface and the registered symbols corresponding to the specific symbols.

8. the interface obtains a plurality of sample output symbols output by applying a plurality of sample input data to the information processing device; The processor: If the number of the specific symbols included in the plurality of sample output symbols is equal to or greater than a predetermined value, the specific symbols are determined to be the registered symbols; 2 . The information processing apparatus according to claim 1 , wherein, when the number of the specific symbols included in the plurality of sample output symbols is less than the predetermined value, a symbol expressed differently from the specific symbols is determined as the registered symbol.

9. the trained model is a model trained based on a plurality of training data which are combinations of training input data and training output symbols, the interface receives the plurality of training output symbols; The processor: If the number of the specific symbols included in the plurality of learning output symbols is equal to or greater than a predetermined value, the specific symbols are determined to be the registered symbols; 2. The information processing apparatus according to claim 1, wherein, when the number of the specific symbols included in the plurality of learning output symbols is less than the predetermined value, a symbol expressed differently from the specific symbols is determined as the registered symbol.

10. The information processing device according to claim 1 , wherein the increasing process does not increase the likelihood of a candidate symbol that does not include the registered symbol among the at least one candidate symbol.

11. The information processing device according to claim 1 , wherein the registered symbol is either the same as the specific symbol, a different expression from the specific symbol, or a symbol that has the same pronunciation as the specific symbol but a different expression.

12. The information processing device according to claim 10 , wherein the specific symbol is expressed in one of hiragana, katakana, Roman letters, English, and the international phonetic alphabet, and the registered symbol is expressed in another one of them.

13. the input data is audio data, The information processing apparatus according to claim 1 , wherein the output symbols are text of the speech represented by the speech data or text summarizing the content of the speech represented by the speech data.

14. the input data is any one of sound data, still image data, and video data; The information processing apparatus according to claim 1 , wherein the output symbol is text indicating a description of an object indicated by the input data.

15. A learning device for updating a model, an interface for acquiring training data that is a combination of training input data and first training output symbols that are text; a processor; The processor: extracting features from the learning input data; applying the features to the model to obtain output symbols; generating second training output symbols by performing preprocessing on the first training output symbols; updating the model so as to reduce an error between the output symbol and a second learning output symbol; The pretreatment is generating a first symbol included in the first training output symbols and a second symbol representing the first symbol in an expression designated by a user; A learning device that performs a predetermined process on the text to divide it into unit symbols and selects the unit symbols as the first symbols.

16. A learning device for updating a model, comprising: an interface for acquiring training data that is a combination of training input data and first training output symbols; a processor; The processor: extracting features from the learning input data; applying the features to the model to obtain output symbols; generating second training output symbols by performing preprocessing on the first training output symbols; updating the model so as to reduce an error between the output symbol and a second learning output symbol; The pretreatment is generating a first symbol included in the first training output symbols and a second symbol representing the first symbol in an expression designated by a user; A learning device, wherein if the number of unit symbols included in the first learning output symbol is less than a predetermined value, the unit symbol is selected as the first symbol.

17. A learning device for updating a model, comprising: an interface for acquiring training data that is a combination of training input data and first training output symbols; a processor; The processor: extracting features from the learning input data; applying the features to the model to obtain output symbols; generating second training output symbols by performing preprocessing on the first training output symbols; updating the model so as to reduce an error between the output symbol and a second learning output symbol; The pretreatment is generating a first symbol included in the first training output symbols and a second symbol representing the first symbol in an expression designated by a user; A learning device that performs processing to assign information indicating the probability that the first symbol will be converted into the second symbol to the first symbol.

18. 16. The learning device according to claim 15, wherein the second symbol is one of the following: the same as the first symbol; a specific symbol having a different expression; and a specific symbol having the same reading but a different expression.

19. The learning device according to claim 15 , wherein the unit symbol is a noun or a proper noun.

20. An information processing method for converting input data into output symbols and outputting the symbols, comprising: The bias module is a module that increases the likelihood of a candidate symbol containing a registered symbol. The combination table is a table showing combinations of specific symbols designated by a user and registered symbols corresponding to the specific symbols, extracting features from the input data; applying the feature quantity to a trained model to estimate at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol; performing an increasing process using the bias module to increase the likelihood of a candidate symbol including a registered symbol corresponding to a specific symbol designated by a user, among the at least one candidate symbol; performing an increasing process using the bias module to increase the likelihood of the candidate symbol including the registered symbol among the at least one candidate symbol; determining a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increasing process has been performed; and if the provisional output symbols include the registered symbol, performing a conversion process to convert the registered symbol into the specific symbol corresponding to the registered symbol by referring to the combination table, and outputting the provisional output symbol after the conversion process as the output symbol.

21. A learning method for updating a model, comprising: acquiring training data that is a combination of training input data and a first training output symbol that is text; extracting features from the learning input data; applying the features to the model to obtain output symbols; generating second training output symbols by performing preprocessing on the first training output symbols; updating the model so as to reduce an error between the output symbol and a second learning output symbol; The pretreatment is generating a first symbol included in the first training output symbols and a second symbol representing the first symbol in an expression designated by a user; A learning method that divides text into unit symbols by executing a predetermined process on the text, and selects the unit symbols as the first symbols.

22. A learning method for updating a model, comprising: acquiring training data that is a combination of training input data and a first training output symbol; extracting features from the learning input data; applying the features to the model to obtain output symbols; generating second training output symbols by performing preprocessing on the first training output symbols; updating the model so as to reduce an error between the output symbol and a second learning output symbol; The pretreatment is generating a first symbol included in the first training output symbols and a second symbol representing the first symbol in an expression designated by a user; A learning method comprising a process of selecting a unit symbol as a first symbol when the number of unit symbols included in the first learning output symbol is less than a predetermined value.

23. A learning method for updating a model, comprising: acquiring training data that is a combination of training input data and a first training output symbol; extracting features from the learning input data; applying the features to the model to obtain output symbols; generating second training output symbols by performing preprocessing on the first training output symbols; updating the model so as to reduce an error between the output symbol and a second learning output symbol; The pretreatment is generating a first symbol included in the first training output symbols and a second symbol representing the first symbol in an expression designated by a user; A learning method comprising a process of assigning information indicating a probability that the first symbol will be transformed into the second symbol to the first symbol.

24. A non-transitory recording medium storing a program for causing a computer to convert input data into output symbols and output the converted symbols, The bias module is a module that increases the likelihood of a candidate symbol containing a registered symbol. The combination table is a table showing combinations of specific symbols designated by a user and registered symbols corresponding to the specific symbols, The program causes the computer to: extracting features from the input data; applying the feature quantity to a trained model to estimate at least one candidate symbol and a likelihood representing the likelihood of each of the at least one candidate symbol; performing an increasing process using the bias module to increase the likelihood of a candidate symbol including a registered symbol corresponding to a specific symbol designated by a user, among the at least one candidate symbol; performing an increasing process using the bias module to increase the likelihood of the candidate symbol including the registered symbol among the at least one candidate symbol; determining a provisional output symbol based on the likelihood of each of the at least one candidate symbol after the increasing process has been performed; a non-transitory recording medium that, when the provisional output symbols include the registered symbols, performs a conversion process of converting the registered symbols into the specific symbols corresponding to the registered symbols by referring to the combination table, and outputs the provisional output symbols that have been converted as the output symbols.

25. A non-transitory recording medium storing a program for causing a computer to execute the learning method described in any one of claims 21 to 23.

Images