Server supporting device for voice recognition and method for operating the server

Abstract

A server and a method for operating the server for supporting speech recognition in a device. The server and method identify multiple estimated strings from a first string, obtain a second string based on the multiple estimated strings, and transmit the second string to the device. The first string is output via speech recognition of a speech signal input to the device.

Description

Technical Field

[0001] The present disclosure relates to a server for supporting a device to perform speech recognition and a method for operating the server. More specifically, the present disclosure relates to a method for enhancing speech recognition results by using server-side post-processing. Background Art

[0002] With the development of electronic devices that perform various complex functions, electronic devices that include voice recognition functions to improve the accessibility of devices have been released on the market. In the voice recognition function, the user's voice can be recognized without any special key operation or contact with a touch module, thereby making it easy to control the device.

[0003] According to this voice recognition function, for example, calls or message sending can be performed in portable terminals such as smartphones and home appliances such as televisions and refrigerators without special key operations, and various functions such as route finding, Internet search and alarm setting can be easily set in portable terminals and home appliances.

[0004] Recently, with the development of artificial intelligence (AI) technology, this artificial intelligence (AI) technology has even been associated with speech recognition, making it possible to quickly and accurately recognize various utterances.

[0005] On-device speech recognition that performs local automatic speech recognition (ASR) is applicable even in situations where latency is very low and there is no network connection. However, in server-based speech recognition, speech recognition is performed based on information stored in the server. Summary of the Invention

[0006] [Technical Issues]

[0007] A method for a device to selectively use device-side speech recognition and server-based speech recognition is provided.

[0008] [Technical solution]

[0009] According to an embodiment of the present disclosure, a server is provided. The server includes: a memory storing one or more computer-readable instructions; a processor configured to execute the one or more computer-readable instructions stored in the memory; and a communication interface configured to receive, from a device, a first string obtained by the device performing speech recognition on a speech signal input to the device. The processor is further configured to execute the one or more instructions to: identify an estimated string based on the first string, the estimated string being used to replace a portion of the first string; and control the communication interface to send a second string to the device, in which the portion of the first string is replaced with the estimated string. The first string is output from the speech signal input to the device via speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a diagram used to compare and explain device-side speech recognition and server-based speech recognition;

[0012] Figure 2a is a block diagram of a speech recognition system according to an embodiment of the present disclosure;

[0013] Figure 2b is a block diagram of a speech recognition system according to an embodiment of the present disclosure;

[0014] Figure 2c is a block diagram of a speech recognition system according to an embodiment of the present disclosure;

[0015] Figure 3 is a structural block diagram of a device according to an embodiment of the present disclosure;

[0016] Figure 4a is a detailed structural block diagram of a device according to an embodiment of the present disclosure;

[0017] Figure 4b is a detailed structural block diagram of a device according to an embodiment of the present disclosure;

[0018] Figure 5a is a diagram for explaining a method, performed by a device, for determining to perform device-side voice recognition according to an embodiment of the present disclosure;

[0019] Figure 5b is a diagram for explaining a method for determining to perform server-based speech recognition, performed by a device, according to an embodiment of the present disclosure;

[0020] Figure 6 is a diagram for explaining a frame synchronization string according to an embodiment of the present disclosure;

[0021] Figure 7 is a block diagram of a server according to an embodiment of the present disclosure;

[0022] Figure 8a is a diagram for explaining a method for supporting a device to perform speech recognition, which is executed by a server according to an embodiment of the present disclosure;

[0023] Figure 8b is a diagram for explaining a method, performed by a server, of determining a replacement character string by obtaining a likelihood of each character corresponding to each speech signal frame according to an embodiment of the present disclosure;

[0024] Figure 9is a detailed block diagram of a server according to an embodiment of the present disclosure;

[0025] Figure 10a The structure of an artificial intelligence recurrent neural network (RNN) for calculating posterior probability according to an embodiment of the present disclosure is shown;

[0026] Figure 10b An example of a confusion matrix for calculating likelihood according to an embodiment of the present disclosure is shown;

[0027] Figure 11a is a diagram for explaining a process performed by a server to calculate a likelihood matrix associated with a replacement character to be substituted for each character in a first character string received from a device according to an embodiment of the present disclosure;

[0028] Figure 11b is a diagram for explaining a process performed by a server to calculate a likelihood matrix associated with a replacement character to be substituted for each character in a first character string received from a device according to another embodiment of the present disclosure;

[0029] Figure 12 is a block diagram of an apparatus for selectively using two speech recognition modules according to an embodiment of the present disclosure;

[0030] Figure 13 is a flowchart of a method for performing speech recognition executed by a device according to an embodiment of the present disclosure;

[0031] Figure 14 is a detailed flowchart of a method for performing speech recognition executed by a device according to an embodiment of the present disclosure;

[0032] Figure 15 is a flow chart of a method for operating a server according to an embodiment of the present disclosure;

[0033] Figure 16 is a detailed flow chart of a method for operating a server according to an embodiment of the present disclosure;

[0034] Figure 17 is a diagram for explaining weighted finite state transducer (WFST) decoding performed by a service according to an embodiment of the present disclosure;

[0035] Figure 18 A screen showing a device for displaying a speech recognition result according to an embodiment of the present disclosure is shown; and

[0036] Figure 19 is a detailed block diagram of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the embodiments presented in the disclosure.

[0038] According to an embodiment of the present disclosure, a server is provided. The server includes: a memory storing one or more computer-readable instructions; a processor configured to execute the one or more computer-readable instructions stored in the memory; and a communication interface configured to receive, from a device, a first string obtained by the device performing speech recognition on a speech signal input to the device. The processor is further configured to execute the one or more instructions to: identify an estimated string based on the first string, the estimated string being used to replace a portion of the first string; and control the communication interface to send a second string to the device, in which the portion of the first string is replaced with the estimated string. The first string is output from the speech signal input to the device via speech recognition.

[0039] According to an embodiment of the present disclosure, a method for operating a server is provided. The method includes: receiving, from a device, a first string obtained by the device performing speech recognition on a speech signal input to the device; and sending, to the device, a second string in which a portion of the first string is replaced with an estimated string. The first string is output from the speech signal input to the device via speech recognition.

[0040] According to an embodiment of the present disclosure, a device is provided. The device includes: a memory storing one or more computer-readable instructions; a processor configured to execute the one or more instructions stored in the memory; and a communication interface configured to communicate with a server. The processor is further configured to execute the one or more instructions to: obtain a first string by performing speech recognition on a speech signal; determine whether to replace a portion of the first string with another string; based on the determination, control the communication interface to send the first string to the server; and control the communication interface to receive a second string from the server, the second string obtained by the server replacing a portion included in the first string with an estimated string.

[0041] According to an embodiment of the present disclosure, a method for operating a device is provided. The method includes: obtaining a first string by performing speech recognition on a speech signal; determining whether to replace a portion of the first string with another string; based on the determination, transmitting the first string to a server; and receiving a second string from the server, the second string obtained by the server replacing the portion included in the first string with an estimated string.

[0042] [Invention Mode]

[0043] Although currently widely used general terms are selected to describe the present disclosure in consideration of its functions, these general terms may vary depending on the intentions of those skilled in the art, case precedents, the emergence of new technologies, and the like. Terms arbitrarily selected by the applicant of the present disclosure may also be used in specific cases. In such cases, their meanings need to be specified in the detailed description. Therefore, the terms must be defined based on their meanings and the content of the entire specification, rather than simply stating the terms.

[0044] Throughout this disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0045] When the terms "comprises" and / or "comprising" or "includes" and / or "including" are used in this specification, the presence of the elements is specified, but the presence or addition of one or more other elements is not excluded. When the terms "unit", "device" and "module" are used in this specification, they refer to a unit that performs at least one function or operation, and can be implemented as hardware, software, or a combination of hardware and software.

[0046] In the present disclosure, the term "character" refers to a character that is used to write a human language in a visual form. Examples of characters may include Korean characters, alphabetic characters, Chinese characters, numbers, phonetic symbols, punctuation marks, and other symbols.

[0047] In this disclosure, the term "string" refers to a sequence of characters.

[0048] In the present disclosure, the term "phoneme" is the smallest unit representing a sound and is composed of at least one character. For example, in an alphabetic notation system, a character may be a phoneme. Accordingly, a character in the present disclosure may be referred to as a phoneme, and a string in the present disclosure may represent a sequence of phonemes. In the present disclosure, a string may also be referred to as text or a text string.

[0049] The term "morpheme" is the smallest meaningful unit composed of at least one phoneme. The term "word" is the smallest basic unit of language composed of at least one morpheme and can be used alone or to express a grammatical function. The term "phoneme" is the sound unit that distinguishes one word from another in human language.

[0050] According to the speech recognition model of the embodiment of the present disclosure, a speech signal may be converted into a character string and output the character string. The character string output by the speech recognition model according to the embodiment of the present disclosure may be a "frame-synchronized character string". The term "frame" may refer to a unit for processing a speech signal by dividing the speech signal at preset time intervals, or the divided speech signal itself. In the present disclosure, a "frame-synchronized character string" refers to a character string that includes characters corresponding to speech signal frames respectively when a speech signal is converted into a character string by a speech recognition model and a character string is generated as an output.

[0051] For example, the speech recognition model may receive a speech signal corresponding to "baseball" uttered by a user and may output a frame-synchronized string [b, b, a, a, a, s, s, e, b, b, a, a, l].

[0052] In this disclosure, when a speech recognition model generates and outputs a string of characters from a speech signal, a "confidence score for a string of characters" refers to the accuracy of the string of characters output by the speech recognition model. For example, the confidence score for a string of characters can be calculated using a predetermined equation based on, for example, a likelihood obtained from the string of characters, a partial likelihood output when estimating the string of characters, or a posterior probability value. As the confidence score for a string of characters increases, it can be determined whether the string of characters has been accurately estimated by the speech recognition model.

[0053] In the present disclosure, "evaluation information for a particular string" may refer to information about a particular string used by a server based on a recommendation, with the server outputting another string with a greater confidence score than the particular string. For example, the evaluation information for a particular string may include the likelihoods of multiple estimated strings obtained from the particular string. A server according to an embodiment of the present disclosure may select and output the string with the highest likelihood or confidence from among the multiple estimated strings.

[0054] In this disclosure, “likelihood” may refer to probability or possibility, and thus “likelihood of event B relative to event A” may refer to the conditional probability P(B|A) representing the likelihood of event B occurring when event A occurs.

[0055] In the present disclosure, when a speech recognition model generates and outputs a character string from a speech signal, "likelihood obtained from the character string" refers to the likelihoods of multiple estimated character strings estimated based on the character string. The multiple estimated character strings estimated based on the character string may refer to multiple character strings obtained by replacing at least one character within the character string with another character.

[0056] More specifically, a character string output when speech recognition is accurately performed is referred to as a ground truth character string, and a "likelihood obtained from a certain character string" may refer to the likelihood of a certain character string being estimated as a speech recognition result when each of a plurality of estimated character strings is assumed to be a ground truth character string. According to an embodiment of the present disclosure, the "likelihood obtained from a certain character string" may include a likelihood matrix associated with replacement characters that will replace each character within the certain character string.

[0057] According to an embodiment of the present disclosure, the likelihood obtained from a character string can be used to identify replacement characters that are phonetically similar to each character within the character string. Based on the identified replacement characters, an estimated character string is determined, in which at least one character within the character string is corrected to another character. Furthermore, the most appropriate estimated character string can be selected from the determined estimated character strings based on pre-stored information such as a language model and dictionary information. This can be a recommended character string rather than a specific character string.

[0058] When a speech recognition model performs speech recognition, the results of previously performed speech recognition may affect the results of subsequently performed speech recognition. When a character is mistakenly recognized as another character with a similar pronunciation, the language information is incorrectly determined due to the misrecognition, which may increase the probability that the character following the character will also be misrecognized. In other words, when a character is mistakenly recognized as another character, the word determined by combining the character with the subsequent character may become different from the word determined by combining the other misrecognized character with the subsequent character.

[0059] Therefore, the device or server according to an embodiment of the present disclosure may obtain a replacement character string by decoding the character string in consideration of pronunciation information and language information about the character string, using the likelihood obtained from the character string.

[0060] In the present disclosure, "the likelihood matrix obtained for a certain character" may refer to a likelihood value matrix including replacement characters to be used to replace a certain character. "The likelihood value of the replacement character to be used to replace a certain character" may refer to the probability that a certain character is estimated as a result of speech recognition when a certain character is assumed to be a basic true character. For example, for the character "a" included in the character string obtained through speech recognition, a likelihood matrix [0.4 0.01 0.010.01 0.2 ... 0.01] including the probability that the true character is "a", the probability that the true character is "b", the probability that the true character is "c" ... and the probability that the true character is "z" may be obtained. When a likelihood matrix including the likelihood values ​​of the replacement characters corresponding to each character included in the character string is obtained, a high likelihood value may be assigned to the replacement character that is similar in pronunciation to each character.

[0061] In the present disclosure, "likelihood obtained from a certain character string" can be obtained from the likelihood value associated with the replacement character corresponding to each character in the certain character string. The likelihood value associated with the replacement character corresponding to each character in the certain character string can be calculated by taking into account the characters accumulated before each character. However, the embodiments of the present disclosure are not limited thereto, and the likelihood value associated with the replacement character corresponding to each character in the certain character string can be calculated by taking into account only each character without taking into account the characters accumulated before each character.

[0062] According to an embodiment of the present disclosure, the "likelihood obtained from a certain string considering the characters accumulated before each character in the string" can be calculated based on the "posterior probability of each character included in a certain string" and the "character sequence probability" of a certain string.

[0063] The "posterior probability" of event A refers to the conditional probability of expecting event A when taking into account events, observations, or background knowledge related to event A.

[0064] In the present disclosure, when a speech recognition model generates a character string from a speech signal and outputs the character string, the "posterior probability of a character within the character string" may include the probability that the speech recognition model has accurately predicted the character when considering the characters before the character within the character string and the probability that the speech recognition model has incorrectly predicted the character as another character.

[0065] In the present disclosure, when a speech recognition model generates a character string from a speech signal and outputs the character string, "character sequence probability" may refer to the probability that the characters are arranged according to the character string.

[0066] According to an embodiment of the present disclosure, the "likelihood obtained from a character string by considering only each character within the character string" can be calculated by a "confusion matrix" that includes the probability that a character has been incorrectly predicted. In the present disclosure, the "confusion matrix" is also referred to as an error matrix, and when a speech recognition model converts a speech signal into a character string and outputs a character string, the confusion matrix includes the probability that the speech recognition model has accurately predicted a character included in the character string and the probability that the speech recognition model has incorrectly predicted a character as another character. For example, for a character that is similar in pronunciation to a character, the probability that the speech recognition model has incorrectly predicted a character that is similar in pronunciation to a character as a character can be weighted more heavily.

[0067] In the present disclosure, an "acoustic model" may refer to a model that includes information that determines which character or phonetic symbol a speech signal matches in units of phonemes. For example, a device according to an embodiment of the present disclosure may calculate the probability that each character matches a speech signal based on the acoustic model.

[0068] In the present disclosure, "dictionary information" may include mapping information between a plurality of words and characters included in each of the plurality of words. A "language model" may be an artificial intelligence (AI) model that has learned the relationship between words to estimate the probability of subsequent words when assigning a specific word string.

[0069] In this disclosure, "artificial neural network" is a general term for computing systems based on neural networks in the human or animal brain. An artificial neural network is a detailed method of machine learning and is therefore a network of multiple neurons, which are nerve cells. Artificial neural networks can be implemented using hardware, but are primarily implemented using computer software. An artificial neural network is a network of several neurons connected in weighted links, with each neuron being a basic computing unit. Weighted links can adjust weights to adapt to a given environment.

[0070] Artificial neural networks are a general term for various models such as self-organizing maps (SOMs), recurrent neural networks (RNNs), and convolutional neural networks (CNNs), and there are many types of artificial neural networks.

[0071] In this disclosure, a group of words related to a certain attribute is referred to as a domain of the certain attribute.

[0072] In the present disclosure, “an operation of correcting a first character string” may refer to an operation of recommending and outputting a second character string having a confidence score greater than that of the first character string by replacing at least one character included in the first character string with another character. Therefore, in the present disclosure, the expressions “correction of a character string,” “correction of a character,” “a certain character is replaced with another character,” “recommending another character instead of a certain character,” “a certain character string is replaced with another character string,” and “recommending another character string instead of a certain character string” may be used interchangeably.

[0073] The device or server included in the voice recognition system according to an embodiment of the present disclosure may be a voice assistant device and / or provide a "voice assistant service". The voice assistant service may be a service that provides a dialogue with the user. The voice assistant service may provide a response message to the user to imitate a person talking directly to the user, taking into account the user situation, the device situation, etc. The voice assistant service may appropriately generate the information required by the user and provide the information to the user, like a personal assistant for the user. The voice assistant service may be linked to access various services, such as broadcast services, content sharing services, content provision services, power management services, game provision services, chat services, document editing services, search services, call services, photo services, traffic recommendation services, and movie playback services, and may provide the user with necessary information or necessary functions.

[0074] This disclosure is described in detail with reference to the accompanying drawings so that the present disclosure can be easily implemented by those skilled in the art. However, the present disclosure can be implemented in many different forms and should not be construed as limited to the examples set forth herein.

[0075] Embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings.

[0076] Figure 1 A diagram comparing and explaining on-device speech recognition and server-based speech recognition.

[0077] Device-side speech recognition refers to local speech recognition performed by the device 100 on the utterance of the user 10, and server-based speech recognition refers to remote speech recognition performed by the server 200 on the utterance of the user 10 received by the device 100. That is, device-side speech recognition may not require the device 100 to be connected to the server 200, while server-based speech recognition may provide speech recognition using the device 100 and the server 200 in a client-server relationship.

[0078] With the development of end-to-end speech recognition and compression technology, device-side speech recognition technology has gradually developed, so the performance difference between device-side speech recognition and server-based speech recognition is gradually narrowing. Specifically, in speech recognition of open-domain speech that is not restricted to a specific domain or general dictation, the difference between device and server performance is negligible. General dictation refers to writing down speech that does not correspond to the domain-oriented named entity. Named entities may include specific place names, specific personal names, specific device names, specific trademark names, etc. When describing a domain, a group of words related to a certain attribute is called the domain of a certain attribute.

[0079] Device-based speech recognition offers latency of less than approximately 50 milliseconds, which may be significantly less than the hundreds of milliseconds of latency associated with server-based speech recognition. Therefore, device-based speech recognition may be more suitable for certain environments where device 100 operates, such as suburban areas, aircraft interiors, or shadowed areas where network accessibility is limited or unavailable. Device-based speech recognition offers advantages in terms of security and privacy concerns and can reduce server management costs.

[0080] Server-based speech recognition is implemented in a server that can store more named entities (such as place names, personal names, and brand names) than a device.

[0081] Therefore, according to server-based speech recognition, words related to new buzzwords or new song titles can be assigned higher weights, and when words are not recognized by speech, a hotfix operation can be performed to correct speech recognition defects by adding the words to the dictionary. Rescoring of speech recognition results can be performed by using, for example, a language model and dictionary information optimized for a third-party application running on the server.

[0082] Therefore, general speech recognition such as dictation, general commands, and subtitle generation is performed by the device, but when speech recognition needs to be performed by using, for example, a language model and dictionary information corresponding to a specific domain, hybrid speech recognition performed by the server is required.

[0083] At this time, when the entire speech recognition process is divided and executed by the device and the server, a dependency relationship may be generated between the device and the server.

[0084] For example, a method can be used in which a device performs calculations to apply an acoustic model to an utterance, and a server performs decoding calculations to apply a language model and dictionary information to intermediate values ​​extracted from the acoustic model. Because this method creates a dependency between the device and the server, it cannot be used between incompatible devices and servers.

[0085] For another example, in end-to-end speech recognition involving both encoding and decoding calculations, a method can be used in which only the device performs the encoding calculations, and the server performs the decoding calculations on the encoded data. Because the decoding calculations require dictionary information about the encoding method, a dependency is created between the device performing the encoding and the server performing the decoding. Therefore, even this method cannot be used between incompatible devices and servers.

[0086] To solve this problem, Figure 2a A speech recognition system according to an embodiment of the present disclosure is shown.

[0087] According to an embodiment of the present disclosure, device 100 may perform device-side speech recognition to convert a speech signal into a first string. Device 100 may determine whether the device-side speech recognition has failed based on a confidence score of the first string. If device 100 determines that the device-side speech recognition has failed, device 100 may transmit the first string corresponding to the device-side speech recognition result to server 200.

[0088] According to an embodiment of the present disclosure, the device 100 may transmit information about the voice signal to the server 200 in the form of a character string, so that the server 200 may process the character string regardless of the device-side voice recognition used by the device 100 .

[0089] According to an embodiment of the present disclosure, the first character string sent by the device 100 to the server 200 may be a frame synchronization character string.

[0090] A "frame" may refer to a unit for processing a speech signal by segmenting it into time intervals, or the segmented speech signal itself. A "frame-synchronized character string" refers to a character string that includes characters corresponding to each speech signal frame when a speech signal is converted into a character string by a speech recognition model and the character string is generated as output.

[0091] The apparatus 100 according to an embodiment of the present disclosure may generate a frame-synchronized character string as a speech recognition result by using an algorithm such as a recurrent neural network transformer (RNN-T) or connectionist temporal classification (CTC).

[0092] However, the embodiments of the present disclosure are not limited thereto. Even when the speech recognition result of the apparatus 100 according to the embodiment of the present disclosure is not frame-synchronized, the apparatus 100 may generate a frame-synchronized string by performing forced alignment. Figure 6 A detailed description of the frame synchronization string and a detailed method of generating the frame synchronization string through forced alignment is given.

[0093] When the confidence score of the result of performing voice recognition by using the device-side voice recognition is sufficiently high, the device 100 according to an embodiment of the present disclosure may use the result of performing voice recognition without change.

[0094] On the other hand, when the device 100 according to an embodiment of the present disclosure determines that the confidence score of the result of voice recognition performed via device-side voice recognition is not high enough, the device 100 may send a character string corresponding to the voice recognition result to the server 200 .

[0095] Therefore, when the device 100 according to an embodiment of the present disclosure determines that the confidence score of the result of performing speech recognition via device-side speech recognition is greater than the confidence threshold, the device 100 does not send the voice signal to the server 200 to prevent the server 200 from restarting the speech recognition process from the beginning, thereby reducing processing time.

[0096] When the device 100 according to an embodiment of the present disclosure determines that the confidence score of the result of voice recognition performed via device-side voice recognition is less than the confidence threshold, the device 100 can send the character string corresponding to the voice recognition result to the server 200 in units of sentences, words, phrases or frames.

[0097] When the device 100 according to an embodiment of the present disclosure performs speech recognition and thus obtains a character string constituting a sentence or phrase, the device 100 may transmit all characters included in the sentence or phrase to the server 200, or transmit only some characters included in the sentence or phrase to the server 200. The device 100 may transmit some characters having low confidence scores to the server 200 based on the confidence score of the character string.

[0098] The device 100 according to an embodiment of the present disclosure may receive the character string corrected by the server 200, and may combine the character string that was not sent to the server 200 due to determination that no correction was required with the corrected character string. The device 100 according to an embodiment of the present disclosure may output the combined character string based on the interpretation result of the combined character string, or may provide a voice assistant service.

[0099] The server 200 according to an embodiment of the present disclosure may receive a character string corresponding to the voice recognition result from the device 100 in units of sentences, words, phrases, or frames.

[0100] According to an embodiment of the present disclosure, server 200 may correct errors in a received first character string using a language model and dictionary information stored in server 200. Server 200 may obtain a second character string from the first character string using a language model in server 200 that includes more information than the language model stored in device 100. Server 200 may obtain the second character string by replacing at least one character included in the first character string with another character. The second character string may be a character string obtained by correcting errors included in the first character string.

[0101] In the present disclosure, the server 200 according to an embodiment of the present disclosure may correct the first character string by replacing at least one character included in the first character string received from the device 100 with another character, and may transmit the corrected first character string to the device 100 .

[0102] “An operation of correcting a first character string” may refer to an operation of recommending and outputting a second character string having a higher confidence score than the first character string. Therefore, in the present disclosure, the expressions “correction of a character string,” “correction of a character,” “a certain character is replaced with another character,” “recommending another character instead of a certain character,” “a certain character string is replaced with another character string,” and “recommending another character string instead of a certain character string” may be used interchangeably.

[0103] When the server 200 according to an embodiment of the present disclosure obtains a string of characters constituting a sentence or phrase from the device 100, the server 200 may correct the characters included in the sentence or phrase, or may correct some characters included in the sentence or phrase. The server 200 may correct some characters with low confidence scores based on the confidence score of the string of characters.

[0104] The server 200 according to the embodiment of the present disclosure may combine the character string that was not corrected based on the determination that no correction is required with the corrected character string. The device 200 according to the embodiment of the present disclosure may send the combined character string to the device 100.

[0105] The server 200 according to the embodiment of the present disclosure can perform decoding on the received character string by using different dictionary information and different language models for different domains. According to the embodiment of the present disclosure, since the dictionary information is stored in the server 200, new terms or new named entities can be easily hot-fixed.

[0106] According to an embodiment of the present disclosure, server 200 may receive a string from device 100 and select a domain associated with the received string. For example, server 200 may receive information about the domain associated with the string along with the string from device 100, and may determine a domain to decode the string based on the received information. For example, the domain may be a subject area of ​​the string, such as baseball, weather, biology, etc. For another example, server 200 may determine a domain associated with the string received from device 100 based on the received string. According to an embodiment of the present disclosure, server 200 may decode the received string using dictionary information and a language model corresponding to the determined domain.

[0107] Therefore, the server 200 according to an embodiment of the present disclosure can output a speech recognition result with improved speech recognition accuracy by re-decoding a character string received from the device 100. For example, the server 200 can receive a first character string from the device 100 and decode the first character string by using a language model and dictionary information within the server 200, thereby outputting a second character string in which at least one character included in the first character string is corrected.

[0108] Server 200 may transmit the second character string to device 100. Device 100 may improve speech recognition accuracy performance by receiving the second character string having a greater confidence score than the first character string from server 200, and improve speech recognition of the user's speech input using the second character string.

[0109] When server 200 according to an embodiment of the present disclosure obtains a string of characters comprising a sentence from device 100, server 200 may correct errors in the entire sentence or in some of the characters included in the sentence. Server 200 may correct errors in some characters with low confidence scores based on the confidence score of the string. Server 200 according to an embodiment of the present disclosure may combine the string that was not corrected based on the determination that no correction is necessary with the corrected string to obtain a second string.

[0110] refer to Figure 2a , according to the embodiment of the present disclosure, the server 200 may send the second character string as a speech recognition result to the device 100. However, the embodiment of the present disclosure is not limited to Figure 2a .

[0111] refer to Figure 2b and Figure 2c According to an embodiment of the present disclosure, the server 200 may determine the user's utterance intention with respect to the second character string and send information related to the voice assistant service to the device 100 based on the second character string.

[0112] The server 200 according to an embodiment of the present disclosure can provide various types of voice assistant services to the device 100 by using the second character string obtained from the first character string. The voice assistant service can be a service that provides a dialogue with the user. The voice assistant service can provide a response message to the user, imitating a person directly talking to the user, taking into account the user's situation, the device situation, etc. The voice assistant service can appropriately generate the information requested by the user and provide the information to the user to respond to the user's request, acting like a personal assistant for the user.

[0113] In this case, in order to provide the voice assistant service, the server 200 may provide information for performing a conversation with the user to the device 100 based on a character string by using a natural language understanding (NLU) model, a dialogue manager (DM) model, a natural language generation (NLG) model, etc. within the server 200.

[0114] For example, the server 200 may control the device 100 or another device (eg, a smart home appliance or a wearable device) based on a result of interpreting the second character string.

[0115] refer to Figure 2b , the server 200 according to an embodiment of the present disclosure may generate a control command for controlling the device 100 or a control command for enabling the device 100 to control another device based on a result of interpreting the character string, and provide the generated control command to the device 100.

[0116] refer to Figure 2c The server 200 according to an embodiment of the present disclosure may provide a voice assistant service related to various services that may be provided by other controlled devices such as televisions, home appliances, etc., for example, in a home network setting. For example, the voice assistant service may be linked to various services such as a broadcast service, a content sharing service, a content providing service, a power management service, a game providing service, a chat service, a document editing service, a search service, a call service, a photo taking service, a traffic recommendation service, and a movie playing service, and may provide the user with necessary information or necessary functions.

[0117] The server 200 according to an embodiment of the present disclosure may transmit information related to the voice assistant service to the device 100 based on the second character string. The information related to the voice assistant service may include a response message provided to the user or information requested by the user, so as to simulate a person directly talking to the user in consideration of the user situation, the device situation, etc.

[0118] Server 200 can determine the user's utterance intent based on the second character string and request service provider server 201 to provide the service requested by the user. Service provider server 201 can provide at least one of a broadcast service, a content sharing service, a content provision service, a power management service, a game provision service, a chat service, a document editing service, a search service, a call service, a photography service, a traffic recommendation service, or a movie playback service. Service provider server 201 can be implemented in one or more controlled devices such as televisions and home appliances, for example, in a home network setting.

[0119] Although the server 200 providing the voice assistant service is connected to Figure 2c , but embodiments of the present disclosure are not limited thereto. For example, according to embodiments of the present disclosure, server 200 may be connected to multiple service providing servers and may determine the service requested by the user based on the user's speech intention. Server 200 may select a service providing server corresponding to the determined service and may send a service provision request to the selected service providing server.

[0120] The service providing server 201 according to an embodiment of the present disclosure may provide information related to the requested service based on a service request received from the server 200 providing the voice assistant service. For example, the service providing server 201 may provide broadcasts, content, applications, traffic recommendation information, search results, etc. as information related to the requested service. The service providing server 201 may provide information related to the requested service to the server 200 providing the voice assistant service or the device 100.

[0121] Now, the apparatus 100 for selectively sending a character string as a speech recognition result to the server 220 to request the server 200 to correct the character string according to an embodiment of the present disclosure and the corresponding structure and corresponding operation method of the server 200 for correcting the received character string will be described in detail.

[0122] Figure 3 1 is a structural block diagram of the device 100 according to an embodiment of the present disclosure.

[0123] The device 100 according to an embodiment of the present disclosure may be a fixed terminal or a mobile terminal implemented as a computer device. The device 100 may be, but is not limited to, at least one of a smartphone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcast terminal, an AI speaker, a speaker, a personal digital assistant (PDA), a portable multimedia player (PMP), or a tablet personal computer (PC). The device 100 may communicate with another device and / or a server via a network using a wireless or wired communication method.

[0124] refer to Figure 3 , the device 100 may include a receiver 110 , a processor 120 , a communication interface 130 , a memory 140 , and an output interface 150 . Figure 3 One or more components of the illustrated apparatus 100 may be omitted or combined. Figure 3 Additional components beyond those shown may comprise the apparatus 100. For example, referring to Figure 19 , the device 100 according to some embodiments of the present disclosure may further include a user input interface 2100 , a sensing unit 2400 , and an audio / video (A / V) input interface 2600 .

[0125] The receiver 110 according to an embodiment of the present disclosure may receive a voice signal from a user. For example, the receiver 110 may receive a voice signal by converting external sound into electrical sound data via a microphone. Figure 3 In the embodiment, the receiver 110 is included in the device 100. However, according to another embodiment of the present disclosure, the receiver 110 may be included in a separate device and may be connected to the device 100 in a wired or wireless manner.

[0126] The memory 140 according to an embodiment of the present disclosure may store instructions for performing voice recognition, and various models, neural networks, dictionary information, etc. used in the voice recognition.

[0127] The processor 120 according to an embodiment of the present disclosure may perform voice recognition by executing one or more instructions stored in the memory 140 and loaded for execution.

[0128] The processor 120 according to an embodiment of the present disclosure may obtain a first character string as a speech recognition result of a speech signal.

[0129] For example, the first string may be a frame synchronization string including characters corresponding to speech signal frames obtained by dividing the speech signal at predetermined time intervals. Alternatively, the first string may be a string obtained by including each character emitted by the speech signal one by one according to a tag synchronization method.

[0130] Next, the processor 120 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string, and may transmit the first character string to the server 200 via the communication interface 130 based on the determination. The processor 120 according to an embodiment of the present disclosure may transmit the first character string to the server 200 in units of sentences, words, phrases, or frames. When the processor 120 according to an embodiment of the present disclosure performs speech recognition and thus obtains a character string constituting a sentence or phrase, the processor 120 may transmit all characters included in the sentence or phrase to the server 200, or transmit only some characters included in the sentence or phrase to the server 200. Based on the confidence score of the character string, the processor 120 may transmit characters with low confidence scores to the server 200.

[0131] Determining whether to replace the first string with another string may mean determining that voice recognition has failed and thus determining to replace the first string with another string. Alternatively, determining whether to replace the first string with another string may mean determining whether to replace the first string with another string obtained by performing voice recognition in the server.

[0132] For example, the processor 120 may determine a confidence score of a first character string, and may determine whether to replace the first character string with another character string based on the determined confidence score.

[0133] The confidence score of the first character string may be calculated based on at least one of likelihoods of a plurality of estimated character strings obtained from the first character string or a posterior probability that at least one character within the first character string is replaced by another character.

[0134] For example, the processor 120 may calculate the confidence score based on the likelihood output as a result of Viterbi decoding. Alternatively, the processor 120 may calculate the confidence score based on the posterior probability output from a softmax layer in an end-to-end speech recognition model.

[0135] Alternatively, the processor 120 according to an embodiment of the present disclosure may determine a plurality of estimated character strings estimated during speech recognition of a speech signal, and may calculate a confidence score for the first character string based on correlations between the plurality of estimated character strings. As the correlations between the plurality of estimated character strings including the first character string increase, the confidence score for the first character string may increase.

[0136] For another example, the processor 120 may determine whether to replace the first string with another string based on a result of comparing the first string with a keyword pre-stored in the device 100. For example, when the first string does not include the pre-stored keyword, the processor 120 may determine whether to replace the first string with another string.

[0137] For another example, the processor 120 may determine whether to replace the first string with another string based on whether the named entity is included in a domain related to the first string or the first string. For example, when it is determined that the first string is associated with a named entity-oriented domain or is determined that the first string is not associated with an open domain, the processor 120 may determine to replace the first string with another string.

[0138] When it is determined that the first character string is to be replaced with another character string, the processor 120 according to an embodiment of the present disclosure may control the communication interface 130 to transmit the first character string to the server 200 based on the determination.

[0139] The communication interface 130 according to an embodiment of the present disclosure can communicate with an external device, apparatus or server via wired or wireless communication. The communication interface 130 may include a short-range communication module, a wired communication module, a mobile communication module, a broadcast receiving module, etc.

[0140] When the result of performing speech recognition on the speech signal is not a frame synchronization string, the processor 120 according to an embodiment of the present disclosure may generate a frame synchronization string by forcibly aligning the first string and transmit the generated frame synchronization string to the server 200 .

[0141] The processor 120 according to an embodiment of the present disclosure may identify a portion of a speech signal in which each character included in the first character string is pronounced, and may identify multiple speech frames included in the identified portion of the speech signal. The processor 120 may obtain a frame-synchronized string by continuously arranging the characters multiple times based on the identified speech frames.

[0142] For example, when the pronunciation period of a certain character included in the first character string is n frames (where n is a natural number), the processor 120 may obtain a frame synchronization character string by continuously arranging n certain characters (n is greater than or equal to 2).

[0143] Communication interface 130 may receive a second string from server 200. The second string is a string obtained by server 200 by replacing at least one character in the first string with another character. Communication interface 130 may receive a response message generated by server 200 based on server 200's interpretation of the second string.

[0144] When it is determined that the first string does not need to be corrected, the processor 120 according to the embodiment of the present disclosure may determine that the first string will not be replaced by another string. When the first string is not replaced by another string, the processor 120 according to the embodiment of the present disclosure may output the first string through the output interface 150.

[0145] On the other hand, when it is determined that the first string needs to be corrected, the processor 120 may determine that the first string will be replaced by another string. When it is determined that the first string will be replaced by another string, the output interface 150 may output the second string received from the server 200 instead of the first string.

[0146] According to an embodiment of the present disclosure, the first character string obtained by apparatus 100 may be a character string obtained based on first dictionary information and a first language model. According to an embodiment of the present disclosure, the second character string obtained by apparatus 200 may be a character string obtained based on second dictionary information and a second language model stored in server 200.

[0147] The second dictionary information and the second language model stored in server 200 may include a greater amount of information than the first dictionary information and the first language model. Therefore, the confidence score of the second character string received from server 200 may be greater than the confidence score of the first character string. Device 100 may improve speech recognition performance by receiving the second character string from server 200 having a higher confidence score than the first character string, and using the second character string to provide output in response to the user's speech input.

[0148] The output interface 150 according to the embodiment of the present disclosure may output the first string or the second string unchanged, or may output a word string obtained from the first string or the second string. For example, when the first string is a frame synchronization string, the output interface 150 may output a word string obtained from the first string.

[0149] The output interface 150 according to an embodiment of the present disclosure may output a speech recognition result performed based on the first character string or the second character string. The output interface 150 may notify the user of the speech recognition result or may send the speech recognition result to an external device (e.g., a smartphone, a home appliance, a wearable device, or a server). For example, the output interface 150 may include a speaker capable of outputting an audio signal or a display capable of outputting a video signal.

[0150] Alternatively, the output interface 150 may perform an operation corresponding to the speech recognition result. For example, the device 100 may interpret the first character string or the second character string and determine the function of the device 100 corresponding to the interpretation result. The device 100 may output a screen image for executing the function through the output interface 150. Alternatively, the device 100 may transmit keywords corresponding to the interpretation result to an external server, receive information related to the transmitted keywords from the external server, and output the received information on the screen through the output interface 150. Alternatively, the device 100 may generate a response message to the speech signal based on the interpretation result, and may output the response message through the output interface 150.

[0151] The device 100 according to an embodiment of the present disclosure may determine the user's utterance intention by performing natural language processing on the first character string or the second character string, and output information related to the voice assistant service through the output interface 150. The device 100 may use, for example, an NLU model, a DM model, and an NLG model within the device 100 to provide the voice assistant service based on the first character string or the second character string.

[0152] Alternatively, the output interface 150 may receive information related to the voice assistant service from the server 200 based on the second character string, and may output the received information. For example, the information related to the voice assistant service based on the second character string may include a control command for controlling the device 100 or another device, which is generated based on the result of interpreting the user's speech intention through natural language processing on the second character string. Alternatively, for example, the information related to the voice assistant service based on the second character string may include a service or information required by the user, which is provided based on the result of interpreting the user's speech intention through natural language processing on the second character string.

[0153] When the processor 120 according to an embodiment of the present disclosure transmits only some characters included in a sentence or phrase to the server 200, the processor 120 may combine the corrected character string received from the server 200 with the character string that was not transmitted to the server 200 based on the determination that no correction is required. The processor 120 may output the combined character string, output a speech recognition result based on the combined character string, or provide a voice assistant service based on a result of interpreting the combined character string.

[0154] Now refer to Figure 4a and Figure 4b The method of operation of the apparatus 100 is described in detail.

[0155] Figure 4a Detailed structural block diagram of a device according to an embodiment of the present disclosure.

[0156] like Figure 4a As shown, the automatic speech recognition (ASR) module 121 of the processor 120 may receive the speech signal obtained by the receiver 110 and perform speech recognition on the speech signal.

[0157] Figure 4a The ASR module 121 can perform speech recognition on the speech signal using an end-to-end approach. An end-to-end approach is a speech recognition method that uses a trained deep neural network (DNN) to directly map the speech signal to a string or word sequence. Compared to other speech recognition methods that use multiple models such as acoustic models and language models, the end-to-end approach can simplify the speech recognition process by using a single trained DNN. Examples of low-level end-to-end speech recognition models include RNN-T models and CTC models.

[0158] The ASR module 121 may extract a feature vector from the speech signal and may output a first character string from the feature vector by using the DNN 144 stored in the memory 140 .

[0159] The determiner 125 of the processor 120 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string based on the confidence score of the first character string output by the ASR module 121. The determiner 125 may receive confidence score information about the first character string from the ASR module 121.

[0160] The determiner 125 according to an embodiment of the present disclosure may receive a posterior probability value as confidence score information about the first character string output by the softmax layer of the ASR module 121. The determiner 125 may calculate a confidence score based on the posterior probability value associated with the first character string.

[0161] For example, when the confidence score is equal to or greater than a threshold, the determiner 125 may determine that the first character string does not need to be corrected and may output the first character string through the output interface 150. On the other hand, when the confidence score is less than the threshold, the determiner 125 may determine that the first character string needs to be corrected and may transmit the first character string to the server 200 through the communication interface 130.

[0162] For ease of explanation, Figure 4a The example in which the first character string is outputted via the output interface 150 is shown. However, the embodiments of the present disclosure are not limited thereto. The device 100 according to the embodiment of the present disclosure can determine the user's utterance intention by performing natural language processing on the first character string, and output information related to the voice assistant service via the output interface 150.

[0163] The device 100 may provide a voice assistant service based on the first character string using, for example, an NLU model, a DM model, and an NLG model within the device 100 .

[0164] For example, the processor 120 of the device 100 may generate a response message to the first string and output it to the user via the output interface 150, thereby simulating a person directly conversing with the user, taking into account the user's situation, the device's situation, and the like. Alternatively, for example, the processor 120 may generate information requested by the user based on the first string and provide the generated information to the user via the output interface 150. Alternatively, for example, the processor 120 may determine the user's utterance intent based on the first string and request a service provider server to provide the service requested by the user. The output interface 150 may output the information received from the service provider server.

[0165] The output interface 150 of the device 100 according to an embodiment of the present disclosure may receive information related to the voice assistant service from the server 200 and may output the received information. The information related to the voice assistant service may be information generated by the server 200 based on the first character string or the second character string obtained by correcting the first character string. For example, the information related to the voice assistant service may include a response message to a user's voice signal, a service required by the user, or information required by the user.

[0166] Figure 4b is a detailed structural block diagram of a device according to another embodiment of the present disclosure.

[0167] like Figure 4b As shown, the ASR module 121 of the processor 120 can receive the speech signal obtained by the receiver 110 and perform speech recognition on the speech signal. The phoneme string acquirer 122 can obtain a phoneme string from the speech signal by using the acoustic model 141 stored in the memory 140. The acoustic model 141 can segment the waveform of the speech signal and estimate the phoneme string including the phonemes by using a hidden Markov model, a Gaussian mixture model, Bayesian inference, a multi-layer neural network, etc.

[0168] The character string acquirer 123 of the processor 120 may estimate a word from the phoneme string based on the dictionary information 142 and the language model 143 stored in the memory 140 and output a character string including the estimated word.

[0169] The determiner 125 of the processor 120 according to an embodiment of the present disclosure may calculate a confidence score of the first string output by the ASR module 121 and may determine whether to replace the first string with another string based on the calculated confidence score. The determiner 125 may receive confidence score information about the first string from the ASR module 121.

[0170] The determiner 125 according to an embodiment of the present disclosure may calculate a confidence score based on a partial likelihood in the first character string output as confidence score information about the first character string by the Viterbi decoder of the ASR module 121 .

[0171] When the confidence score is equal to or greater than the threshold, the determiner 125 according to an embodiment of the present disclosure may determine that the first character string does not need to be corrected, and may output the first character string through the output interface 150. On the other hand, when the confidence score is less than the threshold, the determiner 125 may determine that the first character string needs to be corrected, and may transmit the first character string to the server 200 through the communication interface 130. For convenience of explanation, Figure 4bThe example in which the first character string is outputted via the output interface 150 is shown. However, the embodiments of the present disclosure are not limited thereto. The device 100 according to the embodiment of the present disclosure can determine the user's utterance intention by performing natural language processing on the first character string, and output information related to the voice assistant service via the output interface 150.

[0172] The device 100 may provide a voice assistant service based on the first character string using, for example, an NLU model, a DM model, and an NLG model within the device 100 .

[0173] For example, the processor 120 of the device 100 may generate a response message to the first string and output it to the user via the output interface 150, thereby simulating a person directly conversing with the user, taking into account the user's situation, the device's situation, and the like. Alternatively, for example, the processor 120 may generate user-requested information based on the first string and provide the generated information to the user via the output interface 150. Alternatively, for example, the processor 120 may determine the user's utterance intention based on the first string and request a service provider server to provide the user's requested information. The output interface 150 may output the information received from the service provider server.

[0174] The output interface 150 of the device 100 according to an embodiment of the present disclosure may receive information related to the voice assistant service from the server 200 and may output the received information. The information related to the voice assistant service may be information generated by the server 200 based on the first character string or a second character string obtained by correcting the first character string. For example, the information related to the voice assistant service may include a response message to a user's voice signal, a service required by the user, or information required by the user.

[0175] As described above, the device 100 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string based on the confidence score of the speech recognition result of the speech signal. However, the embodiment of the present disclosure is not limited thereto. According to another embodiment of the present disclosure, the device 100 may determine whether to replace the first character string with another character string based on the comparison result of the keyword pre-stored in the device 100 and the first character string. Alternatively, the device 100 according to another embodiment of the present disclosure may determine whether to replace the first character string with another character string based on the domain associated with the first character string. Alternatively, the device 100 according to another embodiment of the present disclosure may interpret the meaning of the first character string via natural language understanding, and may determine whether to replace the first character string with another character string based on the interpretation result.

[0176] Figure 5a 1 is a diagram for explaining a method, performed by the device 100 , for determining to perform device-side voice recognition according to an embodiment of the present disclosure.

[0177] For example, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string based on a comparison result of a keyword pre-stored in the device 100 with the first character string.

[0178] When the first character string includes at least one pre-stored keyword, the processor 125 according to an embodiment of the present disclosure may determine not to replace the first character string with another character string. Therefore, the device 100 can use the result of the speech recognition performed by the ASR module 121 of the device 100 without the intervention of the server 200.

[0179] For example, when the first character string output by the ASR module 121 is “Read me my text,” the determiner 125 may determine that the first character string includes the pre-stored keyword “text,” and may determine not to replace the first character string with another character string.

[0180] For another example, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may determine whether to replace the first string with another string based on a domain associated with the first string or whether the first string includes a named entity.

[0181] When it is determined that the first character string is not associated with the named entity-oriented domain and the first character string is associated with the open domain, the processor 125 according to an embodiment of the present disclosure may determine not to replace the first character string with another character string. Therefore, the device 100 can use the result of the speech recognition performed by the ASR module 121 of the device 100 without the intervention of the server 200.

[0182] For example, when the first character string output by the ASR module 121 is “take a picture”, the determiner 125 may determine that the first character string is associated with the open domain and may determine not to replace the first character string with another character string.

[0183] When it is determined that the named entity is included in the first character string, the processor 125 according to an embodiment of the present disclosure may determine to replace the first character string with another character string.

[0184] The determiner 125 according to an embodiment of the present disclosure may determine whether at least one of the named entities stored in the memory 140 is included in the first string. Alternatively, in the absence of dictionary information for the named entity, the determiner 125 according to an embodiment of the present disclosure may determine whether the named entity is included in the first string. For example, the determiner 125 may identify the named entity included in the first string by performing part-of-speech (POS) tagging on the words identified from the first string.

[0185] For example, when the first character string output by the ASR module 121 is “photograph”, the determiner 125 may determine that the first character string does not include a named entity and may determine not to replace the first character string with another character string.

[0186] For another example, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may interpret the meaning of the first character string via natural language understanding, and may determine whether to replace the first character string with another character string based on the interpretation result.

[0187] When the interpretation result indicates that the voice signal is a general command associated with the operation of the device 100, the processor 125 according to an embodiment of the present disclosure may determine not to replace the first character string with another character string. Therefore, the device 100 can use the result of the voice recognition performed by the ASR module 121 of the device 100 without the intervention of the server 200.

[0188] For example, when the first character string output by the ASR module 121 is “Do I have a new voicemail?”, the determiner 125 may determine that the first character string is a general command associated with confirming a text message and may determine not to replace the first character string with another character string.

[0189] Figure 5b is a diagram for explaining a method of determining to perform server-based speech recognition, performed by the apparatus 100 , according to an embodiment of the present disclosure.

[0190] like Figure 5b As shown, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may determine that the first character string needs to be replaced with another character string, and may send the first character string to the server 200 based on the determination.

[0191] Figure 5b It shows a situation where the ASR module 121 of the apparatus 100 receives a speech signal corresponding to “Cardinals Baseball Team” spoken by a user and mistakenly obtains a first character string “Cat Deer Baseball Team”.

[0192] For example, since the first character string does not include a pre-stored keyword, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may determine that the first character string is to be replaced with another character string.

[0193] For another example, when it is determined that the first character string is related to the sports field or includes a named entity, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may determine to replace the first character string with another character string.

[0194] The determiner 125 according to an embodiment of the present disclosure may determine whether at least one of the named entities stored in the memory 140 is included in the first string. Alternatively, in the absence of dictionary information for the named entity, the determiner 125 according to an embodiment of the present disclosure may determine whether the named entity is included in the first string. For example, the determiner 125 may identify the named entity included in the first string by performing part-of-speech (POS) tagging on the words identified from the first string. However, embodiments of the present disclosure are not limited thereto, and various types of named entity recognition (NER) methods may be used.

[0195] For another example, the determiner 125 of the processor 120 of the device 100 according to an embodiment of the present disclosure may determine that the voice signal is not a general command as a result of interpreting the first character string, and may determine to replace the first character string with another character string.

[0196] like Figure 5b As shown, the determiner 125 of the device 100 according to an embodiment of the present disclosure may determine that a first character string needs to be replaced with another character string, and may send the first character string to the server 200 based on this determination. The server 200 may receive the first character string from the device 100 and perform decoding by using a language model and dictionary information (e.g., dictionary information in the sports field) available to or stored in the server 200. The server 200 may obtain a second character string in which at least one character included in the first character string has been corrected as a decoding result. The device 100 may improve the accuracy of speech recognition by receiving the second character string from the server 200 and using the received second character string.

[0197] When the device 100 according to an embodiment of the present disclosure performs speech recognition and thus obtains a character string constituting a sentence or phrase, the device 100 may transmit all characters in the sentence or phrase to the server 200, or transmit only some characters included in the sentence or phrase to the server 200. The determiner 125 of the processor 120 of the device 100 may determine, based on the confidence score of the character string, to transmit some characters having a low confidence score to the server 200.

[0198] The device 100 according to an embodiment of the present disclosure may receive the corrected string from the server 200, and may combine the string that was not sent to the server 200 based on the determination that no correction is required with the corrected string. The device 100 according to an embodiment of the present disclosure may output the combined string, output a speech recognition result based on the combined string, or provide a voice assistant service based on the result of interpreting the combined string.

[0199] According to an embodiment of the present disclosure, device 100 may provide server 200 with information about the domain associated with a first string of the device 100, while simultaneously requesting server 200 to correct the first string. Domain information is information used to indicate or identify a domain and may, for example, include a domain name and a domain identifier, but embodiments of the present disclosure are not limited thereto. Device 100 may identify the domain associated with the first string based on a domain confidence score for the first string output by the ASR model of device 100. The domain confidence score may be a number that indicates the strength of association of at least a portion of the first string with a particular domain. For example, device 100 may calculate a confidence score that indicates the strength of association of the first string output by the ASR model with a domain previously registered in device 100. Device 100 may identify the domain associated with the first string based on the calculated domain confidence score. Device 100 may identify the domain associated with the first string based on a rule, or may obtain the domain confidence score associated with the first string using an AI model trained for domain identification.

[0200] Figure 6 is a diagram for explaining a frame synchronization string according to an embodiment of the present disclosure.

[0201] like Figure 6 As shown, the ASR module 121 of the apparatus 100 according to an embodiment of the present disclosure may output a frame synchronization character string 603 , which includes characters corresponding to speech signal frames F obtained by dividing the speech signal 601 at preset time intervals.

[0202] For example, the ASR module 121 may receive a voice signal corresponding to the user's utterance "baseball" and may output a frame synchronization string [b, b, a, a, a, s, s, e, b, b, a, a, l].

[0203] However, the embodiments of the present disclosure are not limited thereto, and the ASR module 121 according to the embodiments of the present disclosure may output a non-frame synchronization string (i.e., a tag synchronization string) as a result of speech recognition. Even in this case, the device 100 may generate a frame synchronization string by performing forced alignment on the string obtained from the speech signal.

[0204] The processor 120 of the apparatus 100 according to an embodiment of the present disclosure may identify a portion of a speech signal in which each character included in the first character string is pronounced, and may identify a plurality of speech frames included in the identified portion of the speech signal. The processor 120 may obtain a frame-synchronized string by sequentially arranging the characters multiple times according to the identified speech frames.

[0205] For example, the ASR module 121 may output a first string [b, a, s, e, b, a, l, l], which is a non-frame synchronization string. In this case, the processor 120 may continuously arrange each character included in the first string multiple times based on the time period in which each character is pronounced. Thus, the processor 120 may obtain a frame synchronization string [b, b, a, a, a, s, s, e, b, b, a, a, l].

[0206] The device 100 according to an embodiment of the present disclosure may output the frame synchronization string 603 to the server 200. The server 200 may decode the frame synchronization string 603 received from the device 100 and may transmit a second string obtained based on the decoding result to the device 100.

[0207] Figure 7 is a block diagram of a server according to an embodiment of the present disclosure.

[0208] The server 200 according to an embodiment of the present disclosure may be connected to the device 100 in a wired or wireless manner as long as the server 200 is communicably connected to the device 100 .

[0209] refer to Figure 7 , the server 200 may include a communication interface 210, a processor 220 and a memory 230. Figure 7 Alternative components than those shown may make up server 200 . Figure 7 One or more components of the servers shown may be omitted or combined together.

[0210] The memory 230 of the server 200 according to an embodiment of the present disclosure may store instructions for performing speech recognition, as well as various models, neural networks, dictionary information, etc. used in speech recognition.

[0211] The processor 220 according to an embodiment of the present disclosure may perform voice recognition by executing one or more instructions stored in the memory 230 and loaded for execution.

[0212] The communication interface 210 according to an embodiment of the present disclosure may communicate with an external device or apparatus via wired or wireless communication. The communication interface 210 may include a short-range communication module, a wired communication module, a mobile communication module, a broadcast receiving module, and the like.

[0213] The communication interface 210 of the server 200 according to an embodiment of the present disclosure may receive a first character string from the device 100. The first character string may be outputted by the device 100 performing voice recognition on a voice signal input to the device 100.

[0214] For example, the first character string received by the server 200 may be a frame synchronization character string including characters corresponding to voice signal frames obtained by dividing the voice signal at preset time intervals. In another example, the first character string received by the server 200 may be a non-frame synchronization character string.

[0215] When the first character string received from the device 100 is a non-frame synchronization character string, the processor 220 according to an embodiment of the present disclosure can obtain a frame synchronization character string from the first character string. The processor 220 can obtain the frame synchronization character string by continuously arranging at least one character included in the first character string multiple times in units of frames.

[0216] The processor 220 of the server 200 according to an embodiment of the present disclosure may obtain a second character string from a first character string by replacing at least one character included in the first character string with another character.

[0217] Processor 220 according to an embodiment of the present disclosure may identify a replacement character having a pronunciation similar to that of each character included in the first character string, and determine an estimated character string in which at least one character in the first character string has been corrected to another character based on the identified replacement character. Processor 220 may select the most appropriate estimated character string from the determined estimated character strings based on pre-stored information such as a language model and dictionary information, and may obtain the most appropriate estimated character string as the second character string.

[0218] The method of obtaining the second character string performed by the processor 220 according to an embodiment of the present disclosure will now be described in more detail.

[0219] First, processor 220 may identify multiple estimated character strings from a first character string. Processor 220 may calculate a likelihood matrix associated with replacement characters that would replace each character in the first character string. Processor 220 may identify multiple estimated character strings in which at least one character in the first character string has been replaced with another character based on the likelihood values ​​in the likelihood matrix.

[0220] The processor 220 according to an embodiment of the present disclosure may calculate the likelihood of multiple estimated character strings based on the first character string. The processor 220 may calculate the likelihood of multiple estimated character strings based on the likelihood values ​​in the likelihood matrix associated with the replacement character to replace each character in the first character string.

[0221] The likelihood obtained from the first string may refer to the likelihood that the first string is estimated as the speech recognition result when each of the multiple estimated strings is assumed to be a substantially true string. According to an embodiment of the present disclosure, the processor 220 may identify a replacement character that has a pronunciation similar to that of each character in the first string, and may use the likelihood obtained from the first string based on the identified replacement character to determine an estimated string in which at least one character in the first string has been corrected to another character.

[0222] Processor 220 may obtain a second character string as one of the plurality of estimated character strings based on the likelihood, dictionary information, and language model. Processor 220 may determine whether to replace the first character string with the second character string based on the calculated likelihood. Based on this determination, processor 220 may obtain the second character string from the first character string by replacing at least one character included in the first character string with another character.

[0223] The processor 220 according to an embodiment of the present disclosure may calculate the likelihood according to the first character string through a process described later in the present disclosure.

[0224] For example, the processor 220 may calculate the posterior probability of each character in the first string based on the preceding characters of each character. When the preceding characters of a character are taken into account, the posterior probability of a character in the first string may include the probability that the character is replaced by multiple other characters. In other words, when the preceding characters of a character in the first string are taken into account, the posterior probability of a character may include the probability that the ASR module of the processor 120 of the device 100 has accurately predicted the character and the probability that the ASR module has incorrectly predicted the character as another character.

[0225] Next, processor 220 may calculate a character sequence probability for the first character string. The character sequence probability of a character string may refer to the probability of the characters being arranged in the character string. The character sequence probability may be calculated based on the characters accumulated before each character in the character string. Processor 220 may calculate the likelihood of the multiple estimated character strings obtained from the first character string based on the posterior probability of each character and the character sequence probability.

[0226] In order to calculate the posterior probability, the processor 220 according to the embodiment of the present disclosure may use a recurrent neural network (RNN) including a plurality of long short-term memory (LSTM) layers and a softmax layer. Figure 10a Describe the RNN used to compute the posterior probability in more detail.

[0227] For another example, the processor 220 may calculate the posterior probability of each character in the first character string based on a predetermined confusion matrix. The processor 220 may calculate the likelihood of multiple estimated character strings obtained from the first character string based on the posterior probability of each character. Figure 10b Describe in more detail the confusion matrix used to calculate the posterior probability.

[0228] For another example, the processor 220 may calculate the posterior probability of each character in the first string based on a predetermined probability value. The processor 220 may determine the probability that the first character included in the first string is actually the first character as P, P may be a predetermined value, and P may be a value between 0 and 1. The processor 220 may determine the probability that the first character included in the first string is actually a character other than the first character as (1-P) / (N-1), N represents the number of characters, and N may be a natural number. In other words, the processor 220 may determine the probability that the ASR module of the processor 120 of the device 100 has accurately predicted the first character in the first string as P, and may determine the probability that the ASR module has incorrectly predicted another character as the first character as (1-P) / (N-1).

[0229] For example, the processor 220 may determine the probability that the first character included in the first character string is actually the first character as 0.9, and determine the probability that the first character is actually another character as 0.1 / (N-1).

[0230] Processor 220 according to an embodiment of the present disclosure may include a likelihood calculator that calculates likelihoods for a plurality of estimated character strings obtained from a first character string. Processor 220 may also include a decoder that uses dictionary information and a language model to obtain a second character string from the likelihoods. Processor 220 may obtain the second character string by re-decoding the likelihoods obtained from the first character string using the dictionary information and the language model.

[0231] For example, the decoder of processor 220 may obtain the second character string based on the dictionary information and language model stored in server 200. When the likelihood of multiple estimated character strings obtained from the first character string is input, the decoder may output the second character string. For example, the decoder of processor 220 may include a weighted finite state transducer (WFST) decoder.

[0232] When the processor 220 performs WFST decoding, the server 200 according to an embodiment of the present disclosure can construct and decode a search space by using WFST based on the relationship T between characters, dictionary information L including mapping information between words and characters, and a language model G that estimates the probability of the next word when assigning a specific word string.

[0233] For another example, the decoder of processor 220 may recalculate the likelihoods of multiple estimated character strings obtained from the first character string based on the dictionary information and the language model. The decoder may then determine, from the multiple estimated character strings, the second character string with the highest recalculated likelihood. For example, the decoder of processor 220 may include a Viterbi decoder. Taking into account the dictionary information and the language model, the Viterbi decoder may search for the character string with the highest likelihood relative to the first character string as the second character string.

[0234] The communication interface 210 according to an embodiment of the present disclosure may transmit the second character string to the device 100. Alternatively, the communication interface 210 may transmit a response message to the voice signal generated by the processor 220 to the device 100. The processor 220 may interpret the second character string by using an NLU model and may generate a response message to the voice signal based on the interpretation result.

[0235] The processor 220 may determine the type of the response message by applying the DM model to the interpretation result. The processor 220 may generate a response message of the determined type by using the NLG model and transmit it to the device 100.

[0236] Alternatively, the communication interface 210 may transmit information related to the voice assistant service generated based on the second character string to the device 100. In order to provide the voice assistant service, the processor 220 may provide information for performing a dialogue with the user based on the second character string to the device 100 by using an NLU model, a DM model, an NLG model, etc. within the server 200. The processor 220 may generate a control command for controlling the device 100 or another device based on the result of interpreting the second character string, and provide the generated control command to the device 100.

[0237] Now refer to Figure 8a A method for supporting the device 100 to perform speech recognition, which is performed by each component of the server 200 according to an embodiment of the present disclosure, is described. Figure 8a A situation is shown where the user of device 100 says "Cardinals baseball team."

[0238] First, the device 100 may estimate a first character string [Cat Deer Baseball Team] by performing voice recognition on a user's voice signal.

[0239] The device 100 may determine whether to replace the first character string with another character string based on the confidence score of the first character string, the domain associated with the first character string, the interpretation result of the meaning of the first character string, or whether the first character string includes a named entity. Figures 4a to 5b A detailed method of determining whether to perform server-based speech recognition to replace the first character string, performed by the apparatus 100, is described, and thus a redundant description thereof will be omitted.

[0240] exist Figure 8a In the example, the device 100 may determine that the first character string needs to be replaced with another character string, and may send the first character string [Cat Deer Baseball Team] to the server 200.

[0241] When sending the first string to the server 200, the device 100 according to an embodiment of the present disclosure may send information related to the voice signal along with the first string. The device 100 according to an embodiment of the present disclosure may also send information related to the length of the voice signal frame represented by each character in the first string along with the first string. For example, the device 100 may send the first string synchronized with the voice signal frame to the server 200. A string synchronized with the voice signal frame may refer to a string including characters corresponding to voice signal frames obtained by dividing the voice signal at preset time intervals.

[0242] However, the embodiments of the present disclosure are not limited to the embodiment of the present disclosure in which the device 100 sends a frame synchronization string to the server 200. The device 100 according to the embodiment of the present disclosure may send a non-frame synchronization first string to the server 200. The non-frame synchronization first string may refer to a string obtained by including each character emitted by the voice signal one by one according to the tag synchronization method.

[0243] When transmitting the non-frame-synchronized first character string to server 200, device 100 according to an embodiment of the present disclosure may provide information related to the speech signal along with the non-frame-synchronized first character string. Server 200 may generate a frame-synchronized character string based on the information related to the speech signal by forcibly aligning the first character string. For example, the information related to the speech signal may include information about the portion of the speech signal from which the speech recognition model of device 100 obtained the first character string.

[0244] The server 200 according to an embodiment of the present disclosure may receive a first string from the device 100 via the communication interface 210. The server 200 according to an embodiment of the present disclosure may receive a frame-synchronized first string. However, as described above, the server 200 may receive a non-frame-synchronized first string. In this case, the server 200 may receive information related to the speech signal from the device 100 together with the first string obtained by the device 100 from the speech signal. The server 200 may generate a frame-synchronized first string by forcibly aligning the first string based on the information related to the speech signal.

[0245] The processor 220 may identify a plurality of estimated character strings from the first character string, and may obtain a second character string based on the plurality of estimated character strings.

[0246] According to an embodiment of the present disclosure, processor 220 may identify a replacement character that has a pronunciation similar to that of each character in the first character string, and determine an estimated character string in which at least one character in the first character string has been corrected to another character based on the identified replacement character. Processor 220 may select the most appropriate estimated character string from the determined estimated character strings based on pre-stored information such as a language model and dictionary information, and may obtain the most appropriate estimated character string as the second character string.

[0247] The method of obtaining the second character string performed by the processor 220 according to an embodiment of the present disclosure will now be described in more detail.

[0248] First, the processor 220 may calculate likelihoods of a plurality of estimated character strings obtained from the first character string.

[0249] Considering the language model and dictionary information stored in device 100, a first character string estimated by device 100 from the speech signal is obtained based on the probability distribution of each speech signal frame corresponding to any character. Server 200 may remove deviations related to the language model and dictionary information of device 100 from the first character string estimated by device 100, and may perform re-decoding using the language model and dictionary information stored in server 200.

[0250] The server 200 may calculate likelihoods of the plurality of estimated character strings obtained from the first character string to remove deviations associated with the language model and dictionary information of the device 100 from the first character string.

[0251] The processor 220 may perform decoding to obtain the second character string by applying the dictionary information and the language model stored in the memory 230 to the likelihood obtained from the first character string. When the processor 220 performs decoding by applying the dictionary information and the language model stored in the memory 230 of the server 200, the processor 220 may use the dictionary information and the language model including many named entities, thereby improving the accuracy of speech recognition.

[0252] For example, the named entity "Cardinals" may not be stored in the language model of the memory of the apparatus 100. Therefore, the apparatus 100 may incorrectly estimate the first character string [Cat Deer Baseball Team] from the speech signal "Cardinals Baseball Team".

[0253] However, if Figure 8a As shown, the named entity "Cardinals" in the sports field may be stored in the memory 230 of the server 200. Therefore, the processor 220 of the server 200 may determine that the "Cat and Deer" estimated by the device 100 is actually likely to be the name of the baseball team "Cardinals".

[0254] The processor 220 may identify a replacement character that has a pronunciation similar to that of each character in the first string, and obtain a second string in which at least one character in the first string has been corrected to another character based on the identified replacement character. Thus, the processor 220 may obtain a second string [Cardinals Baseball Team], in which "Cat Deer" in the first string has been replaced by "Cardinals." This will be referred to later. Figure 17 A detailed method for obtaining a second character string [Cardinals Baseball Team] from a first character string [Cat Deer Baseball Team] by using a WFST decoding method is described.

[0255] The server 200 may transmit the second character string to the device 100. The device 100 may replace the first character string estimated by the device 100 with the second character string received from the server 200, and may output the second character string. Figure 8a As shown, for example, the confidence score of the first character string [Cat Deer Baseball Team] may be 0.1, and the confidence score of the second character string [Cardinals Baseball Team] may be 0.5. The apparatus 100 according to an embodiment of the present disclosure can improve speech recognition performance by receiving a second character string having a higher confidence score than the first character string from the server 200, and providing a more accurate response to the user input using the received second character string.

[0256] As described above, the server 200 according to an embodiment of the present disclosure may receive a frame synchronization string from the device 100 or may generate a frame synchronization string from a string received from the device 100. The server 200 may determine a replacement string by obtaining the likelihood of each character corresponding to each speech signal frame. The server 200 may simultaneously receive an entire string including a plurality of characters, or may sequentially receive at least some of the characters included in the string.

[0257] Now refer to Figure 8b The method of determining a replacement character string by obtaining the likelihood of each character corresponding to each speech signal frame, performed by the server 200 according to an embodiment of the present disclosure, is described in more detail.

[0258] The server 200 according to an embodiment of the present disclosure may receive the first character string for frame synchronization from the device 100 , or may generate the first character string for frame synchronization from the character string received from the device 100 .

[0259] For example, the communication interface 210 of the server 200 may receive information related to the speech signal from the device 100 together with the character string obtained from the speech signal by the device 100. The server 200 may generate a frame-synchronized first character string by forcibly aligning the character strings based on the information related to the speech signal.

[0260] The character string evaluator 221 of the server 200 may calculate a likelihood matrix 813 for each character in the first character string of the frame sync, the likelihood matrix 813 being related to a replacement character to replace each character in the first character string of the frame sync.

[0261] According to an embodiment of the present disclosure, the likelihood matrix of a certain character calculated by the string evaluator 221 may refer to a matrix including likelihood values ​​of replacement characters used to replace the certain character. The likelihood value of the replacement character used to replace the certain character may refer to the probability of the certain character being estimated as a speech recognition result when assuming that the replacement character is a basic true character.

[0262] For example, for the character "a" included in the character string obtained as a result of speech recognition, a likelihood matrix [0.4 0.01 0.01 0.01 0.2 ... 0.01] including a probability value that the actual character is "a", a probability value that the actual character is "b", a probability value that the actual character is "c", ... and a probability value that the actual character is "z" can be obtained. When a likelihood matrix including likelihood values ​​of replacement characters corresponding to each character included in the character string is obtained, a high likelihood value can be assigned to a replacement character that is similar in pronunciation to each character.

[0263] The decoder 223 of the server 200 may select one estimated character string in which at least one character within the frame-synchronized first character string has been replaced from the plurality of estimated character strings based on the likelihood matrix 813 and may obtain the selected estimated character string as the second character string.

[0264] For example, decoder 223 may recalculate likelihood matrix 813 based on the dictionary information and the language model. Decoder 223 may determine the second string with the highest recalculated likelihood from the plurality of estimated strings. For example, decoder 223 may include a Viterbi decoder. Taking into account the dictionary information and the language model, the Viterbi decoder may search for the string with the highest likelihood relative to the first string as the second string.

[0265] The decoder 223 of the server 200 may obtain a character string 817 having the highest confidence score from the plurality of estimated character strings 815 as a second character string based on the likelihood of the plurality of estimated character strings, the dictionary information, and the language model. The server 200 may transmit the second character string to the device 100. The device 100 may improve speech recognition performance by receiving the second character string having a higher confidence score than the first character string from the server 200 and using the second character string.

[0266] Now refer to Figures 9 to 11b The method of calculating likelihood performed by the server 200 according to various embodiments of the present disclosure is described in detail.

[0267] Figure 9 is a detailed block diagram of a server according to an embodiment of the present disclosure.

[0268] like Figure 9 As shown, the communication interface 210 of the server 200 may receive a first character string from the device 100 .

[0269] The string evaluator 221 of the processor 220 may output evaluation information of the first string so that the decoder 223 can recommend and output a second string having a higher confidence score than the first string. For example, the evaluation information of the first string may include a likelihood calculated based on the first string.

[0270] The string evaluator 221 may calculate the likelihood of a plurality of estimated strings obtained from the first string. The string evaluator 221 may obtain the plurality of estimated strings by replacing each character included in the first string with another character. The likelihood of the plurality of estimated strings may refer to the probability that the speech recognition module estimates the first string when assuming that each of the plurality of estimated strings obtained from the first string is a substantially true string.

[0271] The likelihood obtained from the first string and output by string evaluator 221 can be used to identify replacement characters that are similar in pronunciation to characters of the first string, and determine an estimated string in which at least one character within a string has been corrected to another character based on the identified replacement characters.

[0272] The string evaluator 221 may calculate a likelihood matrix associated with a replacement character to replace each character in the first string, and may identify a plurality of estimated strings based on the likelihood values ​​in the likelihood matrix. The string evaluator 221 may output the likelihood matrix obtained from each character as the likelihood of the plurality of estimated strings.

[0273] The character string evaluator 221 may calculate the likelihood from the first character string by using the likelihood calculation data 231 stored in the memory 230. For example, the likelihood calculation data 231 may include a neural network or a confusion matrix trained for likelihood calculation.

[0274] For example, the string evaluator 221 may calculate a posterior probability for each character in the first string based on characters that appear before each character. The string evaluator 221 may calculate a character sequence probability based on the first string. The string evaluator 221 may calculate the likelihood of multiple estimated strings obtained from the first string based on the posterior probability of each character and the character sequence probability.

[0275] For another example, the string evaluator 221 may calculate the posterior probability of each character in the first string based on a predetermined confusion matrix. The string evaluator 221 may calculate the likelihood of multiple estimated strings obtained from the first string based on the posterior probability of each character.

[0276] After the string evaluator 221 calculates the likelihood, the decoder 223 can obtain a second string based on the calculated likelihood by using the dictionary information and the language model. The decoder 223 can obtain the second string with the maximum likelihood from a plurality of estimated strings obtained by replacing at least one character in the first string with another character.

[0277] Decoder 223 may obtain a second character string in which at least one character in the first character string has been replaced with another character by using dictionary information 232 and language model 233. For example, decoder 223 may include a WFST decoder using likelihood as input, or a Viterbi decoder using typical token passing.

[0278] According to an embodiment of the present disclosure, the dictionary information stored in server 200 may be dictionary information that stores the relationship between words and character strings, rather than general dictionary information that stores the relationship between phoneme strings and words. The language model may be an AI model that has learned the relationship between words so that it can estimate the probability of the next word when assigning a specific word string. For example, the language model may be a neural network such as an RNN or an n-gram statistical model.

[0279] The communication interface 210 may transmit the second character string to the device 100. However, the embodiments of the present disclosure are not limited to the embodiment of the present disclosure in which the second character string is transmitted to the device 100. The server 200 according to the embodiment of the present disclosure may determine the user's utterance intention by performing natural language processing on the second character string, and transmit information related to the voice assistant service based on the second character string to the device 100 via the communication interface 210.

[0280] According to various embodiments of the present disclosure, Figure 2b and Figure 2c The information related to the second character string transmitted by the server 200 to the device 100 is described, and thus a redundant description thereof will be omitted.

[0281] The decoder 223 of the server 200 according to the embodiment of the present disclosure can decode the first character string by using different pieces of dictionary information and different language models for different domains. Therefore, the server 200 according to the embodiment of the present disclosure can output a speech recognition result with improved speech recognition accuracy by re-decoding the first character string received from the device 100.

[0282] The processor 220 of the server 200 may receive a first character string from the device 100 and determine a domain related to the first character string. The decoder 223 of the server 220 may decode the first character string using dictionary information and a language model corresponding to the determined domain.

[0283] For example, processor 220 of server 200 may receive domain information related to the first character string from device 100 along with the first character string, and may determine a domain for performing decoding on the first character string based on the received domain information. For example, processor 220 may determine a domain that is the same as or similar to the domain identified by the domain information received from device 100 as the domain for performing decoding.

[0284] For another example, the processor 220 of the server 200 may determine a domain related to the first character string received from the device 100 based on the received first character string. Figure 9 Although not shown, the server 200 may store a domain recognition model of an AI model trained for performing domain recognition in the memory 230. By using the domain recognition model, the processor 220 may output a domain confidence score using the first string as an input value. The processor 220 may determine the domain associated with the first string based on the domain confidence score indicating the confidence of the recognized domain. According to an embodiment of the present disclosure, the string evaluator 221 or the decoder 223 of the server 200 may determine the domain associated with the first string received from the device 100 based on the received first string.

[0285] For example, the processor 223 of the server 200 may determine a domain related to the first character string received from the device 100 based on the received first character string. The decoder 223 according to an embodiment of the present disclosure may decode the received first character string by using dictionary information and a language model specific to the determined domain.

[0286] The decoder 223 according to an embodiment of the present disclosure may be a secondary decoder. The secondary decoder may perform primary decoding on the evaluation information on the first character string received from the character string evaluator 221, and then may perform secondary decoding by using the result of the primary decoding.

[0287] In this case, the decoder 223 according to an embodiment of the present disclosure may perform decoding based on a general dictionary and language model by using a primary decoder. The decoder 223 according to an embodiment of the present disclosure may perform decoding based on a dictionary and language model specific to a determined domain of the received first character string by using a secondary decoder.

[0288] For another example, the communication interface 210 of the server 200 according to an embodiment of the present disclosure may receive information from the device 100 for determining a domain associated with a first character string, along with the first character string. For example, the information received from the device 100 for determining the domain may include contextual information. For example, the contextual information may include at least one of information about an application currently used by the user on the device 100 or the server 200, conversation history information, information about the situation surrounding the device 100, or trend information. The processor 220 of the server 200 may determine the domain in which to decode the first character string based on the contextual information. A detailed method for determining the domain based on the contextual information will now be described.

[0289] For example, processor 220 may determine a domain based on the application currently being executed by the user. When a user is accessing a map application on device 100 or server 200, processor 220 may determine the domain associated with the map as the domain to be used for decoding when determining the domain of a string obtained from the user's utterance. For example, processor 220 may determine the domain to be decoded by giving a higher weight to the map domain, or may determine the map domain as the domain to be decoded.

[0290] Alternatively, for example, processor 220 may determine the domain based on conversation history information. If it is determined that the user's conversation history is associated with "music," processor 220 may determine the domain associated with music as the domain to be decoded when determining the domain of the string obtained from the user's utterance. For example, processor 220 may determine the domain to be decoded by giving a higher weight to the music domain, or may determine the music domain as the domain to be decoded.

[0291] Alternatively, for example, processor 220 may determine the domain based on situation information of the surrounding environment of device 100 sensed by sensors installed on device 100. Processor 220 may determine the domain based on the location of device 100 identified using global positioning system (GPS) information of device 100. When a user attempts to search for a restaurant, processor 220 may determine the domain associated with the location of device 100 as the domain to perform decoding. When device 100 is located near a movie theater, processor 220 may determine the domain associated with movies as the domain to perform decoding.

[0292] Alternatively, for example, the processor 220 may determine the domain based on trend information.The processor 220 may determine a domain associated with a headline news or a real-time search for a keyword through a portal website as a domain on which decoding is to be performed.

[0293] A case where the character string evaluator 221 of the server 200 according to an embodiment of the present disclosure obtains likelihood based on characters accumulated before each character in the first character string will now be described in detail.

[0294] The communication interface 210 of the server 200 according to the embodiment of the present disclosure may receive the first string y of the frame synchronization from the device 100. o [0:1+1]. As mentioned above, reference has been made to Figure 6 The frame synchronization string is described, so its redundant description will be omitted.

[0295] In the following description, y o [L] may be a frame synchronization character estimated by the device-side speech recognition module from the speech signal. The frame synchronization character may refer to a character estimated from a speech frame included in the speech signal. o [L] is included in V, where V is the set of all characters.

[0296] y o [0:L+1] means when 0≤L'≤L o [L']. L and L' are the indices of the string.

[0297] The communication interface 210 may simultaneously receive an entire character string including a plurality of characters, or may sequentially receive some characters included in the character string.

[0298] y p [L] refers to the estimated frame synchronization character used by the server to post-process the character string obtained by the device. p [L] is included in V, where V is a set of characters. W i is a word string. i are the words included in D, where D is a group of words.

[0299] The string evaluator 221 of the server 200 may calculate the character according to the first string y o The probability of the character sequence P(y o [0:L+1]). Character sequence probability P(y o [0:L+1]) can be calculated based on the character-level language model.

[0300] The string evaluator 221 can calculate when the first string y o [0:L+1] has been estimated by the device 100 that the Lth character is actually y p The posterior probability P(y p [L]|y o [0:L+1]). The string evaluator 221 may be based on the first string y o [0:L+1] to calculate character y o The posterior probability P(y p [L]|y o [0:L+1]). In other words, the string evaluator 221 can be based on the first string y o[0:L+1] to calculate that the device 100 has accurately estimated the character y o [L] and the probability that the device 100 has incorrectly estimated the character y o The probability of [L].

[0301] The string evaluator 221 according to an embodiment of the present disclosure may calculate the posterior probability of each character of the first string based on the first string by using a neural network. In other words, the calculation of the posterior probability of each character of the first string based on the first string may be performed using a trained neural network for calculating the posterior probability of each character of the first string based on the first string.

[0302] The string evaluator 221 according to an embodiment of the present disclosure may be configured by using Figure 10a The RNN 1000 of the LSTM layer 1010 and the softmax layer 1030 is used to calculate the posterior probability of each character within the first string.

[0303] Figure 10a The LSTM layer 1010 may include multiple stacked LSTM layers. Figure 10a , the first character string may be input to the LSTM layer 1010 , data output from the LSTM layer 1010 may be input to the softmax layer 1030 , and the softmax layer 1030 may output the posterior probability of each character of the first character string.

[0304] According to an embodiment of the present disclosure, a neural network that calculates the posterior probability of each character in a string can be trained by learning a base true string and an error string output from a speech recognition module. Specifically, when the neural network receives an error string output from the speech recognition module, the neural network can be trained so that the output value approaches the base true string.

[0305] The AI ​​model used by the string evaluator 221 to obtain the posterior probability according to an embodiment of the present disclosure may be trained based on the speech recognition results of multiple speech recognition modules to prevent the posterior probability from overfitting the speech recognition results of a specific speech recognition module.

[0306] The string evaluator 221 of the processor 220 may be configured to calculate the string value of the character string based on the posterior probability P(y p [L]|y o [0:L+1]) and character sequence probability P(y o [0:L+1]) to calculate the likelihood P(y o [0:L+1]|y p [L]).

[0307] Based on the posterior probability P(y p [L]|y o[0:L+1]) and character sequence probability P(y o [0:L+1]), the likelihood P(y o [0:L+1]|y p [L]).

[0308] [Equation 1]

[0309]

[0310] In [Equation 1], P(y p [L]) represents y p [L] The prior probability of a character y p The prior probability of [L] can be a value pre-calculated based on the usage frequency statistics of a certain character.

[0311] The decoder 223 of the server 200 according to the embodiment of the present disclosure can use the dictionary information 232 and the language model 233 to obtain the likelihood P(y o [0:L+1]|y p [L]) Estimate the second string W i The second character string may be a character string obtained by replacing at least one character of the first character string with another character. The communication interface 210 may replace the second character string W i Sent to the device 100. Although the server 200 receives the frame synchronization string y from the device 100 o [0:L+1], but the server 200 may send a second character string W having a word string shape to the device 100 i .

[0312] According to another embodiment of the present disclosure, the string evaluator 221 of the server 200 may calculate the likelihood by considering only each character without considering the characters accumulated before each character. According to another embodiment of the present disclosure, the string evaluator 221 may calculate the likelihood by considering only the character y o [L] instead of the frame synchronization string y o [0:L+1] to calculate the likelihood. When only considering character y o [L] instead of the frame synchronization string y o When [0:L+1], the server 200 can have a very simple structure, and the calculation process can be simplified because only the character-level confusion matrix is ​​stored and used instead of the neural network.

[0313] The communication interface 210 of the server 200 may receive the first string y of the frame synchronization from the device 100. o [0:L+1]. Referenced above Figure 6The frame synchronization character string is described, and thus a redundant description thereof will be omitted. The communication interface 210 may simultaneously receive the entire character string including a plurality of characters, or may sequentially receive some characters included in the character string.

[0314] When the first character yo[L] in the first character string has been estimated by the device 100, the string evaluator 221 of the server 200 according to another embodiment of the present disclosure may obtain that the Lth character is actually y p The posterior probability P(y p [L]|y o [L]). The string evaluator 221 can be based on the first character y o [L] to get the first character y o The posterior probability P(y p [L]|y o [L]). In other words, the posterior probability calculator 221 can be based on the first character y o [L] to obtain the first character y that the device 100 has accurately estimated o [L] and the probability that the device 100 has incorrectly estimated the first character y o The probability of [L].

[0315] The character string evaluator 221 according to an embodiment of the present disclosure may obtain a posterior probability of each character of the first character string according to the first character string by using a confusion matrix.

[0316] Figure 10b An example of a confusion matrix 1001 for calculating a posterior probability according to an embodiment of the present disclosure is shown.

[0317] The confusion matrix 1001 includes the probability that the speech recognition module of the apparatus 100 has accurately predicted a certain character included in the character string and the probability that another character is incorrectly predicted as the certain character.

[0318] For example, since the characters "a" and "e" have similar pronunciations, the probability that the speech recognition module will incorrectly estimate the actual character "a" as the character "e" may be relatively high. On the other hand, since the characters "a" and "b" have very different pronunciations, the probability that the speech recognition module will incorrectly estimate the actual character "a" as the character "b" may be relatively low.

[0319] Therefore, if Figure 10b As shown, the probability that the speech recognition module of the apparatus 100 incorrectly estimates the actual character "a" as the character "e" may be 0.23, and the probability that the speech recognition module incorrectly estimates the actual character "a" as the character "b" may be 0.01.

[0320] When the character estimated by the device 100 is the first character y o[L], the string evaluator 221 according to the embodiment of the present disclosure may be Figure 10b Search the confusion matrix 1001 and get the actual character is y p The posterior probability P(y p [L]|y o [L]).

[0321] The string evaluator 221 can be based on the obtained posterior probability P(y p [L]|y o [L]) to calculate the likelihood P(y o [L]|y p [L]).

[0322] Based on the posterior probability P(y p [L]|y o [L]), the likelihood P(y o [L]|y p [L]).

[0323] [Equation 2]

[0324]

[0325] In [Equation 2], P(y p [L]) represents y p [L] The prior probability of a character y p The prior probability of [L] can be a value pre-calculated based on the usage frequency statistics of a certain character.

[0326] The decoder 223 of the server 200 can use the dictionary information 232 and the language model 233 to calculate the likelihood P(y o [L]|y p [L]) Estimate the second string W i The second character string may be a character string obtained by replacing at least one character of the first character string with another character. The communication interface 210 may replace the second character string W i Sent to the device 100. Although the server 200 receives the frame synchronization string y from the device 100 o [0:L+1], but the server 200 may output a second character string W having a word string shape i .

[0327] As described above, the string evaluator 221 of the server 200 according to an embodiment of the present disclosure may receive a frame synchronization string from the device 100 and obtain the likelihood of each character corresponding to each speech signal frame. For example, the string evaluator 221 may calculate the likelihood of the character y of the index L corresponding to the speech signal frame. oThe likelihood P(y o [0:L+1]|y p [L]) or P(y o [L]|y p [L]).

[0328] Now refer to Figure 11a and Figure 11b A method of obtaining the likelihood of each character corresponding to each speech signal frame according to a character string received from the apparatus 100 , performed by the character string evaluator 221 according to an embodiment of the present disclosure, is described in detail.

[0329] refer to Figure 11a , the string evaluator 221 according to an embodiment of the present disclosure may receive the frame synchronization string 1101. The string evaluator 221 may calculate a likelihood matrix related to a replacement character to replace each character.

[0330] like Figure 11b As shown, the likelihood matrix associated with the replacement characters calculated by the string evaluator 221 according to the embodiment of the present disclosure can be represented as a matrix including the likelihood of a certain character being each of the arbitrary characters. Figure 11b As shown in Table 1105 , each of the arbitrary characters can be mapped to each of the indices of the likelihood matrix.

[0331] For example, the value of index a1 in likelihood matrix 1103 may represent the likelihood that a character is replaced by the character "a" corresponding to index a1. The value of index a2 in likelihood matrix 1103 may represent the likelihood that a character is replaced by the character "b" corresponding to index a2. The value of index a3 in likelihood matrix 1103 may represent the likelihood that a character is replaced by the character "c" corresponding to index a3.

[0332] The string evaluator 221 according to an embodiment of the present disclosure may calculate a likelihood matrix 1107 associated with a replacement character for each character in the string to be replaced. The string evaluator 221 may output the likelihood matrix 1107 calculated as the likelihood of a plurality of estimated strings, in which at least one character in the first string has been replaced, to the decoder 223.

[0333] The decoder 222 according to an embodiment of the present disclosure may obtain a character string with the highest confidence score as the second character string from among the confidence scores of the plurality of estimated character strings based on the likelihood received from the character string evaluator 221 by using the dictionary information and the language model.

[0334] As described above, the speech recognition system according to various embodiments of the present disclosure may perform device-side speech recognition in some cases, or may perform server-based speech recognition in other cases. However, embodiments of the present disclosure are not limited thereto. The device 300 according to an embodiment of the present disclosure may include multiple speech recognition modules, and may perform device-side speech recognition in a first speech recognition module in some cases, or may perform server-based speech recognition in a second speech recognition module in other cases.

[0335] Figure 12 is a block diagram of an apparatus 300 for selectively using two speech recognition modules according to an embodiment of the present disclosure.

[0336] refer to Figure 12 , the device 300 may include a receiver 310 , a processor 320 , a memory 340 and an output interface 350 . Figure 12 One or more components of the device 300 shown may be omitted or combined. Figure 12 Additional components other than those shown. For example, refer to Figure 19 According to some embodiments of the present disclosure, the device 300 may further include a user input interface 2100 , a sensing unit 2400 , and an A / V input interface 2600 .

[0337] The receiver 310 according to the embodiment of the present disclosure may receive a voice signal from the user. For example, the receiver 310 may receive the voice signal by converting external sound into electrical sound data via a microphone. Figure 12 In the embodiment, the receiver 310 is included in the device 300. However, according to another embodiment of the present disclosure, the receiver 310 may be included in a separate device and may be connected to the device 300 in a wired or wireless manner.

[0338] The memory 340 according to an embodiment of the present disclosure may store instructions for performing voice recognition, and various models, neural networks, dictionary information, etc. used in voice recognition.

[0339] The memory 340 may store various models, neural networks, and dictionary information used in speech recognition. The first data 341 stored in the memory 340 may include at least one of the models, neural networks, and dictionary information used by the first ASR module 321 to perform speech recognition. The second data 342 stored in the memory 340 may include at least one of the models, neural networks, and dictionary information used by the second ASR module 322 to perform speech recognition.

[0340] The processor 320 according to an embodiment of the present disclosure may perform speech recognition by executing one or more instructions stored in the memory 340. The processor 320 according to an embodiment of the present disclosure may include a first ASR module 321 and a second ASR module 322.

[0341] The first ASR module 321 of the processor 320 according to an embodiment of the present disclosure may receive a speech signal obtained by the receiver 310 and perform speech recognition on the speech signal based on first data 341 (e.g., an acoustic model, a neural network, a language model, or dictionary information). The first ASR module 321 may obtain a first string from the speech signal. The first string may be a frame synchronization string.

[0342] because Figure 12 The first ASR module 321 may correspond to Figure 4a ASR module 121 or Figure 4b The ASR module 121 is described below, and thus its redundant description will be omitted.

[0343] The determiner 323 of the processor 320 according to an embodiment of the present disclosure may determine whether to replace the first character string output by the first ASR module 321 with another character string.

[0344] For example, the determiner 323 of the processor 320 may determine a confidence score of the first character string, and may determine whether to replace the first character string with another character string based on the determined confidence score.

[0345] For example, when the confidence score of the first character string is equal to or greater than a threshold value, the determiner 323 of the processor 320 according to an embodiment of the present disclosure may determine that the first character string does not need to be corrected, and may output the first character string through the output interface 350. On the other hand, when the confidence score is less than the threshold value, the determiner 323 of the processor 320 may determine that the first character string needs to be corrected, and may send the first character string to the second ASR module 322.

[0346] For another example, the determiner 323 of the processor 320 may determine whether to replace the first string with another string based on a comparison result between a keyword pre-stored in the device 100 and the first string. For another example, the determiner 323 of the processor 320 may determine whether to replace the first string with another string based on whether the named entity is included in a domain related to the first string or in the first string.

[0347] Regarding the detailed method of determining whether to replace the first character string with another character string, performed by the determiner 323 of the processor 320 according to the embodiment of the present disclosure, the above reference Figures 3 to 5bThe method of determining whether to replace a first character string with another character string, which is executed by the processor 120 of the device 100 according to an embodiment of the present disclosure, will be described. A repeated description thereof will be omitted.

[0348] When it is determined that the first character string does not need to be corrected, the determiner 323 of the processor 320 according to an embodiment of the present disclosure may determine not to replace the first character string with another character string. When the first character string is not replaced with another character string, the determiner 323 of the processor 320 according to an embodiment of the present disclosure may output the first character string through the output interface 350.

[0349] When it is determined that the first character string needs to be replaced with another character string, the determiner 323 of the processor 320 according to an embodiment of the present disclosure may send the first character string to the second ASR module 322 based on the determination.

[0350] The determiner 323 of the processor 320 according to an embodiment of the present disclosure may transmit the first character string to the second ASR module 322 in units of sentences, words, phrases, or frames. When the first ASR module 321 of the processor 320 according to an embodiment of the present disclosure performs speech recognition to obtain a character string constituting a sentence or phrase, the determiner 323 may transmit all characters included in the sentence or phrase to the second ASR module 322, or transmit some characters included in the sentence or phrase to the second ASR module 322. Based on the confidence score of the character string, the determiner 323 may transmit some characters with low confidence scores to the second ASR module 322.

[0351] The second ASR module 322 of the processor 320 according to an embodiment of the present disclosure may receive and process the first string. The second ASR module 322 may obtain a second string by re-decoding the first string based on the language model and dictionary information stored in the second data 342, wherein at least one character in the first string has been replaced.

[0352] The second ASR module 322 may calculate likelihoods for a plurality of estimated character strings based on the first character string. Based on the calculated likelihoods, the second ASR module 322 may determine whether to replace the first character string with a second character string. Based on this determination, the second ASR module 322 may replace at least one character included in the first character string with another character to obtain the second character string from the first character string. The second ASR module 322 may obtain the second character string as one of the plurality of estimated character strings based on the likelihoods, dictionary information, and a language model.

[0353] because Figure 12 The second ASR module 322 may correspond to Figure 7 and Figure 9 The processor 220 is described below, and therefore a redundant description thereof will be omitted.

[0354] The second ASR module 322 may output the second character string via the output interface 350 .

[0355] The output interface 350 according to an embodiment of the present disclosure may output a speech recognition result corresponding to the first character string or the second character string. The output interface 350 may notify the user of the speech recognition result, or may send the speech recognition result to an external device (e.g., a smartphone, a home appliance, a wearable device, or a server). For example, the output interface 350 may include a speaker capable of outputting an audio signal or a display capable of outputting a video signal.

[0356] Alternatively, the device 300 according to an embodiment of the present disclosure may perform an operation corresponding to the result of interpreting the first or second character string. For example, the device 300 may determine the function of the device 300 corresponding to the speech recognition result and output a screen image for executing the determined function through the output interface 350. Alternatively, the device 300 may transmit keywords corresponding to the interpretation result to an external server, receive information related to the transmitted keywords from the external server, and output the received information to the screen through the output interface 350.

[0357] Alternatively, the device 300 according to an embodiment of the present disclosure may determine the user's utterance intention by performing natural language processing on the first character string or the second character string, and output information related to the voice assistant service through the output interface 350. The device 300 may use, for example, an NLU model, a DM model, and an NLG model within the device 300 to provide the voice assistant service.

[0358] For example, the device 300 may generate a response message to the user's voice input based on the first or second character string and output the generated response message, thereby simulating a person directly conversing with the user, taking into account the user's situation, the device's situation, and so on. For another example, the device 300 may generate information requested by the user based on the first or second character string and output the generated information. For another example, the device 300 may determine the user's utterance intent based on the first or second character string and request a service provider server to provide the requested information. The device 300 may transmit the information received from the service provider server via the output interface 350.

[0359] Compared to the first data 341 used by the first ASR module 321, the second ASR module 322 according to an embodiment of the present disclosure can use the second data 342 including many language models and a large amount of dictionary information. Compared to the first data 341, the second data 342 may include many named entities such as place names, personal names, and trademark names. Therefore, according to the speech recognition by the second ASR module 322, the dictionary information and language models including a large number of named entities can be used, and high-precision speech recognition can be performed.

[0360] therefore, Figure 12 Device 300 may perform general speech recognition, such as dictation, general commands, and subtitle generation, in first ASR module 321 to minimize latency. However, if the confidence score of the first character string output by first ASR module 321 is not high enough, device 300 may perform additional processing on the first character string in second ASR module 322. Second ASR module 322 may improve speech recognition accuracy by using second data 342 that includes more information than first data 341.

[0361] The processor 320 of the device 300 according to an embodiment of the present disclosure may obtain the corrected string from the second ASR module 322, and may combine the string not sent to the second ASR module 322 with the corrected string based on the determination that no correction is required. The device 300 according to an embodiment of the present disclosure may output the combined string, output the result of speech recognition performed based on the combined string, or provide a voice assistant service based on the result of interpreting the combined string.

[0362] According to an embodiment of the present disclosure, the determiner 323 of the processor 320 may provide domain information associated with the first character string to the second ASR module 322, while requesting the second ASR module 322 to correct the first character string. The domain information is information for identifying the domain, and may include, for example, a domain name and a domain identifier, but the embodiments of the present disclosure are not limited thereto.

[0363] The determiner 323 of the device 300 may identify the domain associated with the first string based on the domain confidence score of the first string output by the first ASR module 321. The domain confidence score may be a number indicating the strength of association of at least a portion of the first string with a particular domain. For example, the determiner 323 may calculate a confidence score indicating the strength of association between the first string output by the first ASR module 321 and a domain previously registered in the first data 341. The device 300 may identify the domain associated with the first string based on the calculated domain confidence score. The device 300 may identify the domain associated with the first string based on a rule, or may obtain the domain confidence score associated with the first string by using an AI model trained for domain identification.

[0364] The second ASR module 322 according to an embodiment of the present disclosure may decode the first character string using different pieces of dictionary information and different language models for different domains included in the second data 342. Therefore, the second ASR module 322 according to an embodiment of the present disclosure may output a speech recognition result with improved speech recognition accuracy by re-decoding the first character string.

[0365] The second ASR module 322 may receive the first character string from the determiner 323 and determine a domain associated with the first character string. The second ASR module 322 may decode the first character string using dictionary information and a language model corresponding to the determined domain.

[0366] For example, the second ASR module 322 may receive information about a domain associated with the first character string from the determiner 323 along with the first character string, and may determine a domain for performing decoding on the first character string based on the received information. For example, the second ASR module 322 may determine a domain that is the same as or similar to the domain identified based on the information received from the determiner 323 as the domain for performing decoding.

[0367] For another example, the second ASR module 322 may determine a domain associated with the first string received from the determiner 323 based on the received first string. The device 300 may store a domain recognition model in the memory 340, which is an AI model trained for domain recognition. The second ASR module 322 may output a domain confidence score by using the domain recognition model and the first string as an input value. The second ASR module 322 may determine the domain associated with the first string based on the domain confidence score.

[0368] For another example, the second ASR module 322 according to an embodiment of the present disclosure may receive information for determining a domain associated with the first character string from the determiner 323 together with the first character string. The information for determining the domain received from the determiner 323 may include context information. For example, the context information may include at least one of information about an application currently used by the user on the current device 300, conversation history information, situation information around the device 300, or trend information. The second ASR module 322 may determine a domain for performing decoding on the first character string based on the context information. A detailed method for determining a domain based on context information may be used. Figure 9 The operating method of the processor 220 is not described, and thus a redundant description thereof will be omitted.

[0369] The operation method of the device 100 according to the embodiment of the present disclosure will now be described in detail. The operation of the operation method of the device 100 to be described below can be respectively performed by Figure 3 、 Figure 4a and Figure 4b The device shown is used to perform the

[0370] Figure 13 The present invention is a flowchart of a method for performing speech recognition executed by a device according to an embodiment of the present disclosure.

[0371] In operation S1310, the device 100 according to an embodiment of the present disclosure may obtain a first character string by performing voice recognition on a voice signal.

[0372] The apparatus 100 according to an embodiment of the present disclosure may estimate the first character string by performing speech recognition according to various speech recognition methods.

[0373] For example, the apparatus 100 may obtain a character string from a speech signal by using an acoustic model, lexicon information, and a language model. First, the apparatus 100 may obtain a phoneme string included in the speech signal by using the acoustic model. For example, the apparatus 100 may estimate a phoneme string including phonemes by using a hidden Markov model, a Gaussian mixture model, Bayesian inference, a multi-layer neural network, or the like. Based on the lexicon information and the language model, the apparatus 100 may estimate a word from the phoneme string and obtain a first character string including the estimated word.

[0374] For another example, the apparatus 100 may extract a feature vector from the speech signal, and may output a first character string from the feature vector by using a DNN.

[0375] For example, the first character string may be a frame synchronization character string including characters corresponding to voice signal frames obtained by segmenting the voice signal at predetermined time intervals. Alternatively, for example, the first character string may be a character string including each character emitted by the voice signal, obtained using a tag synchronization method.

[0376] When the first string is not frame-synchronized, the apparatus 100 according to the embodiment of the present disclosure may obtain a frame-synchronized string by performing forced alignment. Figure 6 The description is applicable to the frame synchronization string and the detailed method of generating the frame synchronization string through forced alignment. The redundant description of the frame synchronization string and the detailed method of generating the frame synchronization string through forced alignment will be omitted.

[0377] In operation S1330 , the device 100 according to an embodiment of the present disclosure may determine whether to replace the first character string with another character string.

[0378] For example, device 100 according to an embodiment of the present disclosure may determine a confidence score for a first string and, based on the determined confidence score, determine whether to replace the first string with another string. For example, when the confidence score of the first string is equal to or greater than a threshold, device 100 may determine that the first string does not need to be replaced with another string. On the other hand, when the confidence score of the first string is lower than the threshold, device 100 may determine that the first string needs to be replaced with another string.

[0379] The confidence score of the first character string may be calculated based on at least one of likelihoods of a plurality of estimated character strings obtained from the first character string or a posterior probability that at least one character within the first character string is replaced with another character.

[0380] For example, the apparatus 100 may calculate the confidence score based on the likelihood output as a result of Viterbi decoding. Alternatively, the processor 120 may calculate the confidence score based on the posterior probability output from the softmax layer in the end-to-end speech recognition model.

[0381] Alternatively, the apparatus 100 according to an embodiment of the present disclosure may determine a plurality of estimated character strings estimated during speech recognition of a speech signal, and may calculate a confidence score for the first character string based on correlations between the plurality of estimated character strings. As the correlations between the plurality of estimated character strings including the first character string increase, the confidence score for the first character string may increase.

[0382] For another example, the device 100 may determine whether to replace the first string with another string based on a comparison result of a pre-stored keyword with the first string. For example, when the pre-stored keyword is not included in the first string, the device 100 may determine whether to replace the first string with another string.

[0383] For another example, the device 100 may determine whether to replace the first string with another string based on whether the named entity is included in a domain related to the first string or in the first string. For example, when it is determined that the first string is associated with a named entity-oriented domain, the device 100 may determine to replace the first string with another string.

[0384] When it is determined in operation S1330 that the first character string is replaced with another character string, in operation S1340, the device 100 according to the embodiment of the present disclosure may transmit the first character string to the server 200. The device 100 according to the embodiment of the present disclosure may transmit the first character string to the server 200 in frame synchronization. The device 100 may transmit the entire character string including multiple characters at the same time, or may transmit some characters included in the character string sequentially. The device 100 according to the embodiment of the present disclosure may transmit the first character string in units of words or sentences.

[0385] When it is determined in operation S1330 not to replace the first character string with another character string, the device 100 according to the embodiment of the present disclosure may output the first character string in operation S1370. The device 100 according to the embodiment of the present disclosure may output the first character string unchanged, or may output a word string obtained from the first character string.

[0386] In operation S1350, the device 100 according to an embodiment of the present disclosure may receive a second character string from the server 200. The second character string may be a character string obtained by the server 200 by replacing at least one character within the first character string with another character.

[0387] In operation S1360, the device 100 according to an embodiment of the present disclosure may output the second character string. The device 100 according to an embodiment of the present disclosure may output the second character string unchanged, or may output a word string obtained from the second character string. For example, the device 100 may output a modified first character string in which characters of the first character string are replaced with characters of the second character string, as shown in FIG. Figures 8a to 8b described.

[0388] The embodiments of the present disclosure are not limited to Figure 13 In an embodiment of the present disclosure, the device 100 outputs the first character string or the second character string unchanged. According to an embodiment of the present disclosure, the device 100 can determine the user's speech intention by performing natural language processing on the first character string or the second character string, and output information related to the voice assistant service.

[0389] The device 100 may provide a voice assistant service based on the first character string or the second character string using, for example, an NLU model, a DM model, and an NLG model within the device 100 .

[0390] For example, device 100 may generate a response message based on the first or second character string and output the generated response message, thereby simulating a person directly conversing with the user, taking into account the user's situation, the device's situation, and the like. For another example, device 100 may generate information requested by the user based on the first or second character string and output the generated information. For another example, device 100 may determine the user's utterance intent based on the first or second character string and request a service provider server to provide the user's requested information. Device 100 may output the information received from the service provider server.

[0391] The device 100 according to an embodiment of the present disclosure may receive and output information related to a voice assistant service generated based on the second character string, rather than receiving the second character string from the server 200. The information related to the voice assistant service may be information generated by the server 200 based on the second character string obtained by correcting the first character string. For example, the information related to the voice assistant service may include a response message to the user's voice signal, a service required by the user, or information required by the user.

[0392] like Figure 13 As shown, the device 100 according to an embodiment of the present disclosure may determine whether to replace a first character string output by the device-side speech recognition module with another character string, and may selectively use server-based post-processing based on the determination result.

[0393] The device 100 according to an embodiment of the present disclosure may calculate a confidence score of a first character string output by the device-side speech recognition module in units of words (or sentences) spoken by a user, and may determine whether to replace the first character string based on the calculated confidence score.

[0394] Figure 14 yes Figure 13 Detailed flowchart of operation S1310 in the method for performing speech recognition performed by a device according to an embodiment of the present disclosure.

[0395] In operation S1411, the apparatus 100 according to an embodiment of the present disclosure may receive a speech signal. In operation S1413, the apparatus 100 according to an embodiment of the present disclosure may determine whether a word boundary has been detected. The apparatus 100 according to an embodiment of the present disclosure may continue to receive a speech signal including a speech frame until a word boundary is detected.

[0396] For example, the apparatus 100 may detect word boundaries based on pauses detected from a speech signal or prosodic information including stress and intonation.

[0397] When the word boundary is detected, in operation S1415 , the device 100 according to the embodiment of the present disclosure may obtain a first character string from the speech signal.

[0398] In operation S1431, the apparatus 100 according to an embodiment of the present disclosure may calculate a confidence score of the first character string. The confidence score of the first character string may be calculated based on at least one of the likelihoods of a plurality of estimated character strings obtained from the first character string or a posterior probability that at least one character in the first character string is replaced with another character.

[0399] For example, the apparatus 100 may calculate the confidence score based on the likelihood output as a result of Viterbi decoding. Alternatively, the apparatus 100 may calculate the confidence score based on the posterior probability output from the softmax layer in the end-to-end speech recognition model.

[0400] Alternatively, the apparatus 100 according to an embodiment of the present disclosure may determine a plurality of estimated character strings estimated during speech recognition of a speech signal, and may calculate a confidence score for the first character string based on correlations between the plurality of estimated character strings. As the correlations between the plurality of estimated character strings including the first character string increase, the confidence score for the first character string may increase.

[0401] In operation S1433 , the device 100 according to an embodiment of the present disclosure may determine whether the confidence score of the first character string is less than a threshold value.

[0402] When it is determined that the confidence score of the first character string is less than the threshold, in operation S1340, the device 100 according to an embodiment of the present disclosure may transmit the first character string to the server 200. In response to the transmitted first character string, the device 100 may receive a second character string in which at least one character in the first character string has been replaced with another character. The device 100 may output the received second character string.

[0403] On the other hand, when it is determined that the confidence score of the first character string is equal to or greater than the threshold value, in operation S1370, the device 100 according to an embodiment of the present disclosure may output the first character string.

[0404] The device 100 according to an embodiment of the present disclosure may determine the user's utterance intention by performing natural language processing on the first character string or the second character string, and output information related to the voice assistant service, rather than outputting the first character string or the second character string unchanged.

[0405] The device 100 may provide a voice assistant service based on the first character string or the second character string using, for example, an NLU model, a DM model, and an NLG model within the device 100 .

[0406] For example, device 100 may generate a response message based on the first or second character string and output the generated response message, thereby simulating a person directly conversing with the user, taking into account the user's situation, the device's situation, and the like. For another example, device 100 may generate information requested by the user based on the first or second character string and output the generated information. For another example, device 100 may determine the user's utterance intent based on the first or second character string and request a service provider server to provide the user's requested information. Device 100 may output the information received from the service provider server.

[0407] The device 100 according to an embodiment of the present disclosure may receive information related to a voice assistant service generated based on a second character string instead of receiving the second character string from the server 200 in response to the transmitted first character string. The device 100 may output the information received from the server 200. The information related to the voice assistant service may be information generated by the server 200 based on the second character string obtained by correcting the first character string.

[0408] For example, information related to the voice assistant service may include a response message to a user voice signal, a service required by the user, or information required by the user.

[0409] like Figure 14 As shown, device 100 according to an embodiment of the present disclosure may determine whether to replace the first string with another string based on the confidence score of the first string. When the confidence score of the first string is less than a threshold, device 100 may send the first string to server 200. Device 100 may obtain a second string from server 200 by replacing at least one character of the first string with another character based on the dictionary information and language model within server 200. Therefore, device 100 according to an embodiment of the present disclosure may improve the accuracy of speech recognition by receiving a second string from server 200 having a higher confidence score than the first string and using the received second string.

[0410] Figure 14 The present disclosure illustrates an embodiment in which the confidence score of the speech recognition result is calculated based on the words spoken by the user and the determination of whether to replace the first character string is made. However, the embodiments of the present disclosure are not limited thereto. The apparatus 100 according to an embodiment of the present disclosure may calculate the confidence score of the speech recognition result based on the sentences spoken by the user and determine whether to replace the first character string. Various conventional methods can be used to detect the end of a sentence spoken by the user, and a detailed description thereof will be omitted herein.

[0411] Figure 15 is a flow chart of the operating method of the server according to an embodiment of the present disclosure. The operations of the operating method of the server 200 to be described below can be respectively performed by Figure 7 and Figure 9 to execute on the server shown.

[0412] In operation S1510, the server 200 according to an embodiment of the present disclosure may receive a first character string from the device 100. The first character string may be output by the device 100 performing voice recognition on a voice signal.

[0413] For example, the first character string received by the server 200 may be a frame synchronization character string including characters corresponding to voice signal frames obtained by dividing the voice signal at preset time intervals. In another example, the first character string received by the server 200 may be a non-frame synchronization character string.

[0414] When the first character string received from the device 100 is a non-frame synchronization character string, the processor 220 according to an embodiment of the present disclosure can obtain a frame synchronization character string from the first character string. The processor 220 can obtain the frame synchronization character string by continuously arranging at least one character included in the first character string multiple times in units of frames.

[0415] In operation S1520, the server 200 according to an embodiment of the present disclosure may calculate likelihoods of multiple estimated character strings based on the first character string. The server 200 according to an embodiment of the present disclosure may obtain the multiple estimated character strings by replacing each character included in the first character string with another character. The likelihoods of the multiple estimated character strings may refer to the probability that the speech recognition module estimates the first character string when assuming that each of the multiple estimated character strings obtained from the first character string is a substantially true character string.

[0416] According to an embodiment of the present disclosure, the server 200 may identify replacement characters that are similar in pronunciation to each character in the first character string, and may obtain a likelihood obtained from the first character string based on the identified replacement characters to determine an estimated character string in which at least one character in the first character string has been corrected to another character.

[0417] The server 200 according to an embodiment of the present disclosure may calculate a likelihood matrix associated with a replacement character to be substituted for each character in the first character string, and may identify multiple estimated character strings based on the likelihood values ​​in the likelihood matrix. The server 200 may output the likelihood matrix obtained from each character as the likelihood of the multiple estimated character strings.

[0418] For example, the server 200 may calculate the likelihood of the first string based on the characters accumulated before each character in the first string. The server 200 according to an embodiment of the present disclosure may calculate the posterior probability of each character in the first string based on the characters accumulated before each character in the first string. The server 200 may calculate the character sequence probability based on the characters accumulated before each character in the first string. The server 200 may calculate the likelihood of multiple estimated character strings obtained from the first string based on the posterior probability of each character and the character sequence probability.

[0419] For another example, server 200 may calculate the likelihood of the first string by considering only each character in the first string without considering the characters accumulated before each character in the first string. Server 200 according to an embodiment of the present disclosure may calculate the posterior probability of each character in the first string based on a predetermined confusion matrix. Server 200 may calculate the likelihood of multiple estimated strings obtained from the first string based on the posterior probability of each character.

[0420] In operation S1530 , the server 200 according to an embodiment of the present disclosure may obtain a second character string from the first character string by replacing at least one character included in the first character string with another character based on the likelihood calculated in operation S1520 .

[0421] The server 200 according to an embodiment of the present disclosure may identify multiple estimated character strings based on the calculated likelihoods, wherein at least one character in a first character string has been replaced with another character. The server 200 may obtain a second character string from the multiple estimated character strings based on the likelihoods of the identified multiple estimated character strings, a language model, and dictionary information.

[0422] According to an embodiment of the present disclosure, server 200 may determine whether to replace a first character string with a second character string based on the calculated likelihood. Based on this determination, server 200 may obtain a second character string from the first character string by replacing at least one character included in the first character string with another character. Server 200 may select an estimated character string with the highest likelihood from a plurality of estimated character strings based on the likelihood, dictionary information, and a language model. Server 200 may obtain a second character string, in which at least one character in the first character string has been replaced with another character, based on the selected estimated character string.

[0423] For example, the server 200 may obtain the second character string by using a WFST decoder based on the dictionary information and the language model stored in the server 200. When the server 200 performs WFST decoding, the server 200 according to an embodiment of the present disclosure may construct and decode a search space by using a WFST based on the relationship T between characters, the dictionary information L including mapping information between words and characters, and the language model G that estimates the probability of the next word when assigning a specific word string.

[0424] For another example, server 200 may include a Viterbi decoder that recalculates the likelihoods of multiple estimated character strings obtained from a first character string based on lexicon information and a language model. The Viterbi decoder may determine, from the multiple estimated character strings, a second character string with the highest recalculated likelihood. Taking into account the lexicon information and the language model, the Viterbi decoder may search for a character string with the highest likelihood relative to the first character string as the second character string.

[0425] In operation S1540, the server 200 according to an embodiment of the present disclosure may transmit the second character string to the device 100. Alternatively, the server may transmit the one or more characters to be replaced and the one or more replacement characters to the device 100.

[0426] The server 200 according to an embodiment of the present disclosure may interpret the second character string by using an NLU model, and may generate a response message to the user's voice signal based on the interpretation result. The server 200 may generate the response message and also transmit the response message to the device 100.

[0427] The embodiments of the present disclosure are not limited to Figure 15 In an embodiment of the present disclosure, the server 200 sends the second character string to the device 100 without changing it. The server 200 according to an embodiment of the present disclosure can determine the user's speech intention by performing natural language processing on the second character string and send information related to the voice assistant service.

[0428] The server 200 may provide a voice assistant service based on the second character string using, for example, an NLU model, a DM model, and an NLG model within the server 200 .

[0429] For example, server 200 may generate a control command for controlling device 100 or another device based on the result of interpreting the second character string and transmit the generated control command to device 100. For another example, server 200 may generate a response message based on the second character string and transmit the generated response message, simulating a person directly conversing with the user, taking into account the user's situation, the device's situation, and so on. For another example, server 200 may generate information requested by the user based on the second character string and transmit the generated information. For another example, server 200 may determine the user's utterance intent based on the second character string and request a service provider server to provide the service requested by the user. Server 200 may transmit the information received from the service provider server.

[0430] Figure 16 Detailed flowchart of a method for obtaining likelihood from a character string by taking into account characters accumulated before each character in a method for operating a server according to an embodiment of the present disclosure.

[0431] In operation S1510 , the server 200 according to an embodiment of the present disclosure may receive a first character string from the device 100 . Figure 16 Operation S1510 in Figure 15 The operations in are the same, so further redundant description is omitted.

[0432] In operation S1621 , the server 200 according to an embodiment of the present disclosure may obtain a posterior probability of each character within the first character string based on characters accumulated before each character in the first character string.

[0433] For example, the server 200 may calculate the posterior probability of each character in the first character string by using a neural network pre-trained for calculating the posterior probability of a character string.

[0434] In operation S1623 , the server 200 according to an embodiment of the present disclosure may calculate a character sequence probability based on the first character string.

[0435] In operation S1625, the server 200 according to an embodiment of the present disclosure may calculate the likelihoods of the plurality of estimated character strings obtained from the first character string based on the posterior probability calculated in operation S1621 and the character sequence probability calculated in operation S1623. The server 200 according to an embodiment of the present disclosure may calculate a likelihood matrix associated with a replacement character to be substituted for each character in the first character string, and may obtain the likelihoods of the plurality of estimated character strings based on the calculated likelihood matrix.

[0436] The server 200 according to an embodiment of the present disclosure may determine whether the likelihood matrix has been calculated for all characters included in the first character string. The server 200 according to an embodiment of the present disclosure may repeatedly perform operations S1621, S1623 and S1625 until the likelihood matrix has been calculated for all characters included in the first character string.

[0437] Referenced above Figure 9 The detailed process of calculating the likelihood from the first character string is described, and thus a redundant description thereof will be omitted.

[0438] In operation S1627, the server 200 may obtain a second character string from the likelihood calculated in operation S1625 by using the dictionary information and the language model. The second character string may be a character string obtained by replacing at least one character of the first character string with another character.

[0439] For example, the server 200 may obtain the second character string from the plurality of estimated character strings through a WFST decoder using the likelihood as an input or a Viterbi decoder using typical token passing based on the dictionary information, the language model, and the calculated likelihood.

[0440] In operation S1540, the server 200 according to an embodiment of the present disclosure may transmit the second character string to the device 100. The server 200 according to an embodiment of the present disclosure may determine the user's utterance intention by performing natural language processing on the second character string and transmit information related to the voice assistant service, rather than transmitting the second character string unchanged to the device 100. A repeated description thereof will be omitted. Figure 16 Operation 1540 in the Figure 15 The operations in are the same, so further redundant description is omitted.

[0441] Figure 17 is a diagram for explaining WFST decoding according to an embodiment of the present disclosure.

[0442] The server 200 according to an embodiment of the present disclosure may calculate a likelihood based on the first character string received from the device 100, and may perform WFST decoding by using the calculated likelihood as an input. The server 200 according to an embodiment of the present disclosure may perform WFST decoding by using a WFST by modeling each of the likelihoods T of a plurality of estimated character strings obtained from the first character string, the dictionary information L including mapping information between words and characters, and the language model G of the probability of the next word when a specific word string is assigned.

[0443] An example of modeling a language model that stores information about the relationship between the words “the,” “cat,” “and,” “deer,” “is,” “cardinal,” “baseball,” and “to” using a WFST will now be described. Figure 17 A limited number of character strings that can be formed by combining words based on the language model are shown.

[0444] exist Figure 17 In the example, each circle represents a state, and words stored in the language model are represented by arrows. The WFST decoder can calculate a confidence score for each of the multiple strings combined along multiple paths. The confidence score for each string can be calculated based on its likelihood, dictionary information, and the language model. The WFST decoder can select and output the string with the highest confidence score.

[0445] For example, Figure 8a As shown, the server 200 according to an embodiment of the present disclosure may receive a first character string [Cat Deer Baseball Team] from the device 100 .

[0446] Server 200 may calculate likelihoods for the plurality of estimated character strings obtained from the first character string. When the calculated likelihoods are input to a WFST decoder of server 200, the WFST decoder may output a second character string. The WFST decoder may determine the second character string with the highest confidence score from among the confidence scores of the plurality of estimated character strings and may output the determined second character string.

[0447] like Figure 8a As shown, the named entity "Cardinals" in the sports field may be stored in the memory 230 of the server 200. Therefore, the processor 220 of the server 200 may determine that the "Cat and Deer" estimated by the device 100 is actually likely to be the name of the baseball team "Cardinals".

[0448] Therefore, reference Figure 17 According to an embodiment of the present disclosure, the WFST decoder can determine the character string [Cardinals Baseball Team] with the highest confidence as the second character string from multiple estimated character strings [Cat Deer Baseball Team] and [Cardinals Baseball Team], and can output the determined second character string.

[0449] Figure 18 A screen of a device for displaying a speech recognition result according to an embodiment of the present disclosure is shown.

[0450] The device 100 according to an embodiment of the present disclosure may output a word string 1811 "Cat Deer Baseball Team" obtained from a character string estimated from a voice signal received from a user via voice recognition. When performing device-side voice recognition, the device 100 may display an image 1812 on the screen indicating that device-side voice recognition is being performed.

[0451] When the confidence score of the result of performing voice recognition by using the device-side voice recognition is sufficiently high, the device 100 according to an embodiment of the present disclosure may use the result of performing voice recognition without change.

[0452] On the other hand, when the device 100 according to an embodiment of the present disclosure determines that the confidence score of the result of voice recognition performed via device-side voice recognition is not high enough, the device 100 may send a character string corresponding to the voice recognition result to the server 200 .

[0453] The server 200 according to an embodiment of the present disclosure may receive a character string from the device 100 and decode the character string using a language model and dictionary information within the server 200, thereby obtaining the character string "Red Cardinals Baseball Team" in which at least one character included in the character string has been corrected. The server 200 may transmit "Red Cardinals Baseball Team" to the device 100.

[0454] The device 100 according to an embodiment of the present disclosure may output a character string 1821 “Cardinals Baseball Team” received from the server 200. When performing server-based voice recognition, the device 100 may display an image 1822 on the screen indicating that server-based voice recognition is being performed.

[0455] Figure 19 is a detailed block diagram of a device according to an embodiment of the present disclosure.

[0456] Figure 19 The device 100 may include the above reference Figure 3 For example, the same components as those of the device 100 described in Figure 19 The processor 2300 in the component can be used with Figure 3 The processor 120 is the same as that of the processor 120, and the output interface 2220 can be used with Figure 3 The output interface 150 is the same. Although Figure 19 Not shown, but Figure 19 The memory 2700 can store instructions for performing speech recognition and various models, neural networks, dictionary information, etc. used in speech recognition, such as Figure 3 Therefore, repeated description thereof will be omitted herein.

[0457] Figure 19 The device 100 may execute the above reference Figures 3 to 18 Therefore, components of the apparatus 100 not described above will now be described.

[0458] refer to Figure 19 , the device 100 may include a user input interface 2100 , an output interface 2200 , a processor 2300 , a sensing unit 2400 , a communication interface 2500 , an A / V input interface 2600 , and a memory 2700 .

[0459] The user input interface 2100 represents a device through which a user inputs data for controlling the device 100. For example, the user input interface 2100 may include, but is not limited to, a keypad, a dome switch, a touchpad (capacitive type, resistive type, infrared beam type, integral strain gauge type, surface acoustic wave type, piezoelectric type, etc.), a scroll wheel, or a toggle switch. The user input interface 2100 may receive user elements required to generate session information to be provided to the user.

[0460] The output interface 2200 may output an audio signal, a video signal, or a vibration signal, and may include a display 2210 , an audio output interface 2220 , and a vibration motor 2230 .

[0461] The vibration motor 2230 may output a vibration signal. For example, the vibration motor 2230 may output a vibration signal corresponding to the output of audio data or video data (eg, a call signal receiving a voice or a message receiving a voice).

[0462] The sensing unit 2400 may sense a state of the device 100 or a state around the device 100 and may transmit information corresponding to the sensed state to the processor 2300 .

[0463] The sensing unit 2400 may include, but is not limited to, at least one selected from the following: a magnetic sensor 2410, an acceleration sensor 2420, a temperature / humidity sensor 2430, an infrared sensor 2440, a gyroscope sensor 2450, a position sensor 2460 (e.g., a global positioning system (GPS)), a pressure sensor 2470, a proximity sensor 2480, and an RGB sensor 2490 (i.e., an illumination sensor). A person of ordinary skill in the art can intuitively understand the functions of most sensors through their names, so detailed descriptions thereof will be omitted herein.

[0464] The communication interface 2500 may include components for communicating with another device. For example, the communication interface 2500 may include a short-range wireless communication interface 2510, a mobile communication interface 2520, and a broadcast receiver 2530.

[0465] Examples of the short-range wireless communication interface 2510 may include, but are not limited to, a Bluetooth communication interface, a Bluetooth low energy (BLE) communication interface, a near field communication (NFC) interface, a wireless local area network (WLAN) (e.g., Wi-Fi) communication interface, a Zigbee communication interface, an infrared data association (IrDA) communication interface, a Wi-Fi Direct (WFD) communication interface, an ultra-wideband (UWB) communication interface, and an Ant+ communication interface.

[0466] The mobile communication interface 2520 may exchange wireless signals with at least one selected from a base station, an external terminal, and a server on a mobile communication network. Examples of wireless signals may include voice call signals, video call signals, or various types of data transmitted according to text / multimedia messages.

[0467] The broadcast receiver 2530 receives a broadcast signal and / or broadcast-related information from an external source via a broadcast channel. The broadcast channel may be a satellite channel, a ground wave channel, etc. According to an embodiment of the present disclosure, the device 100 may not include the broadcast receiver 2530.

[0468] The communication interface 2500 may transmit or receive information required to generate session information to be provided to the user to or from the second interactive electronic device, another device, and a server.

[0469] The A / V input interface 2600 outputs an audio signal or a video signal and may include a camera 2610 and a microphone 2620. The camera 2610 may obtain image frames such as still images or moving images via an image sensor in a video call mode or a shooting mode. Images captured via the image sensor may be processed by the processor 2300 or a separate image processor (not shown).

[0470] The image frame obtained by the camera 2610 may be stored in the memory 2700 or transmitted to the outside via the communication interface 2500. Depending on the configuration of the device 100, at least two cameras 2610 may be included.

[0471] Microphone 2620 receives an external audio signal and converts the external audio signal into an electronic audio signal. For example, microphone 2620 may receive an audio signal from an external device or a speaker. Microphone 2620 may use various noise cancellation algorithms to remove noise generated when receiving the external audio signal.

[0472] The memory 2700 may store programs used by the processor 2300 to perform processing and control, and may also store data input to or output from the device 100 .

[0473] Memory 2700 may include at least one type of storage medium selected from the following: flash memory, hard disk memory, multimedia card micro memory, card memory (for example, secure digital (SD) or extreme digital (XD) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), programmable ROM (PROM), magnetic memory, magnetic disk and optical disk.

[0474] Programs stored in the memory 2700 may be classified into a plurality of modules according to their functions, for example, a user interface (UI) module 2710 , a touch screen module 2720 , and a notification module 2730 .

[0475] The UI module 2710 may provide a UI, a graphical user interface (GUI), etc. that is specific to each application and interoperates with the device 100. The touch screen module 2720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the processor 2300. The touch screen module 2720 according to some embodiments of the present disclosure may recognize and analyze touch codes. The touch screen module 2720 may be configured by separate hardware including a controller and may be included in the display 2210 of the output interface 2200.

[0476] The notification module 2730 may generate a signal for notifying that an event has been generated in the device 100. Examples of events generated in the electronic device 100 may include call signal reception, message reception, key signal input, schedule notification, etc. The notification module 2730 may output the notification signal in the form of a video signal via the display 2210, in the form of an audio signal via the audio output interface 2220, and in the form of a vibration signal via the vibration motor 2230.

[0477] The disclosed embodiments may be implemented as one or more software programs comprising computer-readable instructions stored in a computer-readable storage medium and / or a memory of the apparatus 100 .

[0478] The computer is a device that can call stored instructions from a storage medium and operate according to the embodiments of the present disclosure according to the called instructions, and may include the image transmitting device and the image receiving device according to the embodiments of the present disclosure.

[0479] The computer-readable storage medium may be configured as a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not include a distinction between whether data is stored semi-permanently or temporarily in the storage medium.

[0480] Furthermore, the electronic device or method according to the embodiments of the present disclosure may be provided in a computer program product, which may be traded as a commodity between a seller and a buyer.

[0481] A computer program product may include a software program and a computer-readable storage medium storing the software program. For example, a computer program product may include a product (e.g., a downloadable application) in the form of a software program that is electronically distributed by the manufacturer of an electronic device or an electronic marketplace (e.g., Google Play Store, Apple Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or may be temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic marketplace's server, or a relay server used to temporarily store the software program.

[0482] The computer program product may include the storage medium of a server or the storage medium of a terminal in a system consisting of a server and a terminal (e.g., an image transmitting device or an image receiving device). Alternatively, when a third device (e.g., a smartphone) is present for communication with the server or terminal, the computer program product may include the storage medium of the third device. Alternatively, the computer program product may include the software program itself, which is transmitted from the server to the terminal or third device, or from the third device to the terminal.

[0483] In this case, one of the server, terminal, and third device may run the computer program product to perform the method according to the embodiment of the present disclosure. Alternatively, at least two of the server, terminal, and third device may run the distributed computer program product and perform the method according to the embodiment of the present disclosure.

[0484] For example, a server (eg, a cloud server or an AI server, etc.) may execute a computer program product stored on the server to control a terminal communicating with the server to execute a method according to an embodiment of the present disclosure.

[0485] For another example, the third device may execute a computer program product to control a terminal communicating with the third device to execute a method according to an embodiment of the present disclosure. For example, the third device may remotely control an image sending device to send a package image or an image receiving device to receive a package image.

[0486] When the third device runs the computer program product, the third device may download the computer program product from the server and run the downloaded computer program product. Alternatively, the third device may run the computer program product provided in a preloaded state to perform the method according to an embodiment of the present disclosure.

Claims

1. A server, comprising: a memory storing one or more computer-readable instructions; a processor configured to execute the one or more computer-readable instructions stored in the memory; as well as a communication interface configured to receive a first character string from a device, wherein the first character string is obtained by the device performing voice recognition on a voice signal input to the device, Wherein, when executing the one or more computer-readable instructions, the processor is configured to: identifying an estimated character string based on the first character string, the estimated character string being used to replace a portion of the first character string; and controlling the communication interface to send a second character string to the device, in which the portion of the first character string is replaced with the estimated character string; The step of identifying, based on the first character string, an estimated character string for replacing a portion of the first character string includes: identifying a replacement character corresponding to each character in the portion of the first character string, and determining a plurality of estimated character strings based on at least one of the replacement characters; Calculating a likelihood matrix associated with the replacement character; Calculating likelihoods of the plurality of estimated character strings based on likelihood values ​​in the likelihood matrix; and The estimated character string used to replace the portion in the first character string is selected from the plurality of estimated character strings based on the likelihoods of the plurality of estimated character strings.

2. The server according to claim 1, wherein The replacement characters are characters that are similar in pronunciation to each character in the first character string.

3. The server according to claim 1, wherein The processor, when executing the one or more computer-readable instructions, is further configured to: The estimated character string used to replace a portion of the first character string is selected from the plurality of estimated character strings based on the likelihood, dictionary information, and a language model.

4. The server according to claim 1, wherein The likelihood matrix obtained for each character of the first character string is calculated based on characters of the first character string accumulated before each character of the first character string.

5. The server according to claim 1, wherein The likelihood matrix obtained for each character of the first character string is calculated based on: posterior probabilities calculated based on characters of the first character string accumulated before each character of the first character string; and a character sequence probability calculated based on characters of the first character string accumulated before each character of the first character string. The server according to claim 5 , wherein: The posterior probability is calculated using an artificial intelligence recurrent neural network RNN ​​including multiple long short-term memory (LSTM) layers and a flexible maximum transfer function (softmax) layer.

7. The server according to claim 1, wherein: The likelihood matrix obtained for each character of the first character string is calculated based on a predetermined confusion matrix.

8. The server according to claim 1, wherein The first character string includes characters respectively corresponding to speech signal frames obtained by dividing the speech signal at preset time intervals.

9. The server according to claim 1, wherein: The processor, when executing the one or more computer-readable instructions, is further configured to provide a service associated with the voice signal input to the device based on the second character string.

10. A device comprising: a memory storing one or more computer-readable instructions; a processor configured to execute the one or more computer-readable instructions stored in the memory; as well as a communication interface configured to communicate with a server, Wherein, when executing the one or more computer-readable instructions, the processor is further configured to: obtaining a first character string by performing speech recognition on the speech signal; determining whether to replace a portion of the first character string with another character string; Based on the determination, controlling the communication interface to send the first character string to the server; and controlling the communication interface to receive a second character string from the server, where the second character string is obtained by the server replacing the portion included in the first character string with an estimated character string; The estimated character string used to replace the portion of the first character string is obtained by the server by performing the following steps: identifying a replacement character corresponding to each character in the portion of the first character string, and determining a plurality of estimated character strings based on at least one of the replacement characters; Calculating a likelihood matrix associated with the replacement character; Calculating likelihoods of the plurality of estimated character strings based on likelihood values ​​in the likelihood matrix; and The estimated character string used to replace the portion in the first character string is selected from the plurality of estimated character strings based on the likelihoods of the plurality of estimated character strings.

11. A method for operating a server, the method comprising: receiving a first character string from a device, the first character string being obtained by the device performing speech recognition on a speech signal input to the device; identifying an estimated character string based on the first character string, the estimated character string being used to replace a portion of the first character string; obtaining a second character string based on the estimated character string; sending the second character string to the device, wherein the portion of the first character string is replaced with the estimated character string; The step of identifying, based on the first character string, an estimated character string for replacing the portion of the first character string comprises: identifying a replacement character corresponding to each character in the portion of the first character string, and determining a plurality of estimated character strings based on at least one of the replacement characters; Calculating a likelihood matrix associated with the replacement character; Calculating likelihoods of the plurality of estimated character strings based on likelihood values ​​in the likelihood matrix; and The estimated character string used to replace the portion in the first character string is selected from the plurality of estimated character strings based on the likelihoods of the plurality of estimated character strings.

12. The operating method according to claim 11, wherein The replacement characters are characters that are similar in pronunciation to each character in the first character string.

13. The operating method according to claim 11, wherein: Selecting the estimated character string from the plurality of estimated character strings to replace the portion of the first character string includes: The estimated character string used to replace a portion of the first character string is selected from the plurality of estimated character strings based on the likelihood, dictionary information, and a language model.

Images