Methods and apparatus for speech processing

Abstract

A method and apparatus for speech processing are disclosed. The method includes: determining a target segment from a speech signal; determining a target character sequence corresponding to the target segment by decoding the target segment; identifying a first overlap between the target character sequence and a previous character sequence based on an edit distance; and merging the target character sequence and the previous character sequence based on the first overlap. The cost of the edit distance is determined based on any one or any combination of any two or more of the following: the type of operation performed at the edit distance, whether the character to be operated on is located in the first overlap, and whether the characters to be operated on match.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0051269, filed with the Korean Intellectual Property Office on April 28, 2020, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a method and apparatus for speech processing. Background Technology

[0003] Speech recognition technology can be used to recognize a user's voice. The recognized user's voice can then be converted into text. With the development of speech recognition technology, users increasingly need voice commands to control devices or act as agents. However, due to limited training data for the models used to perform speech recognition or environmental noise in the speech signal, it is difficult to guarantee optimal performance. Summary of the Invention

[0004] The present invention is provided in a simplified form to describe the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a method includes: determining a target segment from a speech signal; determining a target character sequence corresponding to the target segment by decoding the target segment; identifying a first overlap between the target character sequence and a previous character sequence based on an edit distance; and merging the target character sequence and the previous character sequence based on the first overlap. A cost applied to the edit distance is determined based on any one or any combination of the type of operation performed at the edit distance, whether the character to be operated on is located in the first overlap, and whether the characters to be operated on match.

[0006] The cost of edit distance can be applied differently to the first overlapping and non-overlapping parts between the target character sequence and the previous character sequence.

[0007] The insertion cost of the edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence can be less than the insertion cost occurring in the first overlapping portion.

[0008] The matching cost of edit distance can be less than the insertion cost of edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence.

[0009] The insertion cost, deletion cost, and replacement cost of the edit distance occurring in the first overlapping portion can be positive numbers.

[0010] The matching cost of edit distance can be negative, and the insertion cost of edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence can be zero.

[0011] The steps of determining a target segment from a speech signal may include: determining the target segment from the speech signal based on the segment length and the overlap length between segments.

[0012] The overlap length between segments can be determined based on the available resources of the device performing the method.

[0013] The confirmation steps may include: confirming a second overlapping portion between the target character sequence and a partial sequence extracted from a previous character sequence based on the maximum character length spoken by the user during the overlap length between segments.

[0014] The previous character sequence can be determined based on the result of decoding a previous segment that partially overlaps with the target segment in the speech signal.

[0015] The verification steps may include: verifying the third overlapping portion between the target segment and the result of decoding the previous segment, which is included in the previous character sequence.

[0016] The first overlapping portion in the target segment can be less than half the size of the target segment.

[0017] A non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, configure the one or more processors to perform the method.

[0018] In another overall aspect, an apparatus includes: one or more processors configured to: determine a target segment from a speech signal; determine a target character sequence corresponding to the target segment by decoding the target segment; identify an overlap between the target character sequence and a previous character sequence based on an edit distance; and merge the target character sequence and the previous character sequence based on the overlap. A cost applied to the edit distance is determined based on any one or any combination of the type of operation performed at the edit distance, whether the character to be operated on is located in the overlap, and whether the characters to be operated on match.

[0019] The cost of edit distance can be applied differently to the non-overlapping and overlapping portions between the target character sequence and the previous character sequence.

[0020] The matching cost of edit distance can be less than the insertion cost of edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence.

[0021] The insertion, deletion, and replacement costs of the edit distance occurring in the overlap between the target character sequence and the previous character sequence can be positive numbers.

[0022] The one or more processors may be configured to determine a target segment from the speech signal based on the segment length and the overlap length between segments.

[0023] The overlap length between segments can be determined based on the available resources of the device.

[0024] The previous character sequence can be determined based on the result of decoding a previous segment that partially overlaps with the target segment in the speech signal.

[0025] In another general aspect, a method includes: determining a plurality of segments from a speech signal; determining a plurality of character sequences for each of the plurality of segments by decoding the plurality of segments; identifying overlapping portions between adjacent character sequences in the plurality of character sequences based on corresponding edit distances in the overlapping portions; and merging adjacent character sequences in the plurality of character sequences based on corresponding overlapping portions. The cost applied to each edit distance is determined based on the type of operation performed at the corresponding edit distance, whether the character to be operated on is located in the corresponding overlapping portion, and whether the characters to be operated on match, or any combination of any two or more of these factors.

[0026] The cost applied to each edit distance in the corresponding edit distance can be applied differently to the corresponding overlapping portion and the corresponding non-overlapping portion of the overlapping portion between adjacent character sequences in the plurality of character sequences.

[0027] The insertion cost of the corresponding edit distance in the non-overlapping part and the insertion cost of the corresponding edit distance in the overlapping part can be different.

[0028] The insertion, deletion, and replacement costs of the edit distance occurring in the overlapping area can be positive numbers.

[0029] The multiple segments can be determined from the speech signal based on the length of each segment and the overlap length between adjacent segments.

[0030] The overlap length between adjacent segments can be determined based on the available resources of the device performing the method.

[0031] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0032] Figure 1 An example of speech processing based on neural networks is shown.

[0033] Figure 2 and Figure 3An example of speech recognition processing is shown.

[0034] Figures 4 to 6 An example of integrating character sequences is shown.

[0035] Figure 7 An example of editing distance is shown.

[0036] Figure 8 An example of a speech processing method is shown.

[0037] Figure 9 An example of a voice processing device is shown.

[0038] Figure 10 and Figure 11 An example of a voice processing device is shown.

[0039] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0040] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various variations, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may vary as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known upon understanding this disclosure may be omitted.

[0041] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, as will become clear upon understanding the disclosure of this application.

[0042] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, it may be directly "on" another element, directly "connected to," or directly "bonded to" said other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, directly "connected to," or "bonded to" another element, no other elements may be present in between.

[0043] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0044] While terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Therefore, without departing from the teaching of the examples described herein, a first component, assembly, region, layer, or part referred to as such may also be referred to as a second component, assembly, region, layer, or part.

[0045] For ease of description, spatial relative terms such as “above,” “on top of,” “below,” and “under” are used herein to describe the relationship between one element and another as shown in the accompanying drawings. Such spatial relative terms are intended to encompass different orientations of the device in use or operation, in addition to those depicted in the drawings. For example, if the device in the drawings is flipped, the element described as “above” or “on top of” another element will then be “below” or “under” said other element. Thus, depending on the spatial orientation of the device, the term “above” can encompass both above and below orientations. The device may be positioned in other ways (e.g., rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein should be interpreted accordingly.

[0046] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” specify the presence of the features, quantities, operations, components, elements, and / or combinations thereof described, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0047] Unless otherwise defined, all terms used herein (including technical and scientific terms) will be clear upon understanding this application. Unless expressly defined herein, terms (such as those defined in a general dictionary) will be interpreted as having meanings consistent with their meanings in the context of the relevant field and should not be interpreted in an idealized or overly formalized sense.

[0048] As will be clear upon understanding the disclosure of this application, the features of the examples described herein can be combined in various ways. Furthermore, as will be clear upon understanding the disclosure of this application, although the examples described herein have various configurations, other configurations are possible.

[0049] Figure 1 An example of speech processing based on neural networks is shown.

[0050] exist Figure 1 In this device, the neural network for processing speech may include an encoder 110 and a decoder 120. The speech processing device may use the encoder 110 and decoder 120 to recognize a user's speech signal 130 and determine a character sequence 140 as the result. The encoder 110 may generate contextual information of the provided speech signal 130 by encoding it, and the decoder 120 may output the progressively recognized character sequence 140 as the recognition result by decoding the contextual information. The decoder 120 may iteratively perform processing to infer expected subsequent words by providing previous words at predetermined time points as input for subsequent time points until recognition terminates. The encoder 110 and decoder 120 may be speech recognition models trained to output corresponding speech recognition results in response to input from the user's speech signal 130. Note that the use of the term "may" in connection with examples or embodiments (e.g., what an example or embodiment may include or implement) indicates the existence of at least one example or embodiment that includes or implements such features, but all examples and embodiments are not limited thereto.

[0051] As described above, speech processing includes speech recognition based on encoder 110 and decoder 120. Due to the limited training data of the network and the limitation on the length of the training data for training efficiency, recognition accuracy may decrease when the speech signal is longer than the training data. To prevent this, the relatively long speech signal is divided into multiple segments, speech recognition is performed separately for each segment using encoder 110 and decoder 120, and the resulting character sequences are integrated into the recognition result. A detailed description will be provided below with reference to the accompanying drawings.

[0052] Figure 2 and Figure 3 An example of speech recognition processing is shown.

[0053] exist Figure 2 In this process, the speech processing device can determine multiple character sequences 231 to 234 by dividing the long speech signal 210 into multiple segments 221 to 224 and performing speech recognition on each segment 221 to 224 individually. The speech recognition result 240 of the long speech signal 210 is obtained by merging the multiple character sequences 231 to 234. Here, for ease of description, the term "segment" may also be referred to as a "chunk" or a "window." The long speech signal 210 comprises multiple segments.

[0054] The speech processing device can divide a long speech signal 210 into multiple segments 221 to 224. For example, the speech processing device can divide the long speech signal 210 into multiple segments 221 to 224 based on the segment length A and the overlap length between segments B.

[0055] In one example, the overlap length B can be determined to be less than half the segment length A. Since speech recognition is performed on each of the multiple segments 221 to 224 (which will be described in further detail later), the overlapping portions between adjacent segments can be decoded twice. By setting the overlap length B to less than half the segment length A, all portions of the non-long speech signal 210 will be decoded twice, thus minimizing computational overhead. In a non-limiting example, the segment length A can be determined to be "8" seconds, and the overlap length B between segments can be determined to be "2" seconds. However, various other values ​​can be applied.

[0056] The speech processing device can dynamically determine the overlap length B between segments based on available resources. Since each segment to be recognized for speech recognition is obtained from the long speech signal 210 according to predetermined criteria (e.g., segment length A and overlap length B), the accuracy of speech recognition at segment boundaries in the middle of a utterance may be lower than the accuracy of speech recognition between utterances within a segment. To compensate for the potential reduction in accuracy, it is desirable to set the overlap length B between segments to be relatively long. However, as the overlap length B increases, the proportion of the long speech signal 210 that will be decoded twice increases, thereby increasing computational overhead. In examples where resources are fully available and performance at a predetermined level or higher is guaranteed even with increased computational overhead, the speech processing device can set the overlap length B to be longer than a preset threshold length. Conversely, in examples where resources are insufficient, the speech processing device can set the overlap length B to be shorter than the preset threshold length, thereby preventing performance degradation caused by increased computational overhead.

[0057] In the example where the voice processing device is a server for speech recognition, if the server will perform a small number of speech recognition operations, the overlap length B can be set longer than a preset threshold length. Conversely, if the server will perform a large number of speech recognition operations, the overlap length B can be set shorter than the preset threshold length. In another example where the voice processing device is a user terminal (such as a smartphone), if the user terminal does not perform any operations other than speech recognition, the overlap length B can be set longer than the preset threshold length. Conversely, if the user terminal is simultaneously performing speech recognition operations and other operations (e.g., mobile gaming), the overlap length B can be set shorter than the preset threshold length.

[0058] Voice processing devices can be based on Figure 1The encoder-decoder described herein performs speech recognition on each of the multiple segments 221 to 224 and determines multiple character sequences 231 to 234 as the result. More specifically, the first character sequence 231 can be determined as the speech recognition result of the first segment 221, and similarly, the other character sequences 232 to 234 can be determined as the speech recognition results of the second segment 222 to the fourth segment 224, respectively.

[0059] The speech processing device can determine the speech recognition result 240 corresponding to the long speech signal 210 by merging multiple character sequences 231 to 234. The speech processing device can identify overlapping portions between adjacent character sequences based on edit distance and merge adjacent character sequences based on the overlapping portions. Here, edit distance is a technique used to quantify the degree of difference between two character sequences by calculating the minimum operation cost required to transform a single character sequence into another character sequence. In this example, the cost of the edit distance can be determined based on any one or any combination of the operation type (e.g., insertion, alteration, and / or deletion), the position of the character to be operated on, and whether the characters to be operated on match. Since the overlapping portion is the part that is decoded twice, one of the decoding results can be selected and included in the speech recognition result 240. The processes of identifying overlapping portions and merging character sequences will be described in further detail later.

[0060] Figure 3 This is a flowchart illustrating an example of speech recognition processing for long speech signals.

[0061] In operation 310, in one example, in response to the input of a speech signal, the speech processing device may initialize the start time t_start of the target segment to be subject to speech recognition to zero and initialize the history (HISTORY) containing accumulated speech recognition results to a null value. Although the start time t_start of the target segment is initialized to zero in this example, the example is not limited to this.

[0062] In operation 320, the speech processing device can verify whether the speech signal includes spoken utterances within a time period from start time t_start to segment length A, or whether the speech signal ended before this period. Therefore, the speech processing device can obtain the target segment by extracting a portion of the speech signal from start time t_start to segment length A. Optionally, if the speech signal ended before the portion from start time t_start to segment length A was extracted, the speech processing device can obtain the portion extracted from start time t_start to the present as the target segment.

[0063] In operation 330, the speech processing device can obtain the target character sequence D, which serves as the result of decoding the extracted target segment.

[0064] In operation 340, the speech processing device can merge the target character sequence D and the HISTORY, and store the merged result as the HISTORY. First, null values ​​are stored in the HISTORY. Therefore, the target character sequence D obtained in operation 330 can be stored in the HISTORY as is. If it is not the first time, the previous character sequence is stored in the HISTORY. Therefore, the speech processing device can identify the overlapping portion between the target character sequence D and the HISTORY, merge the target character sequence D and the HISTORY based on the identified portion, and store the merged result as the HISTORY. Since the overlapping portion is the portion that is decoded twice, one of the decoding results can be selected and stored in the HISTORY. As described above, the results of performing speech recognition on segments of the speech signal can be sequentially accumulated in the HISTORY. (Refer to later...) Figures 4 to 6 Further examples of processing the merging of character sequences are described.

[0065] In operation 350, the voice processing device can verify whether the voice signal has ended. If the voice signal has not ended, operation 360 can be executed. Conversely, if the voice signal has ended, operation 370 can be executed.

[0066] In Operation 360, the voice processing device can increment the start time t_start of the target segment by a shift time C. The shift time C can be set to be greater than half the segment length A and less than the segment length A, thereby minimizing computational overhead. The shift time C can be dynamically determined based on the available resources of the voice processing device. For example, if sufficient resources are available, the shift time C can be set to approximately half the segment length A; conversely, if insufficient resources are available, the shift time C can be set to approximately the segment length A.

[0067] In operation 370, the speech processing device may return the character sequence accumulated in the HISTORY as the result of recognizing the speech signal.

[0068] Figures 4 to 6 An example of integrating character sequences is shown.

[0069] exist Figure 4 The image shows an example of merging the target character sequence 420 with the previous character sequence 410. Figure 4 An example of the initial operation in a speech processing method is shown. The preceding character sequence 410 may be the result of decoding the first segment extracted from the speech signal, and the target character sequence 420 may be the result of decoding the second segment after the first segment extracted from the speech signal.

[0070] The speech processing device can identify the overlap 430 between the previous character sequence 410 and the target character sequence 420 based on the edit distance. The cost of the edit distance can be determined based on at least one of the type of operation to be performed at the edit distance (e.g., insertion, alteration, or deletion), whether the character to be operated on is located in the overlap between the two character sequences, and whether the characters to be operated on match. (Refer to...) Figure 7 Further describe the cost of editing distance.

[0071] like Figure 4 As shown in the example, the previous character sequence 410 and the target character sequence 420 can be aligned at the character level based on the overlap 430 between them. In this example, the cost of the edit distance between the previous character sequence 410 and the target character sequence 420 is minimized.

[0072] As further described below, the aligned previous character sequence 410 and the target character sequence 420 can be merged.

[0073] First, the remaining portions of the previous character sequence 410 and the target character sequence 420, excluding the overlapping portion 430 (i.e., the non-overlapping portions), can be decoded once, and the corresponding decoding result can be used as is. On the other hand, the overlapping portion 430 is decoded twice, and the corresponding decoding results are included in the previous character sequence 410 and the target character sequence 420, respectively. Therefore, one of the decoding results of the overlapping portion 430 included in the previous character sequence 410 and the target character sequence 420 needs to be selected.

[0074] For example, the speech processing device can identify word boundaries in the overlapping portion 430. Word boundaries can be spaces between words in the decoding result. Furthermore, the speech processing device can designate the identified word boundaries as conversion candidates and select one of the conversion candidates located closest to the center of the overlapping portion 430 as the conversion point. Additionally, the speech processing device can merge the previous character sequence 410 and the target character sequence 420 by connecting all words before the conversion point in the previous character sequence 410 to all words after the conversion point in the target character sequence 420. Since each segment boundary can correspond to a cut-off portion of the speech signal, each segment boundary can have a lower recognition accuracy than the center of the segment. Therefore, the previous character sequence 410 and the target character sequence 420 can be merged based on the conversion point located at the center of the overlapping portion 430 to obtain a merged result with higher recognition accuracy.

[0075] exist Figure 4In the example, the word boundaries in the overlapping portion 430 can be determined between "recognition" and "all" in the previous character sequence 410, and between "recognition" and "algorithm" in the target character sequence 420, and the corresponding portions can be designated as transition candidates. In this example, a single transition candidate is located at the center of the overlapping portion 430, therefore, a single transition candidate can be selected as a transition point. Then, the word "The end-to-end speech recognition" before the transition point in the previous character sequence 410 and the word "algorithm has been widely studied" after the transition point in the target character sequence 420 can be concatenated, and "The end-to-end speech recognition algorithm has been widely studied" can be determined as the merged result. When the speech signal is cut off, the last word "all" in the previous character sequence 410 is obtained by incorrectly recognizing "algorithm" as "all". By using the above method, incorrectly recognized words can be effectively prevented from being included in the merged result.

[0076] Figure 4 The merged result determined in the process can be merged with the result of decoding subsequent target segments, which will be referred to later. Figure 5 describe.

[0077] exist Figure 5 The image shows an example of merging the target character sequence 520 with the previous character sequence 510. Here, the previous character sequence 510 is... Figure 4 The example shows the merged result, and the target character sequence 520 is the result of decoding the third segment extracted from the speech signal.

[0078] The speech processing device can determine the overlap between the previous character sequence 510 and the target character sequence 520 based on the edit distance. For example, in Figure 5In the example, there may be no overlap between the previous character sequence 510 and the target character sequence 520. Therefore, the previous character sequence 510 and the target character sequence 520 can be aligned so that they do not overlap. As described above, by utilizing the previous decoding result as is, the non-overlapping portion can be included in the merged result. Therefore, the previous character sequence 510 and the target character sequence 520 can be concatenated as is, so that "The end-to-end speech recognition algorithm has been widely studied and showed competitive performance to conventional speech" can be determined as the merged result. Figure 5 The merged result determined in the process can be merged with the result of decoding subsequent target segments, which will be referred to later. Figure 6 describe.

[0079] exist Figure 6 The image shows an example of merging the target character sequence 620 with the previous character sequence 610. Here, the previous character sequence 610 is... Figure 5 The example shows the merged result, and the target character sequence 620 is the result of decoding the fourth segment extracted from the speech signal.

[0080] The speech processing device can identify the overlap 630 between the previous character sequence 610 and the target character sequence 620 based on the edit distance. For example, in Figure 6 In the example, the previous character sequence 610 and the target character sequence 620 can be aligned at the character level based on the overlapping portion 630. In this example, the cost of the edit distance between the previous character sequence 610 and the target character sequence 620 is minimized.

[0081] When the overlapping portion 630 is confirmed based on the edit distance, the entire previous character sequence 610 can be considered. However, in some examples, only a portion of the previous character sequence 610 can be considered. For example, since a user can speak a finite number of characters during the overlap length between segments, a partial sequence 640 corresponding to the maximum number of characters the user can speak can be extracted from the previous character sequence 610. By calculating the edit distance between the partial sequence 640 and the target character sequence 620, the overlapping portion 630 can be confirmed even with a relatively small amount of operation. In another example, the previous character sequence 610 is obtained by merging the decoding results of the first to third segments, where the decoding result 650 of the third segment corresponding to the last segment overlaps with the target character sequence 620. Therefore, by calculating the edit distance between the decoding result 650 of the third segment and the target character sequence 620, the overlapping portion 630 can be confirmed even with a relatively small amount of operation. In this example, the third segment is the segment immediately preceding the target segment and will be referred to as the previous segment for ease of description.

[0082] The previous character sequence 610 and the target character sequence 620, aligned based on the overlapping portion 630, can be merged according to the method described above, and "The end-to-end speech recognition algorithm has been widely studied and showed competitive performance to conventional speech recognition methods" can be determined as the merged result. Figure 6 In the example, the first word "conventional" in the overlapping portion 630 of the previous character sequence 610 is different from the first word "convolutional" in the overlapping portion 630 of the target character sequence 620. However, based on the transition point designated as the space between "conventional speech" and the space between "convolutional speech", "conventional" can be included in the merged result.

[0083] Figure 7 An example of editing distance is shown.

[0084] exist Figure 7 The example shown illustrates how to calculate the edit distance between the target character sequence and the previous character sequence. Figure 7In the example shown in the table, “proposed convolution” in the first column is the previous character sequence, “conventional algorithm” in the first row is the target character sequence, and “” included in the first row and first column represents a space.

[0085] Edit distance is a technique used to quantify the degree of difference between two character sequences by calculating the minimum operation cost required to transform a single character sequence into another character sequence. For ease of description, the process of calculating the minimum operation cost can be described by, for example... Figure 7 The table shown is used to represent this.

[0086] The cost of the edit distance is determined based on at least one of the following: the type of operation to be performed at the edit distance (e.g., insertion, alteration, or deletion), whether the characters to be operated on (e.g., the previous character sequence and the target character sequence) are in an overlapping region, and whether the characters to be operated on match (e.g., whether the previous character sequence and the target character sequence match in the overlapping region).

[0087] The cost of edit distance can be applied differently to the overlapping and non-overlapping portions between the target character sequence and the preceding character sequence. In other words, the insertion cost of edit distance occurring in the non-overlapping portion can be less than the insertion cost occurring in the overlapping portion. For example, the insertion cost of edit distance occurring in the non-overlapping portion can be "0", while the insertion cost of edit distance occurring in the overlapping portion can be "2". Figure 7 In the example, the beginning 710 of the previous character sequence can be a non-overlapping portion and has the same cost that remains unchanged as it moves down the table (i.e., even if a character from the previous character sequence is inserted). Similarly, the end 720 of the target character sequence can also be a non-overlapping portion and has the same cost that remains unchanged as it moves to the right of the table (i.e., even if a character from the target character sequence is inserted). Conversely, the cost can increase by "2" each time a character is inserted into an overlapping portion (i.e., the end of the previous character sequence or the beginning of the target character sequence).

[0088] Furthermore, the insertion, deletion, and replacement costs of the edit distance occurring in the overlapping portion can be positive numbers. In this example, each cost can be determined by tuning the speech processing technique. For instance, the insertion and / or deletion costs of the edit distance occurring in the overlapping portion can be greater than the replacement costs of the edit distance. Specifically, the insertion and / or deletion costs of the edit distance occurring in the overlapping portion can be "2", and the replacement costs of the edit distance occurring in the overlapping portion can be "1". This is to prevent a larger error from being identified as a predetermined character when the recognition result is omitted than when the predetermined character is identified as another character due to noise in the speech recognition processing.

[0089] Furthermore, the matching cost of an edit distance can be less than the insertion cost of an edit distance occurring in a non-overlapping region. In other words, the matching cost of an edit distance can be set to a negative number less than "0" (e.g., "-1"), where "0" is the insertion cost of an edit distance occurring in a non-overlapping region. Figure 7 In the example, the cost can be reduced each time a match occurs in the overlapping portion 730. In other words, any one or a combination of both of the segments where the cost of the edit distance continues to decrease and the segments from the point where the cost of the edit distance begins to decrease to the point where the cost reduction stops can be identified as the overlapping portion 730. Furthermore, the characteristic that the overlapping portion 730 at the end of the previous character sequence is detected with a high probability and the characteristic that the overlapping portion 730 at the beginning of the target character sequence is detected with a high probability can be used to identify the overlapping portion 730.

[0090] When the edit distance is calculated by applying cost as described above, it is possible to... Figure 7 The minimum operating cost is calculated in the overall aspect of the table, connecting the starting point at the top left and the ending point at the bottom right. Figure 7 In the example, "convolution" can be identified as the overlapping portion of the previous character sequence "proposed convolution," and "conventional" can be identified as the overlapping portion of the target character sequence "conventional algorithm." Therefore, even if the beginning of the target character sequence and the end of the previous character sequence do not perfectly match, the overlapping portion can be identified based on the minimum operational cost of the edit distance. As mentioned above, the edit distance is calculated in characters; therefore, the edit distance is applicable to any language used for general purposes.

[0091] The cost of edit distance is applied differently to the overlapping and non-overlapping portions between the target character sequence and the preceding character sequence. Furthermore, the overlapping portion can be identified based on the edit distance. For example, based on dynamic programming, the overlapping portion can be identified, and different costs can be applied to both overlapping and non-overlapping portions. Optionally, the overlapping portion between the target character sequence and the preceding character sequence is assumed differently. After calculating the cost of edit distance for examples of various assumptions, the overlapping portion of the assumption with the lowest cost can be determined as the overlapping portion between the target character sequence and the preceding character sequence.

[0092] Figure 8 An example of a speech processing method is shown.

[0093] exist Figure 8 The image shows an example of a speech processing method executed by a processor set in a speech processing device.

[0094] In operation 810, the speech processing device obtains or determines a target segment from the speech signal. The speech processing device may obtain the target segment from the speech signal based on the segment length and the overlap length between segments. The overlap length between segments may be determined based on the available resources of the speech processing device.

[0095] In operation 820, the speech processing device obtains the target character sequence corresponding to the target segment by decoding the target segment.

[0096] In operation 830, the speech processing device identifies the overlap between the target character sequence and the previous character sequence based on the edit distance. The cost of the edit distance is determined based on at least one of the type of operation performed at the edit distance, whether the character to be operated on is located in the overlap, and whether the characters to be operated on match. The cost of the edit distance can be applied differently to the overlap and non-overlapping parts between the target character sequence and the previous character sequence. The matching cost of the edit distance can be negative, the insertion cost of the edit distance occurring in the non-overlapping part can be "0", and the replacement cost and insertion cost of the edit distance occurring in the overlap can be positive.

[0097] In operation 840, the speech processing device merges the target character sequence and the previous character sequence based on the overlapping portion.

[0098] Reference Figures 1 to 7 The provided description also applies Figure 8 Therefore, for the sake of brevity, detailed descriptions will be omitted.

[0099] Figure 9 An example of a voice processing device is shown.

[0100] exist Figure 9 In this device, the voice processing device 900 may include a memory 910, a processor 920, and an input / output interface 930. The memory 910, the processor 920, and the input / output (I / O) interface 930 may communicate with each other via a bus 940.

[0101] The memory 910 may include computer-readable instructions. When the instructions stored in the memory 910 are executed by the processor 920, the processor 920 may perform the aforementioned operations. The memory 910 may include volatile memory or non-volatile memory.

[0102] Processor 920 is one or more means that execute instructions or programs or control voice processing device 900, and may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), etc. Voice processing device 900 can be connected to another device (e.g., a microphone or an external device configured to acquire a user's voice signal) via input / output interface 930 to exchange data with said other device. Furthermore, voice processing device 900 can handle the aforementioned operations.

[0103] User terminals may include various computing devices (such as smartphones, tablets, laptops, and personal computers), various wearable devices (such as smartwatches and smart glasses), various home appliances (such as smart speakers, smart TVs, and smart refrigerators), smart cars, smart self-service terminals, and Internet of Things (IoT) devices.

[0104] Figure 10 and Figure 11 An example of a voice processing device is shown.

[0105] exist Figure 10 In this context, the voice processing device can be implemented as a server 1000.

[0106] Server 1000 is a separate device from the user terminal controlled by the user and can communicate with the user terminal via wired and / or wireless networks. The user's voice signals can be collected by the user terminal and sent to server 1000 via the network. Server 1000 can recognize the voice signals using speech recognition model 1010 according to the speech processing method described above. In this example, speech recognition model 1010 can be used. Server 1000 can then return the speech recognition result to the user terminal. For example, the user terminal may include: various computing devices (such as smartphones, tablets, laptops, and personal computers), various wearable devices (such as smartwatches and smart glasses), various home appliances (such as smart speakers, smart TVs, and smart refrigerators), smart cars, smart self-service terminals, and Internet of Things (IoT) devices.

[0107] The user terminal can simply provide the user with the recognition results received from the server 1000, or perform subsequent operations based on the recognition results. For example, subsequent operations may include, but are not limited to: outputting a response to a user's question about the weather, executing a user's instruction to play music, inputting text into an input box displayed on the user terminal, or interpreting or translating the speech recognition results into another language.

[0108] exist Figure 11 In this context, the voice processing device can be implemented as a user terminal 1100. Figure 11For ease of description, user terminal 1100 is shown as a smartphone. However, any user-controlled device can be used without limitation. User terminal 1100 can directly obtain voice signals from the user and recognize the voice signals using voice recognition model 1110 according to the voice processing method described above. The user terminal can then simply provide the recognition result to the user or perform subsequent operations based on the recognition result.

[0109] The operations described in this application are performed using hardware components. Figures 1 to 11The hardware components, including encoder 110, decoder 120, speech signal 130, character sequence 140, speech processing device 900, memory 910, processor 920, input / output interface 930, server 1000, speech recognition models 1010 and 1110, and user terminal 1100, are configured to perform the operations described in this application. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memory, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented using one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms “processor” or “computer” may be used in the description of the examples described herein; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented using a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented using one or more processors, or processors and controllers. One or more other hardware components may be implemented using one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0110] Figures 1 to 11 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0111] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0112] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, instructions or software, along with any associated data, data files, and data structures, are distributed across a networked computer system, enabling the instructions or software, along with any associated data, data files, and data structures, to be stored, accessed, and executed in a distributed manner via one or more processors or computers.

[0113] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in this disclosure.

Claims

1. A method for speech processing, the method comprising: Determine the target segment from the speech signal; The target character sequence corresponding to the target segment is determined by decoding the target segment. The cost of editing distance is used to identify the first overlap between the target character sequence and the previous character sequence; as well as The target character sequence and the previous character sequence are merged based on the first overlapping part. Here, edit distance represents the degree of difference between the target character sequence and the previous character sequence. This degree of difference is obtained by quantifying the minimum operation cost required to transform a single character sequence into another character sequence. The cost of the edit distance is determined based on any one or any combination of any two or more of the following: the type of operation performed on the target character sequence and the previous character sequence; whether the target character sequence and the previous character sequence are located in the first overlapping portion; and whether the target character sequence and the previous character sequence match in the first overlapping portion.

2. The method according to claim 1, wherein, The steps for merging the target character sequence and the previous character sequence based on the first overlapping portion include: Identify the word boundaries in the first overlapping portion. Designate the confirmed word boundaries as conversion candidates. Select the transition candidate located closest to the center of the first overlapping portion from the specified transition candidates as the transition point. The previous character sequence and the target character sequence are merged by connecting all words before the transition point in the previous character sequence to all words after the transition point in the target character sequence.

3. The method according to claim 1, wherein, The cost of edit distance is applied differently to the first overlapping and non-overlapping parts between the target character sequence and the previous character sequence.

4. The method according to claim 1, wherein, The insertion cost of the edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence is less than the insertion cost of the edit distance occurring in the first overlapping portion.

5. The method according to claim 1, wherein, The matching cost of edit distance is less than the insertion cost of edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence.

6. The method according to claim 1, wherein, The insertion, deletion, and replacement costs for the edit distance occurring in the first overlapping region are positive.

7. The method according to claim 1, wherein, The matching cost for edit distance is negative, and The insertion cost for edit distances occurring in the non-overlapping portion between the target character sequence and the previous character sequence is zero.

8. The method according to claim 1, wherein, The steps for determining the target segment from the speech signal include: determining the target segment from the speech signal based on the segment length and the overlap length between segments.

9. The method according to claim 8, wherein, The overlap length between segments is determined based on the available resources of the device performing the method.

10. The method according to claim 1, wherein, The confirmation steps include: confirming a second overlapping portion between the target character sequence and a partial sequence extracted from the previous character sequence, based on the maximum character length spoken by the user during the overlap length between segments.

11. The method according to any one of claims 1 to 10, wherein, The preceding character sequence is determined based on the result of decoding the preceding segment that partially overlaps with the target segment in the speech signal.

12. The method according to claim 11, wherein, The confirmation steps include: confirming the third overlapping portion between the target character sequence and the character sequence obtained by decoding the previous segment in the previous character sequence.

13. The method according to claim 11, wherein, The first overlapping portion in the target segment is less than half the size of the target segment.

14. An apparatus for speech processing, the apparatus comprising: One or more processors are configured as follows: Determining the target segment from the speech signal The target character sequence corresponding to the target segment is determined by decoding the target segment. The cost of using edit distance is used to identify the overlap between the target character sequence and the previous character sequence, and The target character sequence and the previous character sequence are merged based on the overlapping part. Here, edit distance represents the degree of difference between the target character sequence and the previous character sequence. This degree of difference is obtained by quantifying the minimum operation cost required to transform a single character sequence into another character sequence. The cost of edit distance is determined based on any one or any combination of the type of operation performed on the target character sequence and the previous character sequence, whether the target character sequence and the previous character sequence are located in the overlapping part, and whether the target character sequence and the previous character sequence match in the overlapping part.

15. The device according to claim 14, wherein, The one or more processors are configured to: Identify word boundaries in the overlapping sections. Designate the confirmed word boundaries as conversion candidates. Select the transformation candidate located closest to the center of the overlapping portion from the specified transformation candidates as the transformation point. The previous character sequence and the target character sequence are merged by connecting all words before the transition point in the previous character sequence to all words after the transition point in the target character sequence.

16. The device according to claim 14, wherein, The cost of edit distance is applied differently to the non-overlapping and overlapping portions between the target character sequence and the previous character sequence.

17. The device according to claim 14, wherein, The matching cost of edit distance is less than the insertion cost of edit distance occurring in the non-overlapping portion between the target character sequence and the previous character sequence.

18. The device according to claim 14, wherein, The insertion, deletion, and replacement costs of the edit distance occurring in the overlapping portion between the target character sequence and the previous character sequence are positive.

19. The device according to claim 14, wherein, The one or more processors are configured to: The target segment is determined from the speech signal based on the segment length and the overlap length between segments.

20. The device according to claim 19, wherein, The overlap length between segments is determined based on the available resources of the device.

21. The device according to any one of claims 14 to 20, wherein, The previous character sequence was determined based on the results of decoding the previous segment that partially overlapped with the target segment in the speech signal.

22. A method for speech processing, the method comprising: Determining multiple segments from the speech signal; Multiple character sequences of the multiple segments are determined by decoding the multiple segments respectively; The overlapping portion between adjacent character sequences in the plurality of character sequences is determined based on the cost of the edit distance between adjacent character sequences in the plurality of character sequences; as well as Adjacent character sequences in the plurality of character sequences are merged based on overlapping portions. Here, edit distance represents the degree of difference between adjacent character sequences among the plurality of character sequences. This degree of difference is obtained by quantifying the minimum operation cost required to transform a single character sequence into another character sequence. The cost of each edit distance is determined based on any one or any combination of the type of operation performed on adjacent character sequences in the plurality of character sequences, whether the adjacent character sequences in the plurality of character sequences are located in the overlapping portion, and whether the adjacent character sequences in the plurality of character sequences match in the overlapping portion.

23. The method according to claim 22, wherein, The steps for merging adjacent character sequences in the plurality of character sequences based on overlapping portions include: Identify word boundaries in the overlapping sections. Designate the confirmed word boundaries as conversion candidates. Select the transformation candidate located closest to the center of the overlapping portion from the specified transformation candidates as the transformation point. The previous character sequence and the target character sequence are merged by connecting all words before the transition point in the previous character sequence to all words after the transition point in the target character sequence.

24. The method according to claim 22, wherein, The cost of each edit distance is applied differently to the overlapping and non-overlapping portions between adjacent character sequences in the plurality of character sequences.

25. The method according to claim 22, wherein, The insertion cost of edit distances occurring in non-overlapping regions is different from that of edit distances occurring in overlapping regions.

26. The method according to claim 22, wherein, The insertion, deletion, and replacement costs for edit distances occurring in overlapping regions are positive.

27. The method according to claim 22, wherein, The multiple segments are determined from the speech signal based on the length of each segment and the overlap length between adjacent segments.

28. The method according to claim 27, wherein, The overlap length between adjacent segments is determined based on the available resources of the device performing the method.

29. A non-transitory computer-readable storage medium storing instructions, which, when executed by one or more processors, configure the one or more processors to perform the method of any one of claims 1 to 13 and 22 to 28.

Images