The Method And System Of Integrated Speech Processing Including Speech-Segment Determination Through Self-Attention-Based Sequence Encoding And CTC-Based Decoding

Abstract

The present invention relates to an integrated speech processing method and system comprising the calculation of utterance segments through self-attention-based sequence encoding and CTC-based decoding, and more specifically, to an integrated speech processing method and system comprising the calculation of utterance segments through self-attention-based sequence encoding and CTC-based decoding, wherein time and frequency information based on an input speech signal is generated and projected into a fixed dimension, a computation is performed through a convolutional neural network to generate high-dimensional features, a self-attention computation is performed with a mask applied to the features to calculate a logit, and CTC-based decoding is performed again to generate a final speech recognition text and calculate the start and end points of the utterance.

Description

Technology Field

[0001] The present invention relates to an integrated speech processing method and system comprising the calculation of utterance segments through self-attention-based sequence encoding and CTC-based decoding, and more specifically, to an integrated speech processing method and system comprising the calculation of utterance segments through self-attention-based sequence encoding and CTC-based decoding, wherein time and frequency information based on an input speech signal is generated and projected into a fixed dimension, a computation is performed through a convolutional neural network to generate high-dimensional features, a self-attention computation is performed with a mask applied to the features to calculate a logit, and CTC-based decoding is performed again to generate a final speech recognition text and calculate the start and end points of the utterance. Background Technology

[0003] With the recent rapid advancement of artificial intelligence technology, STT speech recognition technology is establishing itself as a core interface across various industries, including AI contact centers, voice assistants, and kiosks. In particular, beyond simply converting user speech into text, the precision of the process for accurately detecting meaningful utterance segments within the voice signal and converting them into text is becoming critical in order to understand user intent and provide appropriate responses.

[0004] Conventional technology has primarily utilized a method of operating the VAD speech segment detection module and the STT speech recognition engine independently to provide speech recognition services. In this structure, the VAD module first determines the start and end of a speech based on signal processing criteria such as energy levels or frequency characteristics, and then transmits only the audio of the detected segment to the STT engine. However, this separated structure frequently resulted in false detections, such as mistaking ambient noise or static for actual speech, or incorrectly interpreting short pauses in the middle of a speech as the end of the utterance, thereby cutting off the sentence. Furthermore, a discrepancy occurs between the physical time segment determined by the VAD and the logical timing at which the STT engine generates text; consequently, the downstream Natural Language Processing (NLP) engine receives incomplete sentences, leading to a decrease in the accuracy of intent recognition. Moreover, there are limitations in that the system complexity increases and integrated optimization is difficult because a separate VAD model must be provided and tuned.

[0005] Under these circumstances, there is a need for an integrated speech processing technology that can accurately calculate the start and end times of a speech by synchronizing them with the text generation time using information generated during the inference process within the speech recognition model without a separate VAD preprocessing module, thereby simplifying the structure of the system while dramatically improving the consistency between speech recognition and speech segment detection. The problem to be solved

[0007] The present invention aims to provide an integrated speech processing method and system comprising the calculation of utterance segments through self-attention-based sequence encoding and CTC-based decoding, and more specifically, to generate time and frequency information based on an input speech signal and project it onto a fixed dimension, perform operations through a convolutional neural network to generate high-dimensional features, perform self-attention operations with a mask applied to the features to calculate logits, and then perform CTC-based decoding again to generate final speech recognition text and calculate the start and end points of the utterance. means of solving the problem

[0009] To solve the above problems, in one embodiment of the present invention, an integrated voice processing method comprising calculating a speech interval through self-attention-based sequence encoding and CTC-based decoding performed in a server system including one or more processors and one or more memories comprises: a first feature information generation step of generating a feature including time and frequency information based on a voice signal received from a user terminal, wherein the feature corresponds to a 2D or 3D array form, and generating a first feature information by projecting the feature onto a fixed dimension and normalizing the range of the first feature information; a second feature information generation step of generating a second feature information corresponding to a high-dimensional intermediate feature from a time and frequency pattern by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; and a mask generation step of converting the second feature information into a sequence form that can be processed by a sequence model and generating a mask based on whether valid data exists. The present invention provides an integrated speech processing method comprising: a self-attention encoding step of inputting the second feature information into a self-attention-based speech recognition encoder, generating speech context information including associations between frames based on the mask, and then projecting the speech context information to calculate frame-unit logits; a CTC decoding step of performing CTC-based decoding based on the mask and the logits to evaluate tokens corresponding to text candidates per frame and remove duplicates between consecutive frames to generate final speech recognition text; and an utterance interval calculation step of analyzing the distributed arrangement structure of tokens appearing in the mask and the logits to calculate the start and end times of an utterance corresponding to a segment where meaningful speech exists.

[0010] In one embodiment of the present invention, the self-attention encoding step encodes the second feature information in the form of a sequence using a plurality of encoding layers, and by reflecting the mask in the attention weight calculation process of the self-attention operation, it restricts meaningless sections including padding sections and invalid frames from contributing to the attention operation and can generate the speech context information in the form of a sequence output that reflects long-term dependencies between frames.

[0011] In one embodiment of the present invention, the self-attention encoding step may include: a step of applying a dense layer that linearly projects the voice context information to a dimension corresponding to a dictionary of characters, subwords, and tokens defined in advance, to calculate a frame-unit logit including a classification score indicating the possibility that each frame corresponds to a specific token; and a step of calculating an effective length indicating the length of an effective logit section excluding padding sections among a frame-unit logit sequence based on the mask, wherein the frame-unit logit sequence corresponds to the frame-unit logit arranged in chronological order.

[0012] In one embodiment of the present invention, the CTC decoding step may include: a step of specifying an effective logit section that is the subject of actual calculation, excluding padding sections from a frame-unit logit sequence based on the mask; a step of performing greedy decoding to select a token having the highest classification score in each frame among a plurality of tokens included in the effective logit section; a step of identifying whether the selected token is a blank token corresponding to the absence of utterance or a non-blank token representing a meaningful character, wherein a space token representing a space is treated as a single character and included in the non-blank token; and a step of generating a final speech recognition text by merging the same non-blank token into a single non-blank token when the same non-blank token is repeated in consecutive frames and removing the blank token within the effective logit section.

[0013] In one embodiment of the present invention, the utterance section calculation step may include: determining a section based on the probability of expression of each frame character included in the frame unit logit, and generating an utterance candidate section by identifying a valid logit section that has a higher classification score in the non-blank token than in the blank token; and determining the utterance candidate section as a valid section in which meaningful utterance exists with noise or meaningless sounds excluded, and generating an utterance valid section by calculating the start time of the utterance and the end time of the utterance, wherein if the time interval between one or more utterance candidate sections is less than a preset time threshold, merging them into a single utterance candidate section.

[0014] To solve the above problems, in one embodiment of the present invention, an integrated voice processing system comprising one or more processors and one or more memories, and including the calculation of a speech interval through self-attention-based sequence encoding and CTC-based decoding, comprises: a first feature information generation unit that generates a feature including time and frequency information based on a voice signal received from a user terminal, wherein the feature corresponds to a two-dimensional or three-dimensional array form, generates first feature information by projecting the feature onto a fixed dimension, and normalizes the range of the first feature information; a second feature information generation unit that generates second feature information corresponding to a high-dimensional intermediate feature from a time and frequency pattern by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; and a mask generation unit that converts the second feature information into a sequence form that can be processed by a sequence model and generates a mask based on whether valid data exists. The present invention provides an integrated speech processing system comprising: a self-attention encoding unit that inputs the second feature information into a self-attention-based speech recognition encoder, generates speech context information including associations between frames based on the mask, and calculates frame-unit logits by projecting the speech context information; a CTC decoding unit that evaluates tokens corresponding to text candidates per frame by performing CTC-based decoding based on the mask and the logits, and generates final speech recognition text by removing duplicates between consecutive frames; and an utterance interval calculation unit that calculates the start and end times of an utterance corresponding to a segment where meaningful speech exists by analyzing the distributed arrangement structure of tokens appearing in the mask and the logits. Effects of the invention

[0016] According to one embodiment of the present invention, by naturally calculating speech segments during the processing of a speech recognition model without independently building a separate speech segment detection module or performing it as a preprocessing step, the complexity of the system can be reduced and the construction and operation costs can be reduced.

[0017] According to one embodiment of the present invention, instead of determining the presence or absence of sound based solely on sound pressure or frequency characteristics, by determining the utterance interval based on the possibility of token expression calculated through a dense layer, it is possible to exclude noise or non-utterance noise and precisely detect only meaningful linguistic utterance intervals.

[0018] According to one embodiment of the present invention, since the time at which the speech recognition text is generated and the time at which the utterance interval is determined are derived from the same feature information, the effect of perfectly matching the temporal consistency between the text result and the utterance timing information can be achieved.

[0019] According to one embodiment of the present invention, by applying a mask based on whether valid data exists during the self-attention operation, padding sections or invalid frames can be prevented from acting as unnecessary noise in context learning, thereby improving the accuracy of speech recognition.

[0020] According to one embodiment of the present invention, by including a preprocessing process that projects input voice features onto a fixed dimension and normalizes them, it is possible to achieve the effect of maintaining consistent performance even if the scale of the input signal changes due to differences in the recording environment or microphone performance.

[0021] According to one embodiment of the present invention, by effectively extracting local features of time and frequency patterns through a multi-stage convolutional neural network (CNN) and transmitting them to a self-attention encoder, it is possible to produce high-quality speech features that reflect short-term characteristics and long-term contextual information of a speech signal.

[0022] According to one embodiment of the present invention, by analyzing frame-unit logit sequences to identify sections where the score of non-blank tokens is higher than that of blank tokens, it is possible to achieve the effect of immediately estimating valid utterance sections without separate VAD tuning, even in new domains or environments that have not been learned.

[0023] According to one embodiment of the present invention, by transmitting text with unnecessary silent segments removed to a subsequent intent classification or answer generation engine based on calculated utterance segment information, it is possible to achieve the effect of improving the analysis accuracy and response quality of the natural language processing (NLP) engine.

[0024] According to one embodiment of the present invention, when the time interval between one or more utterance candidate intervals is less than a preset threshold, they are merged into one interval, thereby preventing a single sentence from being fragmented into multiple pieces due to short pauses or breaks in the middle of utterance.

[0025] According to one embodiment of the present invention, by adopting a CTC-based decoding method to eliminate duplication between frames and generate final text, it is possible to achieve the effect of providing speech recognition results suitable for real-time while ensuring fast inference speed.

[0026] According to one embodiment of the present invention, the dense layer is designed not only to generate classification scores for text output but also to perform a dual function as a key indicator for utterance interval estimation, thereby maximizing the parameter efficiency of the model.

[0027] According to one embodiment of the present invention, even in situations where the length of the input sequence is variable, the effective length is calculated based on a mask and reflected in CTC loss calculation and decoding, thereby enabling stable and robust processing for utterance data of various lengths. Brief explanation of the drawing

[0029] FIG. 1 schematically illustrates the internal configuration of a server system according to one embodiment of the present invention. FIG. 2 schematically illustrates the execution steps of an integrated voice processing method according to one embodiment of the present invention. FIG. 3 schematically illustrates the process of generating first feature information and an example of a feature according to one embodiment of the present invention. FIG. 4 schematically illustrates a comparative example of a process for generating second feature information through a multi-stage convolutional neural network and pooling operation according to an embodiment of the present invention. FIG. 5 schematically illustrates an example of generating a mask according to second feature information converted into a sequence form according to one embodiment of the present invention. FIG. 6 schematically illustrates the process of generating speech context information through a self-attention operation according to one embodiment of the present invention. FIG. 7 schematically illustrates the process of calculating frame-unit logits by applying a density layer to voice context information according to one embodiment of the present invention. FIG. 8 schematically illustrates an example of a process for calculating an effective length according to one embodiment of the present invention. FIG. 9 schematically illustrates an example of the application of a CTC decoding step according to one embodiment of the present invention. FIG. 10 schematically illustrates an example of a process for generating ignition candidate sections in an ignition section calculation step according to an embodiment of the present invention. FIG. 11 schematically illustrates an example of a process for generating an ignition effective section of an ignition section calculation step according to one embodiment of the present invention. FIG. 12 illustrates, in an exemplary manner, the internal configuration of a computing device according to one embodiment of the present invention. Specific details for implementing the invention

[0030] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0031] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0032] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0033] The "user terminal" mentioned below may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, all types of handheld-based wireless communication devices that ensure portability and mobility, such as smartphones, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), and BLE Beacon (Bluetooth Low Energy Beacon) terminals. In addition, the “network” can be implemented as a wired network such as a Local Area Network (LAN), Wide Area Network (WAN), or Value Added Network (VAN), or as any type of wireless network such as a mobile radio communication network or a satellite communication network.

[0035] FIG. 1 schematically illustrates the internal configuration of a server system (1000) according to one embodiment of the present invention.

[0037] In one embodiment of the present invention, the server system (1000) is an integrated voice processing system comprising one or more processors and one or more memories, and including a speech interval calculation through self-attention-based sequence encoding and CTC-based decoding, and may include a first feature information generation unit (100) that generates a feature including time and frequency information based on a voice signal received from a user terminal, wherein the feature corresponds to a two-dimensional or three-dimensional array form, generates a first feature information by projecting the feature onto a fixed dimension, and performs a first feature information generation step of normalizing the range of the first feature information.

[0039] Next, the system may include a second feature information generation unit (200) that performs a second feature information generation step of generating second feature information corresponding to high-dimensional intermediate features from time and frequency patterns by performing multi-stage convolutional neural network and pooling operations on the normalized first feature information, and may include a mask generation unit (300) that performs a mask generation step of converting the second feature information into a sequence form that can be processed by a sequence model and generating a mask based on whether valid data exists.

[0041] Additionally, it may include a self-attention encoding unit (400) that inputs the second feature information into a self-attention-based speech recognition encoder, generates speech context information including associations between frames based on the mask, and then projects the speech context information to calculate logits per frame, and may include a CTC decoding unit (500) that performs a CTC decoding step by performing CTC-based decoding based on the mask and the logits to evaluate tokens corresponding to text candidates per frame and remove duplicates between consecutive frames to generate final speech recognition text.

[0043] And it may include a speech interval calculation unit (600) that performs a speech interval calculation step of calculating the start and end times of speech corresponding to the interval where meaningful voice exists by analyzing the distributed arrangement structure of tokens appearing in the above mask and above logit.

[0045] FIG. 2 schematically illustrates the execution steps of an integrated voice processing method according to one embodiment of the present invention.

[0047] In one embodiment of the present invention, the server system (1000) can generate a feature in the form of a two-dimensional or three-dimensional array including time and frequency information based on a voice signal received from a user terminal, generate a first feature information by projecting the feature onto a fixed dimension, and normalize the range of the first feature information (S10) for stable data operation.

[0049] Next, multi-stage convolutional neural network (CNN) and pooling operations can be performed on the normalized first feature information, and through this process, second feature information corresponding to high-dimensional intermediate features reflecting time and frequency patterns can be generated from simple signal information (S11).

[0051] In addition, the generated second feature information can be converted into a sequence form corresponding to a form that the sequence model can process. At this time, a mask (S12) can be generated based on whether valid data exists in order to identify padding, etc., that occurs as the length of the input voice is variable. The principle of generating the mask and an example thereof will be described later in FIG. 5.

[0053] Subsequently, the above second feature information is input into a self-attention-based encoder, and the above mask can be applied to the attention weight calculation process. Through this, padding intervals and invalid frames can be restricted from contributing to the computation, and speech context information reflecting long-term dependencies between frames can be generated.

[0054] Specifically, a speech recognition system may perform batch processing to process speech signals of various lengths by padding the latter part of a short signal with '0' or a specific value to match the length of the longest signal. In this case, the invalid frames may include sections corresponding to the padding added for length matching that do not contain actual speech information, or sections of simple noise within the speech that have no value for analysis.

[0055] Furthermore, in spoken language, the meaning or context of a word can be determined not only by words immediately adjacent to it but also by words that are temporally separated, which may correspond to long-term dependency. In this case, while the convolutional neural network manages a narrow range, the self-attention system manages the entire sentence at once to identify connections between words that are far apart.

[0057] Next, by applying a dense layer to the above-mentioned speech context information and linearly projecting it onto a dimension corresponding to a dictionary of predefined characters, subwords, and tokens, frame-unit logits (S13) can be calculated to indicate the probability that each frame corresponds to a specific token. Additionally, an effective length corresponding to the length of the effective logit interval can be calculated based on the mask.

[0058] At this time, the above-mentioned density layer can perform the role of linearly projecting the speech context information generated by the encoder into a dimension corresponding to the number of predefined characters or tokens, and can be utilized as a key indicator of VAD for determining the presence or absence of linguistic meaning rather than the physical existence of sound, as described later in Fig. 7.

[0060] Next, based on the mask, a valid logit section excluding the padding section can be identified, and greedy decoding can be performed to select the token with the highest classification score for each frame within that section. Subsequently, the selected token is identified as a blank token corresponding to the absence of utterance and a non-blank token corresponding to a meaningful character, and if the same non-blank token is repeated in consecutive frames, it is merged into one and the blank token is removed to generate a final speech recognition text (S14).

[0061] As such, the CTC algorithm used in step S14 can be a core decoding method that enables learning and inferring sequences without separate alignment information when the number of input frames and the length of the output text are different.

[0063] Finally, a speech candidate segment can be generated by identifying a segment in the frame unit logit where the classification score of the non-blank token is higher than that of the blank token, and a speech valid segment can be generated by excluding noise or meaningless sounds from the speech candidate segment and confirming only meaningful speech.

[0064] At this time, if the time interval between one or more utterance candidate intervals is less than a preset threshold, they can be merged into one interval to calculate the start and end times of the final utterance (S15).

[0066] FIG. 3 schematically illustrates the process of generating first feature information and an example of a feature according to one embodiment of the present invention.

[0068] Schematically, FIG. 3(a) illustrates a process of generating features based on a voice signal input from a user terminal and generating first feature information by projecting the features, and FIG. 3(b) illustrates examples of a two-dimensional array form and a three-dimensional array form corresponding to the features.

[0070] In one embodiment of the present invention, in the first feature information generation step, a feature including time and frequency information is generated based on a voice signal received from a user terminal, wherein the feature corresponds to a two-dimensional or three-dimensional array form, and the first feature information is generated by projecting the feature onto a fixed dimension, and the range of the first feature information can be normalized.

[0072] Specifically, as illustrated in FIG. 3(a), preprocessing such as a Short Time Fourier Transform (STFT) can be performed on the voice signal in analog and digital form received from a user terminal to generate features containing time and frequency information. At this time, the features contain the inherent properties of the voice signal, but their scale or dimensions may be irregular depending on the recording environment or equipment characteristics, and as illustrated in FIG. 3(a), they may correspond to a shape similar to crumpled paper that contains information but has an irregular shape.

[0073] Furthermore, unlike a general Fourier transform where the frequency components of the entire signal are known but the timing of frequency occurrence is unknown, the aforementioned STFT corresponds to a technique that performs a Fourier transform by dividing a long-duration signal into short time units, thereby enabling the analysis of frequency changes over time. Additionally, the STFT can be performed in the order of frame splitting, application of a window function, and transformation, through which the aforementioned features corresponding to a spectrogram can be generated.

[0074] Next, the above features can be projected onto a fixed dimension and normalized to generate the first feature information, and this process corresponds to a process of flattening an irregular data distribution into a form optimized for the model to learn. For example, as illustrated in FIG. 3(a), this corresponds to ironing crumpled paper corresponding to the features to make it stiff and flattened paper corresponding to the first feature information. Through this, the dimensions of the data are aligned and the range of values ​​is standardized, thereby enabling stable input in subsequent neural network operations.

[0076] In one embodiment of the present invention, as shown in FIG. 3(b), the feature may correspond to a spectrogram in the form of a two-dimensional array and a three-dimensional array, and the spectrogram visualizes sound waves, with the X-axis representing time and the Y-axis representing frequency, and the color, brightness, and height of each point may represent the intensity of the corresponding frequency component.

[0077] Preferably, in the present invention, the above features can be extracted by performing a dB log scale transformation while maintaining the linear frequency axis.

[0078] Specifically, the spectrogram in the form of a two-dimensional array may correspond to a planar image composed of two axes of time and frequency, as illustrated in FIG. 3(b), and expresses sound intensity through contour lines and colors, and can be processed as a tensor in the form of (time, frequency) when primarily used as input to a convolutional neural network. In addition, the tensor corresponds to a basic unit of data and may mean a multidimensional array with arbitrary dimensions.

[0079] Next, the spectrogram in the form of a three-dimensional array may visualize the sound intensity on the Z-axis in addition to time on the X-axis and frequency on the Y-axis, as shown in FIG. 3(b), or, from a data processing perspective, may correspond to a form having a three-dimensional tensor structure of (time, frequency, channel).

[0081] FIG. 4 schematically illustrates a comparative example of a process for generating second feature information through a multi-stage convolutional neural network and pooling operation according to an embodiment of the present invention.

[0083] In one embodiment of the present invention, a multi-stage convolutional neural network and pooling operation can be performed on the normalized first feature information in the second feature information generation step to generate second feature information corresponding to high-dimensional intermediate features from time and frequency patterns.

[0085] Specifically, as illustrated in FIG. 4, the normalized first feature information may correspond to the top rectangle, which may correspond to time-frequency plane data that has not yet been processed through computation and may be in a state where there is no depth of information corresponding to the channel.

[0086] Next, an initial feature map can be generated by performing a first-order convolution and pooling operation on the first feature information, which can form a rectangular prism in the shape of a brick by creating a depth called a channel in the planar data. Additionally, the existence of a channel may mean that basic patterns corresponding to edges and tones, etc., have started to be extracted from the speech signal.

[0087] Subsequently, high-dimensional features can be extracted by repeatedly performing multi-stage convolution and pooling on the feature map, and accordingly, the size of the data corresponding to the time and frequency axes can be reduced and the depth corresponding to the channels can be further increased, and as shown in Fig. 4, the rectangular prism can be represented in a form divided into several pieces. In addition, the several pieces can indicate that the local features of the speech have been further subdivided and abstracted.

[0088] Next, the dimensions of the extracted high-dimensional features can be modified to make them easier for the sequence model to process, and a dimension merging process can be performed by combining the frequency and channel axes to flatten the data into a long bar shape arranged along the time axis.

[0089] Finally, the second feature information can be generated by passing through a projection layer and aligning it into a fixed dimension that can be processed by the self-attention encoder corresponding to the subsequent step, and can be represented in the form of a standardized standard box as shown in FIG. 4. Preferably, the self-attention encoder can align it into a fixed dimension corresponding to 512 dimensions.

[0091] In one embodiment of the present invention, as shown in FIG. 4, the transformation from a rectangle to a rectangular prism can be expressed in a way that the simple sound intensity information is enriched into feature information from various perspectives through a convolutional neural network, and may correspond to the model beginning to understand sound as a meaningful signal rather than a simple wave.

[0092] In addition, the transformation from the above rectangular prism to the above multiple pieces may signify a situation where the physical size of the data is reduced through pooling, but the number of channels increases and the density of information increases, and the transformation from the above bars to the above standard boxes may serve as a bridge to transition from an image processing structure (CNN) to a language processing structure (Transformer). This allows complexly intertwined features to be arranged in chronological order like a bar shape and fitted into the above standard boxes that the self-attention encoder can accept, thereby enabling neural networks of different structures to be smoothly combined to generate high-quality speech features.

[0094] In one embodiment of the present invention, the convolutional neural network may correspond to an operation in which a filter slides over input data to find a specific pattern. For example, it may correspond to a process of moving a filter that acts as a magnifying glass over the first feature information corresponding to a rectangle to find and record features corresponding to a 'horizontal line', a 'vertical line', and a 'circle', and as the layer deepens, it may find complex information such as 'vowel shapes' and 'consonant features' starting from simple lines.

[0095] In addition, the above pooling can correspond to an operation that reduces the size of a feature map while retaining only the important information, and is similar to reducing a high-resolution photo to create a thumbnail; while the amount of data is reduced, the fact that important information exists in specific sections is preserved, which can increase computational efficiency.

[0096] And the above dimension merging and projection may correspond to a process of converting a 3D tensor containing time, frequency, and channels into a 2D sequence containing time and feature vectors, and adjusting the magnitude of the vector through a linear transformation. For example, it may correspond to the task of combining multiple stacked information cards into a single long summary, fitting it perfectly into a standard envelope, and passing it on to the next department.

[0098] FIG. 5 schematically illustrates an example of generating a mask according to second feature information converted into a sequence form according to one embodiment of the present invention.

[0100] In one embodiment of the present invention, in the mask generation step, the second feature information is converted into a sequence form that can be processed by a sequence model, and a mask can be generated based on whether valid data exists.

[0102] Specifically, the output second feature information may correspond to a 3-dimensional tensor in the form of (Batch, Channel, Height, Width) or a 4-dimensional tensor in the form of (Batch, Channel, Frequency, Time), and the second feature information may be converted into a 2-dimensional sequence structure in the form of (Batch, Time, Feature_Dimension) to be input into a sequence encoder in a subsequent stage including a transformer.

[0103] In addition, since the length of the input utterance may vary each time, and the deep learning model must fix the length based on the longest input for batch processing, padding can be performed to fill the latter part of the short input with a specific value such as '0', and the padding section may correspond to a box containing X, as shown in Fig. 5.

[0104] For example, as shown in FIG. 5, the length of the second feature information converted into a sequence form can be normalized to a length of '9' to match the second feature information sequence #2, which corresponds to the longest length, and accordingly, padding can be performed on the second feature information sequence #1 and second feature information sequence #3 to fix them to the same length as the second feature information sequence #2.

[0106] In one embodiment of the present invention, the mask generation unit (300) can identify sections within a sequence where actual voice information exists and sections filled with padding. At this time, a '1' corresponding to 'True' can be assigned to frames containing valid data, and a '0' corresponding to 'False' can be assigned to padding sections to generate a binary mask as shown in FIG. 5.

[0107] Specifically, in the operation process of generating the mask above, Valid_Length can be calculated for each sequence within the input batch. For example, if the total sequence length corresponds to L and the length of valid data corresponds to Length_{valid}, '1' can be assigned to the interval where the length to interval i for index i is less than Length_{valid}, and '0' can be assigned to the interval where the length to interval i is Length_{valid} or greater and less than the total sequence length L.

[0109] In addition, if the length of the input speech signal undergoes pooling or stride operations in the convolutional neural network (CNN) during the preprocessing process, the time axis length of the second feature information may be reduced, and it may be necessary to adjust the mask length considering the pooling ratio.

[0110] For example, in the case of a structure where the temporal resolution is reduced to 1 / 4 during the CNN process, the length of the mask may need to be generated based on the temporal length of the compressed second feature information rather than the length of the original speech signal. Therefore, accurate location mapping may be possible only if the mask is generated based on the value obtained by dividing the original valid length corresponding to the original speech signal by the pooling ratio.

[0111] At this time, when generating a mask, it may be necessary to clearly distinguish whether '0' is 0 as a data value or 0 as padding, and thus, instead of simply checking whether the value is 0, a mask can be generated based on the length information of the original metadata.

[0113] In one embodiment of the present invention, if a mask is not applied, padding values ​​that have no meaning may be included in the self-attention operation, which can distort the probability distribution. By forcibly replacing the attention score of the padding section with '-infinity' or the like through the mask, the model can focus only on the relationship between valid voice frames, thereby increasing the accuracy of voice recognition.

[0114] In addition, even if multiple voice data of different lengths are grouped into a single batch and processed simultaneously, independent context learning can be achieved without mutual interference.

[0116] FIG. 6 schematically illustrates the process of generating speech context information through a self-attention operation according to one embodiment of the present invention.

[0118] In one embodiment of the present invention, in the self-attention encoding step, the second feature information is input to a self-attention-based speech recognition encoder, and speech context information including associations between frames can be generated based on the mask.

[0119] Specifically, the second feature information in the form of a sequence is encoded using a plurality of encoding layers, and by reflecting the mask in the attention weight calculation process of the self-attention operation, meaningless sections including padding sections and invalid frames are restricted from contributing to the attention operation, and the speech context information reflecting long-term dependencies between frames can be generated as a sequence-form output.

[0121] More specifically, as illustrated in FIG. 6, the second feature information can be input into an encoder block composed of a Multi-Head Attention and FeedForward network. Subsequently, the block is stacked N times and executed repeatedly, and residual connection and layer normalization can be performed at each iteration step. Accordingly, the final output data may correspond to the speech context information, which implies not only the acoustic characteristics of individual frames but also the contextual relationships within the entire sequence. At this time, within each block, residual connection, which adds the original input to the operation result, and layer normalization, which adjusts the data distribution, are performed to help ensure stable learning without information loss even as the layers deepen.

[0122] In addition, gradual abstraction can be achieved by repeatedly inputting into the blocks above. In the case of the initial blocks corresponding to the lower layer, one can focus on identifying acoustic features of short segments, such as the relationship between adjacent frames, or the distinction between consonants and vowels and the connection of phonemes. In the case of the later blocks corresponding to the upper layer, the field of view widens as the layers progress, allowing one to identify the flow of the entire sentence or the long-term dependency mentioned above, and to develop simple sound information into high-dimensional semantic information.

[0123] For example, in the case of the ambiguous audio sequence 'Father goes into the bag,' it is possible to clarify what each frame sounds like—such as 'a,' 'beo,' 'ji,' and 'ga'—examine only the relationship with the immediately adjacent frame, and group the organized sounds to create meaningful word candidates. At this stage, consideration can be made regarding whether it is 'Father + ga' or 'Father + bag.' Subsequently, based on the aforementioned word candidates, the entire sentence can be viewed from beginning to end; upon seeing the predicate 'goes into' at the end of the sentence, it is determined that the subject should naturally be the person 'Father' rather than the 'bag,' thereby enabling the output of audio context information such as 'Father in the room.'

[0125] In one embodiment of the present invention, the self-attention operation may correspond to a mechanism that calculates what association each frame in an input sequence has with all other frames, including itself.

[0126] Specifically, for each frame of the second feature information, a weight matrix can be multiplied to generate a query (Q) vector corresponding to the subject asking who I am currently related to, a key (K) vector corresponding to an identifier that can verify whether I am related to them, and a value (V) vector corresponding to the information I actually possess.

[0127] Subsequently, the dot product of the current frame's query and the keys of all frames can be calculated, and a larger dot product value may indicate a higher correlation between the two frames. For example, a query for the word 'love' can yield a high score with the key for the word 'sun'.

[0128] Next, the score calculated through the inner product can be divided by a specific value corresponding to the square root of the dimension so that it does not become too large. At this time, the above mask can be used to make the probability value of the padding interval '0' during the subsequent Softmax application process so that it does not participate in context formation.

[0129] And the softmax function can be applied to the scores, and all scores are converted into probability values ​​between 0 and 1, and the total sum can be 1. At this time, the probability value may represent a ratio that determines how much information the current frame will take from each other frame.

[0130] Finally, the voice context information can be generated by multiplying each frame value vector by the probability value corresponding to the weight and adding them all together. At this time, the voice context information can be generated by reflecting relevant information with a high weight and reflecting it significantly, and reflecting irrelevant information with a low weight.

[0132] For example, let’s assume that four frames of ‘Sa’, ‘Rang’, ‘Hae’, and padding are input, and I will try to understand the context from the perspective of ‘Hae’, which corresponds to the third frame.

[0133] First, 'Hae's' query #3( For ), the key vector of all frames( inside ) and value vector( inside You can prepare ), and the above and each key vector can be compared. At this time, 80 points, is 90 points, 95 points, and It can result in 10 points, and the score corresponding to frame 4, which corresponds to padding, can be removed by applying a mask containing 0 or 1 to each score.

[0134] Next, a softmax function can be applied to each of the above scores to convert them into weights, and the weights may be 0.25 for 'Sa' corresponding to Frame 1, 0.35 for 'Rang' corresponding to Frame 2, and 0.40 for 'Hae' corresponding to Frame 3.

[0135] Finally, by combining the above weights, the final information of the 'solution' corresponding to Frame 3 is ' It can be represented as follows. Through this, the frame of 'hae' can include not only the sound information of 'hae' but also the preceding contextual information of 'sarang,' allowing for a contextual understanding that it is part of 'saranghae' (Love) rather than 'hae (Sun).'

[0137] In one embodiment of the present invention, when calculating the attention score, the mask generated in FIG. 5 must be applied to exclude meaningless padding values ​​from the context, and since the self-attention has the characteristic of ignoring sequence information, position encoding can be added if necessary to prevent dilution of sequence information, and since the amount of computation increases in proportion to the square of the input sequence length, memory efficiency may be considered for excessively long audio inputs.

[0139] FIG. 7 schematically illustrates the process of calculating frame-unit logits by applying a density layer to voice context information according to one embodiment of the present invention.

[0141] In one embodiment of the present invention, in the self-attention encoding step, a dense layer is applied to linearly project the speech context information onto a dimension corresponding to a dictionary of predefined characters, subwords, and tokens, thereby calculating a frame-unit logit that includes a classification score indicating the probability that each frame corresponds to a specific token.

[0142] Specifically, as illustrated in FIG. 7, the speech context information corresponding to abstract feature values ​​implying contextual relationships with surrounding frames can pass through a dense layer, which is a fully connected layer, and the arrows in FIG. 7 can represent weight operations between input nodes and output nodes, and can be calculated in the form of a logit corresponding to a raw value before being converted into a probability based on the weight operations. At this time, the dimension of the calculated logit can be equal to the total number of tokens.

[0144] More specifically, the density layer can perform the role of mapping the speech context information in the abstract feature space to the lexical dictionary space of the characters, subwords, and tokens that we intend to recognize, and this can be called linear projection. In this case, the lexical dictionary space may correspond to a list of all correct answer candidates that the model can predict, and the density layer may represent the closest candidate in the list to which input sound is closest as a logit, and the subword may correspond to a combination of frequently occurring characters grouped into a single unit, and it is efficient to recognize the word 'school' at once without splitting it into 'hak' + 'gyo'.

[0145] In addition, the above-mentioned dense layer may correspond to a linear equation of the logit for the product of the voice context information vector and the weights, and the matrix size of the weights may correspond to the product of the input dimension and the total number of tokens.

[0146] For example, it can be assumed that a user made the sound 'an' in a specific frame t, and a 512-digit numeric vector containing the sound characteristics of 'an' and the surrounding context can be input. Subsequently, scores can be assigned to 3000 predefined characters, and for 'an' corresponding to index 50, the highest score of 12.5 is received, and a classification score can be generated indicating that the probability of it being 'an' in the frame at time t is the highest.

[0148] In one embodiment of the present invention, the logit corresponding to the output of the density layer can be used not merely for text conversion, but as direct basis data for a VAD that determines a speech segment with actual meaning based on the token score, and can mathematically restore the character in a subsequent step by converting it into a score of a promised character system.

[0150] FIG. 8 schematically illustrates an example of a process for calculating an effective length according to one embodiment of the present invention.

[0152] In one embodiment of the present invention, in the self-attention encoding step, an effective length representing the length of an effective section excluding padding sections among a frame-unit logit sequence can be calculated based on the mask, and the frame-unit logit sequence may correspond to the frame-unit logits arranged in chronological order.

[0153] Specifically, the mask vector can have a value of '1' for valid frames and '0' for invalid padding frames, and the number of valid data in the sequence can be immediately derived by simply performing a simple summation operation of adding all elements of the mask vector.

[0154] More specifically, a mask vector generated during the sequence conversion process and passed through the encoder can be verified, and all values ​​within the mask vector can be added. Subsequently, the effective length can be calculated through a process of determining and storing the summed result as the effective length of the corresponding batch.

[0155] For example, as shown in FIG. 8, if there are 0 padding frames among a total of 9 frames, an effective length of 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 9 can be calculated, if there is 1 padding frame, an effective length of 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 = 8 can be calculated, and if there are 3 padding frames, an effective length of 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 = 6 can be calculated.

[0157] FIG. 9 schematically illustrates an example of the application of a CTC decoding step according to one embodiment of the present invention.

[0159] In one embodiment of the present invention, the method may include: a step of specifying an effective logit section that is the subject of actual operation by excluding padding sections from a frame-unit logit sequence based on the mask in the CTC decoding step; a step of performing greedy decoding to select the token having the highest classification score in each frame among a plurality of tokens included in the effective logit section; a step of identifying whether the selected token is a blank token corresponding to the absence of utterance or a non-blank token representing a meaningful character, wherein a space token representing a space is treated as a single character and included in the non-blank token; and a step of generating a final speech recognition text by merging the same non-blank token into a single non-blank token when the same non-blank token is repeated in consecutive frames and removing the blank token within the effective logit section.

[0161] In other words, the final speech recognition text can be generated through the steps of valid logit interval specification, greedy decoding, token identification, token normalization, and final text generation.

[0162] For example, as shown in FIG. 9, a mask array [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0] can be identified, and the front part of the mask array, where the value is '1', can be set as a valid logit interval containing actual voice information, and the back part, where the value is '0', can be excluded from the operation as padding.

[0163] Next, through the greedy decoding above, the token with the highest score for each frame within the valid logit interval can be selected, and for frame 1, 'Sa', for frame 2, 'Sa', for frame 3, a blank token, for frame 4, 'Rang', for frame 5, a blank token, for frame 6, a space token, and for frame 7, 'Hae' can be selected.

[0164] Afterward, it can be identified whether the selected token is a non-blank token corresponding to a meaningful character or a blank token without an utterance, and as shown in FIG. 9, in the case of Frame 1, Frame 2, Frame 4, Frame 6, and Frame 7, it can be identified as a non-blank token, and in the case of Frame 3 and Frame 5, it can be identified as a blank token.

[0165] Finally, identical non-blank tokens in consecutive frames containing 'sa' can be merged into one, all blank tokens can be removed, and the final speech recognition text 'I love you' can be generated.

[0167] In one embodiment of the present invention, the greedy decoding may correspond to a method of selecting the class with the highest probability or score for the input at each time point, and may correspond to a simple and fast method of immediately selecting the most likely sound without considering future time points or the optimal solution of the entire context.

[0169] In one embodiment of the present invention, in speech recognition, whether a person says 'Ah' briefly or 'Ah~' at length, it may be recognized multiple times in frame units, such as 'Ah', 'Ah', 'Ah', 'Ah'. Accordingly, these frames must be combined into one, but a situation may arise where the actual characters are repeated, such as in 'Gangga', making it impossible to distinguish them.

[0170] In this situation, a blank token can be set to distinguish between tokens, and 'sa' + 'sa' can be merged by considering it as a long pronunciation of 'sa', or 'gak' + blank token + 'gak' can be not merged by considering it as two characters, 'gakgak'.

[0171] In addition, by setting a space token corresponding to a non-blank token, whitespace can be treated as meaningful characters, which can help natural language processing engines correctly parse sentences.

[0173] FIG. 10 schematically illustrates an example of a process for generating ignition candidate sections in an ignition section calculation step according to an embodiment of the present invention.

[0175] In one embodiment of the present invention, in the utterance section calculation step, the section is determined based on the possibility of expression of each frame character included in the frame unit logit, and a valid logit section having a higher classification score in the non-blank token than in the blank token can be identified to generate a candidate utterance section.

[0176] At this time, in the step of generating utterance candidate intervals, logits for each time frame calculated through a self-attention encoder and a density layer can be used as input data, and the classification score can correspond to the probability that the frame corresponds to a blank token and the probability that it corresponds to a non-blank token.

[0178] For example, as shown in FIG. 10, time frame inside The entry and exit of utterances can be identified by tracking changes in the score of blank tokens and non-blank tokens according to the flow.

[0179] Specifically, and In this case, since the score of the blank token is significantly higher than the score of the non-blank token, it can be judged as a 'silent interval' where there is no utterance, and In this case, since the score of the non-blank token surpasses the score of the blank token, it can be determined as the 'utterance entry section' where utterance begins. Also, inside In the case of, since the score of the non-blank token is consistently dominant, it can be judged as a 'smoothing maintenance period,' and inside In that case, the score of the blank token increases again while the score of the non-blank token drops sharply, so It can be determined as a 'silence return interval' where the utterance ends at that point.

[0180] Subsequently, based on the identification results of each segment, [corresponding to the consecutive segments where non-blank tokens were dominant , , , ] can be generated by confirming it as a single utterance candidate interval.

[0182] In one embodiment of the present invention, since conventional energy-based VADs simply detect ignition when the sound volume is loud, energy may be emitted even from noise such as the sound of a door closing or clapping.

[0183] On the other hand, since the above-mentioned integrated speech processing method uses a classification score that asks 'Does this sound correspond to a specific character?', even if loud noise occurs, if it does not match a token, the non-blank score is low and it can not be mistaken for a speech, thus enabling the implementation of a semantic-based VAD that is robust against noise.

[0185] FIG. 11 schematically illustrates an example of a process for generating an ignition effective section of an ignition section calculation step according to one embodiment of the present invention.

[0187] In one embodiment of the present invention, in the utterance section calculation step, the utterance candidate section is determined as a valid section in which meaningful utterances exist with noise or meaningless sounds excluded, and the start time of the utterance and the end time of the utterance are calculated to generate a valid utterance section, wherein if the time interval between one or more utterance candidate sections is less than a preset time threshold, the utterance candidate sections may be merged into one.

[0188] Specifically, the utterance candidate segments generated in the process of FIG. 10 can be used as input data to generate utterance valid segments as shown in FIG. 11, and through this process, physically separated segments can be merged according to contextual continuity or meaningless short noises can be removed to finely refine the beginning and end of the utterance.

[0190] For example, as illustrated in FIG. 11, the ignition candidate section [ , , , Based on the length and spacing of ], the above ignition effective section [ , Can generate ]

[0191] Specifically, if the preset time threshold corresponds to 0.3 seconds and the preset minimum length threshold corresponds to 0.1 seconds, The section ends at 2.5 seconds, and The segment starts at 2.6 seconds, and since the interval between the two segments is 0.1 seconds, it is determined to be a short pause, so the two segments can be merged into a single continuous utterance without being separated. Also, The segment length is 0.05 seconds, so it can be removed as it is determined to be short noise. Finally, The segment can be determined as a normal utterance because its length is greater than or equal to the preset minimum length threshold of 1.0 second, and the interval from the preceding segment is also greater than or equal to the preset time threshold of 1.95 seconds. Through this, the merged segment and the normal maintained segment are combined to form the final valid utterance segment [ , Can generate ]

[0193] In one embodiment of the present invention, by merging intervals based on the preset time threshold, the chattering phenomenon in which the logit score temporarily drops in the middle of a speech and the interval is broken into small pieces can be prevented, and the preset minimum length threshold can be carefully set considering linguistic characteristics to prevent the risk of short answers being treated as noise and deleted.

[0195] FIG. 12 illustrates, in an exemplary manner, the internal configuration of a computing device according to one embodiment of the present invention.

[0197] The server system (1000) mentioned in the description of FIG. 1 may include components of the computing device (11000) illustrated in FIG. 12, which will be described later.

[0199] As illustrated in FIG. 12, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0201] Specifically, the memory (11200) may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0202] At this time, access to the memory (11200) from other components, such as the processor (11100) or the peripheral device interface (11300), can be controlled by the processor (11100). The processor (11100) may be composed of a single or multiple units and may include processors in the form of GPUs and TPUs to improve computational processing speed.

[0203] The above peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and the memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing a software module or instruction set stored in the memory (11200).

[0204] The input / output subsystem (11400) may connect various input / output peripheral devices to the peripheral device interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripheral devices such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral device interface (11300). According to another aspect, the input / output peripheral devices may be connected to the peripheral device interface (11300) without passing through the input / output subsystem (11400).

[0205] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0206] The communication circuit (11600) may enable communication with another computing device using at least one external port. Alternatively, as described above, the communication circuit (11600) may enable communication with another computing device by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal, as needed.

[0208] The embodiment of FIG. 12 is merely an example of the computing device (11000), and the computing device (11000) may have some components shown in FIG. 12 omitted, additional components not shown in FIG. 12 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors in addition to the components shown in FIG. 12, and the communication circuit (1160) may include a circuit for RF communication of various communication methods (Wi-Fi, 3G, LTE, 5G, 6G, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0209] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0211] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0212] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be standardized and stored or executed in a standardized manner on a networked computing device. Software and data may be stored on one or more computer-readable recording media.

[0213] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0215] According to one embodiment of the present invention, by naturally calculating speech segments during the processing of a speech recognition model without independently building a separate speech segment detection module or performing it as a preprocessing step, the complexity of the system can be reduced and the construction and operation costs can be reduced.

[0216] According to one embodiment of the present invention, instead of determining the presence or absence of sound based solely on sound pressure or frequency characteristics, by determining the utterance interval based on the possibility of token expression calculated through a dense layer, it is possible to exclude noise or non-utterance noise and precisely detect only meaningful linguistic utterance intervals.

[0217] According to one embodiment of the present invention, since the time at which the speech recognition text is generated and the time at which the utterance interval is determined are derived from the same feature information, the effect of perfectly matching the temporal consistency between the text result and the utterance timing information can be achieved.

[0218] According to one embodiment of the present invention, by applying a mask based on whether valid data exists during the self-attention operation, padding sections or invalid frames can be prevented from acting as unnecessary noise in context learning, thereby improving the accuracy of speech recognition.

[0219] According to one embodiment of the present invention, by including a preprocessing process that projects input voice features onto a fixed dimension and normalizes them, it is possible to achieve the effect of maintaining consistent performance even if the scale of the input signal changes due to differences in the recording environment or microphone performance.

[0220] According to one embodiment of the present invention, by effectively extracting local features of time and frequency patterns through a multi-stage convolutional neural network (CNN) and transmitting them to a self-attention encoder, it is possible to produce high-quality speech features that reflect short-term characteristics and long-term contextual information of a speech signal.

[0221] According to one embodiment of the present invention, by analyzing frame-unit logit sequences to identify sections where the score of non-blank tokens is higher than that of blank tokens, it is possible to achieve the effect of immediately estimating valid utterance sections without separate VAD tuning, even in new domains or environments that have not been learned.

[0222] According to one embodiment of the present invention, by transmitting text with unnecessary silent segments removed to a subsequent intent classification or answer generation engine based on calculated utterance segment information, it is possible to achieve the effect of improving the analysis accuracy and response quality of the natural language processing (NLP) engine.

[0223] According to one embodiment of the present invention, when the time interval between one or more utterance candidate intervals is less than a preset threshold, they are merged into one interval, thereby preventing a single sentence from being fragmented into multiple pieces due to short pauses or breaks in the middle of utterance.

[0224] According to one embodiment of the present invention, by adopting a CTC-based decoding method to eliminate duplication between frames and generate final text, it is possible to achieve the effect of providing speech recognition results suitable for real-time while ensuring fast inference speed.

[0225] According to one embodiment of the present invention, the dense layer is designed not only to generate classification scores for text output but also to perform a dual function as a key indicator for utterance interval estimation, thereby maximizing the parameter efficiency of the model.

[0226] According to one embodiment of the present invention, even in situations where the length of the input sequence is variable, the effective length is calculated based on a mask and reflected in CTC loss calculation and decoding, thereby enabling stable and robust processing for utterance data of various lengths.

[0228] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims.

Claims

Claim 1 An integrated speech processing method comprising calculating utterance segments through self-attention-based sequence encoding and CTC-based decoding performed on a server system comprising one or more processors and one or more memories, wherein the method generates features including time and frequency information based on a speech signal received from a user terminal, wherein the features correspond to a 2D or 3D array form, and generates first feature information by projecting the features onto a fixed dimension and normalizes the range of the first feature information; a first feature information generation step of generating second feature information corresponding to a high-dimensional intermediate feature from the time and frequency pattern by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; a mask generation step of converting the second feature information into a sequence form that can be processed by a sequence model and generating a mask based on whether valid data exists; inputting the second feature information into a self-attention-based speech recognition encoder, generating speech context information including associations between frames based on the mask, and then projecting the speech context information to form a frame-unit An integrated speech processing method comprising: a self-attention encoding step for calculating logits; a CTC decoding step for evaluating tokens corresponding to text candidates per frame by performing CTC-based decoding based on the mask and the logits, and generating final speech recognition text by removing duplicates between consecutive frames; and an utterance segment calculation step for calculating the start and end times of an utterance corresponding to a segment where meaningful speech exists by analyzing the distributed arrangement structure of tokens appearing in the mask and the logits; wherein the self-attention encoding step encodes the second feature information in the form of a sequence using a plurality of encoding layers, and by reflecting the mask in the attention weight calculation process of the self-attention operation, restricts meaningless segments including padding segments and invalid frames from contributing to the attention operation, and generates the speech context information reflecting long-term dependencies between frames as a sequence-form output. Claim 2 delete Claim 3 An integrated speech processing method comprising calculating utterance segments through self-attention-based sequence encoding and CTC-based decoding performed on a server system comprising one or more processors and one or more memories, wherein the method generates features including time and frequency information based on a speech signal received from a user terminal, wherein the features correspond to a 2D or 3D array form, and generates first feature information by projecting the features onto a fixed dimension and normalizes the range of the first feature information; a first feature information generation step of generating second feature information corresponding to a high-dimensional intermediate feature from the time and frequency pattern by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; a mask generation step of converting the second feature information into a sequence form that can be processed by a sequence model and generating a mask based on whether valid data exists; inputting the second feature information into a self-attention-based speech recognition encoder, generating speech context information including associations between frames based on the mask, and then projecting the speech context information to form a frame-unit A self-attention encoding step for calculating logits; a CTC decoding step for evaluating tokens corresponding to frame-specific text candidates and removing duplicates between consecutive frames by performing CTC-based decoding based on the mask and the logits to generate final speech recognition text; and an utterance segment calculation step for calculating the start and end times of an utterance corresponding to a segment where meaningful speech exists by analyzing the distributed arrangement structure of tokens appearing in the mask and the logits; wherein the self-attention encoding step comprises a step of calculating frame-unit logits including classification scores indicating the probability that each frame corresponds to a specific token by applying a dense layer that linearly projects the speech context information onto a dimension corresponding to a predefined lexical dictionary of characters, subwords, and tokens;An integrated speech processing method comprising: a step of calculating an effective length representing the length of an effective logit section excluding padding sections among a frame-unit logit sequence based on the above mask, wherein the frame-unit logit sequence corresponds to the frame-unit logits arranged in chronological order. Claim 4 An integrated speech processing method comprising calculating utterance segments through self-attention-based sequence encoding and CTC-based decoding performed on a server system comprising one or more processors and one or more memories, wherein the method generates features including time and frequency information based on a speech signal received from a user terminal, wherein the features correspond to a 2D or 3D array form, and generates first feature information by projecting the features onto a fixed dimension and normalizes the range of the first feature information; a first feature information generation step of generating second feature information corresponding to a high-dimensional intermediate feature from the time and frequency pattern by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; a mask generation step of converting the second feature information into a sequence form that can be processed by a sequence model and generating a mask based on whether valid data exists; inputting the second feature information into a self-attention-based speech recognition encoder, generating speech context information including associations between frames based on the mask, and then projecting the speech context information to form a frame-unit A self-attention encoding step for calculating logits; a CTC decoding step for evaluating tokens corresponding to frame-specific text candidates and removing duplicates between consecutive frames by performing CTC-based decoding based on the mask and the logits to generate final speech recognition text; and an utterance interval calculation step for calculating the start and end times of an utterance corresponding to a segment containing meaningful speech by analyzing the distributed arrangement structure of tokens appearing in the mask and the logits; wherein the CTC decoding step comprises: a step of specifying a valid logit interval subject to actual computation by excluding padding intervals from a frame-unit logit sequence based on the mask; and a step of performing greedy decoding to select the token having the highest classification score in each frame among a plurality of tokens included within the valid logit interval.An integrated speech processing method comprising: identifying whether the selected token is a blank token corresponding to the absence of an utterance or a non-blank token representing a meaningful character, wherein a space token representing a space is treated as a single character and included in the non-blank token; and, if the same non-blank token is repeated in consecutive frames, merging it into a single non-blank token and removing the blank token within the valid logit interval to generate a final speech recognition text. Claim 5 An integrated speech processing method comprising calculating utterance segments through self-attention-based sequence encoding and CTC-based decoding performed on a server system comprising one or more processors and one or more memories, wherein the method generates features including time and frequency information based on a speech signal received from a user terminal, wherein the features correspond to a 2D or 3D array form, and generates first feature information by projecting the features onto a fixed dimension and normalizes the range of the first feature information; a first feature information generation step of generating second feature information corresponding to a high-dimensional intermediate feature from the time and frequency pattern by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; a mask generation step of converting the second feature information into a sequence form that can be processed by a sequence model and generating a mask based on whether valid data exists; inputting the second feature information into a self-attention-based speech recognition encoder, generating speech context information including associations between frames based on the mask, and then projecting the speech context information to form a frame-unit A self-attention encoding step for calculating logits; a CTC decoding step for evaluating tokens corresponding to frame-specific text candidates and removing duplicates between consecutive frames by performing CTC-based decoding based on the mask and the logits to generate final speech recognition text; and an utterance interval calculation step for calculating the start and end times of an utterance corresponding to a segment containing meaningful speech by analyzing the distributed arrangement structure of tokens appearing in the mask and the logits; wherein the CTC decoding step comprises: a step of specifying a valid logit interval subject to actual computation by excluding padding intervals from a frame-unit logit sequence based on the mask; and a step of performing greedy decoding to select the token having the highest classification score in each frame among a plurality of tokens included within the valid logit interval.The method comprises: identifying whether the selected token is a blank token corresponding to the absence of an utterance or a non-blank token representing a meaningful character, wherein a space token representing a blank space is treated as a single character and included in the non-blank token; and, if the same non-blank token is repeated in consecutive frames, merging it into a single non-blank token and removing the blank token within the valid logit interval to generate a final speech recognition text; wherein the utterance interval calculation step comprises determining the interval based on the probability of occurrence of a character per frame included in the frame-unit logit, and identifying a valid logit interval having a higher classification score in the non-blank token than in the blank token to generate an utterance candidate interval; An integrated speech processing method comprising: a step of determining the above-mentioned speech candidate segment as a valid segment in which meaningful speech exists with noise or meaningless sounds excluded, and generating a speech valid segment by calculating the start time and end time of the speech, wherein if the time interval between one or more speech candidate segments is less than a preset time threshold, the speech candidate segments are merged into a single speech candidate segment.; Claim 6 An integrated speech processing system comprising one or more processors and one or more memories, and including the calculation of utterance segments through self-attention-based sequence encoding and CTC-based decoding, wherein the system generates features including time and frequency information based on a speech signal received from a user terminal, wherein the features correspond to a 2D or 3D array form, and generates first feature information by projecting the features onto a fixed dimension and normalizes the range of the first feature information; a first feature information generation unit that generates second feature information corresponding to high-dimensional intermediate features from time and frequency patterns by performing a multi-stage convolutional neural network and pooling operation on the normalized first feature information; a mask generation unit that converts the second feature information into a sequence form that can be processed by a sequence model and generates a mask based on whether valid data exists; inputs the second feature information into a self-attention-based speech recognition encoder, generates speech context information including associations between frames based on the mask, and then projects the speech context information to form a frame-unit An integrated speech processing system comprising: a self-attention encoding unit that calculates logits; a CTC decoding unit that evaluates tokens corresponding to text candidates per frame by performing CTC-based decoding based on the mask and the logits, and generates final speech recognition text by removing duplicates between consecutive frames; and an utterance segment calculation unit that calculates the start and end times of an utterance corresponding to a segment where meaningful speech exists by analyzing the distributed arrangement structure of tokens appearing in the mask and the logits; wherein the self-attention encoding unit encodes the second feature information in the form of a sequence using a plurality of encoding layers, and by reflecting the mask in the attention weight calculation process of the self-attention operation, restricts meaningless segments including padding segments and invalid frames from contributing to the attention operation, and generates the speech context information reflecting long-term dependencies between frames as a sequence-form output.

Images