DEVICE FOR LANGUAGE PROCESSING AND METHOD FOR IT

Abstract

In one embodiment, a speech processor that determines a target embedding containing a feature of target data determines a context vector and response text, estimated as text of the target data, by applying the target data associated with a language to a speech processing model trained to output a feature of a signal as a number. It then determines whether an operation of determining the response text is complete by applying the target embedding and the context vector to a second decoder trained to determine whether an embedding is complete, and outputs the response text based on whether the response text and the target embedding are complete.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a speech processing device and a method thereof, and in particular to a technology for generating sentences by applying embedding with respect to a language to a transformer-based language model. BACKGROUND OF THE INVENTION

[0002] Modern robotics and conversational systems with artificial intelligence (AI systems) rely heavily on speech-to-text conversion technologies to recognize a user's speech and generate an appropriate response based on the recognized result. These technologies operate by converting a speech signal into text and then generating sentences or determining the user's utterance intent based on the converted text.

[0003] The process of converting a speech signal into text takes time, which can make it difficult to immediately process user requests in robots and conversational AI systems that require real-time processing. Furthermore, there are frequent instances where the text converted from the speech signal is interpreted differently than the actual utterance. For example, if the user says "Tell me the price of Ionic 5," the text conversion process might mistakenly interpret it as "Tell me the price of Ayuni 5." This error can prevent the user from obtaining the intended result.

[0004] Furthermore, if an EPD technology that cuts and delivers the speech signal terminates too early, only a portion of the speech data can be processed, leading to inaccurate results. For example, if the EPD terminates at "Open" when the user says "Open ---- the door," the output may differ from the actual intention. This is due to imperfections in the text conversion process and limitations in the signal processing method. In particular, this can be a serious limitation in systems (e.g., robot control, voice assistants, and conversational AI) where real-time responsiveness and accuracy are critical.

[0005] To solve these problems, it is desirable to develop a technology that eliminates a text conversion stage and directly uses embedding data extracted from the speech signal in real time. SUMMARY

[0006] Embodiments of the present disclosure were carried out to solve the aforementioned problems that arise in the prior art, while retaining the advantages achieved through the prior art.

[0007] One embodiment of the present disclosure provides a speech processing device that performs real-time processing without a text conversion process by using embeddings extracted directly from a speech signal, and a method thereof.

[0008] An embodiment of the present disclosure provides a speech processing device that simultaneously performs text generation and speech presence / absence determination by a single embedding by converting a speech signal into an embedding form and simultaneously extracting text and status information based on the converted result by using a text decoder and a speech activity decoder, and a method thereof.

[0009] The technical problems to be solved by the present disclosure are not limited to the problems mentioned above, and all other technical problems not mentioned here will be clearly understandable to the person skilled in the art, to whom the present disclosure relates, from the following description.

[0010] According to one embodiment of the present disclosure, a speech processing device may include a memory that stores a computer-executable instruction and a processor that executes the instruction by accessing the memory.The processor can determine a target embedding that includes a feature of target data by applying the target data, which pertains to a language, to a language processing model trained to output a feature of a signal as a number, a context vector, and response text estimated as the text of the target data; by applying the target embedding to a first decoder trained to output a probability of text based on an attention mechanism; by determining whether an operation of determining the response text is complete by applying the target embedding and the context vector to a second decoder trained to determine whether an embedding is complete; and can output the response text based on whether the response text and the target embedding are complete.

[0011] In one embodiment, the processor can determine training noise data based on at least one of first sub-noise data determined in the target space, or second sub-noise data generated based on a standard normal distribution, or any combination thereof; can determine training speech data based on at least one of first sub-speech data recorded by a user, or second sub-speech data generated based on a speech synthesis model, or any combination thereof; can generate training target data by signal-noise ratio (SNR) mixing of the training noise data and the training speech data; and can determine a training target embedding by applying the training target data to the speech processing model.

[0012] In one embodiment, the processor can determine a first temporary output by applying the training target embedding to a text decoder, determine a second temporary output by applying the training target embedding to a speech activity decoder, and train the speech processing model by a first loss based on the first temporary output and the text decoder, and a second loss based on the second temporary output and the speech activity decoder.

[0013] In one embodiment, the processor can identify a first training response text, determined by the first decoder at a time point preceding a target time point, if the target time point is the time at which the training target embedding is applied to the first decoder. The processor can then determine a training context vector and a second training response text by applying the training target embedding and the first training response text to the first decoder. The second training response text can be applied to a training input of the first decoder at a time point following the target time point.

[0014] In one embodiment, the processor can determine a termination probability regarding whether the determination of the second training response text has finished by applying the training target embedding and the training context vector to the second decoder, and can determine whether an operation of determining the second training response text should be terminated based on a comparison between the termination probability and a predetermined value.

[0015] In one embodiment, the processor can determine a first decoder loss based on the first decoder and the second training response text, and a second decoder loss based on the second decoder and the termination probability, based on a response generation model that includes the first decoder and the second decoder, and can train the response generation model based on the first decoder loss and the second decoder loss.

[0016] In one embodiment, the processor can determine a temporary output by applying the target embedding and context vector to the second decoder, and can determine whether to terminate the process of determining the response text based on a comparison between the temporary output and a predetermined value.

[0017] In one embodiment, the processor can convert the response text into a language and output the language to a user who enters the target data.

[0018] In one embodiment, the processor can determine an utterance intent of the response text by applying the response text to an utterance intent prediction model based on the determination of the response text, can determine action data by applying the utterance intent to an action database according to the utterance intent, and can transmit the action data to a robot connected to the speech processing device.

[0019] According to one embodiment of the present disclosure, a language processing method may include: determining a target embedding containing a feature of target data by applying the target data, which relates to a language, to a language processing model trained to output a feature of a signal as a number; determining a context vector and response text, estimated as text of the target data, by applying the target embedding to a first decoder trained to output a probability of text based on an attention mechanism; determining whether an operation of determining the response text is complete by applying the target embedding and the context vector to a second decoder trained to determine whether an embedding is complete; and outputting the response text based on whether the response text and the target embedding are complete.

[0020] In one embodiment, outputting the response text may include: determining training noise data based on at least one of first sub-noise data determined in the target space, or second sub-noise data generated based on a standard normal distribution, or any combination thereof; determining training speech data based on at least one of first sub-speech data recorded by a user, or second sub-speech data generated based on a speech synthesis model, or any combination thereof; generating target training data by SNR mixing of the training noise data and the training speech data; and determining a target training embedding by applying the target training data to the speech processing model.

[0021] In one embodiment, outputting the response text may involve: determining a first temporary output by applying the training target embedding to a text decoder, determining a second temporary output by applying the training target embedding to a speech activity decoder, and training the speech processing model by a first loss based on the first temporary output and the text decoder, and a second loss based on the second temporary output and the speech activity decoder.

[0022] In one embodiment, outputting the response text can involve: identifying a first training response text, determined by the first decoder at a time point preceding a target time point, if the target time point is the time at which the training target embedding is applied to the first decoder; and determining a training context vector and a second training response text by applying the training target embedding and the first training response text to the first decoder. The second training response text can be applied to a training input of the first decoder at a time point following the target time point.

[0023] In one embodiment, outputting the response text may involve: determining a termination probability regarding whether the retrieval of the second training response text has finished, by applying the training target embedding and the training context vector to the second decoder, and determining whether an operation of retrieving the second training response text should be terminated, based on a comparison between the termination probability and a predetermined value.

[0024] In one embodiment, outputting the response text may involve: determining a loss of the first decoder based on the first decoder and the second training response text, and a loss of the second decoder based on the second decoder and the termination probability, based on a response generation model that includes the first decoder and the second decoder, and training the response generation model based on the loss of the first decoder and the loss of the second decoder.

[0025] In one embodiment, outputting the response text may involve: determining a temporary output by applying the target embedding and context vector to the second decoder, and determining whether to terminate the response text determination process based on a comparison between the temporary output and a predetermined value.

[0026] In one embodiment, outputting the response text may involve converting the response text into a language and outputting the language to a user who enters the target data.

[0027] In one embodiment, outputting the response text may involve: determining the utterance intent of the response text by applying the response text to an utterance intent prediction model based on the determination of the response text, determining action data by applying the utterance intent to an action database according to the utterance intent, and transmitting the action data to a robot connected to a speech processing device. ABBREVIATION OF THE DRAWINGS

[0028] The above and other embodiments, features and advantages of the present disclosure will become clearer from the following detailed description in conjunction with the accompanying drawings: Fig. 1 is a block diagram illustrating a speech processing device according to an embodiment of the present disclosure; Fig. 2 is a flowchart for describing a speech processing method according to an embodiment of the present disclosure; Fig. Figure 3 is a diagram illustrating components included in a processor in a speech processing device according to an embodiment of the present disclosure; Fig. Figure 4 is a flowchart for describing the training of a model in a speech processing device according to an embodiment of the present disclosure; Fig. Figure 5 is a diagram illustrating a method for processing a target embedding in a speech processing device according to an embodiment of the present disclosure; Fig. Figure 6 is a flowchart describing a method for determining response text in a speech processing device according to an embodiment of the present disclosure; Fig. Figure 7 is a flowchart describing a method for converting response text into speech and outputting the speech in a speech processing device according to an embodiment of the present disclosure; and Fig. Figure 8 is a diagram illustrating a computing system relating to a speech processing device or speech processing method according to one embodiment of the present disclosure.

[0029] With regard to the description of the drawings, the same or similar components are identified by the same or similar reference numerals. DETAILED DESCRIPTION OF THE ILLUSTRATIVE EXECUTION FORMS

[0030] In the following, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. When adding reference numerals to components in each drawing, it should be noted that the same components should have the same reference numerals even if they are shown on a different drawing. Furthermore, when describing embodiments of the present disclosure, detailed descriptions associated with known functions or configurations are omitted if they might unnecessarily obscure the subject matter of the present disclosure. Various embodiments of the present disclosure can be described below with reference to the accompanying drawings.Accordingly, the person skilled in the art will recognize that modifications, equivalents, and / or alternatives to the various embodiments described herein can be made in various ways without deviating from the scope and basic concept of this disclosure. With regard to the description of the drawings, similar components may be identified by similar reference numerals.

[0031] In describing elements of an embodiment of the present disclosure, the terms first, second, A, B, (a), (b), and the like may be used here. These terms are used only to distinguish one element from another, but do not limit the corresponding elements regardless of their nature, order, or priority. Furthermore, unless otherwise defined, all terms used herein, including technical and scientific terms, are to be interpreted in the manner customary in the field to which the present disclosure belongs. It is understood that terms used herein are to be interpreted in a way that includes a meaning consistent with their meaning in the context of the present disclosure and the relevant field, and are not to be interpreted in an idealized or overly formal sense unless expressly defined herein.For example, terms used here, such as "first," "second," and the like, can refer to different elements of different embodiments of the present disclosure, but do not limit the elements. For example, "a first user feature" and "a second user feature" can denote different user features regardless of their order or priority. For example, without departing from the scope of the present disclosure, a first feature can be referred to as a second feature, and likewise, a second feature can be referred to as a first feature.

[0032] In this description, the terms “possess”, “can possess”, “include” and “comprise” or “can include” and “can encompass” indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations, or components), but do not exclude the presence of additional features.

[0033] It is understood that when an element (e.g., a first element) is described as "(operationally or communicatively) coupled with" or "connected to" another element (e.g., a second element), it may be directly coupled or connected to the other element, or an intervening element (e.g., a third element) may be present. Conversely, when an element (e.g., a first element) is described as "directly coupled with" or "directly connected to" another element (e.g., a second element), it is understood that there is no intervening element (e.g., a third element).

[0034] Depending on the situation, the expression “configured to” used here can, for example, be used as the expression “suitable for”, “with the capability to”, “designed to”, “adapted to”, “made to” or “capable of”.

[0035] The term “configured to” is not intended to mean only “specifically designed to” in hardware. Instead, the term “a device that is configured to” can mean that the device is “capable” of working together with another device or other components. For example, a “processor that is configured to (or set to) perform A, B, and C” can mean a dedicated processor (e.g., an embedded processor) for performing a corresponding operation, or a general-purpose processor (e.g., a CPU or application processor) that performs corresponding operations by executing one or more software programs stored in a memory device. The terms used in the description are employed only to describe a specific embodiment and are not intended to limit the scope of this disclosure.The terms of a singular form may include plural forms unless otherwise specified. All terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person skilled in the art. It is further understood that terms defined in a dictionary and in common use are to be interpreted in the manner customary in the relevant field and not in an idealized or overly formal sense, unless expressly defined herein in different embodiments of the present disclosure. In some cases, terms, although defined in the description, may not be interpreted in such a way as to exclude embodiments of the present disclosure.

[0036] In the present disclosure, the expressions “A or B”, “at least one of A and / or B”, or “one or more of A and / or B”, and the like, used herein, can include any and all combinations of one or more of the associated listed elements. For example, the term “A or B”, “at least one of A and B”, or “at least one of A or B” can refer to all of the following: (1) case, in which at least one A is included; (2) case, in which at least one B is included; or (3) case, in which both at least one A and at least one B are included.Furthermore, when describing a component of an embodiment of the present disclosure, the expressions "at least one of A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", or "at least one of A, B or C or any combination thereof" can include any and all combinations of one or more of the associated listed elements. In particular, the expressions "at least one of A, B or C or any combination thereof" can include A, B or C or any combination thereof, such as AB, ABC, or the like.

[0037] In the following, embodiments of the present disclosure are described with reference to the Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7 to Fig. 8 described in detail.

[0038] Fig. Figure 1 is a block diagram illustrating a speech processing device according to an embodiment of the present disclosure.

[0039] A speech processing device 100 according to one embodiment may include a processor 110, a memory 120 containing instructions 122, and a communication device 130.

[0040] The speech processing device 100 can specify a device that applies an embedding with respect to a language to a transformer-based language model to generate a sentence.

[0041] For example, the speech processing device 100 can convert features of an input speech signal into an embedding, which is a numerical vector, using a speech processing model. The speech processing device 100 can then use this embedding, which contains key information from the speech signal, for text generation and training.

[0042] For example, the speech processing device 100 can generate response text corresponding to speech data by inputting the embedding into a first decoder trained using an attention mechanism. The speech processing device 100 can predict text by reflecting the context of the input speech through a first decoder.

[0043] For example, the speech processing device 100 can determine whether a response text generation process ends by inputting an embedding and a context vector into a second decoder. The speech processing device 100 can determine the termination of a conversation by comparing a termination probability with a predetermined reference value.

[0044] For example, the Speech Processing Device 100 can generate signal-to-noise ratio (SNR)-based training data by mixing noise data and speech data. The Speech Processing Device 100 can train a model that operates robustly even in noisy environments.

[0045] For example, the Speech Processing Device 100 can train a speech processing model based on a loss value calculated by each of a text decoder and a speech activity decoder. The Speech Processing Device 100 can optimize the performance of the speech processing model by combining the losses of the text decoder and the speech activity decoder.

[0046] For example, the Speech Processing Device 100 can convert the generated text response into speech using text-to-speech (TTS) technology. In this way, the Speech Processing Device 100 can provide a user with a natural speech response.

[0047] For example, the Speech Processing Device 100 can analyze the user's utterance intention by applying response text to an utterance intention prediction model. The Speech Processing Device 100 can then perform necessary operations by generating action data based on the analyzed utterance intention in devices such as robots.

[0048] For example, the Speech Processing Device 100 can instantly generate a response without converting the speech signal to text by processing it in real time. In this way, the Speech Processing Device 100 can maintain accuracy even in various noisy environments and can provide high flexibility and real-time information to determine when a conversation is ending.

[0049] Processor 110 can execute software and can control at least one other component (e.g., a hardware or software component) connected to Processor 110. Processor 110 can also perform various data processing operations. For example, Processor 110 can store target data, target embeddings, or response text in Memory 120. For reference, Processor 110 can perform all the operations performed by Speech Processing Device 100. Therefore, for ease of description in this document, an operation or process performed by Speech Processing Device 100 will primarily be described as an operation or process performed by Processor 110.

[0050] Furthermore, for the sake of clarity, processor 110 is primarily described as a single processor in this description, but is not limited to this. For example, the speech processing device could include 100 processors. Each of the processors can perform all operations related to generating a sentence by applying language embedding to a transformer-based language model.

[0051] Memory 120 can temporarily and / or permanently store various data elements and / or information required to generate sentences by applying language embedding to the Transformer-based language model. For example, memory 120 can store target data, target embeddings, or response text.

[0052] The communication device 130 can support communication between the speech processing device 100 and a server 140. For example, the communication device 130 can include one or more components for communication between the speech processing device 100 and the server 140. For example, the communication device 130 can include a wireless short-range communication device, a microphone, or the like. In this case, short-range communication technologies include, but are not limited to, wireless LAN (Wi-Fi), Bluetooth, ZigBee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), Bluetooth Low Energy (BLE), and near-field communication (NFC).

[0053] Fig. Figure 2 is a flowchart for describing a language processing method according to an embodiment of the present disclosure.

[0054] According to one embodiment, a processor (e.g., the 110 processor from) can be installed in S210. Fig. 1) Determine a target embedding that includes features of target data by applying the target data with respect to a language to a language processing model that is trained to output features of a signal as numbers.

[0055] For example, the target data can be data relating to a language and can refer to an inputted speech signal or data related to the inputted speech signal. The target data can include data extracted from a speech signal input by a user, the signal itself, a mixed signal containing noise, or physical properties of the speech (e.g., time-domain information or frequency-domain information). The target data can be used as an input to the speech processing model and can be used to digitize the main features of the speech signal.

[0056] For example, the natural language processing model can refer to a machine learning (or deep learning) model trained to output features of a signal as numbers. The natural language processing model can process the input speech data and convert key information from the speech into a form (i.e., embedding) expressed numerically. The natural language processing model can be based on various neural network structures, such as convolution, transformers, or RNNs. The natural language processing model can be designed to extract robust features by using noise data and SNR mixture data during a training process. The natural language processing model can generate embedding data required for subsequent processing (e.g., text generation by a decoder) by extracting important features (e.g., feature vectors) from the input speech data.

[0057] For example, the target embedding can be the output of the speech processing model and can represent data in the form of a vector derived by digitizing the main features of the input speech data. The target embedding can be expressed by compressing the context, time, and frequency information of speech data. The target embedding can be in the form of a multidimensional vector that can be used in the subsequent decoder (a first decoder or a second decoder). The target embedding can contain essential information from speech data without text conversion. The target embedding can be an intermediate representation that specifies properties of the input speech signal and can provide task-based data, such as generating response text and predicting speech activity.

[0058] In S220, the processor can determine a context vector and response text, estimated as the text of the target data, by applying target embedding to the first decoder trained to output the probability of the text based on the attention mechanism.

[0059] For example, the first decoder can be a decoder trained on the attention mechanism and can generate a probability distribution for the response text by receiving the target embedding and the previous response text as inputs. In other words, the first decoder can be a component that generates the response text by interpreting the input embedding data. The first decoder consists of multiple stages (layers) and can compute the interaction between the input embedding and context information using the attention mechanism. Specifically, the first decoder can model the relationship between the embedding and the previous response text by using trainable weights.The first decoder can mean a decoder that outputs a probability distribution to generate text that matches the target data (speech data), selects the next text token by softmax, and generates the entire response text by repeating the operations.

[0060] For example, the response text can be the text generated by the first decoder (i.e., the output decoder) and can indicate results in a character format that corresponds to the target data. The response text can consist of a word (or token) with the highest value in the probability distribution output by the first decoder. The first decoder operates in a process of sequentially generating words and predicting the next word based on the previous one. As a result of reflecting the user's speech input, the generated text can eventually be delivered to the user. In detail, if the target data of "What is the weather?" is entered by the user, a processor can generate response text such as "The weather today is clear" to provide information that corresponds to the target data. The response text can be used for an operation (e.g.,Text-to-speech conversion, utterance intent analysis, or similar functions of the processor may be used.

[0061] For example, the context vector, generated by the first decoder, can represent a vector that numerically expresses the correlation between the input data (target embedding) and the previous response text. The context vector can be generated by the attentional mechanism and can be expressed by integrating key information from the input embedding and the previous response text. The context vector can be used in all layers of the first decoder and can contain information necessary to predict the next word. The context vector can be calculated based on an attentional score and a trainable weight.

[0062] In S230, the processor can determine whether an operation to determine the response text is complete by applying the target embedding and the context vector to the second decoder, which is trained to determine whether the embedding is complete.

[0063] For example, the second decoder can specify a decoder trained to determine whether a response text generation operation terminates, taking the target embedding and context vector as inputs. The second decoder can output a probability indicating whether the operation terminates. For example, an output of 0.85 could mean that the response text generation operation terminates with an 85% probability. The second decoder is trained using a binary cross-entropy loss based on a difference from the actual termination state (i.e., a ground-truth label) in a training process.

[0064] In S240, the processor can output the response text based on whether the response text and target embedding are complete. For example, the processor can output the response text to a screen via a display or a GUI interface. The processor can convert the generated response text into speech using text-to-speech (TTS) technology and output the speech to the user.

[0065] Fig. Figure 3 is a diagram illustrating components included in a processor in a speech processing device according to an embodiment of the present disclosure.

[0066] Fig. Figure 3 illustrates a structural block diagram showing a configuration of a speech processing device (e.g., the speech processing device 100 of Fig. 1) and shows the interactions between modules. The speech processing device can receive a speech signal as input, can generate response text or train speech embedding data, and can perform various speech processing tasks.

[0067] For example, a signal receiver module 310 can receive speech data from an external source, convert the speech data into digital data, and deliver the digital data to a speech processing model 313. That is, the signal receiver module 310 can be responsible for collecting and preprocessing the speech data.

[0068] For example, a Signal Receiver 311 can receive a speech signal uttered, i.e., spoken, by a user in an external environment. The Signal Receiver 311 can convert the received speech signal into digital data in a form capable of analyzing the signal's characteristics.

[0069] For example, the speech processing model 313 can convert the input speech signal into embedding data (i.e., a target embedding) obtained by digitizing key features through analysis of the input speech signal. The speech processing model 313 can generate embedding data expressed in vector form by compressing time and frequency features of speech data.

[0070] For example, a response generation module 320 can generate response text to be delivered to a user, based on the embedding data received by the natural language processing model 313. The response generation module 320 can convert the generated response text into speech and output the speech to the user.

[0071] For example, a response text generator 321 can receive the embedding data and generate a response in text format. The response text generator 321 can contextually analyze the input embedding data based on an attention mechanism, predict the next word, and generate a final text (i.e., a response text).

[0072] For example, a 323 text-to-speech (TTS) converter can convert the generated text response into speech and deliver the speech to the user. Specifically, the 323 TTS converter can output natural language using TTS technology.

[0073] For example, an embedding training module 330 can provide a function necessary to train the language processing model 313. Specifically, the embedding training module 330 can convert the input language data into an embedding, train the embedding, and optimize its performance.

[0074] For example, a text decoder 331 can specify a decoder that predicts text based on the embedding by using the embedding data generated by the language processing model 313 as an input. The text decoder 331 can perform training by calculating a loss based on a difference between an actual ground-truth text and text predicted during training.

[0075] For example, a speech activity decoder 333 can refer to a decoder that determines whether data contains speech (activity) by using the embedding data of a speech processing model as an input. The speech activity decoder 333 can be used to determine a segment in which a speech signal is present or to determine whether a conversational operation has ended. The speech activity decoder 333 can be trained to determine whether the conversational operation has ended by using a binary cross-entropy loss.

[0076] For example, the signal receiving module 310 can collect speech data that is entered from outside, can supply the speech data to the speech processing model 313, and can generate embedding data that is determined by digitizing key features.

[0077] For example, the Response Generation Module 320 can generate response text based on the embedding data generated by the Signal Receiving Module 310. The Response Generation Module 320 can generate text suitable for information or services requested by the user and can then convert the generated text into speech for output to the user.

[0078] For example, the embedding training module 330 can receive the embedding data generated by the signal receiving module 310 and can perform training and optimization of the speech processing model 313. In particular, the embedding training module 330 can train the text decoder 331 and the speech activity decoder 333 to perform a text generation and speech presence / absence detection task.

[0079] Fig. Figure 4 is a flowchart for describing the training of a model in a speech processing device according to an embodiment of the present disclosure.

[0080] Fig. Figure 4 is a flowchart depicting a training and response generation process of a speech processing device (e.g., the speech processing device 100 of Fig. 1) shows. Each of the in Fig. The 4 illustrated operations (processes) can be described as a process in which a processor trains a language processing model and a response generation model, or outputs text based on generated data.

[0081] In S405, the processor (e.g., the 110 processor from Fig. 1) Identify a training target model according to one embodiment. For example, the training target model may include the language processing model and the response generation model.

[0082] The response generation model can specify a model designed to generate response text to be delivered to a user, or to determine, based on speech or embedding data, whether response generation has ended. The response generation model can generate a response by including a first decoder (i.e., a text decoder) and a second decoder (i.e., a conversation-ending decoder).

[0083] The first decoder can compute a probability distribution of the response text by using the embedding data (i.e., target embedding) as an input, and can generate the response text. The first decoder can generate a text response that corresponds to the input speech data through target embedding. The second decoder can determine whether a response text generation operation has finished. The second decoder can compute a completion probability by using the target embedding and the context vector as inputs, and can use this completion probability to determine whether the response text generation operation is complete.

[0084] In the S410, the processor can generate noise data. For example, the processor can generate noise data to be mixed with speech data, and can generate noise data based on various noise conditions (e.g., café noise, keyboard noise, parking lot noise, or the like) that may occur in real-world environments.

[0085] In S415, the processor can identify input text. For example, the processor can select input text data to use for training.

[0086] In the S420, the processor can generate voice actor recording data and speech synthesis data. For example, the processor can generate speech data from voice actor speech data recorded by a user and a speech synthesis model, and can use the speech data as training data.

[0087] In S425, the processor can generate speech data. For example, the processor can generate speech data that corresponds to text data and can use the speech data as training-specific speech data.

[0088] In the S430, the processor can perform signal-to-noise ratio (SNR) mixing. For example, the processor can mix the generated speech data and the noise data using SNR mixing. The processor can perform SNR mixing to train a model that is robust against various acoustic conditions that can occur in real-world environments.

[0089] In S435, the processor can determine target data. For example, the processor can determine that data generated as a result of SNR mixing is the target data, which serves as a training target. The target data can specify data that includes the main characteristics of a speech signal.

[0090] In S440, the processor can generate voice presence / absence data. At any given point in time, the processor can generate voice presence / absence data indicating whether a language is present at that particular time.

[0091] In S445, the processor can generate conversation termination data. For example, the processor can generate conversation termination data that indicates whether a conversation ends and can use this data during training.

[0092] In S450, the processor can perform embedding training. For example, the processor can train a natural language processing model that generates an embedding based on the trained data by using a text decoder and a speech activity decoder.

[0093] The processor can determine training noise data based on at least one of first sub-noise data determined in the target space, or second sub-noise data generated based on a standard normal distribution, or any combination thereof.

[0094] For example, the target space can represent an environment where source data is collected, which is needed to gather or synthesize training data. The target space can include speech and noise data collected in a real-world environment.

[0095] For example, the initial sub-noise data can refer to actual ambient noise data collected in the target space. This initial sub-noise data can include noise recorded by an actual microphone or physical signals from the actual environment, such as keyboard sounds, wind noise, or vehicle noise.

[0096] For example, the second sub-noise data can refer to artificial noise data synthesized based on a mathematical procedure (e.g., a standard normal distribution). This second sub-noise data can be generated to simulate a noise signal from a real-world environment.

[0097] For example, training noise data can represent training noise generated by combining the first and second sub-noise data. The training noise data can be designed to maximize diversity by combining real and synthetic noise data, and to enable the speech processing model to operate robustly in various noise environments.

[0098] The processor can determine training speech data based on at least one of first subspeech data recorded by a user, or second subspeech data generated based on a speech synthesis model, or any combination thereof.

[0099] For example, the first sublanguage data may refer to actual language data that is actually recorded by the user, and may include raw language data collected based on a person's utterance, natural utterance patterns collected based on a person's utterance, and features of a language.

[0100] For example, the second sublanguage data can specify synthetic speech data generated by a speech synthesis model, can be generated by using deep-learning-based speech synthesis technology (TTS), can include acoustic properties similar to actual speech, and can express various utterance patterns, intonation, and pitch.

[0101] For example, the training language data can represent training-specific language data that is generated independently or by combining the first sublanguage data and the second sublanguage data.

[0102] The processor can generate training target data by SNR mixing training noise data and training speech data.

[0103] The SNR mixing can be expressed based on Equation 1 below. SNR=PsignalPnoise=E[S2]E[N2]

[0104] The processor can determine a training target embedding by applying training target data to the language processing model.

[0105] The processor can determine an initial temporary output by applying the training target embedding to the text decoder.

[0106] For example, the text decoder can specify a decoder that generates response text based on the appropriate embedding by using the training target embedding as an input. The text decoder can analyze embedding data and context information using an attention mechanism, can calculate a probability distribution of the next word, and can sequentially generate text based on this probability distribution.

[0107] The output of the text decoder can be expressed based on Equation 2 and Equation 3 below. yu=P(zu,t|x,t,y1….u−1)=softmax(Linear(tanh(Joint))) joint=Linear(x)+LSTM(y1…u−1)

[0108] The processor can determine a second temporary output by applying the training target embedding to the speech activity decoder.

[0109] For example, the speech activity decoder can specify a decoder that determines whether the embedding contains speech by taking the training target embedding as an input. The speech activity decoder can consist of multiple linear layers and can output the presence / absence of speech as a probability value between 0 and 1.

[0110] The output of the speech activity decoder can be expressed based on equation 4 below. P(actt|xt)=Linear(xt)

[0111] For example, the processor can train the speech processing model through a first loss based on the first temporary output and the text decoder, and a second loss based on the second temporary output and the speech activity decoder.

[0112] The first loss can be expressed based on the equation 5 below. LT=−∑i log P(yi|xi)

[0113] The second loss can be expressed based on the equation 6 below. LA=BinaryCrossEntropyLoss(P(actt|xt),actt)

[0114] Here, [act] _t can denote voice presence / absence data.

[0115] The loss required to handle the language processing model can be expressed based on Equation 7 below. L=LT+LA

[0116] In S455, the processor can identify input text. The processor can identify input text data to be used for training.

[0117] In S460, the processor can determine whether input text contains an utterance intent. The processor can then generate response text according to this intent by determining whether the utterance intent is present in the input text. Alternatively, if no utterance intent is present, the processor can identify the response text from a separate database and / or a large-language model.

[0118] In S465, the processor can generate response text based on an utterance intention obtained from a conversational language model or the large language model.

[0119] In S470, the processor can obtain response text from the database, where other response text is stored in the utterance intent.

[0120] The response text generated in operations S455 to S470 can represent ground truth data required for training the response generation model.

[0121] In S475, the processor can train a response text generator based on the generated text data.

[0122] For example, if a time point at which the training target embedding is applied to the first decoder is a target time point, the processor can identify an initial training response text that is determined by the first decoder at a time point preceding the target time point.

[0123] For example, the target time point can refer to a specific point in time at which the training target embedding is applied to the first decoder. The target time point is a reference point in each stage where the response text is generated and refers to a time step at which the first decoder produces text by processing the input data.

[0124] For example, the first training response text can specify the text generated by the first decoder before the target time. This first response text can then be used as key data to establish the context for the text to be generated after the target time.

[0125] For example, the processor can determine a training context vector and a second training response text by applying the training target embedding and the first training response text to the first decoder.

[0126] For example, the second training response text can specify the text generated by the first decoder after the target time. The second training response text can be generated based on the training target embedding and the first training response text input at the target time.

[0127] For example, the second training response text can be applied to a training input from the first decoder at a time following the target time.

[0128] For example, the processor can determine the completion probability of whether the acquisition of the second response text has finished by applying the training goal embedding and the training context vector to the second decoder. The processor can then determine whether to terminate an operation to acquire the second response text based on a comparison between the completion probability and a predetermined value.

[0129] For example, based on the fact that the first decoder and the second decoder are included in the response generation model, the processor can determine the loss of the first decoder based on the first decoder and the second training response text, and the loss of the second decoder based on the second decoder and the termination probability.

[0130] For example, the processor can train a response generation model based on the loss of the first decoder and the loss of the second decoder.

[0131] For example, the loss of the first decoder could mean the loss indicating a difference between a ground truth text and the text output value generated by the first decoder. The loss of the second decoder could mean the loss indicating a difference between the termination probability output by the second decoder and whether an actual termination occurs (ground truth label).

[0132] For example, the processor can determine a temporary output by applying the target embedding and context vector to the second decoder. The processor can then determine whether to terminate the response text retrieval operation based on a comparison between the temporary output and a predetermined value.

[0133] For example, the response generation model is based on a transformer structure and consists of a first decoder and a second decoder. The response generation model can be trained to generate text and determine conversation termination through deep learning-based training and can provide a response that meets a user's request through real-time inference.

[0134] For example, the training process of a response generation model may include a process for generating input data and ground-truth data. Specifically, the input data may include embedding data (i.e., a training target embedding) generated by the language processing model and an initial training response text (i.e., text generated before the target time).

[0135] For example, training target embedding can be applied to be delivered to the first decoder of the response generation model to analyze a relationship between elements of input data and generate an appropriate response.

[0136] For example, the processor can generate the second training response text based on the training target embedding and the first training response text. Specifically, the processor can generate the second training response text by contextually analyzing input data using the attention mechanism of the response generation model. The processor can compute the cross-entropy loss between the generated response text (i.e., the second training response text) and the ground truth text and update the training weight of the first decoder with this loss value.

[0137] For example, the processor can output the termination probability by applying the training target embedding and context vector to the second decoder. The processor can calculate the binary cross-entropy loss based on the difference between the termination probability and the actual termination (i.e., ground-truth data is a label).

[0138] For example, the processor can obtain a final loss by adding the loss of the first decoder and the loss of the second decoder. The processor can update the weight of the response generation model by using an optimization algorithm (e.g., Adam or AdamW).

[0139] Fig. Figure 5 is a diagram illustrating a method for processing a target embedding in a speech processing device according to an embodiment of the present disclosure.

[0140] Fig. Figure 5 is a diagram showing interactions between main components and a data processing flow of a speech processing device (e.g., the speech processing device 100 of Fig. 1) according to one embodiment.

[0141] Target data consists of an externally inputted speech signal and includes a speech signal captured in an actual environment or synthesized speech data. The target data can be applied to a speech processing model 313.

[0142] The natural language processing model 313 transforms the input target data into a digitized embedding. The natural language processing model 313 is based on neural network structures such as Transformers, RNNs, and Convolutional Neural Networks (CNNs).

[0143] The embedding training module 330 refers to a module that receives embedding data supplied by the language processing model 313 and performs text generation and speech presence / absence tasks. The embedding training module 330 consists of two decoders (e.g., a text decoder and a speech activity decoder).

[0144] A Text Decoder 331 generates response text based on the input embedding data. For example, if the input speech signal is "Hello", the Text Decoder 331 can generate a response text of "Hello".

[0145] A Speech Activity Decoder 333 determines speech presence / absence data based on the input embedding data. The Speech Activity Decoder 333 outputs a binary value (e.g., 0 or 1) indicating whether speech is present at any given time.

[0146] Fig. Figure 6 is a flowchart describing a method for determining response text in a speech processing device according to an embodiment of the present disclosure.

[0147] In S610, a processor according to one embodiment (e.g., the 110 processor from Fig. 1) Identify a target embedding. The processor can identify the target embedding generated by a speech processing model from target data (i.e., a speech signal).

[0148] In S620, the processor can apply target embedding to a first decoder. The processor can calculate a probability distribution for generating a text response by the first decoder and can determine whether a conversation will be terminated in the subsequent stage.

[0149] In S630, the processor can determine whether the conversation is terminated by a second decoder. The processor inputs the target embedding and a context vector generated by the first decoder into the second decoder. The processor can calculate a conversation termination probability based on the input data from the second decoder and determine whether the conversation is terminated (i.e., 0 or 1) based on this probability.

[0150] In S640, the processor can determine whether the conversation is complete. If the termination status is "1", the processor can determine that the conversation is complete based on a conversation termination value received from the second decoder and can terminate the process. Conversely, if the termination status is "0", the processor can determine that the conversation is still ongoing and can proceed to the next operation.

[0151] In S650, the processor can receive a response text from the first decoder. The processor can generate the response text using the first decoder. The first decoder can complete the response text by sequentially generating the next word based on the target embedding and previously generated text data (i.e., response text generated before the target embedding is applied to the first decoder).

[0152] Fig. Figure 7 is a flowchart describing a method for converting response text into speech and outputting the speech in a speech processing device according to an embodiment of the present disclosure.

[0153] According to one embodiment, a processor (e.g., the 110 processor from) can be installed in the S710. Fig. 1) Receive a signal that includes noise, silence, and speech. The input signal can be received by a collection device, such as a microphone, and can be gathered in various noisy environments. For example, the processor can receive speech data, such as "hello," along with ambient noise (e.g., keyboard sounds, vehicle noise, or the like).

[0154] In the S720, the processor can determine a target embedding based on a speech processing model. For example, the speech signal of "hello" can be converted into embedding data (i.e., a target embedding) in the form of a feature vector.

[0155] In the S730, the processor can determine a response text based on the target embedding. For example, if the input signal is "Hello", the processor can generate the response text, such as "Hello, how can I help you?", using a response generation model.

[0156] In S740, the processor can convert the response text into speech and output the speech. Additionally, the processor can determine the utterance intent of the response text by applying it to an utterance intent prediction model based on the response text's determination. The processor can determine action data by applying the utterance intent to an action database according to the utterance intent. The processor can then transmit the action data to a robot equipped with a speech processing device (e.g., the Speech Processing Device 100 of S740). Fig. 1) is connected.

[0157] Fig. Figure 8 is a diagram illustrating a computing system relating to a speech processing device or speech processing method according to one embodiment of the present disclosure.

[0158] With reference to Fig.8. A computing system 1000 relating to a speech processing device or speech processing method may include at least one processor 1100, one memory 1300, one user interface input device 1400, one user interface output device 1500, one memory 1600 and one network interface 1700, which are interconnected via a bus 1200.

[0159] The 1100 processor can be a CPU or a semiconductor device that processes instructions stored in the 1300 memory and / or the 1600 memory. Each of the 1300 and 1600 memory modules can contain different types of volatile or non-volatile storage media. For example, the 1300 memory module can contain both read-only memory (ROM) and random-access memory (RAM).

[0160] Accordingly, the operations (processes or steps) of the method or algorithm described in connection with the embodiments disclosed in the description can be implemented directly with a hardware module, a software module, or a combination of the hardware module and the software module, executed by the processor 1100. The software module can be located on a storage medium (i.e., the memory 1300 and / or the memory 1600), such as random-access memory (RAM), flash memory, read-only memory (ROM), erasable and programmable ROM (EPROM), electrical EPROM (EEPROM), a register, a hard disk drive, a removable disk, or a compact disc ROM (CD-ROM).

[0161] The storage medium can be coupled to the 1100 processor. The 1100 processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated with the 1100 processor. The processor and the storage medium can be implemented using an application-specific integrated circuit (ASIC). The ASIC can be provided in a user terminal. Alternatively, the processor and the storage medium can be implemented with separate components in the user terminal.

[0162] The above description is merely an example of the technical idea of ​​the present disclosure, and various modifications and alterations can be made by a person skilled in the art without departing from the essential features of the present disclosure.

[0163] The embodiments described above can be implemented using hardware elements, software elements, and / or a combination of both. For example, the devices, methods, and components described in embodiments of this disclosure can be implemented using general-purpose or specialized computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any device capable of executing instructions and responding. A processing device can execute an operating system (OS) or a software application running on the OS.Furthermore, the processing unit can access, store, manipulate, process, and generate data in response to the execution of software. The person skilled in the art will understand that, although a single processing unit may be illustrated for clarity, the processing unit can include a multitude of processing elements and / or a multitude of processing element types. For example, the processing unit may include a multitude of processors or a processor and a controller. Additionally, the processing unit may include a different processing configuration, such as a parallel processor.

[0164] Software can comprise computer programs, code, instructions, or one or more combinations thereof, and configure a processing device to operate in a desired manner or to control the processing device independently or collectively. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical equipment, virtual equipment, computer storage media or units, or transmitted signal waves to be interpreted by the processing device or to provide instructions or data to the processing device. Software can be distributed across computer systems connected by networks and stored or executed in a distributed manner. Software and data can be recorded on a computer-readable storage medium.

[0165] The methods according to the embodiments described above can be recorded on a computer-readable medium containing program instructions that can be executed by various computer devices. The computer-readable medium can also contain program instructions, data files, data structures, and the like, individually or in combination. The program instructions recorded on the medium can be specifically designed and configured for the embodiments of the present disclosure or can be known and available to those skilled in the art of computer software. The computer-readable medium can include hardware devices specifically configured to store and execute program instructions, such as magnetic media (e.g., a hard disk, a floppy disk, or a magnetic tape), optical recording media (e.g., CD-ROM and DVD), magneto-optical media (e.g., optical fibers), or optical optical fibers.Optical disks), read-only memory (ROMs), random access memory (RAMs), and flash memory. Examples of computer programs include not only machine language code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or similar device.

[0166] The hardware setup described above can be configured to operate as one or more software modules to perform the operations of the embodiments described above in the present disclosure, or vice versa.

[0167] Even though the embodiments are described with reference to limited drawings, it may be obvious to a person skilled in the art that the embodiments can be modified or altered in various ways based on the above description. For example, adequate effects can be achieved even if the preceding processes and procedures are carried out in a different order than described above and / or the elements mentioned above, such as systems, structures, devices, or circuits, are combined or coupled in different forms and modes than described above, or are replaced or exchanged by other components or equivalents.

[0168] Therefore, other implementations, other embodiments and equivalents to claims are within the scope of the following claims.

[0169] Accordingly, embodiments of the present disclosure are not intended to limit, but rather to illustrate, the technical idea of ​​the present disclosure, and the scope and fundamental concept of the present disclosure are not limited by the embodiments described above. The scope of protection of the present disclosure should be interpreted by the attached claims, and all equivalents thereof should be interpreted as being included within the scope of the present disclosure.

[0170] Descriptions of a speech processing device according to an embodiment of the present disclosure and of a method for it are as follows.

[0171] According to at least one embodiment of the present disclosure, a speech processing device can perform real-time processing without a text conversion process by using embeddings that are extracted directly from a speech signal.

[0172] Furthermore, according to at least one embodiment of the present disclosure, a speech processing device can simultaneously perform text generation and speech presence / absence determination by a single embedding by converting a speech signal into an embedding form and simultaneously extracting text and status information based on the converted result by using a text decoder and a speech activity decoder.

[0173] Furthermore, a variety of effects, understood directly or indirectly through the present disclosure, can be provided.

[0174] Although the present disclosure has been described above with reference to exemplary embodiments and the accompanying drawings, the present disclosure is not limited thereto, but can be modified and altered in various ways by the person skilled in the art to whom the present disclosure relates, without deviating from the basic idea and scope of the present disclosure claimed in the following claims. 330 Embedding training module 1000 computing system

Claims

A speech processing device comprising: a memory configured to store a computer-executable instruction; and a processor configured to execute the instruction by accessing the memory, wherein the processor is configured to: determine a target embedding containing a feature of target data by applying the target data associated with a language to a speech processing model trained to output a feature of a signal as a number; determine a context vector and response text estimated as text of the target data by applying the target embedding to a first decoder trained to output a probability of text based on an attention mechanism;Determine whether a response text retrieval process is complete by applying the target embedding and context vector to a second decoder trained to determine if an embedding is complete; and output the response text based on whether the response text and target embedding are complete. Speech processing device according to claim 1, wherein the processor is configured to: determine training noise data based on at least one of first sub-noise data determined in the target space, or second sub-noise data generated based on a standard normal distribution, or any combination thereof; determine training speech data based on at least one of first sub-speech data recorded by a user, or second sub-speech data generated based on a speech synthesis model, or any combination thereof; generate training target data by signal-to-noise ratio (SNR) mixing of the training noise data and the training speech data; and determine a training target embedding by applying the training target data to the speech processing model. Speech processing device according to claim 2, wherein the processor is configured to: determine a first temporary output by applying the training target embedding to a text decoder; determine a second temporary output by applying the training target embedding to a speech activity decoder; and train the speech processing model by a first loss based on the first temporary output and the text decoder and a second loss based on the second temporary output and the speech activity decoder. Speech processing device according to claim 2, wherein the processor is configured to: identify a first training response text determined by the first decoder at a time preceding a target time, if a time at which the training target embedding is applied to the first decoder is the target time; and determine a training context vector and a second training response text by applying the training target embedding and the first training response text to the first decoder, wherein the second training response text is applied to a training input of the first decoder at a time following the target time. Speech processing device according to claim 4, wherein the processor is configured to: determine a termination probability associated with whether the retrieval of the second training response text is complete, by applying the training target embedding and the training context vector to the second decoder; and determine whether an operation of retrieving the second training response text should be terminated, based on a comparison between the termination probability and a predetermined value. Speech processing device according to claim 5, wherein the processor is configured to: determine a loss of the first decoder based on the first decoder and the second training response text and a loss of the second decoder based on the second decoder and the termination probability, based on a response generation model that includes the first decoder and the second decoder; and train the response generation model based on the loss of the first decoder and the loss of the second decoder. Speech processing device according to claim 1, wherein the processor is configured to: determine a temporary output by applying the target embedding and the context vector to the second decoder; and determine whether to terminate the process of determining the response text based on a comparison between the temporary output and a predetermined value. Speech processing device according to claim 1, wherein the processor is configured to: convert the response text into a speech and output the speech to a user who enters the target data. Speech processing device according to claim 1, wherein the processor is configured to: determine an utterance intent of the response text by applying the response text to an utterance intent prediction model based on receiving the response text; determine action data by applying the utterance intent to an action database according to the utterance intent; and transmit the action data to a robot connected to the speech processing device. A language processing procedure, wherein the procedure comprises: determining a target embedding that includes a feature of target data by applying the target data, which is associated with a language, to a language processing model trained to output a feature of a signal as a number; determining a context vector and response text, which is estimated as text of the target data, by applying the target embedding to a first decoder trained to output a probability of text based on an attention mechanism; determining whether an operation of determining the response text is complete by applying the target embedding and the context vector to a second decoder trained to determine whether an embedding is complete; and outputting the response text based on whether the response text and target embedding are complete. The method of claim 10, wherein outputting the response text comprises: determining training noise data based on at least one of first sub-noise data determined in the target space, or second sub-noise data generated based on a standard normal distribution, or any combination thereof; determining training speech data based on at least one of first sub-speech data recorded by a user, or second sub-speech data generated based on a speech synthesis model, or any combination thereof; generating target training data by SNR mixing of the training noise data and the training speech data; and determining a target training embedding by applying the target training data to the speech processing model. The method of claim 11, wherein the output of the response text comprises: determining a first temporary output by applying the training target embedding to a text decoder; determining a second temporary output by applying the training target embedding to a speech activity decoder; and training the speech processing model by a first loss based on the first temporary output and the text decoder and a second loss based on the second temporary output and the speech activity decoder. The method of claim 11, wherein the output of the response text comprises: identifying a first training response text determined by the first decoder at a time point preceding a target time point, if a time point at which the training target embedding is applied to the first decoder is the target time point; and determining a training context vector and a second training response text by applying the training target embedding and the first training response text to the first decoder, wherein the second training response text is applied to a training input of the first decoder at a time point following the target time point. The method of claim 13, wherein outputting the response text comprises: determining a termination probability associated with whether the determination of the second training response text is complete, by applying the training target embedding and the training context vector to the second decoder; and determining whether an operation of determining the second training response text should be terminated, based on a comparison between the termination probability and a predetermined value. The method of claim 14, wherein the output of the response text comprises: determining a loss of the first decoder based on the first decoder and the second training response text and a loss of the second decoder based on the second decoder and the termination probability, based on a response generation model that includes the first decoder and the second decoder; and training the response generation model based on the loss of the first decoder and the loss of the second decoder. The method of claim 10, wherein the output of the response text comprises: determining a temporary output by applying the target embedding and the context vector to the second decoder; and determining whether the process of determining the response text should be terminated based on a comparison between the temporary output and a predetermined value. Method according to claim 10, wherein outputting the response text comprises: converting the response text into a language and outputting the language to a user who enters the target data. The method of claim 10, wherein the output of the response text comprises: determining an utterance intent of the response text by applying the response text to an utterance intent prediction model based on the determination of the response text; determining action data by applying the utterance intent to an action database according to the utterance intent; and transmitting the action data to a robot connected to a speech processing device. A speech processing device comprising: a processor; a memory storing at least instructions; and a communication device; wherein the instructions, when executed by the processor, cause the speech processing device to: receive, via the communication device, target data comprising speech audio input; process the target data using a speech processing model to generate a target embedding that includes numerical features representing the speech audio input; generate, using a first decoder, a context vector and a response text based on the target embedding, wherein the response text is a text interpretation of the speech audio input; determine, using a second decoder and based on both the target embedding and the context vector, a termination indicator;Stopping the generation of additional response text when the termination indicator meets a termination criterion; and outputting the response text via at least one output mechanism of the speech processing device; wherein the speech processing device processes the speech audio input without requiring a full intermediate text transcription of the speech audio input. Speech processing device according to claim 19, wherein the speech processing device is communicatively coupled to a connected device, and wherein the instructions, when executed by the processor, further cause the speech processing device to: determine an utterance intention by analyzing the response text; retrieve action data corresponding to the utterance intention from an action database; and, based on one or more of the response text or the action data, perform at least one of: displaying the response text on a display device, converting the response text into an audio output to be transmitted by the communication device, or executing a command based at least partially on the response text by sending a control signal via the communication device to the connected device.

Images