Audio processing device and method thereof

Abstract

This invention provides an audio processing device and method that utilizes embedded data directly extracted from audio signals to perform processing in real time without the need for text conversion. [Solution] The system includes a memory and a processor that accesses the memory and executes instructions. The processor applies target data relating to speech to a speech processing model, obtains target embeddings containing features of the target data, applies the target embeddings to a first decoder trained to output text probabilities based on an attention mechanism, obtains response text and context vectors estimated as the text of the target data, applies the target embeddings and context vectors to a second decoder trained to determine whether to terminate the embedding, determines whether to terminate the operation to obtain the response text, and outputs the response text based on whether the response text and target embeddings can be terminated.

Description

Technical Field

[0001] The present invention relates to a voice processing apparatus and method, and more particularly, to a technology for generating a sentence by applying embedding related to voice to a transformer-based language model.

Background Art

[0002] Modern robot technology and interactive artificial intelligence systems greatly rely on voice-text conversion technology in recognizing the user's voice and generating an appropriate response based on it. Such technology operates in a manner that converts a voice signal into text and then utilizes the converted text to generate a sentence or determine the user's utterance intention.

[0003] Time is required in the process of converting a voice signal into text, thus making it difficult to immediately process the user's request in robots and interactive artificial intelligence systems that require real-time processing. Also, it frequently occurs that the text converted from a voice signal is recognized differently from the actual utterance. For example, when the user says "Tell me the price of the Ioniq 5", a situation may occur where it is erroneously recognized as "Tell me the price of Ayun O" in the process of text conversion. Such an error may prevent the user from obtaining the intended result.

[0004] Furthermore, if the EPD technology that cuts and transmits a voice signal ends too quickly, only a part of the voice data may be processed, resulting in inaccurate results. For example, when the user says "Open the door ----", if the EPD ends at "Open the door", a result different from the actual intention of "Open the door" may be output. Such problems are caused by the incompleteness in the text conversion stage and the limitations of the signal processing method, and particularly act as serious constraints in systems where real-time responsiveness and accuracy are important (e.g., robot control, voice secretary, interactive AI). To address these issues, it is necessary to develop a technology that eliminates the text conversion step and directly utilizes embedded data extracted from audio signals in real time. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-003773 [Overview of the project] [Problems that the invention aims to solve]

[0006] Embodiments of the present invention aim to provide an audio processing device and method that utilizes embeddings directly extracted from an audio signal and performs processing in real time without the need for text conversion. Furthermore, embodiments of the present invention aim to provide an audio processing device and method that can simultaneously perform text generation and audio presence / absence determination via a single embedding by converting an audio signal into an embedding format and simultaneously extracting text and state information based on this using a text decoder and an audio activity decoder. The technical problems of the present invention are not limited to those mentioned above, and other technical problems not mentioned should be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0007] An embodiment of the present invention includes a memory storing computer-executable instructions, and a processor that accesses the memory and executes the instructions. The processor applies target data relating to speech to a speech processing model trained to output signal features as numerical values, obtains an embedding containing the features of the target data, applies the embedding to a first decoder trained to output text probabilities based on an attention mechanism, obtains a response text and a context vector estimated as the text of the target data, applies the embedding and the context vector to a second decoder trained to determine whether to terminate the embedding, determines whether to terminate the operation to obtain the response text, and outputs the response text based on whether the response text and the target embedding can be terminated.

[0008] In one embodiment, the processor can acquire training noise data based on at least one of first sub-noise data acquired in the target space, second sub-noise data generated based on a standard normal distribution, or any combination thereof; acquire training voice data based on at least one of first sub-voice data recorded by the user, second sub-voice data generated based on a speech synthesis model, or any combination thereof; generate training target data through SNR (Signal-to-Noise Ratio) mixing of the training noise data and the training voice data; apply the training target data to the speech processing model; and acquire training target embeddings.

[0009] In one embodiment, the processor can apply the training target embedding to a text decoder to obtain a first temporary output, apply the training target embedding to a speech activity decoder to obtain a second temporary output, and perform training of the speech processing model via a first loss based on the first temporary output and the text decoder, and a second loss based on the second temporary output and the speech activity decoder.

[0010] In one embodiment, if the time at which the training target embedding is applied to the first decoder is a target time, the processor identifies a first training response text obtained from the first decoder at the time of transition to the target time, applies the training target embedding and the first training response text to the first decoder, obtains a training context vector and a second training response text, and the second training response text may be applied as a training input to the first decoder at a time subsequent to the target time.

[0011] In one embodiment, the processor can apply the training target embedding and the training context vector to the second decoder, obtain a termination probability regarding whether or not the acquisition of the second training response text can be terminated, and determine whether or not the operation to acquire the second training response text can be terminated based on a comparison of the termination probability with a predetermined value.

[0012] In one embodiment, the processor can obtain, based on the inclusion of the first decoder and the second decoder in the response generation model, the loss of the first decoder based on the first decoder and the second training response text, and the loss of the second decoder based on the second decoder and the termination probability, and can perform training of the response generation model based on the loss of the first decoder and the loss of the second decoder.

[0013] In one embodiment, the processor can obtain a temporary output by applying the target embedding and the context vector to the second decoder, and determine whether to terminate the operation of obtaining the response text based on a comparison of the temporary output with a predetermined value. In one embodiment, the processor can convert the response text into speech and output it to the user who input the target data.

[0014] In one embodiment, the processor can, based on having acquired the response text, apply the response text to a speech intent prediction model to acquire the speech intent of the response text, apply the speech intent to an action database based on the speech intent to acquire action data, and transmit the action data to a robot connected to the speech processing device.

[0015] An embodiment of the present invention provides an audio processing method that includes: applying target data relating to speech to an audio processing model trained to output signal features numerically to obtain an embedding containing the features of the target data; applying the embedding to a first decoder trained to output text probabilities based on an attention mechanism to obtain a response text and context vector estimated as the text of the target data; applying the embedding and context vector to a second decoder trained to determine whether to terminate the embedding to determine whether to terminate the operation of obtaining the response text; and outputting the response text based on whether the response text and the target embedding can be terminated.

[0016] In one embodiment, the operation to output the response text may include: acquiring training noise data based on at least one of first sub-noise data acquired in the target space, second sub-noise data generated based on a standard normal distribution, or any combination thereof; acquiring training voice data based on at least one of first sub-voice data recorded by the user, second sub-voice data generated based on a speech synthesis model, or any combination thereof; generating target training data through SNR (Signal-to-Noise Ratio) mixing of the training noise data and the training voice data; and applying the target training data to the speech processing model to acquire target training embeddings.

[0017] In one embodiment, the operation to output the response text may include the operation of applying the training target embedding to a text decoder to obtain a first temporary output, the operation of applying the training target embedding to a speech activity decoder to obtain a second temporary output, and the operation of training the speech processing model via a first loss based on the first temporary output and the text decoder, and a second loss based on the second temporary output and the speech activity decoder.

[0018] In one embodiment, the operation to output the response text includes, if the time at which the training target embedding is applied to the first decoder is a target time, the operation to identify the first training response text obtained from the first decoder at the time of transition to the target time, and the operation to apply the training target embedding and the first training response text to the first decoder to obtain a training context vector and a second training response text, wherein the second training response text may be applied as a training input to the first decoder at a time subsequent to the target time.

[0019] In one embodiment, the operation of outputting the response text may include applying the training target embedding and the training context vector to the second decoder to obtain an end probability regarding whether to end the acquisition of the training second response text, and determining whether to end the operation of obtaining the training second response text based on a comparison between the end probability and a predetermined value.

[0020] In one embodiment, the operation of outputting the response text may include, based on the first decoder and the second decoder being included in the response generation model, obtaining the loss of the first decoder based on the first decoder and the training second response text, and the loss of the second decoder based on the second decoder and the end probability, and performing learning of the response generation model based on the loss of the first decoder and the loss of the second decoder.

[0021] In one embodiment, the operation of outputting the response text may include applying the target embedding and the context vector to the second decoder to obtain a temporary output, and determining whether to end the operation of obtaining the response text based on a comparison between the temporary output and a predetermined value.

[0022] In one embodiment, the operation of outputting the response text may include converting the response text into voice and outputting it to the user who input the target data.

[0023] In one embodiment, the operation of outputting the response text may include, based on obtaining the response text, applying the response text to a speech intention prediction model to obtain the speech intention of the response text, applying the speech intention to an action database according to the speech intention to obtain action data, and transmitting the action data to a robot connected to a voice processing device.

Effects of the Invention

[0024] Regarding the effects of the voice processing apparatus and its method according to the present invention, it is as follows. According to at least one of the embodiments of the present invention, it is possible to utilize the embedding directly extracted from the voice signal and perform processing in real time without the process of text conversion. Further, according to at least one of the embodiments of the present invention, by converting the voice signal into an embedding form and simultaneously extracting text and state information based on this using a text decoder and a voice activity decoder, it is possible to simultaneously perform text generation and voice presence determination through a single embedding. In addition to this, various effects directly or indirectly grasped through this document are provided.

Brief Description of Drawings

[0025] [Figure 1] It is a diagram illustrating a block diagram of a voice processing apparatus according to an embodiment of the present invention. [Figure 2] It is a flowchart for explaining a voice processing method according to an embodiment of the present invention. [Figure 3] It is a diagram illustrating components included in a processor in a voice processing apparatus according to an embodiment of the present invention. [Figure 4] It is a flowchart for explaining the learning of a model in a voice processing apparatus according to an embodiment of the present invention. [Figure 5] It is a diagram illustrating a method of processing a target embedding in a voice processing apparatus according to an embodiment of the present invention. [Figure 6] It is a flowchart for explaining a method of obtaining a response text in a voice processing apparatus according to an embodiment of the present invention. [Figure 7] It is a flowchart for explaining a method of converting a response text into voice and outputting it in a voice processing apparatus according to an embodiment of the present invention. [Figure 8] This figure illustrates a computer system for a speech processing device or a speech processing method according to one embodiment of the present invention. In relation to the description of the drawing, the same or similar reference numerals may be used for the same or similar components. [Modes for carrying out the invention]

[0026] Hereinafter, some embodiments of the present invention will be described in detail with reference to illustrative drawings. It should be noted that, in assigning reference numerals to the components in each drawing, the same reference numerals will be used for identical components, even if they appear in other drawings, whenever possible. Furthermore, in describing embodiments of the present invention, if a specific description of a related known configuration or function is deemed to interfere with understanding the embodiments of the present invention, such detailed description will be omitted. In particular, various embodiments of this specification will be described with reference to the accompanying drawings. However, this should not be understood as an attempt to limit the technology described herein to any particular embodiment, but rather as including various modifications, equivalents, and / or alternatives to the embodiments described herein. With regard to the description of the drawings, similar reference numerals may be used for similar components.

[0027] In describing the components of embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc., may be used. Such terms are used solely to distinguish a component from other components, and do not limit the nature, order, or sequence of the component in question. Furthermore, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which the present invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and should not be interpreted as ideal or overly formal unless explicitly defined in this application. For example, expressions such as "first," "second," "first," or "second" used herein can modify a variety of components regardless of order and / or importance, and are used solely to distinguish one component from others, without limiting the component in question. For example, the first user device and the second user device refer to different user devices, regardless of order or importance. For example, without exceeding the scope of rights described herein, the first component may be named the second component, and similarly, the second component may be named in place of the first component.

[0028] In this specification, expressions such as “having,” “may have,” “include,” or “may include” indicate the existence of the relevant feature (e.g., numerical values, functions, operations, or components such as parts), and do not exclude the existence of additional features. When it is mentioned that a component (e.g., component 1) is "operally or communicatively coupled with / to" or "connected to" another component (e.g., component 2), it should be understood that the component can be directly coupled to the other component or connected via another component (e.g., component 3). On the other hand, when it is mentioned that a component (e.g., component 1) is "directly coupled" or "directly connected" to another component (e.g., component 2), it may be understood that there is no other component (e.g., component 3) between the component and the other component.

[0029] The expression "configured to" as used herein may be replaced, depending on the context, with other expressions such as "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of."

[0030] The term "configured (or set up) to..." does not necessarily mean only something that is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "device configured to..." can mean that the device "can..." together with other devices or components. For example, the phrase "processor configured (or set up) to perform A, B, and C" can mean a dedicated processor for performing the operations in question (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) that can perform the operations in question by running one or more software programs stored in a memory device. The terms used herein are used solely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions can include plural expressions unless the context clearly indicates otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as those generally understood by a person with ordinary skill in the art described herein.

[0031] Terms used herein that are defined in general dictionaries may be interpreted to have the same or similar meaning as they have in the context of the relevant technology, and not to be interpreted in an ideal or overly formal sense unless explicitly defined herein. Where applicable, even terms defined herein may not be interpreted in a way that excludes the embodiments described herein.

[0032] In this specification, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A and / or B” can include all possible combinations of the items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can refer to all cases where (1) at least one A is included, (2) at least one B is included, or (3) at least one A and at least one B are included. Furthermore, when describing the components of embodiments of the present invention, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” “at least one of A, B, or C,” and “at least one of A, B, C, or any combination thereof” can include any one of the items listed together with the phrase in question, or any possible combination of all of them. In particular, phrases such as "at least one of A, B, C, or any combination thereof" can include A, B, C, or any combination thereof, such as AB or ABC.

[0033] The embodiments of the present invention will be described in detail below with reference to Figures 1 to 8.

[0034] Figure 1 is a block diagram illustrating an audio processing device according to one embodiment of the present invention. An audio processing device 100 according to one embodiment includes a processor 110, a memory 120 containing instruction words 122, and a communication unit 130.

[0035] The speech processing device 100 is a device that generates text by applying speech-related embedding to a language model on a transformer base.

[0036] For example, the speech processing device 100 uses a speech processing model to convert the features of the input speech signal into an embedded vector, which is called an embedding. The speech processing device 100 uses the embedded vector, which contains the main information of the speech signal, for text generation and learning.

[0037] For example, the speech processing device 100 inputs the embedding to a first decoder that has been trained using an attention mechanism, and generates a response text corresponding to the speech data. The speech processing device 100 predicts the text, reflecting the context of the input speech, via the first decoder.

[0038] For example, the speech processing device 100 inputs the embedding and context vector to the second decoder and determines whether to terminate the operation of generating the response text. The speech processing device 100 compares the termination probability with a preset threshold value to determine the termination of the dialogue.

[0039] For example, the speech processing device 100 mixes noise data and speech data to generate SNR (Signal-to-Noise Ratio) based training data. The speech processing device 100 then trains a model that operates well even in noisy environments.

[0040] For example, the speech processing device 100 performs training of a speech processing model based on the loss values ​​calculated by the text decoder and the speech activity decoder, respectively. The speech processing device 100 optimizes the performance of the speech processing model by combining the losses of the text decoder and the speech activity decoder.

[0041] For example, the speech processing device 100 converts the generated text response into speech using text-to-speech (TTS) technology. Through this, the speech processing device 100 provides the user with a natural-sounding speech response.

[0042] For example, the voice processing device 100 applies the response text to a speech intent prediction model to analyze the user's speech intent. Based on the analyzed speech intent, the voice processing device 100 generates behavioral data in a device such as a robot and performs the necessary actions.

[0043] For example, the voice processing device 100 processes the voice signal in real time and generates an immediate response without text conversion. Through this, the voice processing device 100 maintains accuracy even in diverse noisy environments and provides high real-time capability and flexibility in determining whether or not to terminate the conversation.

[0044] The processor 110 executes software and controls at least one other component (e.g., hardware or software component) connected to the processor 110. The processor 110 also performs a variety of other data processing or calculations. For example, the processor 110 stores target data, target embedding, or response text in the memory 120. For reference, the processor 110 performs all the operations performed by the speech processing device 100. Therefore, for the sake of clarity, the operations performed by the speech processing device 100 are primarily described as operations performed by the processor 110.

[0045] Furthermore, for the sake of clarity, this specification primarily describes processor 110 as a single processor, but is not limited thereto. For example, the speech processing device 100 includes multiple processors. Each processor performs all operations related to the operation of applying speech-related embeddings to a language model on a transformer base to generate text.

[0046] Memory 120 temporarily and / or permanently stores various data and / or information required to generate text by applying speech-related embeddings to the language model of the transformer platform. For example, memory 120 stores target data, target embeddings, or response text.

[0047] The communication unit 130 assists in performing communication between the voice processing device 100 and the server 140. For example, the communication unit 130 includes one or more components that enable communication between the voice processing device 100 and the server 140. For example, the communication unit 130 includes a short-range wireless communication unit, a microphone, etc. In this case, short-range communication technologies include, but are not limited to, wireless LAN (Wi-Fi®), Bluetooth®, Zigbee®, WFD (Wi-Fi® Direct), UWB (ultra-wideband), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth® Low Energy), and NFC (Near Field Communication).

[0048] Figure 2 is a flowchart illustrating an audio processing method according to one embodiment of the present invention.

[0049] In one embodiment, the processor (e.g., processor 110 in Figure 1) applies the target data relating to speech to a speech processing model that has been trained to output signal features numerically during operation S210, and obtains a target embedding that includes the features of the target data.

[0050] For example, the target data refers to audio data, meaning an input audio signal or data related thereto. The target data includes data extracted from the audio signal input by the user, the signal itself or a mixed signal containing noise, or the physical characteristics of the audio (e.g., time domain, frequency domain information). The target data may be used as input to an audio processing model and as part of a digitization operation of the main features of the audio signal.

[0051] For example, a speech processing model refers to a machine learning (or deep learning) based model trained to output signal features numerically. The speech processing model processes input speech data and converts the core information of the speech into a numerical form (i.e., embedding). The speech processing model may be designed based on various neural network structures such as convolution, transformer, or RNN. The speech processing model may be designed to extract robust features by utilizing noise data and SNR mixing data during the learning process. The speech processing model extracts important features (e.g., feature vectors) from the input speech data and generates embedding data necessary for subsequent processing (e.g., text generation via a decoder).

[0052] For example, the target embedding is the output of a speech processing model and represents vector data that quantifies the main features of the input speech data. The target embedding may be represented by compressing the contextual, temporal, and frequency information of the speech data. The target embedding is in a multidimensional vector form and may be used as a subsequent decoder (first decoder or second decoder). The target embedding contains essential information of the speech data without text conversion. The target embedding is an intermediate representation that shows the characteristics of the input speech signal and provides foundational data for tasks such as response text generation and speech activity prediction.

[0053] In operation S220, the processor applies the target embedding to a first decoder trained to output text probabilities based on an attention mechanism, and obtains the response text and context vector estimated as the text of the target data.

[0054] For example, the first decoder is a decoder learned based on an attention mechanism, which takes the target embedding and previous response text as input and generates a probability distribution for the response text. That is, the first decoder represents a component that interprets the input embedding data and generates the response text. The first decoder is composed of multiple layers and uses an attention mechanism to compute the interaction between the input embedding and contextual information. Specifically, the first decoder utilizes learnable weights to model the relationship between the embedding and previous response text. The first decoder means a decoder that outputs a probability distribution to generate text corresponding to the target data (audio data), selects the next text token via softmax, and repeats this to generate the entire response text.

[0055] For example, the response text is text generated by the first decoder (i.e., the output decoder) and represents the resulting characterized output of the target data. The response text may consist of the word (or token) with the highest value, which is a probability distribution output from the first decoder. The first decoder operates by sequentially generating words and predicting the next word based on the previous word. The generated text is a result that reflects the user's voice input and is ultimately communicated to the user. Specifically, to provide information corresponding to the target data, the processor generates a response text such as "The weather today is sunny" when the user inputs the target data "Tell me the weather." The response text may be used in the processor's operations (e.g., text-to-speech conversion, speech intent analysis, etc.).

[0056] For example, a context vector is a vector generated by the first decoder that numerically represents the correlation between the stored data (target embedding) and the previous response text. The context vector may be generated by an attention mechanism and may represent the integration of important information from the input embedding and the previous response text. The context vector is used at all levels of the first decoder and contains the information necessary to predict the next word. The context vector may be calculated based on attention scores and learnable weights.

[0057] In operation S230, the processor applies the target embedding and context vector to a second decoder that has been trained to determine whether to terminate the embedding, and determines whether to terminate the operation to obtain the response text. For example, the second decoder is a decoder that takes the target embedding and context vector as input and is trained to determine whether the response text generation operation has finished or not. The second decoder outputs a probability indicating whether the operation has finished or not; for example, if it outputs a number of 0.85, it means that there is an 85% probability that the response text generation operation has finished. The second decoder is trained using Binary Cross-Entropy Loss based on the difference between the actual completion status (i.e., the correct label) during the training process.

[0058] In operation S240, the processor outputs a response text based on the response text and whether the target embedding can be terminated. For example, the processor outputs the response text to the screen via a display or GUI interface. The processor converts the generated response text into speech using text-to-speech (TTS) technology and outputs it to the user.

[0059] Figure 3 is a diagram illustrating the components included in a processor in an audio processing device according to one embodiment of the present invention. Figure 3 illustrates a structural block diagram showing the configuration of a speech processing device (e.g., the speech processing device 100 in Figure 1) and the interactions between each module. The speech processing device receives a speech signal as input, generates response text, and learns speech embedding data to perform various speech processing tasks.

[0060] For example, the signal receiving module 310 receives audio data from an external source, converts it into digital data, and transmits it to the audio processing model 313. In other words, the signal receiving module 310 is responsible for collecting and pre-processing audio data.

[0061] For example, the signal receiver 311 receives a voice signal spoken by the user in an external environment. The signal receiver 311 is capable of analyzing the characteristics of the signal and converts the received voice signal into digital data.

[0062] For example, the speech processing model 313 analyzes the input speech signal and converts it into embedded data (i.e., target embedding) in which the main features are quantified. The speech processing model 313 compresses the temporal and frequency features of the speech data to generate embedded data represented in vector form.

[0063] For example, the response generation module 320 generates a response text to be transmitted to the user based on the embedded data received from the speech processing model 313. The response generation module 320 converts the generated response text into speech and outputs it to the user.

[0064] For example, the response text generator 321 receives embedded data as input and generates a response in text form. Based on an attention mechanism, the response text generator 321 contextually analyzes the input embedded data, predicts the next word, and generates the final text (i.e., the response text).

[0065] For example, the text-to-speech converter 323 converts the generated text response into speech and transmits it to the user. Specifically, the text-to-speech converter 323 uses TTS (Text-to-Speech) technology to output in natural-sounding speech.

[0066] For example, the embedding learning module 330 provides the necessary functions for training the speech processing model 313. Specifically, the embedding learning module 330 converts the input speech data into an embedding, and then trains on it to optimize its performance.

[0067] For example, text decoder 331 is a decoder that takes embedded data generated by speech processing model 313 as input and predicts text based on the embedded data. Text decoder 331 performs training by calculating a loss based on the difference between the predicted text and the actual correct text during training.

[0068] For example, the voice activity decoder 333 is a decoder that takes the embedded data of a voice processing model as input and determines whether or not there is voice (Activity) in the data. The voice activity decoder 333 can be used to determine the section in which a voice signal exists or to decide whether or not to terminate a dialogue operation. The voice activity decoder 333 learns whether or not to terminate by utilizing Binary Cross-Entropy Loss.

[0069] For example, the signal receiving module 310 collects externally input audio data and transmits the audio data to the audio processing model 313 to generate embedded data in which the main features are digitized.

[0070] For example, the response generation module 320 generates response text based on the embedded data generated by the signal receiving module 310. After generating text suitable for the information or service requested by the user, the response generation module 320 converts the generated text into speech and outputs it to the user.

[0071] For example, the embedding learning module 330 receives the embedding data generated by the signal receiving module 310 as input and performs the learning and optimization of the speech processing model 313. Specifically, the embedding learning module 330 learns text generation and speech presence / absence detection tasks via the text decoder 331 and the speech activity decoder 333.

[0072] Figure 4 is a flowchart illustrating the training of a model in an audio processing device according to one embodiment of the present invention. Figure 4 is a flowchart showing the learning and response generation process of a speech processing device (e.g., the speech processing device 100 in Figure 1). Each operation shown in Figure 4 can be described as a process in which the processor learns a speech processing model and a response generation model, and outputs text based on the generated data.

[0073] In one embodiment, the processor (e.g., processor 110 in Figure 1) identifies the model to be trained in operation S405. For example, the model to be trained includes a speech processing model and a response generation model.

[0074] A response generation model refers to a model designed to generate response text to be conveyed to the user based on audio data or embedded data, and to determine whether response generation can be terminated. The response generation model generates responses by including a first decoder (i.e., a text decoder) and a second decoder (i.e., a dialogue termination decoder).

[0075] The first decoder takes the embedding data (i.e., the target embedding) as input, calculates the probability distribution of the response text, and generates the response text. The first decoder generates a text response corresponding to the input audio data via the target embedding. The second decoder determines whether the operation of generating the response text can be completed. The second decoder takes the target embedding and the context vector as input, calculates the completion probability, and determines whether the response text generation operation has been completed.

[0076] In operation S410, the processor generates noise data. For example, the processor generates noise data to be mixed with the audio data, and generates noise data based on various noise conditions that may occur in a real environment (e.g., noise in a cafe, keyboard sounds, noise in a park, etc.).

[0077] In operation S415, the processor identifies the input text. For example, the processor selects the input text data to be used for training.

[0078] In operation S420, the processor generates voice actor recording data and speech synthesis data. For example, the processor generates speech data using the voice actor voice data recorded by the user and a speech synthesis model, and uses them as training data.

[0079] In operation S425, the processor generates audio data. For example, the processor generates audio data that corresponds to text data and uses it as training audio data.

[0080] In operation S430, the processor performs SNR mixing. For example, the processor mixes the generated speech data with noise data via Signal-to-Noise Ratio (SNR) mixing. The processor performs SNR mixing to train a model that is robust to the diverse acoustic conditions that may occur in real-world environments.

[0081] In operation S435, the processor acquires the target data. For example, the processor determines that the data generated from the SNR mixing result is the target data to be learned. The target data represents data containing the main features of the audio signal.

[0082] In operation S440, the processor generates audio presence / absence data. For each point in time of the audio data, the processor generates audio presence / absence data indicating whether or not that point in time contains audio.

[0083] In operation S445, the processor generates dialogue termination data. For example, the processor generates dialogue termination data indicating whether the dialogue can be terminated and uses it as part of the learning process.

[0084] In operation S450, the processor performs embedding learning. For example, the processor utilizes a text decoder and a speech activity decoder to train a speech processing model that generates embeddings based on the learned data.

[0085] The processor acquires training noise data based on at least one of the following: first sub-noise data acquired in the target space, second sub-noise data generated based on a standard normal distribution, or a combination of either of these.

[0086] For example, the target space represents the environment in which source data necessary for collecting or synthesizing training data is collected. The target space includes speech and noise data collected in a real-world environment.

[0087] For example, the first sub-noise data refers to actual environmental noise data actually collected in the target space. The first sub-noise data includes noise data recorded via an actual microphone or physical signals of the actual environment, such as keyboard sounds, wind noise, or car noise.

[0088] For example, the second sub-noise data refers to artificial noise data synthesized based on mathematical methods (e.g., a standard normal distribution). The second sub-noise data may be generated to simulate the noise signal of the actual environment.

[0089] For example, training noise data refers to training noise data generated by combining the first sub-noise data and the second sub-noise data. The training noise data maximizes diversity by combining actual noise data and synthesized noise data, and is designed so that the speech processing model operates robustly in diverse noise environments.

[0090] The processor acquires training audio data based on at least one of the following: first sub-audio data recorded by the user, second sub-audio data generated based on a speech synthesis model, or a combination of either of these.

[0091] For example, the first sub-voice data refers to actual voice data recorded by the user, and includes original voice data collected based on human speech, as well as natural speech patterns and voice characteristics.

[0092] For example, the second sub-speech data represents synthesized speech data generated via a speech synthesis model. It is generated using deep learning-based speech synthesis technology (TTS) and contains acoustic properties similar to actual speech, expressing diverse speech patterns, intonation, and tone.

[0093] For example, training audio data refers to learning audio data generated by independently or combined from the first sub-audio data and the second sub-audio data. The processor generates the training data through SNR (Signal-to-Noise Ratio) mixing of training noise data and training speech data.

[0094] SNR mixing may be expressed by the following formula 1:

number

[0095] The processor applies the training data to the speech processing model and acquires the training embeddings. The processor applies the training embedding to the text decoder and obtains the first extraordinary output.

[0096] For example, the text decoder is described as a decoder that takes a training embedding as input and generates response text based on that embedding. The text decoder utilizes an attention mechanism to analyze the embedding data and contextual information, calculates the probability distribution of the next word, and generates text sequentially based on this.

[0097] The output of the text decoder may be expressed by the following equations 2 and 3:

number

number

[0098] The processor applies the training embedding to the speech activity decoder and obtains a second extraordinary output. For example, a speech activity decoder is a decoder that takes a training embedding as input and determines whether or not the embedding contains speech. The speech activity decoder is composed of multiple linear layers and outputs the presence or absence of speech as a probability value between 0 and 1.

[0099] The output of the speech activity decoder may be expressed by the following equation 4:

number

[0100] For example, the processor performs training of a speech processing model through a first loss based on a first temporary output and a text decoder, and a second loss based on a second temporary output and a speech activity decoder. The first loss may be expressed by the following equation 5:

number

[0101] The second loss may be expressed by the following equation 6:

number

[0102] The loss required to train a speech processing model can be expressed by the following equation 7:

number

[0103] In operation S455, the processor identifies the input text. The processor identifies the input text data to be used for training.

[0104] In operation S460, the processor determines whether or not there is a speech intent in the input text. By determining whether or not a speech intent exists in the input text, the processor generates a response text based on the speech intent if one exists. Conversely, if there is no speech intent, the processor identifies the response text in a separate database and / or large language model.

[0105] In operation S465, the processor generates response text based on the utterance intent obtained from the dialogue language model or the large language model.

[0106] In operation S470, the processor retrieves a response text from a database containing response texts that differ from the intended speech.

[0107] The response text generated in operations S455 to S470 represents the ground truth data necessary for training the response generation model.

[0108] In operation S475, the processor performs training of the response text generator based on the generated text data.

[0109] For example, if the time at which the training embedding is applied to the first decoder is the target time, the processor identifies the first training response text obtained from the first decoder at the time of transition to the target time.

[0110] For example, the reference time refers to a specific point in time when the training target embedding is applied to the first decoder. The reference time is a reference point in each stage where the response text is generated, indicating the time stage in which the first decoder processes the input data and generates the text.

[0111] For example, the first training response text represents the text generated by the first decoder at the target time shift. The first training response text may be used as primary data to form the context of the text generated after the target time shift.

[0112] For example, the processor applies the training target embedding and the first training response text to the first decoder to obtain the training context vector and the second training response text.

[0113] For example, the second training response text refers to the text generated by the first decoder after the target time point. The second training response text may be generated based on the target embedding and the first training response text input at the target time point.

[0114] For example, the second training response text may be applied as the training input to the first decoder at a time point following the target time point.

[0115] For example, the processor applies the training target embedding and the training context vector to the second decoder and obtains a termination probability regarding whether to terminate the acquisition of the second training response text. Based on a comparison of the termination probability with a predetermined value, the processor decides whether to terminate the operation to acquire the second training response text.

[0116] For example, the processor obtains the loss for the first decoder based on the first decoder and the trained second response text, and the loss for the second decoder based on the second decoder and the termination probability, based on the fact that the first and second decoders are included in the response generation model.

[0117] For example, the processor performs training of the response generation model based on the losses of the first decoder and the second decoder.

[0118] For example, the loss of the first decoder represents the difference between the text output value generated by the first decoder and the correct text. The loss of the second decoder represents the difference between the termination probability output by the second decoder and the actual termination status (correct label).

[0119] For example, the processor obtains a temporary output by applying the target embedding and context vector to a second decoder. Based on a comparison of the temporary output with a predetermined value, the processor decides whether to terminate the operation to obtain the response text.

[0120] For example, the response generation model is configured based on a Transformer structure and consists of a first decoder and a second decoder. The response generation model learns text generation and dialogue termination determination functions through deep learning-based learning and provides responses that correspond to user requests through real-time inference.

[0121] For example, the learning process of a response generation model includes the operation of generating input data and ground truth data. Specifically, the input data includes embedded data generated by the speech processing model (i.e., the training target embedding) and the first training response text (i.e., the text generated at the target time shift).

[0122] For example, the training embedding may be transmitted to the first decoder of a response generation model and applied to analyze the relationships between input data and generate a suitable response.

[0123] For example, the processor generates a second trained response text based on the training embedding and the first trained response text. Specifically, the processor leverages the attention mechanism of the response generation model to contextually analyze the input data and generate the second trained response text. The processor calculates the cross-entropy loss between the generated response text (i.e., the second trained response text) and the ground truth text, and updates the learning weights of the first decoder via the loss value.

[0124] For example, the processor applies the training embedding and context vector to a second decoder to output the termination probability. The processor then calculates the Binary Cross-Entropy loss based on the difference between the termination probability and the actual termination (i.e., the label, which is the correct data).

[0125] For example, the processor obtains the final loss by summing the losses of the first decoder and the second decoder. The processor then updates the weights of the response generation model using an optimization algorithm (e.g., Adam, AdamW).

[0126] Figure 5 illustrates a method for processing target embedding in an audio processing device according to one embodiment of the present invention. Figure 5 shows the data processing follow-up and interactions between the main components of an audio processing device according to one embodiment (e.g., the audio processing device 100 in Figure 1).

[0127] The target data consists of externally input audio signals, including audio signals captured in a real environment or synthesized audio data. The target data may be applied to the audio processing model 313. The speech processing model 313 converts the input target data into digitized embeddings. The speech processing model 313 is designed based on neural network structures such as Transformers, RNNs, and Convolutional Neural Networks (CNNs).

[0128] The embedding learning module 330 is a module that receives embedded data transmitted from the speech processing model 313 as input and performs text generation and speech presence / absence detection tasks. The embedding learning module 330 consists of two decoders (e.g., a text decoder and a speech activity decoder).

[0129] The text decoder 331 generates response text based on the input embedding data. For example, if the input audio signal is "hello", the text decoder 331 generates the response text "hello".

[0130] The voice activity decoder 333 determines the presence or absence of voice data based on the input embedding data. The voice activity decoder 333 outputs a binary value (e.g., 0 or 1) indicating whether or not voice is present at each point in time.

[0131] Figure 6 is a flowchart illustrating a method for obtaining response text in an audio processing device according to one embodiment of the present invention.

[0132] A processor according to one embodiment (e.g., processor 110 in Figure 1) can identify the target embedding in operation S610. The processor identifies the target embedding generated by the speech processing model using the target data (i.e., the speech signal).

[0133] In operation S620, the processor applies the target embedding to the first decoder. The processor calculates a probability distribution for text response generation via the first decoder and performs a determination of whether the dialogue can be terminated in the subsequent stage.

[0134] In operation S630, the processor obtains whether the interaction can be terminated from the second decoder. The processor inputs the target embedding and the context vector generated by the first decoder to the second decoder. The processor calculates the probability of terminating the interaction based on the input data via the second decoder and obtains whether the interaction can be terminated (i.e., 0 or 1) based on this.

[0135] In operation S640, the processor determines whether the interaction can be terminated. Based on the interaction termination status value obtained from the second decoder, if the termination status is 1, the processor determines that the interaction has ended and terminates the process. Conversely, if the termination status is 0, the processor determines that the interaction is continuing and performs the next operation.

[0136] In operation S650, the processor obtains the response text from the first decoder. The processor generates the response text via the first decoder. The first decoder completes the response text by sequentially generating the next word based on the target embedding and the previously generated text data (i.e., the response text generated at a point prior to the time when the target embedding is applied to the first decoder).

[0137] Figure 7 is a flowchart illustrating a method for converting response text into speech and outputting it in an audio processing device according to one embodiment of the present invention.

[0138] In one embodiment, a processor (e.g., processor 110 in Figure 1) receives a signal containing noise, silence, and voice during operation S710. The input signal is acquired via a collection device such as a microphone and may be collected in a variety of noisy environments. For example, the processor receives both voice data, such as "hello," and ambient noise (e.g., keyboard sounds, car noises, etc.).

[0139] In operation S720, the processor acquires the target embedding based on the speech processing model. For example, the speech signal "hello" may be converted into feature vector-based embedding data (i.e., target embedding).

[0140] In operation S730, the processor obtains response text based on the target embedding. For example, if the input signal is "Hello", the processor generates response text such as "Hello, how can I help you?" via the response generation model.

[0141] In operation S740, the processor converts the response text into speech and outputs it. Furthermore, based on having acquired the response text, the processor applies the response text to a speech intent prediction model to obtain the speech intent of the response text. The processor can then apply the speech intent to a speech intent-based behavior database to acquire behavior data. The processor transmits the behavior data to a robot connected to a speech processing device (e.g., speech processing device 100 in Figure 1).

[0142] Figure 8 illustrates a computer system for a speech processing device or a speech processing method according to one embodiment of the present invention. Referring to Figure 8, the computer system 1000 for the voice processing device or voice processing method includes at least one processor 1100, memory 1300, user interface input device 1400, user interface output device 1500, storage 1600, and network interface 1700, all connected via a bus 1200.

[0143] The processor 1100 may be a semiconductor device that performs processing on instruction words stored in a central processing unit (CPU) or memory 1300 and / or storage 1600. The memory 1300 and storage 1600 include various types of volatile or volatile storage media. For example, the memory 1300 includes ROM (read-only memory) and RAM (random access memory).

[0144] Accordingly, steps of the methods or algorithms described in relation to the embodiments disclosed herein may be directly embodied by hardware, software modules, or a combination thereof, executed by the processor 1100. The software modules reside in storage media such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, and CD-ROMs (i.e., memory 1300 and / or storage 1600).

[0145] An exemplary storage medium is coupled to a processor 1100, which reads information from and writes information to the storage medium. Alternatively, the storage medium may be integrated with the processor 1100. The processor and storage medium reside within an ordered integrated circuit (ASIC). The ASIC resides within a user terminal. Alternatively, the processor and storage medium reside within a user terminal as separate components.

[0146] The above description is merely illustrative of the technical concept of the present invention, and a person with ordinary skill in the art to which the present invention belongs can make various modifications and alterations without departing from the essential characteristics of the present invention.

[0147] The embodiments described above may be embodied in hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus, methods, and components described in the embodiments may be embodied using a general-purpose computer or a special-purpose computer, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPGA (field programmable gate array), PLU (programmable logic unit), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and software applications performed on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to software execution. For convenience of understanding, the processing device may have been described as being one, but a person with ordinary skill in the art will see that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.

[0148] Software includes computer programs, code, instructions, or a combination of one or more of these, which configure a processing unit to operate as desired, or which independently or collectively instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a certain type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. Software may be distributed across a network of connected computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0149] The methods according to the embodiments may be embodied in a form of program instructions that can be executed via various computer means and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination, and the program instructions recorded on the medium may be specifically designed and configured for the embodiments or may be publicly known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code that can be produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0150] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0151] As described above, embodiments have been explained with limited drawings, but a person with ordinary skill in the art can apply a variety of technical modifications and variations therefrom. For example, satisfactory results can be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described systems, structures, devices, circuits, etc. are combined or assembled in a different manner than described, or substituted or replaced by other components or equivalents.

[0152] Therefore, other embodiments, other embodiments, and those equivalent to the claims described below also fall within the scope of the claims.

[0153] Therefore, the embodiments disclosed herein are for illustrative purposes only, and not to limit the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention should be interpreted in accordance with the following claims, and all technical concepts within an equivalent scope should be interpreted as being included within the scope of the rights of the present invention. [Explanation of symbols]

[0154] 100 Voice Processing Devices 110, 1100 processors 120, 1300 memory 122 Imperative 130 Communications Department 140 servers 310 Signal receiving module 311 Signal Receiver 313 Speech Processing Models 320 Response Generation Module 321 Response Text Generator 323 Text-to-Speech Converter 330 Embedding Learning Modules 331 Text Decoder 333 Voice Activity Decoder 1000 Computer Systems 1200 bus 1310 ROM 1320 RAM 1400 User Interface Input Device 1500 User Interface Output Device 1600 storage 1700 Network Interfaces

Claims

1. Memory containing computer-executable instructions, Includes a processor that accesses the memory and executes the instruction word, The aforementioned processor, The target data relating to speech is applied to a speech processing model trained to output signal features numerically, and a target embedding containing the features of the target data is obtained. The target embedding is applied to a first decoder trained to output text probabilities based on an attention mechanism, and a response text and context vector estimated as the text of the target data are obtained. The target embedding and the context vector are applied to a second decoder that has been trained to determine whether to terminate the embedding, and the operation to obtain the response text is terminated. A speech processing device characterized by outputting the response text based on the response text and whether the target embedding can be terminated.

2. The aforementioned processor, Training noise data is acquired based on at least one of the following: first sub-noise data acquired in the target space, second sub-noise data generated based on a standard normal distribution, or any combination thereof. Training audio data is acquired based on at least one of the following: first sub-audio data recorded by the user, second sub-audio data generated based on a speech synthesis model, or any combination thereof. Training target data is generated through SNR (Signal-to-Noise Ratio) mixing of the training noise data and the training voice data. The speech processing apparatus according to claim 1, characterized in that it applies the training target data to the speech processing model and obtains the training target embedding.

3. The aforementioned processor, The aforementioned training target embedding is applied to a text decoder to obtain a first temporary output, The aforementioned training target embedding is applied to the speech activity decoder to obtain a second temporary output, The speech processing apparatus according to claim 2, characterized in that it performs learning of the speech processing model via the first temporary output and a first loss based on the text decoder, and the second temporary output and a second loss based on the speech activity decoder.

4. The aforementioned processor, If the time at which the training target embedding is applied to the first decoder is the target time, the training first response text obtained from the first decoder at the time of transition to the target time is identified. The training target embedding and the first training response text are applied to the first decoder to obtain a training context vector and a second training response text. The aforementioned training second response text is: The audio processing device according to claim 2, characterized in that it is applied as a training input to the first decoder at a time point following the aforementioned target time point.

5. The aforementioned processor, The training target embedding and the training context vector are applied to the second decoder to obtain a termination probability regarding whether the acquisition of the second training response text can be completed. The speech processing device according to claim 4, characterized in that it determines whether or not to terminate the operation to acquire the training second response text based on a comparison of the termination probability with a predetermined value.

6. The aforementioned processor, Based on the inclusion of the first decoder and the second decoder in the response generation model, the loss of the first decoder based on the first decoder and the second training response text, and the loss of the second decoder based on the second decoder and the termination probability are obtained. The speech processing device according to claim 5, characterized in that it performs learning of the response generation model based on the loss of the first decoder and the loss of the second decoder.

7. The aforementioned processor, By applying the target embedding and the context vector to the second decoder, a temporary output is obtained. The voice processing device according to claim 1, characterized in that it determines whether or not to terminate the operation of acquiring the response text based on a comparison of the temporary output with a predetermined value.

8. The aforementioned processor, The voice processing device according to claim 1, characterized in that it converts the response text into speech and outputs it to the user who input the target data.

9. The aforementioned processor, Based on obtaining the response text, the response text is applied to a speech intent prediction model to obtain the speech intent of the response text. The aforementioned utterance intent is applied to the behavior database based on the aforementioned utterance intent, and behavioral data is obtained. The voice processing device according to claim 1, characterized in that it transmits the aforementioned behavioral data to a robot connected to the voice processing device.

10. The operation involves applying target data related to speech to a speech processing model trained to output signal features numerically, and obtaining target embedding that includes the features of the target data. The operation involves applying the aforementioned target embedding to a first decoder trained to output text probabilities based on an attention mechanism, thereby obtaining the response text and context vector estimated as the text of the target data. Applying the aforementioned target embedding and the aforementioned context vector to a second decoder trained to determine whether to terminate the embedding, and determining whether to terminate the operation to acquire the response text, and A speech processing method characterized by including the operation of outputting the response text based on the response text and whether the target embedding can be terminated.

11. The operation to output the aforementioned response text is: An operation to acquire training noise data based on at least one of the following: first sub-noise data acquired in the target space, second sub-noise data generated based on a standard normal distribution, or any combination thereof. An operation to acquire training audio data based on at least one of the following: first sub-audio data recorded by the user, second sub-audio data generated based on a speech synthesis model, or any combination thereof. The operation of generating training target data through SNR (Signal-to-Noise Ratio) mixing of the training noise data and the training voice data, and The speech processing method according to claim 10, characterized in that it includes the operation of applying the training target data to the speech processing model and acquiring the training target embedding.

12. The operation to output the aforementioned response text is: The operation involves applying the aforementioned training target embedding to a text decoder and obtaining a first temporary output. The operation involves applying the aforementioned training target embedding to the voice activity decoder and obtaining a second temporary output, and The speech processing method according to claim 11, characterized in that it includes an operation to perform learning of the speech processing model via a first temporary output and a first loss based on the text decoder, and a second loss based on a second temporary output and the speech activity decoder.

13. The operation to output the aforementioned response text is: If the time at which the training target embedding is applied to the first decoder is the target time, the operation of identifying the training first response text obtained from the first decoder at the time of transition to the target time, and The operation includes applying the training target embedding and the first training response text to the first decoder to obtain a training context vector and a second training response text, The aforementioned training second response text is: The audio processing method according to claim 11, characterized in that it is applied as a training input to the first decoder at a time point following the aforementioned target time point.

14. The operation to output the aforementioned response text is: The operation involves applying the training target embedding and the training context vector to the second decoder to obtain a termination probability regarding whether the acquisition of the second training response text can be completed, and The speech processing method according to claim 13, characterized in that it includes an operation to determine whether or not to terminate the operation to acquire the training second response text based on a comparison of the termination probability with a predetermined value.

15. The operation to output the aforementioned response text is: Based on the inclusion of the first decoder and the second decoder in the response generation model, the operation of obtaining the loss of the first decoder based on the first decoder and the second training response text, and the loss of the second decoder based on the second decoder and the termination probability, and The speech processing method according to claim 14, characterized in that it includes the operation of performing training of the response generation model based on the loss of the first decoder and the loss of the second decoder.

16. The operation to output the aforementioned response text is: The operation of obtaining a temporary output by applying the target embedding and the context vector to the second decoder, and The speech processing method according to claim 10, characterized in that it includes an operation to determine whether or not to terminate the operation to acquire the response text based on a comparison of the temporary output with a predetermined value.

17. The operation to output the aforementioned response text is: The voice processing method according to claim 10, characterized in that it includes the operation of converting the response text into speech and outputting it to the user who input the target data.

18. The operation to output the aforementioned response text is: Based on obtaining the response text, the operation of applying the response text to a speech intent prediction model to obtain the speech intent of the response text, The operation of applying the aforementioned utterance intent to a database of actions based on the aforementioned utterance intent and acquiring action data, and The voice processing method according to claim 10, characterized in that it includes the operation of transmitting the aforementioned behavioral data to a robot connected to a voice processing device.

Images