Electronic device and control method thereof

Abstract

To provide an electronic device and a control method thereof for generating gestures suitable for robot speech based on changes in voice tones.SOLUTION: In an electronic device having a memory and at least one processor that accesses the memory to execute instruction words, a control method identifies a first word segment and a second word segment related to a grammar from a target sentence included in a corpus, determines a target phrase containing the first word segment and the second word segment based on a comparison of a pause time between the first word segment and the second word segment and a predetermined critical time, and determines a voice tone change of the target phrase based on information regarding a pitch contour of voice data obtained by applying the target sentence to a predetermined voice synthesis model and speech time of each word segment contained in the target phrase.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an electronic device and a control method thereof, and more particularly to a technique for generating learning data for a model that generates robot gestures. [Background technology]

[0002] To generate a robot's speech-gestures corresponding to a robot's utterance text input from a dialogue model (e.g., chatbot), a process is required to select and generate appropriate gestures from a database for the section (e.g., including the start and end points of the gesture) to which a gesture can be assigned in the given robot's utterance text, as well as appropriate gestures for the gesture assignment section. Specifically, if gestures are assigned to the entire utterance text, the gestures are displayed from the start to the end of the utterance, which can hinder the user's immersion. Furthermore, if there is a time mismatch between the robot's utterance and gestures, it can cause negative interaction effects between humans and robots (e.g., a decrease in favorability).

[0003] In this regard, the intervals to which gestures can be assigned may vary based on word segments (e.g., clauses) or phrases (e.g., clauses). However, since the speech duration of a word segment is short, it is difficult to ensure sufficient time for gesture generation. Therefore, technologies that assign gestures to phrases have mainly been developed, but because gestures are generated based on grammatical information (e.g., parts of speech) or semantic information (e.g., morphemes) within the spoken sentence, changes in the robot's voice tone and coherence are not taken into consideration. In addition, utilizing models trained by machine learning methods is difficult because it requires the user to build a huge amount of data, which requires time and effort.

[0004] To solve these problems, it is necessary to develop technology to automatically generate training data for a model that generates robot gestures, and technology to assign gestures based on changes in voice tone. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-158193 Summary of the Invention [Problem to be solved by the invention]

[0006] An embodiment of the present invention provides an electronic device and a control method thereof that determines a tone change of a target phrase based on pitch contour information of speech data acquired by applying a target sentence to a speech synthesis model and the speaking time of a word segment, and thereby generates gestures suitable for robot speech based on the tone change.

[0007] Furthermore, an embodiment of the present invention provides an electronic device and a control method thereof that can acquire information about the pitch contour by applying cubic spline interpolation to the voice data based on the fact that information about the pitch contour cannot be identified from the voice data, thereby acquiring information about the pitch contour.

[0008] In addition, the present invention aims to provide an electronic device and a control method thereof that can provide natural interaction with a user through natural speech-gesture generation by a robot by determining, among phrases included in a target sentence, a phrase with the greatest change in voice tone as a candidate for assigning a gesture to the target sentence. The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0009] An electronic device according to an embodiment of the present invention includes a memory storing computer-executable instructions, and at least one processor that accesses the memory and executes the instructions. The at least one processor may identify a first word segment and a second word segment, which are grammatical units, from a target sentence included in a corpus, determine a target phrase including the first word segment and the second word segment based on a comparison of a pause time between the first word segment and the second word segment and a predetermined critical time, and determine a voice tone change of the target phrase based on information about a pitch contour of speech data acquired by applying the target sentence to a predetermined text-to-speech model and the utterance time of each word segment included in the target phrase.

[0010] In one embodiment, the at least one processor may identify a first speech time of the first word segment and a second speech time of the second word segment from the audio data, determine a first speech tone change of the first word segment via information about the first speech time and the pitch contour of the first word segment, and determine a second speech tone change of the second word segment via information about the second speech time and the pitch contour of the second word segment.

[0011] In one embodiment, the at least one processor may determine the first speech tone change by obtaining a speech tone change rate for each predetermined unit time interval from information about the pitch contour of the first word segment corresponding to the first utterance time, and may determine the second speech tone change by obtaining a speech tone change rate for each predetermined unit time interval from information about the pitch contour of the second word segment corresponding to the second utterance time.

[0012] In one embodiment, the at least one processor may determine an average of the first speech tone change and the second speech tone change based on whether the first word segment and the second word segment are included in the target phrase, and determine the value obtained by applying the average to a normalization function as the speech tone change of the target phrase. In one embodiment, the at least one processor includes the first word segment and the second word segment in the target phrase based on the pause time being less than the critical time; One of the first word segment and the second word segment may be included in the target phrase based on the pause time being equal to or greater than the critical time.

[0013] In one embodiment, the at least one processor may determine the positions of the first word segment and the second word segment in the audio data based on the pause time being less than the critical time, identify a third word segment following the second word segment from the audio data based on the second word segment following the first word segment, and determine whether to include the third word segment in the target phrase based on a comparison of the pause time between the second word segment and the third word segment and the critical time.

[0014] In one embodiment, the at least one processor may obtain a first target vector having a predetermined number of dimensions through word embedding of a target window including the first word segment and the second word segment, apply the first target vector to a phrase unit recognition model to obtain an output indicating whether the word segments included in the target window are segmented, and train the phrase unit recognition model based on a first loss obtained by comparing the output with the target sentence.

[0015] In one embodiment, the at least one processor may acquire second target vectors including vectors having a predetermined number of dimensions for each word segment included in the target phrase through word embedding of the target phrase, reduce the dimension of the second target vector by applying the second target vector to an encoder that reduces the dimension of an input target, acquire a temporary phonetic tone change of the target phrase by applying the second target vector applied to the encoder to a phonetic tone change prediction model, and train the phonetic tone change prediction model based on a second loss acquired by comparing the temporary phonetic tone change of the target phrase and the phonetic tone change of the target phrase.

[0016] In one embodiment, the at least one processor may determine, based on the determined voice tone change of each phrase included in the target sentence, a phrase with the largest voice tone change among the phrases included in the target sentence as a gesture assignment candidate corresponding to a gesture execution section of the target sentence, determine a gesture of the gesture assignment candidate based on a gesture type corresponding to the gesture assignment candidate, and correspond the gesture to a speaking time of the gesture assignment candidate to generate a gesture of a robot scheduled to output the target sentence.

[0017] In one embodiment, the at least one processor may, based on failure to identify information about the pitch contour from the audio data, apply cubic spline interpolation to the audio data to identify information about the pitch contour.

[0018] A control method according to an embodiment of the present invention may include an operation of identifying a first word segment and a second word segment as grammatical units from a target sentence included in a corpus; an operation of determining a target phrase including the first word segment and the second word segment based on a comparison of a pause time between the first word segment and the second word segment and a predetermined critical time; and an operation of determining a voice tone change of the target phrase based on information about a pitch contour of speech data acquired by applying the target sentence to a predetermined text-to-speech model and an utterance time of each word segment included in the target phrase.

[0019] In one embodiment, determining the target phrase may include identifying a first speech time of the first word segment and a second speech time of the second word segment from the audio data, determining a first speech tone change of the first word segment via information about the first speech time and the pitch contour of the first word segment, and determining a second speech tone change of the second word segment via information about the second speech time and the pitch contour of the second word segment.

[0020] In one embodiment, the operation of determining the target phrase may include an operation of determining the first speech tone change by obtaining a speech tone change rate for each predetermined unit time interval from information about the pitch contour of the first word segment corresponding to the first utterance time, and an operation of determining the second speech tone change by obtaining a speech tone change rate for each predetermined unit time interval from information about the pitch contour of the second word segment corresponding to the second utterance time.

[0021] In one embodiment, the operation of determining the speech tone change of the target phrase may include an operation of determining an average of the first speech tone change and the second speech tone change based on the inclusion of the first word segment and the second word segment in the target phrase, and an operation of applying the average to a normalization function to determine the value obtained as the speech tone change of the target phrase.

[0022] In one embodiment, the operation of determining the target phrase may include an operation of including the first word segment and the second word segment in the target phrase based on the pause time being less than the critical time, and an operation of including one of the first word segment and the second word segment in the target phrase based on the pause time being greater than or equal to the critical time.

[0023] In one embodiment, the operation of determining the target phrase may include an operation of determining the positions of the first word segment and the second word segment in the audio data based on the pause time being less than the critical time, an operation of identifying a third word segment that follows the second word segment from the audio data based on the second word segment following the first word segment, and an operation of determining whether to include the third word segment in the target phrase based on a comparison of the pause time between the second word segment and the third word segment and the critical time.

[0024] In one embodiment, the control method may further include an operation of obtaining a first target vector having a predetermined number of dimensions through word embedding of a target window including the first word segment and the second word segment; an operation of applying the first target vector to a phrase unit recognition model to obtain an output indicating whether the word segments included in the target window are segmented; and an operation of training the phrase unit recognition model based on a first loss obtained by comparing the output and the target sentence.

[0025] In an embodiment, the control method may further include an operation of acquiring a second target vector including a vector having a predetermined number of dimensions for each word segment included in the target phrase through word embedding of the target phrase; an operation of reducing the dimension of the second target vector by applying the second target vector to an encoder that reduces the dimension of an input target; an operation of applying the second target vector applied to the encoder to a speech tone change prediction model to acquire a temporary speech tone change of the target phrase; and an operation of training the speech tone change prediction model based on a second loss acquired by comparing the temporary speech tone change of the target phrase and the speech tone change of the target phrase.

[0026] In one embodiment, the control method may include an operation of determining, based on the determination of the voice tone change of each phrase included in the target sentence, a phrase with the largest voice tone change among the phrases included in the target sentence as a gesture assignment candidate corresponding to a gesture execution section of the target sentence; an operation of determining a gesture of the gesture assignment candidate based on a gesture type corresponding to the gesture assignment candidate; and an operation of generating a gesture of a robot scheduled to output the target sentence by associating the gesture with a speaking time of the gesture assignment candidate. In one embodiment, the control method may further include applying cubic spline interpolation to the audio data to identify information about the pitch contour based on the failure to identify information about the pitch contour from the audio data. [Effects of the Invention]

[0027] The effects of the electronic device and the control method thereof according to the present invention will be described below. According to at least one embodiment of the present invention, a speech tone change of a target phrase is determined based on information about the pitch contour of speech data acquired by applying a target sentence to a speech synthesis model and the speaking time of word segments, thereby achieving the effect of generating gestures suitable for robot speech based on the speech tone change.

[0028] Furthermore, according to at least one embodiment of the present invention, when information about the pitch contour cannot be identified from the speech data, cubic spline interpolation is applied to the speech data to obtain information about the pitch contour, thereby achieving the effect of obtaining speech tone changes of a word phrase whose pitch contour cannot be identified.

[0029] Furthermore, according to at least one embodiment of the present invention, the phrase with the greatest change in tone among the phrases included in the target sentence is determined as a candidate for assigning a gesture to the target sentence, thereby providing a natural interaction with the user through the robot's natural speech-gesture generation. In addition, various other effects can be provided that can be grasped directly or indirectly through this document. [Brief explanation of the drawings]

[0030] [Figure 1] 1 illustrates an electronic device according to one embodiment of the present invention. [Figure 2] 4 is a flowchart illustrating a control method according to an embodiment of the present invention. [Figure 3] FIG. 1 illustrates a method for generating gesture training data from a corpus in an electronic device according to one embodiment of the present invention. [Figure 4] FIG. 1 illustrates a method for identifying speech times of word segments and pause times between word segments in an electronic device according to one embodiment of the present invention. [Figure 5] FIG. 1 illustrates a method for determining phrase-wise speech tone changes in an electronic device according to one embodiment of the present invention. [Figure 6] FIG. 1 illustrates a method for generating gestures from gesture training data in an electronic device according to an embodiment of the present invention. [Figure 7] 1 illustrates a method for training a phrase-unit recognition model in an electronic device according to an embodiment of the present invention. [Figure 8] FIG. 1 illustrates a method for training an audio tone change prediction model in an electronic device according to an embodiment of the present invention. [Figure 9] 1 illustrates a method for determining gestures on a target sentence in an electronic device according to an embodiment of the present invention. [Figure 10]1 is a diagram illustrating a computing system relating to an electronic device or control method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] Some embodiments of the present invention will be described in detail below with reference to exemplary drawings. When assigning reference numerals to components in each drawing, care should be taken to assign the same numerals to identical components, even if they appear in different drawings. Furthermore, when describing embodiments of the present invention, if a detailed description of related known structures or functions is deemed to hinder understanding of the embodiments of the present invention, such detailed description will be omitted. In particular, various embodiments of the present document will be described with reference to the accompanying drawings. However, this is not intended to limit the technology described in this document to specific embodiments, and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present document. In connection with the description of the drawings, similar reference numerals will be used to refer to similar components.

[0032] When describing components of embodiments of the present invention, terms such as "first," "second," "A," "B," "(a)," and "(b)" are used. These terms are used to distinguish a component from other components and do not limit the nature, order, or sequence of the components. Furthermore, unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. Terms similar to those defined in commonly used dictionaries should be interpreted as meanings consistent with the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined in this application. For example, terms such as "first," "second," "primary," or "secondary" used in this document are used to modify various components regardless of order and / or importance, and are used only to distinguish one component from other components, and do not limit the components. For example, a first user device and a second user device refer to different user devices regardless of order or importance. For example, a first component may be named a second component, and similarly, a second component may be named in place of the first component, without departing from the scope of the rights set forth herein.

[0033] In this document, the terms "have," "may have," "include," or "may include" indicate the presence of a given feature (e.g., a value, a function, an operation, or a component such as a part) and do not exclude the presence of additional features.

[0034] When a component (e.g., a first component) is referred to as being "operatively or communicatively coupled with" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component or may be coupled through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is referred to as being "directly coupled with" or "directly connected to" another component (e.g., a second component), it should be understood that there is no other component (e.g., a third component) between the component and the other component.

[0035] As used in this document, the expression "configured to" may be used alternatively with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of," depending on the context.

[0036] The term "configured to" does not necessarily mean "specifically designed to" hardware. Instead, depending on the context, the phrase "device configured to" may mean that the device, together with other devices or components, is capable of performing. For example, the phrase "a processor configured to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device. Terms used in this document are used merely to describe particular embodiments and may not be intended to limit the scope of other embodiments. Singular terms may include plural terms unless the context clearly dictates otherwise. Terms used herein, including technical and scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art described herein. Among the terms used in this document, terms defined in a general dictionary may be interpreted as meanings identical or similar to the meanings they have in the context of the relevant technology, and unless explicitly defined in this document, they should not be interpreted as idealized or overly formal. In some cases, even terms defined in this document cannot be interpreted as excluding the embodiments of this document.

[0037] In this document, expressions such as "A or B," "at least one of A and / or B," or "one or more of A and / or B" may include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" may refer to (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B. Furthermore, when describing components of embodiments of the present invention, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, or C," and "at least one of A, B, C, or some combination thereof" may include any one of the listed items or all possible combinations thereof. In particular, a phrase such as "at least one of A, B, C, or some combination thereof" may include A, B, or C, or combinations thereof such as AB or ABC.

[0038] Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS. FIG. 1 is a diagram illustrating an electronic device according to one embodiment of the present invention. According to one embodiment, the electronic device 100 includes a processor 110, a memory 120 including an instruction set 122, and a communication unit .

[0039] The electronic device 100 represents an apparatus for generating training data for a model related to robot gesture generation. For example, the electronic device 100 generates, as training data, target sentences including target phrases for which speech tone changes have been determined. Specifically, the electronic device 100 determines word segments included in the target phrase to determine speech tone changes for the target phrase. To determine the word segments included in the target phrase, the electronic device 100 identifies the utterance time of each word segment and the pause time between the word segments. Thus, the electronic device 100 determines the word segments included in the target phrase based on the utterance time of each word segment and the pause time between the word segments. A detailed description of a method for determining a target phrase (i.e., determining the target phrase by determining the word segments included in the target phrase) will be provided below with reference to FIGS. 4 and 5.

[0040] A word segment indicates a phrase included in a target sentence. For example, a word segment is a grammatical unit larger than a word and may include a phrase, which may be a unit of speech. A phrase indicates a clause included in a target sentence. For example, a phrase includes two or more word segments. A speech tone change indicates a change in speech intensity and / or speech volume of a word segment or phrase. A target sentence may be raw data for training a model related to robot gesture generation. For example, the target sentence may be included in a corpus stored on the server 140. The target sentence may include word segments and phrases.

[0041] The processor 110 executes software and controls at least one other component (e.g., a hardware or software component) coupled to the processor 110. The processor 110 also performs various other data processing or calculation functions. For example, the processor 110 stores word segments, phrases, voice tone changes, and target sentences in the memory 120.

[0042] For reference, the processor 110 performs all operations performed by the electronic device 100. Therefore, in this specification, for convenience of explanation, operations performed by the electronic device 100 will be mainly described as operations performed by the processor 110. Also, in this specification, for convenience of explanation, the processor 110 will mainly be described as one processor, but is not limited thereto. For example, the electronic device 100 may include at least one or more processors. Each of the at least one or more processors performs all operations related to the operation of generating learning data for a model for generating robot gestures.

[0043] The memory 120 temporarily and / or permanently stores various data and / or information required to perform the operation of generating training data for the model for robot gesture generation, such as word segments, phrases, voice tone changes, and target sentences.

[0044] The communication unit 130 supports communication between the electronic device 100 and the server 140. For example, the communication unit 130 may include one or more components that enable communication between the electronic device 100 and the server 140. For example, the communication unit 130 may include a short-range wireless communication unit, a microphone, etc. In this case, short-range communication technologies may include, but are not limited to, wireless LAN (Wi-Fi), Bluetooth, ZigBee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), Bluetooth Low Energy (BLE), and near field communication (NFC).

[0045] The server 140 may include a corpus. For example, the server 140 may transmit at least one sentence among a plurality of sentences included in the corpus to the electronic device 100 in response to a request from the electronic device 100. To generate the corpus, the server 140 receives data (e.g., data including sentences) from various web servers. Specifically, the corpus may include a large collection of sentences including various types of sentences stored in a web cloud. To this end, the server 140 generates a corpus including newspaper articles, community bulletin boards, etc., using web crawling technology.

[0046] FIG. 2 is a flowchart illustrating a control method according to an embodiment of the present invention. In one embodiment, an electronic device (e.g., electronic device 100 of FIG. 1 ) identifies a first word segment and a second word segment, which are grammatical units, from a target sentence included in a corpus in operation 210. For example, the electronic device receives a target sentence and / or a corpus from a server. When the electronic device receives the corpus, the electronic device identifies a sentence from the multiple sentences included in the corpus as the target sentence for training a model for robot gesture generation. When the electronic device identifies the target sentence, the electronic device identifies a first word segment and a second word segment from the target sentence. For reference, the first word segment and the second word segment may be word segments located adjacent to each other in the target sentence. However, without being limited thereto, the first word segment and the second word segment may be segments located a predetermined distance apart from each other in the target sentence.

[0047] In operation 220, the electronic device determines a target phrase including the first word segment and the second word segment based on a comparison of the pause time between the first word segment and the second word segment and a predetermined critical time. For example, the pause time indicates the time from the end of the speech time of the first word segment to the start of the speech time of the second word segment when the second word segment follows the first word segment. Also, the pause time indicates the time from the end of the speech time of the second word segment to the start of the speech time of the first word segment when the first word segment follows the second word segment. The electronic device applies the first word segment and the second word segment to a speech synthesis model to identify the pause time between the first word segment and the second word segment. The electronic device compares the magnitude of the pause time with the critical time. Illustratively, the electronic device includes the first word segment and the second word segment in one target phrase if the pause time is less than the critical time. Alternatively, if the pause time is greater than the critical time, the electronic device may include the first and second word segments in different phrases. A method for determining a target phrase by comparing the pause time and the critical time will be described in detail below with reference to FIG. 5.

[0048] In operation 230, the electronic device determines a voice tone change of the target phrase based on information about the pitch contour of voice data acquired by applying the target sentence to a predetermined voice synthesis model and the speaking time of each word segment included in the target phrase. Here, the voice synthesis model refers to a model that, when a sentence is input, outputs speech or voice corresponding to the input sentence. That is, the electronic device acquires voice data by applying the target sentence to the voice synthesis model. However, the voice synthesis model is not limited thereto. For example, the voice synthesis model may be a model that, when a sentence is input, outputs phrases included in the input sentence and each voice tone change of the phrase.

[0049] The electronic device identifies information about the pitch contour of the audio data to obtain the audio tone change. When the electronic device identifies the information about the pitch contour of the audio data, the electronic device determines the audio tone change of the target phrase based on the information about the pitch contour and the speaking time of each word segment. When the audio tone change of the target phrase is determined, the electronic device uses the target sentence including the target phrase for which the audio tone change has been determined as training data for a model for robot gesture generation.

[0050] FIG. 3 illustrates a method for generating gesture training data from a corpus in an electronic device according to one embodiment of the present invention. An electronic device (e.g., electronic device 100 of FIG. 1 ) according to one embodiment includes a first analyzer 320 for speech generation and time analysis in units of word segments, a second analyzer 330 for speech characteristic analysis in units of word segments, and a third analyzer 340 for speech characteristic analysis in units of phrases. The electronic device generates gesture training data 350 from a corpus 310 via the first analyzer 320, the second analyzer 330, and the third analyzer 340.

[0051] Regarding the first analyzer 320, the electronic device identifies a target sentence from the corpus 310. The electronic device applies the target sentence to a speech synthesis model to obtain speech data. The speech data can be used for a speech tone change determination operation via the second analyzer 330. The electronic device then analyzes the word segment unit time included in the target sentence. Specifically, the electronic device identifies the speech time of the word segment. If the speech time of the word segment is identified, the electronic device identifies the pause time between the word segments. The pause time can be used for a target phrase determination operation via the third analyzer 340.

[0052] Regarding the second analyzer 330, the electronic device extracts information about the pitch contour of the speech data. By extracting the information about the pitch contour, the electronic device identifies the degree of tone change of the word segment. The electronic device uses a parselmouth library (e.g., the praat library) to extract the information about the pitch contour. The electronic device performs interpolation on the information about the pitch contour that could not be extracted through the library. Specifically, if the electronic device is unable to identify information about the pitch contour from the speech data, the electronic device applies cubic spline interpolation to the speech data to identify information about the pitch contour of the speech data. For example, the electronic device removes noise from the speech data and segments the speech data into predetermined speech signal analysis unit frames. The electronic device obtains at least one unit frame by segmenting the speech data. The electronic device applies a short-time Fourier transform (STFT) to each unit frame to determine information about the pitch contour of the unit frame. Here, the electronic device applies cubic spline interpolation to unit frames for which information about the pitch contour is not determined.

[0053] Regarding the third analysis unit 340, the electronic device determines a target phrase. Specifically, the electronic device determines the target phrase by comparing the pause time between word segments with the critical time. This will be described in detail later with reference to FIG. 5. When the target phrase is determined, the electronic device determines gesture assignment candidates in the target sentence and assigns gestures to the gesture assignment candidates. When the electronic device determines gestures for the gesture assignment candidates of the target sentence, it generates the target sentence as gesture training data 350.

[0054] FIG. 4 illustrates a method for identifying speech times of word segments and pause times between word segments in an electronic device according to one embodiment of the present invention. According to one embodiment, an electronic device (e.g., electronic device 100 of FIG. 1) identifies speech times of word segments and pause times between word segments in a target sentence. For example, the electronic device identifies speech times of word segments from speech data acquired by applying a predetermined speech synthesis model to the target sentence. Specifically, the electronic device synthesizes speech for the target sentence and calculates speech times on a grapheme-by-grapheme basis. The electronic device identifies speech times of word segments by matching speech speech times on a grapheme-by-grapheme basis to word segment units. In this way, the electronic device calculates speech times while synthesizing speech for the target sentence through the speech synthesis model, thereby reducing errors in speech time calculation or determination. Thereafter, when the speech times of the word segments have been identified, the electronic device identifies pause times between the word segments.

[0055] The electronic device identifies, as a target sentence, the sentence "It was known that Tanaka has a dog" from among the sentences included in the corpus. In the case of the target sentence shown in Figure 4, the target sentence includes a first word segment "Tanaka," a second word segment "dog," a third word segment "keep," a fourth word segment "having," and a fifth word segment "known."

[0056] The electronic device identifies a speech time of a first word segment from the audio data. Illustratively, the speech time of the first word segment includes a section starting at 0 seconds and ending at 0.2 seconds. The electronic device identifies a speech time of a second word segment from the audio data. Illustratively, the speech time of the second word segment includes a section starting at 0.4 seconds and ending at 0.6 seconds. The electronic device identifies a speech time of a third word segment from the audio data. Illustratively, the speech time of the third word segment includes a section starting at 0.9 seconds and ending at 1.2 seconds. The electronic device identifies a speech time of a fourth word segment from the audio data. Illustratively, the speech time of the fourth word segment includes a section starting at 1.3 seconds and ending at 1.5 seconds. The electronic device identifies a speech time of a fifth word segment from the audio data. Illustratively, the speech time of the fifth word segment includes a section starting at 1.8 seconds and ending at 2.1 seconds.

[0057] The electronic device identifies speech times of word segments (e.g., the first word segment to the fifth word segment) and identifies pause times between the word segments. For example, the electronic device determines the interval from the end of speech of the first word segment to the start of speech of the second word segment as a first pause time. The electronic device determines the interval from the end of speech of the second word segment to the start of speech of the third word segment as a second pause time. The electronic device determines the interval from the end of speech of the third word segment to the start of speech of the fourth word segment as a third pause time. The electronic device determines the interval from the end of speech of the fourth word segment to the start of speech of the fifth word segment as a fourth pause time.

[0058] FIG. 5 illustrates a method for determining phrase-wise speech tone changes in an electronic device according to one embodiment of the present invention. According to an embodiment, an electronic device (e.g., the electronic device 100 of FIG. 1) determines a speech tone change for each phrase. For example, the electronic device determines word segments included in a target phrase from among a plurality of word segments included in a target sentence. Then, the electronic device determines a speech tone change for the target phrase based on the speech times of the word segments included in the target phrase.

[0059] The electronic device identifies from the speech data a first speech time (e.g., 0 to 0.2 seconds in FIG. 5 ) of a first word segment (e.g., "Tanaka ga" in FIG. 5 ) and a second speech time (e.g., 0.4 to 0.6 seconds in FIG. 5 ) of a second word segment (e.g., "inu o" in FIG. 5 ). The electronic device determines a first speech tone change (e.g., 0.2) for the first word segment via information about the first speech time and the pitch contour of the first word segment. The electronic device determines a second speech tone change (e.g., 0.2) for the second word segment via information about the second speech time and the pitch contour of the second word segment.

[0060] Specifically, the electronic device determines a first voice tone change by obtaining a voice tone change rate for each predetermined unit time interval from information about the pitch contour of a first word segment corresponding to a first utterance time. For example, the electronic device determines the first voice tone change by adding the change rates of information about the pitch contour of the first word segment for each unit time interval. In other words, the electronic device determines the first voice tone change of the first word segment by adding all absolute values of the voice tone change rates (e.g., slopes) with respect to the utterance time. Similarly, the electronic device determines a second voice tone change by obtaining a voice tone change rate for each predetermined unit time interval from information about the pitch contour of a second word segment corresponding to a second utterance time.

[0061] The electronic device determines a target phrase when it has identified the speech times of the word segments and the pause times between the word segments. For example, the electronic device determines the word segments to include in the target phrase based on a comparison of the pause times between the word segments and a critical time. Specifically, the electronic device determines the target phrase by including the first word segment and the second word segment in the target phrase based on the first pause time (e.g., 0.2) being less than the critical time (e.g., 0.3). Alternatively, the electronic device determines the target phrase by including one of the first word segment and the second word segment in the target phrase based on the first pause time being equal to or greater than the critical time.

[0062] The electronic device determines the positions of the first and second word segments in the speech data based on the first pause being less than the critical time. The electronic device identifies a third word segment (e.g., "keep" in FIG. 5 ) that follows the second word segment from the speech data based on the second word segment following the first word segment (i.e., the second word segment is located after the first word segment). The electronic device determines whether to include the third word segment in the target phrase based on a comparison of the second pause between the second and third word segments (e.g., 0.3) and the critical time (e.g., 0.3). For example, in FIG. 5 , the electronic device may not include the third word segment in the target phrase because the second pause is equal to the critical time. As a result, the electronic device determines a target phrase (e.g., "Tanaka ga inu wo" in FIG. 5 ) that includes the first and second word segments.

[0063] The electronic device determines an average of a first voice tone change (e.g., 0.2) and a second voice tone change (e.g., 0.2) based on whether the first word segment and the second word segment are included in the target phrase. Here, the average may be 0.2 because the first voice tone change and the second voice tone change are identical. The electronic device determines the value obtained by applying the average to a normalization function as the voice tone change of the target phrase. For example, if the determined average is not a value between 0 and 1, the electronic device determines the average included in a section predetermined through the normalization function as the voice tone change of the target phrase. The closer the voice tone change of the target phrase is to 1, the more voice tone change the target phrase may be identified as a section with a greater number of voice tone changes. However, the method by which the electronic device determines the voice tone change of the target phrase is not limited thereto. For example, the electronic device may determine the average of a value obtained by applying a weighted value to the first voice tone change and a value obtained by applying a weighted value to the second voice tone change as the voice tone change of the target phrase.

[0064] FIG. 6 illustrates a method for generating gestures from gesture training data in an electronic device according to one embodiment of the present invention. An electronic device (e.g., electronic device 100 of FIG. 1) according to an embodiment generates gestures from gesture training data 610. For example, the electronic device identifies a target sentence including a target phrase whose voice tone change has been determined using the target phrase determination method described with reference to FIGS. 2 to 5. The electronic device determines the target sentence as gesture training data 610.

[0065] The electronic device applies gesture training data 610 to models scheduled for training 620. For example, the models scheduled for training may include a phrase unit recognition model and a voice tone change prediction model.

[0066] The electronic device trains the model 620 that is scheduled to be trained. Exemplarily, the model 620 that is scheduled to be trained may include a neural network. The neural network may include multiple layers, and each layer may include multiple nodes. The nodes have node values that are determined based on an activation function. Nodes in any layer are connected to nodes (e.g., other nodes) in other layers via links (e.g., connection edges) that have connection weights. The node values of the nodes are propagated to other nodes via the links. In the inference operation of a neural network, node values are propagated forward from the previous layer to the next layer.

[0067] For example, in the model 620 scheduled for training, the forward propagation operation refers to an operation of propagating a node value based on input data from the input layer to the output layer of the model 620 scheduled for training. That is, after being connected to a node via a connecting line, the node value of the node is propagated (e.g., forward propagated) to a node (e.g., a next node) in the layer. For example, a node receives a value weighted by a connection weight from a previous node (e.g., multiple nodes) connected via a connecting line.

[0068] The node value of a node is determined based on applying an activation function to the sum (e.g., weighted sum) of weights previously received from the node. The parameters of the neural network may illustratively include the connection weights described above. The parameters of the neural network are updated so that the objective function value (described below) is changed in a targeted direction (e.g., a direction that minimizes the loss).

[0069] . The trained model refers to a model trained by machine learning. Specifically, the phrase unit recognition model may be a trained machine learning model that outputs a training output (e.g., whether a word segment can be segmented) from a training input (e.g., gesture training data 620). Furthermore, the voice tone change prediction model may be a trained machine learning model that outputs a training output (e.g., voice tone change of a phrase) from a training input (e.g., gesture training data 620).

[0070] The machine learning model (e.g., a trained phrase unit recognition model or a speech tone change prediction model) is generated by machine learning. The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0071] The machine learning model may include multiple artificial neural network layers. Specifically, the model 620 scheduled for training may include a shared layer including at least one convolution operation and multiple classifier layers (e.g., task-specific layers) connected to the shared layer. The artificial neural network may be, but is not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a U-Net for Image Segmentation (U-Net), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination thereof. Furthermore, the model 620 scheduled for training may include, but is not limited to, a support vector machine (SVM), a long short-term memory (LSTM), a bidirectional long short-term memory (Bi-LSTM), or any combination thereof.

[0072] In the case of supervised learning, the machine learning model may be trained based on training data including pairs of training inputs and training outputs mapped to the training inputs. For example, the machine learning model may be trained to output training outputs from training inputs. During training, the machine learning model generates temporary outputs in response to the training inputs and is trained to minimize the loss between the temporary outputs and the training outputs (e.g., training targets). During the training process, parameters of the machine learning model (e.g., connection weights between nodes / layers in a neural network) are updated according to the loss. This training may be performed, for example, in the electronic device where the machine learning model is implemented, or via a separate server. The trained machine learning model (e.g., the trained phrase-unit recognition model or the trained voice tone change prediction model) is stored in a memory (e.g., memory 120 of FIG. 1). A method for training a phrase-unit recognition model will be described in detail below with reference to FIG. 6, and a method for training a voice tone change prediction model will be described in detail below with reference to FIG. 7.

[0073] The electronic device applies the gesture training data 610 to the model 620 to be trained to train a phrase unit recognition model and a voice tone change prediction model. After the training is completed, the electronic device transmits the phrase unit recognition model and the voice tone change prediction model to the robot 630.

[0074] The electronic device identifies a sentence to be output to the user 680 (hereinafter, output sentence) from the dialogue model of the robot 630. The electronic device applies the output sentence to a phrase unit recognition model that determines a gesture allocation interval unit from the trained model 640. The electronic device acquires phrases included in the output sentence based on applying the output sentence to the phrase unit recognition model.

[0075] The electronic device applies each of the phrases included in the output sentence to a speech tone change prediction model based on acquiring the phrases included in the output sentence from the phrase unit recognition model. The electronic device acquires a speech tone change for each of the input phrases based on applying each of the phrases included in the output sentence to the speech tone change prediction model.

[0076] The electronic device performs speech synthesis 650 of the output sentence based on the determined tone change of each phrase included in the output sentence. Specifically, the electronic device generates a voice file to be output by the robot 630 for each phrase for which a tone change has been determined. Then, the electronic device determines the speech start and end points of phrases with high importance by referring to the tone change of each phrase included in the output sentence. Accordingly, the electronic device assigns or provides gestures to phrases with high importance in the output sentence. The method by which the electronic device assigns a gesture to one of multiple phrases will be described in detail below with reference to FIG. 9.

[0077] When a gesture is assigned to a phrase having a high importance in an output sentence, the electronic device generates a gesture file 660 related to the gesture of the phrase. The electronic device controls the robot 630 via the gesture file 660 so that the robot 630 can speak to the user 680 (i.e., so that the robot 630 can output voice and gestures).

[0078] FIG. 7 is a diagram illustrating a method for training a phrase unit recognition model in an electronic device according to an embodiment of the present invention. According to one embodiment, an electronic device (e.g., electronic device 100 of FIG. 1) identifies a window of interest 720 in training data 710, where window of interest 720 includes a first word segment and a second word segment.

[0079] The electronic device obtains a first object vector having a predetermined number of dimensions through word embedding 730 of the object window 720. The first object vector may be a vector including values that abstract features of the object window. For example, the electronic device generates a 100-dimensional vector for each word segment included in the object window 720 through word embedding 730 of the object window 720.

[0080] The electronic device applies the first target vector to the phrase-unit recognition model 740 to obtain an output indicating whether or not the word segments included in the target window have been segmented (e.g., segmentation performed or segmentation not performed). The electronic device trains the phrase-unit recognition model 740 based on a first loss obtained by comparing the output with training data 710 (e.g., the target sentence). Exemplarily, the first loss includes binary cross entropy. Then, the electronic device trains the phrase-unit recognition model 740 by applying a vector obtained through word embedding of a window including the second and third word segments to the phrase-unit recognition model 740.

[0081] FIG. 8 illustrates a method for training a voice tone change prediction model in an electronic device according to an embodiment of the present invention. According to one embodiment, an electronic device (e.g., electronic device 100 of FIG. 1 ) identifies a target phrase 820 in training data 810. The target phrase 820 may be at least one of the phrases included in the training data 810. The electronic device performs word embedding 830 on the target phrase 820 to obtain a second target vector including a vector having a predetermined number of dimensions for each word segment included in the target phrase 820. For example, the second target vector may be a vector including values that abstract the features of the target phrase 820.

[0082] The electronic device applies the second object vector to an encoder 840 that reduces the dimension of an input object, thereby reducing the dimension of the second object vector. Here, the encoder 840 may be one of a PCA (Principal Component Analysis) model or an autoencoder model, but is not limited thereto.

[0083] The electronic device applies the second target vector applied to the encoder (i.e., the second target vector with reduced dimensions) to the voice tone change prediction model 850 to obtain a temporary voice tone change of the target phrase 820. The electronic device trains the voice tone change prediction model 850 based on a second loss obtained by comparing the temporary voice tone change of the target phrase 820 with the voice tone change of the target phrase 820 (e.g., the voice tone change of the phrase in FIG. 5). Illustratively, the second loss may include a mean squared error (MSE) loss.

[0084] FIG. 9 is a diagram illustrating a method for determining gestures on a target sentence in an electronic device according to an embodiment of the present invention. According to one embodiment, an electronic device (e.g., electronic device 100 of FIG. 1) identifies a target sentence 910 that is both an output of a dialogue model and a robot-spoken sentence. The electronic device applies the target sentence 910 to a phrase-unit recognition model (e.g., phrase-unit recognition model 740 of FIG. 7) to perform segmentation 920 of the target sentence 910.

[0085] Based on the segmentation 920 of the target sentence 910, the electronic device determines a speech tone change for each of the phrases included in the target sentence 910. For example, the electronic device applies each of the phrases included in the target sentence 910 to a speech tone change prediction model (e.g., speech tone change prediction model 850 in FIG. 8 ) to determine a speech tone change for each of the phrases included in the target sentence 910.

[0086] Based on the determination of the tone change of each phrase included in the target sentence 910, the electronic device determines the phrase with the greatest tone change among the phrases included in the target sentence 910 as a gesture assignment candidate corresponding to the gesture execution section of the target sentence 910. Referring to FIG. 9, the phrase with the greatest tone change among the phrases included in the target sentence 910 may be "keeping a pet."

[0087] The electronic device determines a gesture of a gesture assignment candidate based on a gesture type corresponding to the gesture assignment candidate. For example, the electronic device determines a gesture type of the gesture assignment candidate "pet." Specifically, the electronic device determines a gesture of a gesture assignment candidate by requesting a server (e.g., server 140 of FIG. 1) to transmit the gesture type. Here, the gesture type is acquired by inputting a keyword included in the gesture assignment candidate into a gesture database stored in the server.

[0088] . The electronic device associates the gestures of the gesture assignment candidates with the speech times of the gesture assignment candidates to generate gestures of the robot scheduled to output the target sentence 910. For example, the electronic device generates a gesture profile 930 related to gestures to be applied to the target sentence 910. The electronic device executes the voice data of the target sentence 910 and the gesture profile 930 through the robot to provide the user with output of the target sentence including gestures.

[0089] FIG. 10 is a diagram illustrating a computing system for an electronic device or control method according to an embodiment of the present invention. Referring to FIG. 10, a computing system 1000 relating to an electronic device or control method includes at least one processor 1100, memory 1300, a user interface input device 1400, a user interface output device 1500, storage 1600, and a network interface 1700, all connected via a bus 1200.

[0090] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that executes processing based on instructions stored in the memory 1300 and / or the storage 1600. The memory 1300 and the storage 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read only memory (ROM) and a random access memory (RAM).

[0091] Thus, the steps of a method or algorithm described in the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by the processor 1100, or in a combination of the two. The software module may reside in a storage medium (i.e., memory 1300 and / or storage 1600) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, or a CD-ROM.

[0092] An exemplary storage medium is coupled to the processor 1100, such that the processor 1100 reads information from and writes information to the storage medium. Alternatively, the storage medium may be integral to the processor 1100. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

[0093] The above description is merely an illustrative example of the technical concept of the present invention, and various modifications and variations can be made by a person having ordinary knowledge in the technical field to which the present invention pertains without departing from the essential characteristics of the present invention.

[0094] The above-described embodiments may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, or other device capable of executing instructions. The processing device executes an operating system (OS) and software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a processing device may be described as being a single device, but those skilled in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

[0095] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may, independently or collectively, instruct a processing device. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by a processing device or to provide instructions or data to a processing device. The software may be distributed across network-coupled computing systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0096] Methods according to embodiments may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the medium may be those specially designed and constructed for the embodiments, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, for example.

[0097] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0098] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations thereto, for example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted by other components or equivalents, and still achieve suitable results. Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

[0099] Therefore, the embodiments disclosed in the present invention are for illustrative purposes only and are not intended to limit the technical idea of the present invention. The scope of protection of the present invention should be interpreted according to the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention. [Explanation of symbols]

[0100] 100 Electronic equipment 110, 1100 processors 120, 1300 memory 122 Imperative 130 Communications Department 140 servers 310 Corpus 320 1st Analysis Department 330 Second Analysis Department 340 Third Analysis Department 350 gesture learning data 610 Gesture Learning Data 620 Learning Scheduled Model 630 Robot 640 trained models 650 Speech Synthesis 660 Gesture File 680 User 710, 810 training data 720 Target Window 730, 830 word embedding 740 Phrase Unit Recognition Model 820 target phrases 840 Encoder 850 Voice Tone Change Prediction Model 910 Target Text 920 Segmentation 930 Gesture Profiles 1000 Computing Systems 1200 Bus 1310 ROM 1320 RAM 1400 User Interface Input Device 1500 User interface output device 1600 Storage 1700 network interface

Claims

1. a memory storing computer-executable instructions; and at least one processor that accesses the memory and executes the instructions; The at least one processor: Identifying a first word segment and a second word segment of a grammatical unit from a target sentence included in a corpus; determining a target phrase including the first word segment and the second word segment based on a comparison of a pause time between the first word segment and the second word segment and a predetermined critical time; and determining a voice tone change of the target phrase based on information about a pitch contour of speech data acquired by applying the target sentence to a predetermined text-to-speech model and an utterance time of each word segment included in the target phrase.

2. The at least one processor: identifying a first speech time of the first word segment and a second speech time of the second word segment from the speech data; determining a first speech tone change of the first word segment via information about the first speech time and the pitch contour of the first word segment; 2. The electronic device of claim 1, further comprising: determining a second speech tone change of the second word segment via information about the second speech time and the pitch contour of the second word segment.

3. The at least one processor: determining the first speech tone change by obtaining a speech tone change rate for each predetermined unit time interval from information about the pitch contour of the first word segment corresponding to the first utterance time; 3. The electronic device of claim 2, wherein the second voice tone change is determined by obtaining a voice tone change rate for each predetermined unit time interval from information about the pitch contour of the second word segment corresponding to the second utterance time.

4. The at least one processor: determining an average of the first speech tone change and the second speech tone change based on the inclusion of the first word segment and the second word segment in the target phrase; 3. The electronic device of claim 2, further comprising applying the average to a normalization function to determine the value obtained as the speech tone change of the target phrase.

5. The at least one processor: including the first word segment and the second word segment in the target phrase based on the pause time being less than the critical time; The electronic device of claim 1 , further comprising: including one of the first word segment and the second word segment in the target phrase based on the pause time being equal to or greater than the critical time.

6. The at least one processor: determining the location of the first word segment and the second word segment in the audio data based on the pause time being less than the critical time; identifying a third word segment from the audio data that follows the second word segment based on the second word segment following the first word segment; 6. The electronic device of claim 5, wherein the electronic device determines whether to include the third word segment in the target phrase based on a comparison of the pause time between the second word segment and the third word segment and the critical time.

7. The at least one processor: obtaining a first target vector having a predetermined number of dimensions through word embedding of a target window including the first word segment and the second word segment; applying the first object vector to a phrase unit recognition model to obtain an output indicating whether a word segment included in the object window is segmented; The electronic device of claim 1 , wherein the phrase unit recognition model is trained based on a first loss obtained by comparing the output and the target sentence.

8. The at least one processor: Obtaining second target vectors including vectors having a predetermined number of dimensions for each word segment included in the target phrase through word embedding of the target phrase; The second target vector is applied to an encoder that reduces the dimension of an input target, thereby reducing the dimension of the second target vector; applying the second target vector applied to the encoder to a voice tone change prediction model to obtain a temporary voice tone change of the target phrase; 2. The electronic device of claim 1, wherein the voice tone change prediction model is trained based on a second loss obtained by comparing the temporary voice tone change of the target phrase with the voice tone change of the target phrase.

9. The at least one processor: Based on the determination of the tone change of each of the phrases included in the target sentence, a phrase having the largest tone change among the phrases included in the target sentence is determined as a gesture assignment candidate corresponding to a gesture execution section of the target sentence; determining a gesture of the gesture assignment candidate based on a gesture type corresponding to the gesture assignment candidate; The electronic device according to claim 1 , wherein the gesture is associated with an utterance time of the gesture assignment candidate to generate a gesture of a robot scheduled to output the target sentence.

10. The at least one processor:

2. The electronic device of claim 1, wherein, based on the fact that information about the pitch contour cannot be identified from the audio data, cubic spline interpolation is applied to the audio data to identify information about the pitch contour.

11. Identifying a first word segment and a second word segment of a grammatical unit from a target sentence included in a corpus; determining a target phrase including the first word segment and the second word segment based on a comparison of a pause time between the first word segment and the second word segment and a predetermined critical time; determining a voice tone change of the target phrase based on information about a pitch contour of speech data acquired by applying the target sentence to a predetermined text-to-speech model and an utterance time of each word segment included in the target phrase.

12. The operation of determining the target phrase includes: identifying a first speech time of the first word segment and a second speech time of the second word segment from the audio data; determining a first speech tone change of the first word segment via information about the first speech time and the pitch contour of the first word segment; and determining a second speech tone change of the second word segment via information about the second speech time and the pitch contour of the second word segment.

13. The operation of determining the target phrase includes: determining the first speech tone change by obtaining a speech tone change rate for each predetermined unit time interval from information about the pitch contour of the first word segment corresponding to the first utterance time; and determining the second voice tone change by obtaining a voice tone change rate for each predetermined unit time interval from information about the pitch contour of the second word segment corresponding to the second utterance time.

14. The operation of determining a speech tone change of the target phrase comprises: determining an average of the first speech tone change and the second speech tone change based on the inclusion of the first word segment and the second word segment in the target phrase; and determining the value obtained by applying the average to a normalization function as the speech tone change of the target phrase.

15. The operation of determining the target phrase includes: including the first word segment and the second word segment in the target phrase based on the pause time being less than the critical time; and including one of the first word segment and the second word segment in the target phrase based on the pause time being equal to or greater than the critical time.

16. The operation of determining the target phrase includes: determining the location of the first word segment and the second word segment in the audio data based on the pause time being less than the critical time; identifying a third word segment from the audio data that follows the second word segment based on the second word segment following the first word segment; and determining whether to include the third word segment in the target phrase based on a comparison of the pause time between the second word segment and the third word segment and the critical time.

17. obtaining a first target vector having a predetermined number of dimensions through word embedding of a target window including the first word segment and the second word segment; applying the first object vector to a phrase unit recognition model to obtain an output indicating whether a word segment included in the object window has been segmented; 12. The control method of claim 11, further comprising: training the phrase unit recognition model based on a first loss obtained by comparing the output and the target sentence.

18. obtaining second target vectors including vectors having a predetermined number of dimensions for each word segment included in the target phrase through word embedding of the target phrase; applying the second target vector to an encoder that reduces the dimension of an input target to reduce the dimension of the second target vector; applying the second target vector applied to the encoder to a tone change prediction model to obtain a temporary tone change of the target phrase; 12. The control method of claim 11, further comprising: training the voice tone change prediction model based on a second loss obtained by comparing the temporary voice tone change of the target phrase and the voice tone change of the target phrase.

19. determining a phrase having the largest tone change among the phrases included in the target sentence as a gesture assignment candidate corresponding to a gesture execution section of the target sentence based on the determined tone change of each phrase included in the target sentence; determining a gesture of the gesture assignment candidate based on a gesture type corresponding to the gesture assignment candidate; The control method according to claim 11, further comprising: an operation of generating a gesture of a robot scheduled to output the target sentence by associating the gesture with the speech time of the gesture assignment candidate.

20. 12. The control method of claim 11, further comprising: applying cubic spline interpolation to the audio data to identify information about the pitch contour based on the fact that information about the pitch contour cannot be identified from the audio data.

Images