Server device for training translation model, electronic device using trained translation model, and methods therefor

Abstract

A server device for training a translation model, an electronic device using the trained translation model, and methods therefor are disclosed. The electronic device comprises: a memory for storing actual speech spoken in a first language, first text corresponding to the actual speech, second text, which is the target language corresponding to the first text, a communication unit, and a translation model; and a processor, wherein the processor generates synthesized speech corresponding to the first text, and can train the translation model by using the actual speech, the first text, the synthesized speech and the second text.

Description

Server device for training a translation model, electronic device using the trained translation model, and methods thereof

[0001] The present invention relates to a server device that trains a translation model for performing translation between different languages, an electronic device that uses the translation model, and methods thereof.

[0002] To facilitate communication between users who speak multiple languages, the use of translation models that translate between different languages ​​is increasing. In particular, interest in multilingual models that translate between two or more languages ​​is also growing.

[0003] Conventional translation models utilize a cascaded approach. Because the cascaded approach involves two translation models, it suffers from error propagation and latency. Therefore, the need for an end-to-end model emerged to address translation errors and accuracy.

[0004] However, even when using an end-to-end model, pairs of actual spoken voice data and text data in the source language are scarce. This makes training a translation model difficult, and a solution is needed.

[0005] A server device according to at least one embodiment of the present disclosure comprises a processor and a memory. The processor is configured to generate a synthetic voice corresponding to text of an actual voice spoken in a first language, and train the translation model using the actual voice spoken in the first language, a first text corresponding to the actual voice, a synthetic voice corresponding to the first text corresponding to the actual voice spoken in the first language, and a second text in a target language corresponding to the first text.

[0006] According to another embodiment of the present disclosure, an electronic device includes a microphone, a communication unit, a display, a memory storing a VQ codebook learned based on an actual voice and a synthetic voice, and a processor. When a voice signal of a first language is input through the microphone, the processor extracts index information of feature information corresponding to the voice signal from among feature information recorded in the VQ codebook, transmits the index information to a server device through the communication unit, and when information on text of a second language corresponding to the index information is transmitted from the server device, the processor controls the display to display text of the second language.

[0007] According to another embodiment of the present disclosure, a method for training a translation model of a server device may include a step of generating a synthetic voice based on a first text corresponding to an actual voice spoken in a first language, and a step of training a translation model using the actual voice, the first text, the synthetic voice, and a second text in a target language corresponding to the first text.

[0008] FIG. 1 is a drawing for explaining an operation of translating using a translation model in at least one electronic device of the present disclosure.

[0009] FIG. 2 is a block diagram illustrating a configuration of a server device according to at least one embodiment of the present disclosure.

[0010] Figure 3 is a diagram illustrating an example of a method for training a translation model in a server device.

[0011] FIG. 4 is a block diagram showing the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating a translation model learning method of a server device according to at least one embodiment of the present disclosure.

[0013] FIG. 6 is a flowchart illustrating a translation process in a server device according to at least one embodiment of the present disclosure.

[0014] FIG. 7 is a flowchart illustrating a translation process in an electronic device according to at least one embodiment of the present disclosure.

[0015] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0016] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0017] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0018] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0019] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0020] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0021] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0022] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0023] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0024] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0025] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0026] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0027] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0028] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0029] FIG. 1 is a diagram illustrating a translation method according to at least one embodiment of the present disclosure. FIG. 1 illustrates a state in which a server device (100) and an electronic device (200) communicate with each other.

[0030] An electronic device (200) can perform translation between different languages ​​using a translation model. FIG. 1 illustrates a case where a user speaks in Korean (i.e., the source language) and the content is translated into English (i.e., the target language).

[0031] Specifically, in FIG. 1, when a user (20) speaks “Hello” in Korean (30-1), an electronic device (200) receives the spoken voice through a microphone and displays “Hello” (30-2), a translation text in English, which is a target language, on the display of the electronic device (200).

[0032] In the embodiment, the case where the source language spoken by the user (20) is Korean and the target language is English is illustrated, but the source language and the target language can vary widely. The language spoken by the user (20) and the language into which the spoken language of the user (20) is translated, i.e., the source language and the target language, are not limited to a specific type of language. For example, it is acceptable even if the spoken language of the user (20), i.e., the source language, is Russian and the target language is German. The types of the source language and the target language may vary depending on the user's settings. However, in order for the translation model to properly translate, learning about translation between various languages ​​must be performed, and to this end, in various embodiments of the present disclosure, not only actual voices for the source language but also synthetic voices are used together to increase the amount of learning data.

[0033] For convenience of explanation, in this disclosure, the source language may be referred to as the first language and the target language may be referred to as the second language.

[0034] In addition, in FIG. 1, the electronic device (200) is depicted as a mobile phone, but this is for convenience of explanation. Therefore, the electronic device (200) is not limited to a mobile phone, and may be implemented as various devices such as a PC, laptop PC, tablet PC, TV, kiosk, wireless speaker, or robot.

[0035] When a user's (20) first language voice is input through a microphone, the electronic device (200) can acquire text in a second language using a translation model. The electronic device (200) can display the acquired text as is, as shown in Fig. 1, or output it through a speaker in the form of an electronic voice.

[0036] According to various embodiments of the present disclosure, the translation model can be trained using both real-world speech and synthetic speech as training data. The specific training method will be described in detail in the following sections.

[0037] Meanwhile, the learned translation model may be installed on the electronic device (200) or on an external device. Alternatively, some of the software modules constituting the translation model may be installed on the electronic device (200), and other software modules may be installed on the external device.

[0038] In Fig. 1, a case in which a translation model is installed in a server device (100) is shown.

[0039] For example, a user (20) may utter the sentence "Hello" (30-1) while holding an electronic device (200) nearby. The electronic device (200) transmits the voice data uttered by the user to a server device (100). The server device (100) may input the received voice data into a translation model to obtain text in a second language.

[0040] If the target language is set to English, the server device (100) can transmit the text "Hello" (30-2) to the electronic device (200). If the target language is set to French, the server device (100) can transmit the text "Bonjour." If the translation model is implemented as a multi-language model, even if the first language is input in a language other than Korean, if the second language is set to French, the server device (100) can transmit an index of the text "Bonjour."

[0041] Meanwhile, if the electronic device (200) includes at least a part of the software module of the translation model, the electronic device (200) may transmit the result value processed by the software module rather than transmitting the voice data itself spoken by the user to the server device (100). For example, if a VQ codebook trained with actual voice and synthetic voice is stored, the electronic device (200) may search for a feature vector corresponding to the voice spoken by the user in the VQ codebook and transmit an index for the searched feature vector to the server device (200). The server device (200) may transmit a text in a second language corresponding to the received index, and the electronic device (100) may display the transmitted text in the second language.

[0042] A VQ codebook (Vector Quantization Codebook) is used after the feature vector extraction step of multiple voice segments included in a voice signal spoken by a user, and can be a database of vector quantized feature information (i.e., voice feature vectors).

[0043] Meanwhile, as in another example, if the electronic device (200) stores its own translation model, the electronic device (200) may perform the above-described translation operation using the translation model even when communication with the server device (100) is not connected. In Fig. 1, an embodiment in which translated text is displayed on the display of the electronic device (200) is shown, but this is only one embodiment, and the electronic device (200) may also output the translated text as a voice using a TTS (Text to Sound) module.

[0044] FIG. 2 is a block diagram illustrating a configuration of a server device (100) according to at least one embodiment of the present disclosure. According to FIG. 2, the server device (100) may include a communication unit (110), a memory (130), and a processor (120).

[0045] The communication unit (110) is a component for performing communication with various external devices. The communication unit (110) may include at least one wireless communication module, at least one wired communication module, etc. Each communication module may be implemented in the form of at least one hardware chip. For example, the wireless communication module may include at least one module among a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules. In addition, the communication unit (110) may include at least one communication chip for performing communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc. The wired communication module may include, for example, at least one among a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, an optical fiber cable, or a UWB (Ultra Wide-Band) module.

[0046] The memory (130) can store an operating system (OS) for controlling the overall operation of components of the server device (100) and instructions or data related to components of the server device (100).

[0047] The memory (130) may store various programs and data for the operation of the translation learning model in the server device (100). Specifically, the memory (130) may store various software modules, such as a database of feature vectors such as a subsampler, a VQ codebook, an encoder, a discriminator module, a normalization module, a decoder, and the like. Alternatively, the memory (130) may store a synthetic voice generation module for generating synthetic voice, a learning module for learning a translation model, and the like.

[0048] The memory (130) may be implemented in various forms, such as volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0049] In addition, although each configuration of FIG. 2 is illustrated individually, at least one of the communication unit (110), processor (120), and memory (130) may be implemented in multiple units depending on the type or design specifications of the server device (100). In addition, some of the memories (130) may be built into the processor (120).

[0050] The processor (120) is electrically connected to the communication unit (110) and the memory (130), and controls the overall operation of the server device (100) using various commands or programs stored in the memory (130). According to one embodiment, the processor (120) can learn a translation model.

[0051] To train a translation model, the processor (120) may be provided with data regarding actual speech spoken in a first language, text corresponding to the actual speech, and a second text in a second language corresponding to the first text, i.e., a target language second text. For convenience of explanation, the text corresponding to the actual speech may be referred to as the first text, and the text in the target language may be referred to as the second text. For example, if the first text is "Hello," the second text may be "Hello."

[0052] The processor (120) may directly receive actual voice, first text, second text, etc. through various input means such as a microphone, keyboard, mouse, joystick, etc., and store them in the memory (130). Alternatively, data such as actual voice, first text, second text, etc. may be received from an external device, etc., through the communication unit (110).

[0053] The processor (120) can generate a synthetic voice corresponding to the first text. The generation of the synthetic voice can be performed in various ways. For example, the processor (120) can generate the synthetic voice using methods such as waveform synthesis, statistical parameter generation, a speech synthesis system (ASR-TTS), and deep learning-based voice conversion. These generation methods are merely examples and are not limited thereto. Since the method of generating the synthetic voice is a well-known technology, the generation method is omitted.

[0054] The processor (120) trains a translation model using an actual speech, a first text, a synthetic speech, and a second text. In this case, the objectives for training may be set as a first objective of making the difference between the output value of the translation model for the actual speech and the output value of the translation model for the synthetic speech less than a preset threshold, a second objective of distinguishing feature information according to the actual speech and feature information according to the synthetic speech for each text in the first language, etc. The processor (120) trains the translation model in a direction in which each loss function becomes smaller based on the first and second loss functions to achieve these objectives.

[0055] FIG. 3 is a configuration for explaining a translation model learning method of a server device according to at least one embodiment of the present disclosure.

[0056] According to FIG. 3, the memory (130) stores various software modules such as a plurality of encoders and subsamplers (310-1, 300-1, 310-2, 300-2), a VQ codebook (320), a regularizer module (330), a discriminator module (340), a shared encoder (350), a decoder (360), etc.

[0057] One encoder and sub-sampler (310-1, 300-1) may be a software module for real speech, and the other encoder and sub-sampler (310-2, 300-2) may be a software module for synthetic speech.

[0058] The encoder and sub-sampler (310-2, 300-2) for synthetic speech, the regularizer module (330), and the discriminator module (340) may be software modules designed for training a translation model. For convenience of explanation, the encoder and sub-sampler (310-1, 300-1) for real speech are referred to as the first encoder and the first sub-sampler, and the encoder and sub-sampler (310-2, 300-2) for synthetic speech are referred to as the second encoder and the second sub-sampler.

[0059] An encoder and sub-sampler (310-1, 300-1), a VQ codebook (320), a shared encoder (350), and a decoder (360) for real speech can constitute a translation model.

[0060] In Fig. 3, the case of translating a first language into a second language is illustrated, but if the types of languages ​​to be translated increase, the number of encoders and sub-samplers may also increase. For example, in the case of a model that translates from Korean to English, one encoder and one sub-sampler are required for processing actual Korean speech, and one encoder and one sub-sampler are required for processing synthesized speech. However, if a model that translates from French to Korean is also added, one encoder and one sub-sampler are additionally required for processing actual French speech, and one encoder and one sub-sampler are additionally required for processing synthesized speech of French text.

[0061] The processor (120) samples an actual voice signal using the first sub-sampler (300-1) and encodes the actual voice sampled by the first sub-sampler (300-1) using the first encoder (310-1).

[0062] Additionally, the processor (120) samples a synthetic speech signal using a second sub-sampler (300-2), and encodes the synthetic speech sampled by the second sub-sampler (300-2) using a second encoder (310-2). The synthetic speech signal may be generated directly from the first text using a TTS module (not shown), or may be provided from an external device.

[0063] The processor (120) repeatedly trains the translation model until the similarity between the output value of the translation model for the actual speech encoded by the first encoder (310-1) and the output value of the translation model for the synthetic speech encoded by the second encoder (310-2) exceeds a preset threshold.

[0064] In other words, since training a translation model using only actual spoken voices is realistically difficult due to insufficient data, efficient training of the translation model can be achieved by generating synthetic voices and training the translation model.

[0065] To train a translation model, a real speech in a first language, a first text corresponding to the real speech, and a second text corresponding to the first text in the target language are required. The synthetic speech can be generated using models such as Tacotron2 or Fastspeech2. However, these models are only some of the examples, and the processor (120) can generate the synthetic speech using various other methods.

[0066] The acquired real speech spoken in the first language and the synthesized speech of the first text are in the form of analog signals, and therefore require processing into digital signals. The actual spoken speech can be converted into frequency units through a Fourier transform process, and this conversion can be performed through a subsampler and an encoder (310-1, 300-1, 310-2, 300-2). The subsamplers (300-1, 300-2) can perform downsampling. Since each sample of a speech signal is typically composed of thousands to hundreds of thousands of samples per second, processing and storing the speech signal can require a lot of resources. Therefore, downsampling the speech signal may be required for efficient processing.

[0067] The processor (120) samples real voice and synthetic voice using a plurality of subsamplers (300-1, 300-2), converts them into digital signals, and obtains feature information for each digital signal using an encoder (310-1, 310-2). The feature information may be in the form of a feature vector or a list of the vectors.

[0068] Specifically, when the actual voice to be translated and the training voice are acquired, the processor (120) can segment each voice into at least one voice segment using voice activity detection (VAD) technology. The processor (120) can encode the voice segments segmented from each voice using encoders (310-1, 310-2) to acquire feature information thereof. The encoding result of the encoders (310-1, 310-2) of FIG. 3 may be referred to as the output of the encoder. The output of the encoder may be expressed as a hidden layer vector. The processor (120) can extract feature information (i.e., feature vectors) in the same manner not only for the actual voice but also for the synthetic voice.

[0069] However, since the feature vectors, which are the encoding results of each of the actual voice and the synthetic voice, have a very high degree of freedom, there is a need to change them into a discrete form. That is, a VQ codebook (Vector Quantization codebook) (320) may be required to express the changed discrete form.

[0070] In the speech recognition process, vector quantization can be a method that enables the function of compressing data by corresponding the feature vector of the recognized speech to the vector with the closest distance in the codebook, and the function of searching to find the group to which the speech feature vector should belong.

[0071] That is, the feature vectors, which are the encoded results of each of the actual spoken voice and the synthetic voice, can be corresponded to code words within the VQ codebook. Here, a code word means at least one vector within the VQ codebook.

[0072] The processor (120) can update feature vectors extracted from actual spoken speech and synthetic speech with codewords within a VQ codebook. In this case, the processor (120) performs learning based on the first and second loss functions described above so that translation results based on actual and synthetic speech can be similar.

[0073] The processor (120) can learn a translation model by updating the VQ codebook so that the feature vectors of the actual voice and the synthetic voice correspond to similar code words based on a generative adversarial network (GAN)-based method.

[0074] When based on an adversarial generative neural network-based method, the processor (120) can use a generator module (not shown) and a discriminator module (340).

[0075] The discriminator module (340) is trained to distinguish whether the input voice is an actual spoken voice or a synthetic voice, and the generator module is a software module trained to make the characteristic information of the actual voice and the synthetic voice similar enough that the discriminator module cannot determine that it is a synthetic voice.

[0076] That is, the discriminator module is trained to determine whether the input voice is an actual spoken voice or a synthetic voice based on the voice input, and if the input voice is not an actual spoken voice, determine the current learning result as a mismatch, and if the input voice is an actual spoken voice, determine the current learning result as a match.

[0077] In FIG. 3, the processor (120) extracts a codeword corresponding to a feature vector extracted from an actual voice from a VQ codebook (320), and extracts a codeword corresponding to a feature vector extracted from a synthetic voice from a VQ codebook (320). Not only a feature vector but also a codeword may be included in the above-described feature information.

[0078] The processor (120) can update the VQ codebook by generating feature information adjusted so that the feature difference between the actual voice and the synthetic voice is less than a preset threshold based on feature information corresponding to the actual voice and feature information corresponding to the synthetic voice. In this case, a discriminator module (340) may also be used.

[0079] The processor (120) can input a first output value of a translation model that receives actual voice as input and a second output value of a translation model that receives synthetic voice as input to the discriminator module (340). The discriminator module (340) can output a value corresponding to the difference between the first and second output values. The processor (120) can re-update the VQ codebook (340) included in the translation model by comparing the output value of the discriminator module (340) with a threshold value until the output value of the discriminator module (340) becomes less than a threshold value.

[0080] Through this learning, the difference between the feature information corresponding to the actual voice and the feature information corresponding to the synthetic voice can be reduced below a preset threshold, making them similar to each other.

[0081] The processor (120) repeatedly performs the above-described comparison and update operations using the discriminator module (340) and the generator module, which perform opposing operations, thereby increasing the similarity of the translation results of each of the actual voice and the synthetic voice.

[0082] An example of a first loss function that ensures that the difference between the output value of the translation model for actual speech and the output value of the translation model for synthetic speech is less than a preset threshold can be set as follows.

[0083] <Mathematical Formula 1>

[0084]

[0085] First loss function L GAN (i.e. GAN loss function) D is the output value of the discriminator module, is the encoder output value of the synthesized voice, is the encoder output value of the actual voice, is the actual voice, means synthetic voice.

[0086] The processor (120) can learn repeatedly until the value of the first loss function, such as mathematical expression 1, becomes lower than a preset value, so that the feature vectors of the actual spoken voice and the synthetic voice can have similar values.

[0087] As the learning process is repeated N times, the generator module is trained to generate and output synthetic voices similar to actual spoken voices, and the discriminator module is trained to increase the accuracy of distinguishing between actual spoken voices and synthetic voices.

[0088] Meanwhile, when learning solely through the GAN method, there is a possibility that the feature information of the synthesized speech for different texts will be identical. Accordingly, the processor (120) can update the VQ codebook (320) using the normalization module (330) to have different feature information for different texts. The normalization module (330) is a module for learning feature information by classifying it for each text in the first language.

[0089] The processor (120) can input a first output value obtained by actual speech for each text in the first language and a second output value obtained by synthetic speech to the normalization module (330). Based on the output value of the normalization module (330), the processor (120) can update the VQ codebook (320) so that each text has different feature information.

[0090] Specifically, the regularization module (330) can operate in the direction of reducing the second loss function as follows.

[0091] <Mathematical Formula 2>

[0092]

[0093] S represents a function that calculates the cosine similarity between the feature vector of the synthetic voice and the feature vector of the actual spoken voice, and V represents a codeword in the VQ codebook. Specifically, the normalization module (330) is LContrastive By learning so that the value of is less than a preset value, model collapse can be prevented. Model collapse refers to a case where, when learning is performed using the GAN method, texts with different contents are input but correspond to the same codeword in the VQ codebook (320). For example, it refers to a case where texts with different contents, that is, "Hello" and "World of Bugs", are input but correspond to the same codeword in the VQ codebook. Learning by the second loss function is as follows. For example, if the text of an actual spoken voice is "Hello" and the text of a synthetic voice is "Nice to meet you", "Hello" and "Nice to meet you" are texts with different meanings, but it is difficult to view them as texts with completely different meanings in terms of similarity of meaning. Therefore, it is learning so that the text of an actual spoken voice and the text of a synthetic voice correspond to codewords at positions that are not too far apart. As a result of learning by the second loss function described above, a uniform distribution of code words can be achieved.

[0094] At this time, the purpose of the first loss function is to make it difficult to distinguish between actual spoken voice and synthetic voice, and the purpose of the second loss function is to distinguish between feature information according to actual voice and feature information according to synthetic voice according to text of the first language. Therefore, the processor (120) can perform learning based on the first and second loss functions in any order.

[0095] That is, after learning by the first loss function, learning can be performed by the second loss function, or after learning by the second loss function, learning can be performed by the first loss function. Alternatively, learning by the first loss function and learning by the second loss function can be performed sequentially. In addition, in learning by the first loss function, learning of the generator module can be performed, and after learning of the discriminator module, learning of the generator module and discriminator module can be performed alternately.

[0096] The following formula represents an example of a third loss function for optimizing a VQ codebook.

[0097] <Mathematical Formula 3>

[0098]

[0099] According to Equation 3, the Gumbel-softmax loss function (L) is used to optimize the VQ codebook. gumbel-softmax ) and diversity loss function (L diversity ) can be added. When using the Gumbel-softmax loss function, a discrete representation ( by the VQ (vector quantization) module) ) can be a one-hot vector. A one-hot vector is a vector that has 1 only in the position of the corresponding item and fills the remaining items with 0. For example, "apple" can be expressed as [1,0,0,0], and "strawberry" can be expressed as [0,1,0,0].

[0100] Additionally, the diversity loss function can prevent the concentrated selection of specific indices in the VQ codebook. The diversity loss function is a function that is trained to maximize the distance between codewords when corresponding to the VQ codebook.

[0101] The above-described learning method is also applicable to learning about new words. For example, even if a new new word is to be added to the translation model, it may not be realistically possible to train the model using actual human speech for all new words. Therefore, the processor (120) can generate a synthetic voice for the text of the new word using a TTS module. The processor (120) can extract feature information of the generated synthetic voice using the above-described method and perform updates such as adding it to the VQ codebook. When the VQ codebook has been trained for a new word, if the user (20) inputs the new word by speaking it near the microphone of the electronic device (200), the codeword stored in the VQ codebook can be extracted by the synthesized voice. Therefore, even if the user (20) does not speak the new word directly and retrain it each time it occurs, learning can be performed based on the synthesized voice, thereby enabling a rapid provision of a translation service for the new word.

[0102] As described above, the processor (120) can update the VQ codebook (320) using actual speech and synthetic speech, thereby training a translation model including the VQ codebook (320).

[0103] Once training for the VQ codebook (320) is completed, the processor (120) may perform training for the entire translation model including the shared encoder (350), decoder (360), etc.

[0104] That is, as described above, the translation model can be implemented in a form including a first sub-sampler (300-1), a first encoder (310-1), a VQ codebook (320), a shared encoder (350), and a decoder (360).

[0105] The shared encoder (350) can encode speech based on feature information included in a VQ codebook.

[0106] The decoder (360) can extract text in a second language by decoding the feature vector output from the shared encoder (350). The decoder (360) can extract text corresponding to the feature vector using separately prepared dictionary data. While described as a decoder (360), it can also be a shared decoder.

[0107] The processor (120) can perform learning by inputting learning data including actual speech, synthetic speech, text in a first language, text in a second language, etc. for the entire translation model including the shared encoder (350) and decoder (360) in a state where the update of the VQ codebook (320) is completed.

[0108] The translation model learned as described above can be used for translation between multiple languages. While such a translation model may be used while mounted on a server device (100), it is not necessarily limited thereto. The translation model may also be directly mounted on an electronic device (200), or may be mounted on a server device other than the server device (100) that performed the training, and used. In this case, the processor (120) may transmit the trained translation model to the electronic device (200) or other external devices via the communication unit (110).

[0109] Alternatively, only some software modules or VQ codebooks (320) within the translation model may be loaded onto the electronic device (200), and translation may be performed through linkage between the electronic device (200) and the server device (100). In this case, the processor (120) may support translation services by transmitting and receiving various data and signals with the electronic device (200) via the communication unit (110).

[0110] Specifically, assuming that there is at least one electronic device having a translation model including a VQ codebook installed, the processor (120) can receive index information transmitted by the electronic device through the communication unit (110). The index information will be described in detail in the following section.

[0111] When index information is received, the processor (120) extracts feature information corresponding to the index information from a VQ codebook (320) stored in the memory (130), and obtains text in a second language corresponding to the extracted feature information using a shared encoder (350) and a decoder (360). The processor (120) can transmit the obtained text to at least one electronic device via the communication unit (110). The electronic device that receives the text can display it or output it as an electronic voice via a speaker.

[0112] FIG. 4 is a block diagram illustrating the configuration of an electronic device (200) according to at least one embodiment of the present disclosure. As described above, the electronic device (200) may be implemented as various devices such as a mobile phone, a PC, a laptop PC, a tablet PC, a TV, a kiosk, a wireless speaker, a robot, and the like.

[0113] According to FIG. 4, the electronic device (200) includes a communication unit (210), a microphone (240), a display (250), a memory (230), a speaker (260), and a processor (220). However, the present invention is not limited thereto, and the electronic device (200) may be implemented in a form in which some components are excluded, or may be implemented in a form in which other components are further included. Specific examples of the communication unit (210), the processor (220), and the memory (230) have been described in FIG. 2, and therefore, a redundant description thereof will be omitted.

[0114] The microphone (240) can receive the spoken voice when the user (20) actually speaks. The microphone (240) is activated under the control of the processor (220) and can receive the voice signal spoken by the user (20) and various other audio signals. Here, activation may include the overall operation of applying power to the microphone (240) and loading and executing software for performing the microphone function into the memory (230).

[0115] The actual voice is not necessarily limited to the voice spoken by the user (20) near the electronic device (200), but may also be a voice signal output from other devices such as a speaker or telephone. Alternatively, it may be music with lyrics.

[0116] For example, a user (20) can say “hello” into the microphone (240) of an electronic device. The microphone (240) can input the “hello” sound spoken by the user (20).

[0117] The memory (230) may store a translation model including a first subsampler (300-1), a first encoder (310-1), a VQ codebook (320), and other software and data.

[0118] The processor (220) can extract feature information from the voice of the user (20) using the first subsampler (300-1) and the first encoder (310-1). The processor (220) can obtain index information of a codeword corresponding to the extracted feature information based on the VQ codebook (320), and then transmit the obtained index information to the server device (100) through the communication unit (210). The index information may be the codeword itself, or may be a unique index value that can identify the codeword, such as the location of the codeword.

[0119] For example, it can be assumed that the text of the first language spoken by the user (20) is 'world of worms'. If the target language is English, the target text can be 'world of worms'. The processor (220) can obtain index information corresponding to 'world of worms'. That is, if the index value of the codeword corresponding to 'wor' in the VQ codebook (320) is 1, the index value of the codeword corresponding to 'ld' is 2, the index value of the codeword corresponding to 'of' is 3, and the index value of the codeword corresponding to 'ms' is 4, the processor (220) can generate index information in the form of a list [1,2,3,1,4] from the speech 'world of worms'.

[0120] The processor (220) transmits the generated index information to the server device (100) via the communication unit (210). The server device (100) can transmit data on text in a second language corresponding to the transmitted index information.

[0121] When data regarding text in a second language is received through the communication unit (210), the processor (220) can control the display (250) to display text in the second language based on the received data.

[0122] Alternatively, the processor (220) may convert the received data into an electronic voice signal using a TTS module and then output the converted electronic voice signal through a speaker (260).

[0123] The speaker (260) is configured to output various audio signals under the control of the processor (220).

[0124] For example, if a user (20) utters the voice “Hello” and the target language is English, the processor (220) can receive the text “Hello” through the communication unit (210). The processor (220) can control the speaker (260) to output the voice “Hello” or control the display (250) to display it.

[0125] In the above, the case where the processor (220) transmits index information including an index value obtained from a VQ codebook to the server device (100) and then directly receives a text in a second language corresponding to the index information from the server device (100) has been described, but it is not necessarily limited thereto.

[0126] That is, the processor (220) may receive index information corresponding to the text of the second language from the server device (100) through the communication unit (210). Alternatively, if the entire translation model is stored in the memory (230), the processor (220) may directly complete the translation through the VQ codebook stored in the memory (230) without communication with the server device.

[0127] Among these, the process of receiving and translating index information based on the VQ codebook of the server device (100) can be implemented as follows. For example, it can be assumed that one of the index information received from the server device (100) has an index value of 1. Assuming that the vectors of the VQ codebook are codebook vector 1: [0.1, 0.2, 0.3], codebook vector 2: [0.5, 0.6, 0.7], and codebook vector 3: [0.8, 0.9, 0.4], the processor (220) can obtain vector 1 corresponding to the received index value 1.

[0128] The processor (220) can perform encoding and decoding based on the acquired vector 1: [0.1, 0.2, 0.3] to obtain text in a second language.

[0129] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be implemented together in a single product by being combined in whole or in part with at least one other embodiment.

[0130] FIG. 5 is a flowchart illustrating a translation model learning method of a server device according to at least one embodiment of the present disclosure.

[0131] According to one embodiment of the present disclosure, in order to train a translation model of a server device, a synthetic voice can be generated based on a first text. (S510) As described above, the generation of the synthetic voice can use a model such as Fastspeech2. For example, if the actual spoken voice is "Hello" and the corresponding first text is stored, the server device can generate a synthetic voice corresponding to the first text using a model such as Fastspeech2.

[0132] After the synthetic voice is generated, the translation model can be trained using the actual voice, the first text, the synthetic voice, and the second text (S520). That is, the translation model can be trained using the voice actually spoken by the user (20) as “Hello,” the first text as “Hello,” the voice synthesized from the text as “Hello,” and the second text as “Hello” when the target language is English. The training method is described in detail in Fig. 3, and thus, a description of the training method is omitted.

[0133] FIG. 6 is a flowchart illustrating a translation process in a server device according to at least one embodiment of the present disclosure.

[0134] The server device (100) can receive index information of the VQ codebook from the electronic device. (S610) For example, if index information for the voice "Hello" is received from the electronic device, the server device

[0135] A feature vector corresponding to the received index can be obtained from the VQ codebook of the server device (100). (S620) That is, since the VQ codebooks installed in the server device (100) and the electronic device (200) are the same, even if only index information is received from the electronic device (200), the server device (100) can obtain a feature vector based on the received index information.

[0136] The server device can acquire text in the target language by processing the acquired feature vector using a shared encoder and decoder (S640, 650). This process has been described in detail in the above section, so a detailed explanation will be omitted.

[0137] When a text in the target language is obtained, the server device can transmit the text in the target language or index information corresponding to the text to the electronic device. (S650) In FIG. 6 and above, the calculation process of S610 to S640 is described as being performed in the server device (100), but is not limited thereto, and a case may also be included in which a decoder module and a translation module are mounted in the electronic device (200) to obtain a text translated into the target language without communication with the server device (100).

[0138] FIG. 7 is a flowchart illustrating a translation process in an electronic device according to at least one embodiment of the present disclosure.

[0139] In the electronic device (200), an actual voice can be acquired through a microphone. (S710) The electronic device can extract an index of a VQ codebook corresponding to the actual voice. (S720) For example, if a user speaks “hello” into the microphone of the electronic device, the voice can be corresponded to [0.2, 0.3, 0.4] in the VQ codebook. If the vectors of the VQ codebook are codebook vector 1: [0.1, 0.2, 0.3], codebook vector 2: [0.5, 0.6, 0.7], and codebook vector 3: [0.8, 0.9, 0.4], [0.2, 0.3, 0.4] is closest to codebook vector 1, and thus can correspond to codebook vector 1. That is, [0.2, 0.3, 0.4] can have an index value of 1 in the VQ codebook vector. In other words, the index information of the VQ codebook corresponding to the actual speech “Hello” can be 1.

[0140] The electronic device can transmit the index of the VQ codebook to the server device. (S730)

[0141] Since the server device (100) is equipped with the same VQ codebook as the electronic device (200), it can obtain a feature vector from the VQ codebook based on index information received from the electronic device. After obtaining the feature vector, the server device can obtain text in the target language through encoding and decoding processes. Accordingly, when the server device transmits text or index information therefor, the electronic device can receive it. (S740)

[0142] An electronic device can display or output text in the target language through a display or speaker based on the translated text or its index information. (S750) In this case, if the target language is English, the received text may be "Hello." If the target language is French, the received text may be "Bonjour." Of course, there are no restrictions on the type of language.

[0143] The various methods described in FIGS. 5, 6, and 7 can be performed by the server device of FIG. 2 and the electronic device of FIG. 4 described above, but are not necessarily limited thereto, and can also be performed by devices of various configurations.

[0144] Meanwhile, as described above, learning of a translation model and translation using the translation model can be performed by a processor (120) of a server device or a processor (220) of an electronic device.

[0145] These processors (120, 220) may be composed of one or more processors. Each processor (120, 220) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processors described above.

[0146] CPUs are general-purpose processors capable of performing not only general calculations but also artificial intelligence calculations. Their multi-layered cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. General-purpose processors are not limited to the examples described above, except where specifically identified as CPUs.

[0147] A GPU is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where they are specifically referred to as GPUs.

[0148] An NPU is a processor specialized in artificial intelligence computation using artificial neural networks, and each layer of the artificial neural network can be implemented in hardware (e.g., silicon). Since NPUs are designed specifically according to the company's specifications, they have less freedom than CPUs or GPUs, but can efficiently process the AI ​​computations requested by the company. Meanwhile, as a processor specialized in AI computation, an NPU can be implemented in various forms, such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), or a Vision Processing Unit (VPU). Except as specifically designated as an NPU, an AI processor is not limited to the examples described above.

[0149] Additionally, each processor (120, 220) may be implemented as a System on Chip (SoC). In this case, the SoC may further include, in addition to one or more processors, a memory, and a network interface such as a bus for data communication between the processor and the memory.

[0150] When a plurality of processors are included in a SoC (System on Chip) included in an electronic device (200), the electronic device (200) may perform operations related to artificial intelligence (for example, operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors. For example, the electronic device (200) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors. However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor.

[0151] In addition, the processor (120) of the server device may be implemented as a multi-core processor (e.g., dual core, quad core, etc.) included in a single processor. Accordingly, various tasks, such as learning a translation model as described above or translation using the same, can be performed. In particular, the server device (100) can perform artificial intelligence operations, such as convolution operations and matrix multiplication operations, in parallel using the multi-core processor (120).

[0152] The translation model used in various embodiments of the present disclosure may be an end-to-end model. Unlike cascaded methods, end-to-end models perform translation based on a single model, thereby improving translation speed and reducing error propagation, a problem inherent in cascaded methods. According to various embodiments of the present disclosure, not only real speech but also synthetic speech can be used as training data to perform the learning necessary to improve the performance of the translation model. This allows for effective training of the translation model, ultimately improving translation accuracy.

[0153] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0154] A learning algorithm is a method for training a target device using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0155] In addition, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user (20) devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0156] The methods according to various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored from the storage medium and operate according to the called commands, and may include a server device or an electronic device according to the disclosed embodiments.

[0157] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory readable recording medium. Here, the term "non-transitory readable recording medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0158] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0159] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. For server devices, A memory storing an actual speech spoken in a first language, a first text corresponding to the actual speech, a second text corresponding to the first text in a second language, and a translation model; and Processor; including; The above processor, Generate a synthetic voice corresponding to the first text above, A server device that trains the translation model using the actual voice, the first text, the synthetic voice, and the second text.

2. In paragraph 1, The above translation model includes a VQ codebook (Vector Quantization codebook) that includes feature information of the voice, A server device in which the processor generates feature information adjusted so that a feature difference between the real voice and the synthetic voice is less than a preset threshold based on feature information corresponding to the real voice and feature information corresponding to the synthetic voice, thereby updating the VQ codebook.

3. In paragraph 2, The above memory is, A discriminator module for distinguishing the difference between the above real voice and the above synthetic voice is stored, The above processor, The first output value of the translation model that has received the actual voice as input and the second output value of the translation model that has received the synthetic voice as input are input to the discriminator module, A server device that compares the output value of the discriminator module with the threshold value until the output value of the discriminator module becomes less than the threshold value, and re-updates the VQ codebook included in the translation model.

4. In paragraph 3, The above memory is, A regularizer module is stored to learn the above feature information by classifying it by text of the first language. The above processor, A server device that inputs the first output value and the second output value obtained for each text of the first language into the normalization module, and updates the VQ codebook so that each text has different feature information based on the output value of the normalization module.

5. In paragraph 4, The above memory is, A first sub-sampler for sampling the actual speech signal; A first encoder for encoding real speech sampled from the first sub-sampler; A second sub-sampler for sampling a synthetic voice signal converted from the above first text using a TTS module; A second encoder for encoding the synthetic voice sampled from the second sub-sampler; storing; The above processor, A server device that repeatedly trains the translation model until the similarity between the output value of the translation model for the real speech encoded by the first encoder and the output value of the translation model for the synthetic speech encoded by the second encoder exceeds a preset threshold.

6. In paragraph 5, The above translation model is, A shared encoder for encoding speech based on feature information included in the above VQ codebook; and Further comprising a decoder for extracting text in a second language by decoding a feature vector output from the shared encoder based on the above dictionary data; The above processor, A server device that trains the translation model using the actual voice and the synthetic voice, after the update of the above VQ codebook is completed.

7. In paragraph 6, Including the Department of Communications, The above processor, When index information is received through the communication unit from at least one electronic device having a translation model including the above VQ codebook installed, A server device that extracts feature information corresponding to the index information from the VQ codebook stored in the memory, generates text in a second language corresponding to the extracted feature information using the shared encoder and the decoder, and transmits the generated text to the at least one electronic device through the communication unit.

8. In electronic devices, mike; Department of Communications; display; A memory storing a VQ codebook learned based on real and synthetic voices; and Processor; including; The above processor, When a voice signal of the first language is input through the microphone, index information of feature information corresponding to the voice signal is extracted from the feature information recorded in the VQ codebook, and the index information is transmitted to the server device through the communication unit. An electronic device that controls the display to display text in the second language when information about text in a second language corresponding to the index information is transmitted from the server device.

9. In a learning method of a server device for learning a translation model, A step of generating a synthetic voice based on a first text corresponding to an actual voice spoken in a first language; and A learning method, comprising: a step of training a translation model using the actual voice, the first text, the synthetic voice, and a second text in a target language corresponding to the first text.

10. In paragraph 9, The above translation model includes a VQ codebook (Vector Quantization codebook) that includes feature information of the voice, The above learning steps are: A learning method, comprising: a step of updating the VQ codebook by generating feature information adjusted so that a feature difference between the real voice and the synthetic voice is less than a preset threshold based on feature information corresponding to the real voice and feature information corresponding to the synthetic voice.

11. In paragraph 10, The above learning steps are: A step of obtaining a first output value of the translation model that has received the actual voice as input, and a second output value of the translation model that has received the synthetic voice as input, respectively; A step of inputting the first output value and the second output value to a discriminator module for distinguishing the difference between the real voice and the synthetic voice; A learning method, comprising: a step of comparing the output value of the discriminator module with the threshold value and re-updating the VQ codebook included in the translation model until the output value of the discriminator module becomes less than the threshold value.

12. In paragraph 11, The above learning steps are: A learning method further comprising: a step of inputting the first output value and the second output value obtained for each text of the first language into a normalization module, and updating the VQ codebook so that each text has different feature information based on the output value of the normalization module.

13. In paragraph 12, The above learning step comprises: a step of sampling an actual voice signal; A step of encoding the sampled real voice using a first encoder; A step of sampling the synthesized voice converted from the first text using a TTS module; A step of encoding the sampled synthetic voice using a second encoder, and A learning method, comprising: a step of repeatedly training the translation model until the similarity between the output value of the translation model for the real speech encoded by the first encoder and the output value of the translation model for the synthetic speech encoded by the second encoder exceeds a preset threshold.

14. In paragraph 13, The above translation model is, A shared encoder for encoding speech based on feature information included in the above VQ codebook; and Further comprising a decoder for extracting text in a second language by decoding a feature vector output from the shared encoder based on the above dictionary data; The above learning steps are: A learning method, comprising: a step of learning the translation model using the actual voice and the synthetic voice, after the update of the VQ codebook is completed.

15. A non-transitory readable recording medium storing a program for performing a learning method of a server device that trains a translation model, The above learning method is, A step of generating a synthetic voice based on a first text corresponding to an actual voice spoken in a first language; and A non-transitory readable recording medium, comprising: a step of training a translation model using the actual voice, the first text, the synthetic voice, and a second text in a target language corresponding to the first text.

Images