Method and apparatus for lip reading model training using tokenizer evaluation

Abstract

A method of operation of an electronic device for training an artificial intelligence model that performs lip reading according to an embodiment of the present invention comprises the steps of: the electronic device extracting a portion of the total data configured for training the artificial intelligence model as representative data and generating a first model using the representative data; and the electronic device generating a second model using the total data. The step of generating the first model may include the steps of applying a plurality of tokenizers to evaluate the performance of each sample model trained and selecting a sample model according to the evaluation result; transferring weights of a standard model stored in advance to the selected sample model; and generating the first model by further training the selected sample model using the representative data.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for learning a lip reading model based on tokenizer evaluation. Background Technology

[0002] Speech recognition technology is being utilized in various industrial fields following recent advancements in artificial intelligence (AI) and deep learning. However, existing speech recognition systems rely on receiving voice data through microphones, which limits recognition accuracy when there is significant ambient noise or when the user's pronunciation is unclear. To address this issue, lip reading technology, which predicts speech by analyzing the user's lip movements, is being researched. Lip reading technology can be applied in diverse fields, such as assistive technology for the hearing impaired, supplementary speech recognition in noisy environments, and the analysis of non-verbal communication in security and surveillance systems.

[0003] Recently, such lip-reading operations have been primarily studied using techniques based on deep learning models composed of encoders and decoders, as shown in Fig. 1. When speech and lip shape images are input into the lip-reading model, the encoder model captures local and global features regarding the visual and speech, and based on these captured features, the decoder model predicts the spoken sentence corresponding to the speaker's lip shape. The sentence prediction operation involves a token indicating the start of the sentence ( <sos>From ) to the end of the sentence token( <eos>The token prediction process can be repeated until ) appears. When predicting each token, greedy decoding, which selects the token with the highest probability at each step, or beam search, which maintains multiple token candidates and predicts tokens step by step to ultimately select the token path with the highest probability, are mainly used. In the sentence prediction process, in addition to token prediction, since the lengths of the input and output data differ, a connectionist temporal classification (CTC) loss is calculated to predict tokens at each step even when location information of each token is unavailable, thereby enabling stable prediction of tokens corresponding to mouth shapes.

[0004] While research is actively underway to improve the accuracy of speech recognition by training these lip reading models with large amounts of data primarily in English environments, there is a problem of performance degradation in other languages ​​due to a lack of publicly available large-scale data.

[0005] The decoder model of the lip reading model sequentially estimates the tokens constituting the sentence using the Beam Search algorithm to achieve high accuracy during sentence prediction. The Beam Search algorithm maintains k high-probability tokens as candidates at each step and predicts the sentence at the end by connecting the tokens with the highest cumulative probability to date. Most other languages ​​represent sentence tokens in subword units and use the Beam Search algorithm to predict sentences based on lip movements. In contrast, because the lip shape recognition model separates sentences into consonants and vowels for Korean, it primarily uses a greedy decoding method that prioritizes selecting the consonants and vowels with the highest probability at each step. The greedy decoding method has the disadvantage that an incorrect prediction in a previous step significantly impacts subsequent steps, leading to a decrease in overall prediction accuracy.

[0006] Since the vocabulary used in recent everyday conversations is limited, research is being published on representing sentences in Korean at the subword level, similar to other languages. However, depending on the subword representation of sentences separated by a tokenizer, the performance of lip-reading models in predicting sentences may degrade, or the lip-reading models may not be properly trained. For example, since speech appears as a continuous signal, it is common to construct tokens at the level of graphemes, phonemes, or characters. On the other hand, since video is recognized based on lip movements, separating sentences at the subword level rather than the phoneme level demonstrates higher recognition efficiency. Therefore, there is a need to explore tokenizers that can appropriately predict sentences corresponding to speech and lip shapes in the Korean language environment. Tokenizer performance can be evaluated by training a lip-reading model with sentences tokenized by each tokenizer and then testing the results. However, training the entire lip-shape video for tokenizer evaluation has limitations in that it requires a significant amount of time and cost.

[0007] Therefore, the present invention proposes a method for recognizing mouth shapes with high accuracy by finding an appropriate subword separation algorithm with only a small amount of data to train a lip reading model, and utilizing a lip reading model trained on a large scale in this process. Prior art literature

[0008] Republic of Korea Registered Patent No. 10-1092820 (Multimodal Interface System Integrating Lip Reading and Voice Recognition) The problem to be solved

[0009] An embodiment of the present invention is designed to solve the problem of reduced prediction performance of lip reading operations due to the existing lack of training data and differences in transformation methods according to the tokenizer, and to build a lip reading model with high performance even with a small amount of data by effectively utilizing existing large-scale trained weights.

[0010] The purposes of the present disclosure are not limited to those mentioned above, and other purposes and advantages of the present disclosure not mentioned may be understood from the following description and will be more clearly understood from the embodiments of the present disclosure. Furthermore, it will be readily apparent that the purposes and advantages of the present disclosure can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0011] A method of operation for an electronic device that trains an artificial intelligence model performing lip reading according to an embodiment of the present invention comprises the steps of: the electronic device extracting a portion of the entire data configured for training the artificial intelligence model as representative data and generating a first model using the representative data; and the electronic device generating a second model using the entire data. The step of generating the first model may include the steps of applying a plurality of tokenizers to evaluate the performance of each trained sample model and selecting a sample model according to the evaluation result; transferring weights of a standard model stored in advance to the selected sample model; and generating a first model by further training the selected sample model using the representative data.

[0012] And the step of the electronic device selecting a sample model according to the evaluation result may include selecting a preset amount of small data from the representative data and tokenizing sentences corresponding to the selected small data by applying each of a plurality of tokenizers; training a plurality of sample models using the tokenized data as training data; and evaluating the performance of the sample models by comparing the weights of the plurality of sample models with the weights of a standard model stored in advance.

[0013] And the step of the electronic device evaluating the performance of the sample models may include the step of extracting the weights of the final encoder from each of the sample models, the step of extracting the weights of the final encoder from the standard model, and the step of evaluating the performance of each of the sample models based on the result of calculating a distance function between the weights of the final encoder extracted from each of the sample models and the weights of the final encoder extracted from the standard model.

[0014] At this time, the distance function may include cosine similarity, Manhattan distance, and Euclidean distance. Furthermore, when the electronic device selects a sample model based on the evaluation result, it may select a model among a plurality of sample models that has the highest value for cosine similarity and the lowest values ​​for Manhattan distance and Euclidean distance.

[0015] According to various embodiments, the electronic device according to an embodiment of the present invention may generate a first model by applying a plurality of tokenizers to evaluate the performance of each of a plurality of sample models learned in the step of generating the first model, and after the step of selecting a sample model according to the evaluation result, further learning the selected sample model using the representative data, and performing transfer learning by applying the encoder weights of a standard model stored in advance to the encoder of the additionally learned sample model.

[0016] In addition, the electronic device may perform the steps of: evaluating the performance of each of a plurality of sample models learned by applying a plurality of tokenizers in the step of generating the second model, selecting a sample model according to the evaluation result, transferring the weights of the first model to the selected sample model, and generating the second model by further training the selected sample model using the entire data.

[0017] In generating the above second model, the step of selecting a sample model based on the evaluation result may include: selecting a preset amount of small data from the total data and tokenizing sentences corresponding to the selected small data by applying a plurality of tokenizers to each of them; training a plurality of sample models using the tokenized data as training data; and evaluating the performance of the sample models by comparing the distance function between the image and voice feature vectors of the plurality of sample models and the image and voice feature vectors of the standard model.

[0018] And in generating a second-order model, the distance function includes cosine similarity, Mahalanobis distance, and Wasserstein distance, and when evaluating the performance of a sample model for generating a second-order model, the electronic device can select a model among the sample models that has the highest value for cosine similarity and the lowest values ​​for Mahalanobis distance and Wasserstein distance. Effects of the invention

[0019] The present invention can improve the prediction performance of a model by applying a plurality of tokenizers to compare and evaluate a learned sample model, and by selecting the optimal sample model to proceed with training. In addition, by transferring the encoder weights of a large-scale learning model, a more sophisticated Rip Reading model can be trained even with a small amount of data, and the training speed can be shortened and data utilization can be maximized by effectively utilizing existing learned weights. Brief explanation of the drawing

[0020] Figure 1 is a diagram illustrating the basic operation process of a general lip reading model. Figure 2 is a diagram illustrating the structure of a general lip reading model. FIG. 3 is a diagram illustrating the configuration of an electronic device according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a data processing flow according to an embodiment of the present invention. FIGS. 5 to 10 are drawings for explaining a representative data learning operation according to an embodiment of the present invention. FIGS. 11 to 13 are drawings for explaining the overall data learning operation according to an embodiment of the present invention. Specific details for implementing the invention

[0021] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the present invention, and the present invention is defined only by the scope of the claims.

[0022] The terms used in this specification are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other components in addition to the components mentioned. Throughout the specification, the same reference numerals refer to the same components, and "and / or" includes each of the mentioned components and all combinations of one or more. Although terms such as "first," "second," etc., are used to describe various components, these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, the first component mentioned below may be the second component within the technical scope of the invention.

[0023] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0024] The terms “part” or “module” as used in the specification refer to software or hardware components, such as FPGAs or ASICs, and “part” or “module” perform certain roles. However, “part” or “module” is not limited to software or hardware. “Part” or “module” may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, by example, “part” or “module” includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and “parts” or “modules” may be combined into a smaller number of components and “parts” or “modules,” or further separated into additional components and “parts” or “modules.”

[0025] Spatially relative terms such as "below," "beneath," "lower," "above," and "upper" may be used to facilitate the description of the relationship between one component and other components as illustrated in the drawings. Spatially relative terms should be understood as encompassing different orientations of components during use or operation, in addition to the orientations depicted in the drawings. For example, if a component depicted in a drawing is inverted, a component described as "below" or "beneath" of another component may be placed "above" of that component. Therefore, the exemplary term "below" may encompass both the lower and upper directions. Components may also be oriented in other directions, and accordingly, spatially relative terms may be interpreted according to the orientation.

[0026] Before describing the operation of the present invention, we will refer to FIG. 2 to briefly explain the configuration of a general lip reading model.

[0027] As illustrated in FIG. 2, a general lip reading model can be configured to include an encoder and a decoder. In this case, the encoder performs the role of converting input data corresponding to lip movements and spoken speech into numbers that the AI ​​can understand. Specifically, the encoder can perform the operation of converting input data (lip shape images, spoken speech) into feature vectors. Taking lip shape images as an example, the encoder can utilize a Convolutional Neural Network (CNN) to extract features of lip shape and movement, and utilize an LSTM (or Transformer) to learn lip change patterns over time, thereby generating a feature vector containing the features of lip movement. Accordingly, the encoder can finally output a feature vector that quantifies the input data. The feature vector thus output is used as an input value when the decoder predicts text.

[0028] And the decoder can perform the operation of finally generating text (sentence) based on the information obtained from the encoder. At this time, the decoder can utilize a language model to predict the first word (token). Language models include, for example, LSTM, Transformer, etc., and can calculate and provide information on the probability of each word or sentence appearing. In this process, the decoder uses a search algorithm such as beam search or greedy decoding to predict the most natural sentence based on the probabilities provided by the language model.

[0029] For reference, greedy decoding is a method that immediately selects the next token (word) with the highest probability at each step, whereas beam search is a method that maintains multiple high-probability token (word) candidates at each step and continuously selects the group of candidates most likely to have the highest probability in the next step. Beam search operates by maintaining the cumulative probabilities of the words selected in the previous step and ultimately selecting the word sequence with the highest cumulative probability. Accordingly, while greedy decoding has the advantage of faster computation speed, it does not guarantee an optimal sentence, whereas beam search is evaluated as having a higher likelihood of generating relatively natural and contextually appropriate sentences.

[0030] In existing research, lip reading models for a specific language (e.g., Korean) have mainly applied greedy decoding methods, and research applying beam search has been limited. The present invention overcomes these limitations and proposes a method for training a more natural and accurate lip reading model for a specific language (Korean) by utilizing beam search.

[0031] Hereinafter, a learning method for a lip reading model according to an embodiment of the present invention will be described with reference to the drawings from Fig. 3 onwards.

[0032] The learning method of a lip reading model according to an embodiment of the present invention is intended to improve the accuracy of lip shape sentence estimation compared to existing methods in both Visual Speech Recognition (VSR) and Audio-Visual Speech Recognition (AVSR) by utilizing a relatively small amount of data images to find an appropriate subword separation method and utilizing a large-scale English model during the learning process. The electronic device for performing the learning method of the present invention may be configured as shown in FIG. 3.

[0033] FIG. 3 is a diagram illustrating the configuration of an electronic device that performs learning of an artificial intelligence model according to an embodiment of the present invention.

[0034] The configuration illustrated in FIG. 3 may refer to the processor of the electronic device. In addition to the processor, the electronic device (100) may be configured to further include memory (not shown) and a network interface (not shown).

[0035] The above electronic device (100) is a device for performing lip reading model learning and may be configured to include a data preprocessing module (110), a representative data learning module (120), and a whole data learning module (130).

[0036] The above data preprocessing module (110) can perform the operation of preprocessing data used for training an improved lip reading model according to an embodiment of the present invention.

[0037] The above data preprocessing module (110) can perform preprocessing operations on image, voice, and text data, as shown in 410 of FIG. 4.

[0038] First, the data preprocessing module (110) can extract lip movement features from image data (lip shape image). For example, the data preprocessing module (110) can extract only the lip portion from the image (using a CNN model, etc.) for training a lip reading model and convert it into a trainable form (e.g., frame conversion, pixel value normalization, etc.).

[0039] In addition, the data preprocessing module (110) can preprocess not only video data but also voice data together and utilize them for lip reading model training. When training an AVSR model, the data preprocessing module (110) can perform operations to organize audio data recorded together with the video (e.g., voice sampling, noise removal, etc.) and convert it into a form suitable for model training (e.g., converting it into a feature vector).

[0040] Additionally, the data preprocessing module (110) can perform preprocessing operations on text data to enable learning on spoken sentences. The preprocessing operations on text data can generate datasets for each tokenizer method by applying various tokenizers to transform the text. The data preprocessing module (110) can separately store the data transformed for each tokenizer method to prepare for training various models. For reference, an example of a sentence being tokenized through a tokenizer is illustrated in FIG. 6.

[0041] At this time, the tokenizer may be configured to include a byte pair encoding (BPE) or unigram algorithm that divides sentences into appropriate units, which is mainly used in existing lip shape recognition models. BPE tokenizes sentences into new subwords by merging pairs of characters with high frequency, while unigram uses a method of pre-defining a set of words overall and selecting sentence tokens from combinations of characters or words with the highest probability of occurrence. At this time, the tokenizer can represent the same sentence as a combination of different tokens by applying various token sizes and tokenization methods that can constitute the sentence. For example, the data preprocessing module (110) can generate a total of six tokenized text sets based on a BPE tokenizer with token sizes of 1200, 1300, and 1400 and a unigram tokenizer with token sizes of 1200, 1300, and 1400 for the same sentence.

[0042] Next, the electronic device (100) may include a representative data learning module (120).

[0043] The above representative data learning module (120) can perform the representative data-based model learning operation (420) shown in FIG. 4.

[0044] As illustrated in 420 of FIG. 4, the representative data learning module (120) can perform sample model (VSR / AVSR) learning (423) based on multiple tokenized data generated by applying multiple tokenizers to a preset number (e.g., 100 to 500) of data (small amount of data) belonging to a representative dataset (421).

[0045] When training with six tokenizers, a total of six sample models can be generated, one for each tokenizer. This is explained with reference to FIG. 7. FIG. 7 illustrates that two types of tokenized data (label 1, label 2) are generated by tokenizing target text labels using the same tokenizer model (BPE) or two types of tokenizers with different token sizes of 1200 and 1500. The representative data training module (120) can be configured so that the tokenized label data acts as the correct answer in the training of each sample model. Accordingly, each sample model adjusts its model weights in the direction of deriving the correct answer.

[0046] In this way, when a sample model is trained based on a small amount of data among the representative data, the representative data training module (120) can select one sample model among the trained sample models based on performance evaluation and identify the tokenizer that generated the training data (tokenized data) used to train the selected sample model as the most appropriate tokenizer.

[0047] At this time, the representative data learning module (120) can perform an operation (425) of calculating and comparing the weights of the sample models and the weights of the standard model (424, large-scale English learning model) as a method of evaluating the performance of the sample models. And the representative data learning module (120) can select a specific sample model that satisfies a preset criterion for the result of the distance function comparison as a model with excellent performance evaluation.

[0048] Subsequently, the representative data learning module (120) can further train (426) the selected sample model using the remaining data, excluding the small amount of data used for training among the representative data. Then, the representative data learning module (120) can generate a first model by transferring the encoder weights of the standard model (424) to the additionally trained sample model.

[0049] To explain the performance evaluation operation of the above sample model in more detail, we will refer to Fig. 5.

[0050] As illustrated in Fig. 5, 530, the representative data learning module (120) can first extract the weights of the final encoder from each model (sample model) trained with a small amount of data to evaluate the performance of each model, and extract the weights of the final encoder from the standard model.

[0051] For reference, the final encoder can be understood as the encoder layer at the very end of the data processing stage among the multiple encoder layers formed by each model, as illustrated in FIG. 8. That is, the representative data learning module (120) can extract the weights of the final encoder layer among the multiple encoder layers formed by each sample model and standard model.

[0052] Subsequently, the representative data learning module (120) can evaluate the performance of each of the sample models based on the result of calculating the distance function between the weights of the final encoder extracted from each of the sample models and the weights of the final encoder extracted from the standard model.

[0053] That is, the representative data learning module (120) can derive a value calculated by a distance function between the encoder weight of each sample model and the encoder weight of the standard model, as illustrated in FIG. 9. For example, the representative data learning module (120) can derive a value calculated by a distance function between the encoder weight of the first sample model and the encoder weight of the standard model, and a value calculated by a distance function between the encoder weight of the second sample model and the encoder weight of the standard model. Then, the representative data learning module (120) can select the most suitable model among the sample models that satisfies a preset criterion by comparing the values ​​calculated by the distance function derived for each sample model.

[0054] At this time, the distance function may include cosine similarity, Manhattan distance, and Euclidean distance. And the representative data learning module (120) can select the model with the shortest distance between the encoder weights and the encoder weights of the standard model (the model with the highest similarity between encoder weights) among the sample models trained with a small amount of data within the representative data as the most suitable model.

[0055] The term "model with the shortest distance" may refer to a model in which the cosine similarity (an indicator of semantic similarity) in the calculated distance function is highest (close to 1), and the Manhattan distance and Euclidean distance (an indicator of overall similarity) are lowest (close to 0). Accordingly, in the operation step of performing a performance evaluation (530) of the sample model of FIG. 5, the representative data learning module (120) may select a model in which the cosine similarity is closest to 1 and the Manhattan distance and Euclidean distance are closest to 0.

[0056] Subsequently, the representative data learning module (120) can generate a primary model (representative data learning model) by transferring the encoder model weights of the standard model to the sample model selected according to performance evaluation, as shown in 540 of FIG. 5, and training the remainder of the representative data excluding the small amount of data used to train the sample model.

[0057] To explain in more detail the operation of transferring the encoder weights of a standard model to a selected sample model (a model trained with small amounts of data), we will refer to Fig. 10.

[0058] As illustrated in FIG. 10, when one of the sample models is selected based on performance evaluation among a plurality of sample models, the encoder weights of the standard model can be transferred to the encoder of the selected sample model (Model 1). Then, the sample model to which the encoder weights of the standard model have been transferred can be formed into a first model by performing additional training based on the remaining data among the representative data that was not used for training the sample model.

[0059] Meanwhile, the operation of transferring the encoder weights of the standard model to the encoder weights of the sample model can mean that the sample model copies and overwrites the standard model's encoder weights exactly. Furthermore, the encoder of each model consists of a single large encoder model and multiple small encoder layer models, and during transfer learning, the weights (a concept including weights and biases) of all small encoder layer models can be transferred and used. In this case, the structure of the encoder model from which the weights are copied (corresponding to the encoder of the standard model) and the target model to which the copied weights are applied (corresponding to the sample model for generating the first-order model) must be identical.

[0060] The representative data learning module (120) of FIG. 3 and the representative data learning operation (420) illustrated in FIG. 4 have been described above. Next, the overall data learning module (130) of FIG. 3 and the overall data learning operation (430) of FIG. 4 will be described. Furthermore, a detailed description of each process of the overall data learning operation (430) will be provided with reference to FIG. 11 to FIG. 13.

[0061] The entire data learning module (130) according to an embodiment of the present invention performs the entire data learning operation shown in 430 of FIG. 4, and accordingly can generate a second model, which is a lip reading model according to an embodiment of the present invention.

[0062] First, the entire data learning module (130) can generate multiple sample models by training (433) a model using text data (432) that has been tokenized based on a tokenizer for a small amount of data corresponding to a portion of the entire text set (431). Then, the entire data learning module (130) can calculate (434) the distribution distance between the mouth shape / voice feature vector of the standard model and the mouth shape / voice feature vector of the sample model to select a specific sample model similar to the standard model among the generated multiple sample models, and based on the calculation result, select and train (435) the sample model with the shortest distribution distance among the multiple sample models. Through this process, the entire data learning module (130) can construct (436) a second model, which is a lip reading model according to an embodiment of the present invention.

[0063] To explain the content illustrated in 430 of FIG. 4 more specifically, looking at FIG. 11, the entire data learning module (130) can perform model learning using lip shape data, voice data, and tokenized text data extracted from the entire image by the data preprocessing module (110) as learning data.

[0064] The entire data learning module (130) can generate multiple sample models (1120) by performing learning based on a small amount of data (e.g., 100 to 500) of a preset number of data from the entire data. Similar to the operation of generating sample models performed by the representative data learning module (120), the entire data learning module (130) can also perform model learning by classifying text labels to be used as training data according to the natural language token size. Accordingly, a number of sample models corresponding to the types of text labels used as training data can be generated.

[0065] Afterwards, the entire data learning module (130) can perform a performance evaluation operation (1130) of the generated sample model, and can select a specific model based on the performance evaluation (an evaluation of whether each model expresses the mouth shape and voice characteristics well).

[0066] The evaluation operation (1130) of the sample model performed by the above-mentioned entire data learning module (130) may specifically include the operation of extracting mouth shape / voice feature vectors for each sample model trained with a small amount of data, and extracting mouth shape / voice feature vectors from the standard model. The evaluation operation may also include the operation of calculating a distance function (cosine similarity, Mahalanobis distance, Wasserstein distance) between the feature vectors extracted from each sample model and the feature vectors extracted from the standard model as shown in FIG. 12, and identifying and selecting the model with the smallest difference in data distribution (meaning the distribution of images and voice feature vectors output from the encoding model) as the suitable model based on the calculated value.

[0067] Specifically, the entire data learning module (130) can identify and select a model as a fit model among a plurality of sample models that has the highest cosine similarity value (close to 1) and the lowest values ​​for Mahalanobis distance and Wasserstein distance (close to 0).

[0068] And when a specific sample model is selected according to performance evaluation, the above-mentioned entire data learning module (130) can transfer the encoder weights of the first model as in 1140 of FIG. 11 and learn the entire data to create a second model.

[0069] In the case where a second model is created by transferring the encoder weights of a first model to a sample model, the weights of the first model (which can be interpreted as including weights and biases) can be transferred (copied and pasted) to a target model (a sample model for creating a second model), just as in the transfer operation that occurs when creating the first model, and additional training can be performed after the weight transfer is completed.

[0070] Specifically, the entire data learning module (130) can learn a selected sample model (e.g., Model 1 of FIG. 13) using the remaining entire data that was not used for learning the sample model among the entire data, as illustrated in FIG. 13. During the process of learning the selected sample model, the entire data learning module (130) can fine-tune the weights of the sample model by transferring the encoder weights of the standard model to the sample model through transfer learning, and accordingly, the sample model can utilize the learning performance regarding mouth shape and speech features learned by the standard model.

[0071] The secondary model generated by this method corresponds to a lip reading model according to an embodiment of the present invention. By providing a tokenizer suitable for text for training a lip shape recognition model, the model can be trained to flexibly generate sentences corresponding to lip shapes. Furthermore, since the lip reading model according to an embodiment of the present invention is trained by transferring the weights of a standard model trained with large-scale data, it can utilize the lip shape and speech expression performance of existing models, thereby allowing for higher accuracy than existing lip reading models.

[0073] In summary, the method of operation of an electronic device for training an artificial intelligence model that performs lip reading according to an embodiment of the present invention comprises the steps of: the electronic device extracting a portion of the total data configured for training the artificial intelligence model as representative data and generating a first model using the representative data; and the electronic device generating a second model using the total data. The step of generating the first model may include the steps of applying a plurality of tokenizers to evaluate the performance of each trained sample model and selecting a sample model according to the evaluation result; transferring the weights of a standard model stored in advance to the selected sample model; and generating a first model by further training the selected sample model using the representative data.

[0074] And the step of the electronic device selecting a sample model according to the evaluation result may include selecting a preset amount of small data from the representative data and tokenizing sentences corresponding to the selected small data by applying each of a plurality of tokenizers; training a plurality of sample models using the tokenized data as training data; and evaluating the performance of the sample models by comparing the weights of the plurality of sample models with the weights of a standard model stored in advance.

[0075] And the step of the electronic device evaluating the performance of the sample models may include the step of extracting the weights of the final encoder from each of the sample models, the step of extracting the weights of the final encoder from the standard model, and the step of evaluating the performance of each of the sample models based on the result of calculating a distance function between the weights of the final encoder extracted from each of the sample models and the weights of the final encoder extracted from the standard model.

[0076] At this time, the distance function may include cosine similarity, Manhattan distance, and Euclidean distance. Furthermore, when the electronic device selects a sample model based on the evaluation result, it may select a model among a plurality of sample models that has the highest value for cosine similarity and the lowest values ​​for Manhattan distance and Euclidean distance.

[0077] According to various embodiments, the electronic device according to an embodiment of the present invention may generate a first model by applying a plurality of tokenizers to evaluate the performance of each of a plurality of sample models learned in the step of generating the first model, and after the step of selecting a sample model according to the evaluation result, further learning the selected sample model using the representative data, and performing transfer learning by applying the encoder weights of a standard model stored in advance to the encoder of the additionally learned sample model.

[0078] In addition, the electronic device may perform the steps of: evaluating the performance of each of a plurality of sample models learned by applying a plurality of tokenizers in the step of generating the second model, selecting a sample model according to the evaluation result, transferring the weights of the first model to the selected sample model, and generating the second model by further training the selected sample model using the entire data.

[0079] In generating the above second model, the step of selecting a sample model based on the evaluation result may include: selecting a preset amount of small data from the total data and tokenizing sentences corresponding to the selected small data by applying a plurality of tokenizers to each of them; training a plurality of sample models using the tokenized data as training data; and evaluating the performance of the sample models by comparing the distance function between the image and voice feature vectors of the plurality of sample models and the image and voice feature vectors of the standard model.

[0080] And in generating a second-order model, the distance function includes cosine similarity, Mahalanobis distance, and Wasserstein distance, and when evaluating the performance of a sample model for generating a second-order model, the electronic device can select a model among the sample models that has the highest value for cosine similarity and the lowest values ​​for Mahalanobis distance and Wasserstein distance.

[0082] An electronic device (100) according to an embodiment of the present invention may include a processor, memory, a communication interface, etc.

[0083] An electronic device may be a server, a user terminal, or a device that performs various other operations and functions. If the electronic device is a server, it may be an FTP (File Transfer Protocol) server, a web server, a database server, or a cloud server, but is not limited thereto. A server may perform one or more functions and operations, and may be implemented as a single device or distributed across multiple devices to implement each function and operation.

[0084] The processor controls the overall operation of the electronic device. Specifically, the processor is connected to the configuration of the electronic device including memory as described above, and can control the overall operation of the electronic device by executing at least one instruction stored in the memory as described above. In particular, the processor can be implemented as a single processor as well as as a plurality of processors.

[0085] A processor may be implemented in various ways. For example, one or more processors may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors may control one or any combination of other components of an electronic device and may perform operations or data processing related to communication. One or more processors may execute one or more programs or instructions stored in memory. For example, one or more processors may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory.

[0086] In the embodiments of the present disclosure, a processor may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0087] Memory stores various programs or data temporarily or non-temporarily and transmits the stored information to the processor upon the processor's call. In addition, memory can store various information necessary for the processor's computation, processing, or control operations in an electronic format.

[0088] Memory may include, for example, at least one of main memory and secondary memory. Main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM. ROM may include, for example, conventional ROM, EPROM, EEPROM and / or MASK-ROM, etc. RAM may include, for example, DRAM and / or SRAM, etc. Secondary memory may be implemented using at least one storage medium capable of storing data permanently or semi-permanently, such as a flash memory device, SD (Secure Digital) card, Solid State Drive (SSD), Hard Disc Drive (HDD), optical recording media such as a magnetic drum, Compact Disc (CD), DVD or laser disc, magnetic tape, magneto-optical disc and / or floppy disk, etc.

[0089] The communication interface may include at least one of a wireless communication interface, a wired communication interface, and an input / output interface. The wireless communication interface may communicate with various external devices using wireless communication technology or mobile communication technology. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared data association (IrDA), or near field communication (NFC), and mobile communication technologies may include 3GPP, Wi-Max, LTE (Long Term Evolution), 5G, etc.

[0090] A wireless communication interface can be implemented using an antenna, a communication chip, a substrate, etc., capable of transmitting electromagnetic waves to the outside or receiving electromagnetic waves transmitted from the outside.

[0091] A wired communication interface can communicate with various devices based on a wired communication network. Here, the wired communication network can be implemented using physical cables, such as, for example, pair cables, coaxial cables, fiber optic cables, or Ethernet cables.

[0092] The input / output interface may be configured to be connectable to another device provided separately from the electronic device, such as an external storage device. For example, the input / output interface may be a Universal Serial Bus (USB) terminal, and may also be any one of the following interfaces: HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), or DVI (Digital Visual Interface). The input / output interface may input or output at least one of audio and video signals. Depending on the implementation example, the input / output interface may include separate ports for inputting and outputting only audio signals and for inputting and outputting only video signals, or it may be implemented as a single port for inputting and outputting both audio and video signals.

[0093] Although the present invention has been described in detail with reference to the examples above, those skilled in the art may make modifications, changes, and variations to these examples without departing from the scope of the invention. In short, it should be noted that in order to achieve the intended effect of the present invention, it is not necessary to separately include all functional blocks shown in the drawings or follow all sequences shown in the drawings exactly as shown, and that such matters may fall within the technical scope of the invention as described in the claims. Explanation of the symbols

[0094] 100 : Electronic device 110 : Data Preprocessing Module 120: Representative Data Training Module 130: Full Data Training Module< / eos> < / sos>

Claims

Claim 1 A method of operation for an electronic device that performs learning of an artificial intelligence model that performs lip reading, wherein the electronic device comprises the steps of: extracting a portion of the total data configured for learning the artificial intelligence model as representative data and generating a first model using the representative data; and generating a second model using the total data; wherein the step of generating the first model comprises: evaluating the performance of each learned sample model by applying a plurality of tokenizers and selecting a sample model according to the evaluation result; transferring weights of a standard model previously stored to the selected sample model; and generating the first model by further training the selected sample model using the representative data. Claim 2 A method of operation according to claim 1, wherein the step of selecting a sample model according to the evaluation result comprises: selecting a preset amount of small data among the representative data and tokenizing sentences corresponding to the selected small data by applying each of a plurality of tokenizers; training a plurality of sample models using the tokenized data as training data; and evaluating the performance of the sample models by comparing the weights of the plurality of sample models with the weights of a standard model stored in advance. Claim 3 A method of operation according to claim 2, wherein the step of evaluating the performance of the sample models comprises: a step of extracting the weights of the final encoder from each of the sample models; a step of extracting the weights of the final encoder from the standard model; and a step of evaluating the performance of each of the sample models based on the result of calculating a distance function between the weights of the final encoder extracted from each of the sample models and the weights of the final encoder extracted from the standard model. Claim 4 In paragraph 3, the distance function includes cosine similarity, Manhattan distance, and Euclidean distance, and the step of selecting a sample model according to the evaluation result includes selecting a model among the plurality of sample models having the highest value for cosine similarity and the lowest values ​​for Manhattan distance and Euclidean distance; a method of operation. Claim 5 A method of operation according to claim 1, wherein the step of generating the second model comprises: evaluating the performance of each of a plurality of sample models trained by applying a plurality of tokenizers and selecting a sample model according to the evaluation result; transferring the weights of the first model to the selected sample model; and generating the second model by further training the selected sample model using the entire data. Claim 6 In claim 5, the step of selecting a sample model according to the evaluation result comprises: selecting a preset amount of small data from the entire data and tokenizing sentences corresponding to the selected small data by applying each of a plurality of tokenizers; training a plurality of sample models using the tokenized data as training data; and evaluating the performance of the sample models by comparing the image and voice feature vectors of the plurality of sample models with the image and voice feature vectors of the standard model. Claim 7 A method of operation according to claim 6, wherein the distance function includes cosine similarity, Mahalanobis distance, and Wasserstein distance, and the step of evaluating the performance of the sample model includes the step of selecting a model among the sample models having the highest value for the cosine similarity and the lowest values ​​for the Mahalanobis distance and Wasserstein distance. Claim 8 A method of operation for an electronic device that trains an artificial intelligence model performing lip reading, wherein the electronic device comprises: a step of extracting a portion of all data configured for training the artificial intelligence model as representative data and generating a first model using the representative data; and a step wherein the electronic device generates a second model using the all data; wherein the step of generating the first model comprises: a step of generating the first model by transferring the weights of a standard model previously stored to a sample model selected according to the evaluation result for each sample model trained by applying a plurality of tokenizers; and the step of generating the second model comprises: a step of generating the second model by transferring the weights of the first model to a sample model selected according to the evaluation result for each sample model trained by applying a plurality of tokenizers. Claim 9 A method of operation for an electronic device that trains an artificial intelligence model performing lip reading, wherein the electronic device comprises: a step of extracting a portion of the total data configured for training the artificial intelligence model as representative data and generating a first model using the representative data; and a step wherein the electronic device generates a second model using the total data; wherein the step of generating the first model comprises: a step of evaluating the performance of each sample model trained by applying a plurality of tokenizers and selecting a sample model according to the evaluation result; a step of transferring the weights of a standard model previously stored in the selected sample model; and a step of generating the first model by further training the selected sample model using the representative data; and wherein the step of generating the second model comprises: a step of evaluating the performance of each of a plurality of sample models trained by applying a plurality of tokenizers and selecting a sample model according to the evaluation result; a step of transferring the weights of the first model to the selected sample model; and a step of generating the second model by further training the selected sample model using the total data. Claim 10 A computer program stored on a computer-readable recording medium, which is executed by a processor of an electronic device to cause the electronic device to perform the operation method of claim 1.

Images