Adaptive Visual Speech Recognition

By using embedded vectors and visual speech recognition neural network to train adaptive models, the problems of high computational intensity and model over-fitting and under-fitting when adapting to new speakers are solved, and fast adaptation and low-cost visual speech recognition model adaptation are achieved. match.

CN117121099BActive Publication Date: 2024-12-03GDM HOLDINGS LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202280025056.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-06-18
Filing Date
2022-06-15
Publication Date
2024-12-03
Estimated Expiration
2042-06-15

AI Technical Summary

Technical Problem

The existing visual speech recognition neural networks require a large amount of training data when adapting to new speakers, and the calculation intensity is high, making it difficult to implement on consumer hardware. The multi-speaker model often overfits and underfits the training data.

Method used

By using multiple embedding vectors and visual speech recognition neural networks to train adaptive models, learn the embedding vectors of new speakers, and optionally fine-tune neural network parameters, the adaptation process takes only a few minutes of video data, reduces the computational intensity, and is suitable for Consumer hardware.

Benefits of technology

It realizes rapid adaptation of the new speaker model, reduces computational costs, avoids overfitting and underfitting problems of the multi-speaker model, and can perform the adaptation process on low-power hardware.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117121099B_ABST
    Figure CN117121099B_ABST
Patent Text Reader

Abstract

Methods, systems and apparatus for processing video data using an adaptive visual speech recognition model, including a computer program encoded on a computer storage medium. One of the methods includes: receiving a video including a plurality of video frames depicting a first speaker; obtaining a first embedding representing the first speaker; and processing a first input including (i) the video and (ii) the first embedding using a visual speech recognition neural network having a plurality of parameters, wherein the visual speech recognition neural network is configured to process the video and the first embedding according to training values ​​of the parameters to generate a speech recognition output defining a sequence of one or more words spoken by the first speaker in the video.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] This specification relates to visual speech recognition neural networks.

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. In addition to an output layer, some neural networks include one or more hidden layers. The output of each hidden layer is used as input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters.

[0003] One example of a neural network is a visual speech recognition neural network. A visual speech recognition neural network decodes speech based on the movements of a speaker's mouth. In other words, a visual speech recognition neural network takes as input a video of a speaker's face and generates as output text representing the words spoken by the speaker depicted in the video.

[0004] An example of a visual speech recognition neural network is LipNet. LipNets were originally described in LipNet: End-to-End Sentence-Level Lipreading by Assael et al. in arXiv preprint arXiv:1611.01599 (2016), available at arxiv.org. LipNet is a deep neural network that maps variable-length sequences of video frames to text using spatiotemporal convolutions and recurrent neural networks.

[0005] Another example of a visual speech recognition neural network is described in Large-Scale Visual Speech Recognition by Shillingford et al., in arXiv preprint arXiv:1807.05612 (2018), available at arxiv.org. Large-Scale Visual Speech Recognition describes a deep visual speech recognition neural network that maps lip videos to a sequence of phoneme distributions and a speech decoder that outputs a sequence of words based on the phoneme distribution sequence generated by the deep neural network. Summary of the invention

[0006] This specification describes a system implemented as a computer program on one or more computers in one or more locations that can generate a sample-efficient and adaptive visual speech recognition model. In this context, being sample-efficient and adaptive means that the model can be customized to recognize the speech of a new speaker with far less training data than was used to train the adaptive model. For example, while training the adaptive model may require several hours of video recording of each individual speaker, adapting the model for a new speaker may only require a few minutes of video recording of the new speaker.

[0007] The training system can train a visual speech recognition model using multiple embedding vectors and a visual speech recognition neural network for corresponding individual speakers. Due to the computationally intensive nature of the training process, the training can be performed by a distributed computing system, such as a data center, having hundreds or thousands of computers.

[0008] The output of the training process is an adaptive visual speech recognition model that can be efficiently adapted to new speakers. Adapting the model typically involves learning new embedding vectors for the new speaker, and may optionally involve fine-tuning the parameters of the neural network for the new speaker. The adaptation data may be just a few seconds or minutes of video of the new speaker and the corresponding transcription to text. For example, the video may be a video of the speaker as the speaker speaks the text on a text prompt presented to the user on the user device.

[0009] The adaptation process is therefore much less computationally intensive than the original training process. As a result, the adaptation process can be performed on much less powerful hardware - such as a mobile phone or another wearable device, a desktop or laptop computer, or another Internet-enabled device installed in the user's home, to name just a few examples.

[0010] In one aspect, a method includes: receiving a video including a plurality of video frames depicting a first speaker; obtaining a first embedding characterizing the first speaker; and processing a first input including (i) the video and (ii) the first embedding using a visual speech recognition neural network having a plurality of parameters, wherein the visual speech recognition neural network is configured to process the video and the first embedding according to training values ​​of the parameters to generate a speech recognition output defining a sequence of one or more words spoken by the first speaker in the video.

[0011] In some implementations, the visual speech recognition neural network is configured to: generate additional input channels from the first embedding; and combine the additional channels with one or more frames in the video before processing the frames in the video to generate the speech recognition output.

[0012] In some implementations, the visual speech recognition neural network includes a plurality of hidden layers, and wherein the neural network is configured to, for at least one of the hidden layers: generate an additional hidden channel from the first embedding; and combine the hidden channel with an output of the hidden layer before providing the output of the hidden layer for processing by another hidden layer of the visual speech recognition neural network.

[0013] In some implementations, the method further includes obtaining adaptation data for the first speaker, the adaptation data comprising one or more videos of the first speaker and a corresponding true value transcription for each of the videos; and determining a first embedding for the first speaker using the adaptation data.

[0014] In some embodiments, the method further comprises obtaining pre-trained values ​​of model parameters that have been determined by training a visual speech recognition neural network on training data including training examples corresponding to multiple speakers different from the first speaker, wherein determining the first embedding comprises determining the first embedding using the pre-trained values ​​and the adaptation data.

[0015] In some embodiments, determining the first embedding includes: initializing the first embedding; and updating the first embedding by repeatedly performing the following operations: processing each video clip in one or more video clips in the adaptation data and the first embedding using a visual speech recognition neural network according to current values ​​of parameters to generate a corresponding speech recognition output for each of the one or more video clips; and updating the first embedding to minimize a loss function that measures, for each of the one or more video clips, a corresponding error between a true value transcription of the video clip and a corresponding speech recognition output for the video clip.

[0016] In some embodiments, updating the first embedding to minimize a loss function that measures, for each of the one or more video clips, a corresponding error between a true value transcription of the video clip and a corresponding speech recognition output for the video clip includes: back-propagating a gradient of the loss function through a visual speech recognition neural network to determine a gradient of the loss function with respect to the first embedding; and updating the first embedding using the gradient of the loss function with respect to the first embedding.

[0017] In some implementations, the current value is equal to the pre-trained value and is equal to the trained value, and wherein the model parameters are fixed when determining the first embedding.

[0018] In some implementations, the operations further include updating current values ​​of parameters of the visual speech recognition neural network based on a gradient of the loss function relative to the parameters of the visual speech recognition neural network, and wherein the training value is equal to the current value after determining the first embedding vector.

[0019] In some implementations, the method further includes applying a decoder to the speech recognition output for the video to generate a sequence of one or more words spoken by the first speaker in the video.

[0020] In some implementations, the speech recognition output includes, for each of the video frames, a corresponding probability distribution over the vocabulary of text elements.

[0021] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0022] The adaptive visual speech recognition model as described in this specification can be used to quickly adapt to new speakers using orders of magnitude less data than used to train the model. This enables the adaptation process to be performed by the end user's consumer hardware rather than in a data center.

[0023] Furthermore, when trained on large datasets of videos representing multiple speakers, multi-speaker visual speech recognition models tend to underfit a large number of data samples from the training data. This can be due to small imbalances in the collected video data, or even due to the limited capacity of any model to capture all the different scenes represented in the large video dataset. The described techniques address these issues by first training a speaker-conditioned visual speech recognition model conditioned on (i) the speaker's video and (ii) the speaker's embeddings, and then adapting the speaker-conditioned visual speech recognition model by learning embeddings for new speakers (and optionally fine-tuning the model's weights).

[0024] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a diagram illustrating an example architecture for training an adaptive visual speech recognition model.

[0027] Figure 2 is a diagram illustrating an example architecture for adapting an adaptive visual speech recognition model to new individual speakers.

[0028] Figure 3 is a flow chart of an example process for building and using an adaptive visual speech recognition model.

[0029] Like reference numerals and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0030] Figure 1 is a diagram illustrating an example architecture 100 for training an adaptive visual speech recognition model.

[0031] The architecture 100 includes a visual speech recognition neural network 110a that is trained using an embedding table 120 that stores embedding vectors for a plurality of different corresponding individual speakers.

[0032] The visual speech recognition neural network 110a may be any suitable visual speech recognition neural network that receives as input a video of a speaker and an embedding vector for the speaker as described below, and processes the video and embedding vector to generate as output a speech recognition output representing a predicted transcription of speech spoken by the speaker in the video.

[0033] As used in this specification, "video" includes only a sequence of video frames and does not include any corresponding audio for the sequence of video frames. Therefore, the visual speech recognition neural network 110a generates speech recognition output without access to any audio data of the speech actually spoken by the speaker.

[0034] An example of a visual speech recognition neural network is LipNet. Lipnets were originally described in LipNet: End-to-End Sentence-Level Lipreading by Assael et al. in arXiv preprint arXiv:1611.01599 (2016), available at arxiv.org. LipNet is a deep neural network that maps variable-length sequences of video frames to text using spatiotemporal convolutions and recurrent neural networks.

[0035] Another example of a visual speech recognition neural network is described in Large-Scale Visual Speech Recognition by Shillingford et al., in arXiv preprint arXiv:1807.05612 (2018), available at arxiv.org. Large-Scale Visual Speech Recognition describes a deep visual speech recognition neural network that also utilizes spatiotemporal convolutions and recurrent neural networks to map lip videos to a sequence of distributed phonemes.

[0036] In general, either of the two visual speech recognition neural network architectures described above, or any other visual speech recognition neural network architecture, can be modified to accept as input an embedding vector and a video depicting a speaker, such as the speaker's face or just the speaker's mouth or lips, at each of a plurality of time steps. The architecture can be modified to process the embedding vector in any of a variety of ways.

[0037] As an example, the system can generate additional input channels using an embedding vector having the same spatial dimension as a video frame, and then combine the additional channels with the video frame, e.g., by concatenating the additional channels to each video frame along the channel dimension (e.g., as if the input channels were intensity values ​​of colors other than the colors (e.g., RBG) of the video frame). For example, the system can generate the additional channels using the embedding vector by applying a broadcasting operation to the values ​​in the embedding vector to generate a two-dimensional spatial mapping having the same dimension as the video frame.

[0038] As another example, the system can generate an additional hidden channel using an embedding vector having the same spatial dimension as an output of a particular hidden layer in the hidden layers of the neural network, such as one of the spatiotemporal convolutional layers in the neural network, and then combine the additional hidden channel with the output of the particular hidden layer, such as by concatenating the additional channel to the output of the hidden layer along the channel dimension, by adding the additional channel to the output of the hidden layer, by element-wise multiplication of the additional channel with the output of the hidden layer, or by applying a gating mechanism between the output of the hidden layer and the additional channel.

[0039] Figure 1 The components shown in may be implemented by a distributed computing system including multiple computers coordinating to train the visual speech recognition neural network 110a.

[0040] The computing system may train the visual speech recognition neural network 110a on training data 130 including a plurality of training examples 132. Each training example 132 corresponds to a respective speaker and includes (i) a video of the respective speaker 140 and (ii) a respective ground truth transcription 150 of speech spoken by the respective speaker in the video.

[0041] Each speaker corresponding to one or more training examples 132 has a corresponding embedding vector stored in embedding table 120. An embedding vector is a vector of numerical values, such as floating point values ​​or quantized floating point values, having a fixed dimension (number of components).

[0042] During training, the embedding vector for a given speaker can be generated in any of a number of ways.

[0043] As an example, an embedding vector may be generated based on one or more characteristics of a speaker.

[0044] As a specific example, a computer system may use an embedding neural network—e.g., a convolutional neural network that has been trained to generate embeddings of faces that can be used to distinguish between people or reflect other attributes of people's faces—to process one or more images of a speaker's face, e.g., images cropped from a video of the speaker in corresponding training examples, to generate a facial embedding vector for the speaker's face. For example, a computer system may use an embedding neural network to process multiple images of the speaker's face to generate a corresponding image embedding vector for each image, and then combine, e.g., average, the image embedding vectors to generate a facial embedding vector for the speaker's face.

[0045] As another specific example, a computer system may measure certain attributes of a speaker's appearance while speaking, and map each measured attribute to a corresponding attribute embedding vector, e.g., using a predetermined mapping. One example of such an attribute is how often the mouth opens while speaking. Another example of such an attribute is the maximum degree to which the mouth opens while speaking. Yet another example of such an attribute is the average degree to which the mouth opens while speaking.

[0046] When the system generates a corresponding embedding vector for each of multiple characteristics of a speaker—e.g., a facial embedding vector and one or more corresponding attribute embedding vectors—the system can combine, e.g., average, sum, or concatenate, the corresponding embedding vectors for the multiple characteristics to generate an embedding vector for the speaker.

[0047] As another example, the system may randomly initialize each speaker embedding (embedding vector) in embedding table 120, and then update the speaker embeddings in conjunction with the training of neural network 110a.

[0048] At each iteration of training, the system samples a mini-batch of one or more training examples 132, and for each training example 132, uses the neural network 110a to process the corresponding speaker video 140 in the training example and the embedding vector for the corresponding speaker from the embedding table 120 to generate a predicted speech recognition output for the training example 132, such as a probability distribution over a set of text elements, such as characters, phonemes, or word fragments.

[0049] The system then trains the neural network 110a using a gradient-based technique, such as stochastic gradient descent, Adam, or rmsProp, to minimize a loss function that measures, for each training example 132 in the mini-batch, the respective error between the true value transcription 150 of the speech for the training example 132 and the predicted speech recognition output for the training example 132. For example, the loss function can be a connectionist temporal classification (CTC) loss function. The CTC loss is described in more detail in Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks by Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber, International Conference on Machine Learning, 2006, pp. 369-376.

[0050] In some embodiments, the system also updates the embedding vectors for the speaker videos in the mini-batch, for example, by back-propagating gradients through the neural network 110a and into the appropriate embedding vectors. More specifically, when the embedding vectors in the embedding table 120 are randomly initialized, the system also updates the embedding vectors. When the embedding vectors in the embedding table 120 are generated based on characteristics or attributes of the corresponding speaker, in some embodiments, the system keeps the embedding vectors fixed during training, while in other embodiments, the system fine-tunes the embedding vectors by updating the embedding vectors in the embedding table 120 in conjunction with the training of the neural network 110a.

[0051] After training, to generate a transcription for a new speaker video, the system (or another system) can process the new speaker video and the new embedding for that speaker as input to generate a predicted speech recognition output for the new speaker video. Optionally, the system can then apply a decoder to the predicted speech recognition output, such as a beam search decoder or a finite state transformer (FST) based decoder to map the speech recognition output to a sequence of words.

[0052] However, while the characteristics of a speaker are used to generate embeddings at training time, these characteristics will generally not be available for a new speaker. Therefore, before using the trained neural network 110a to generate transcriptions for the new speaker, the system can adapt the trained neural network 110a to the new speaker.

[0053] Figure 2 is a diagram illustrating an example architecture 200 for adapting an adaptive visual speech recognition model to a new individual speaker. During the adaptation process, the embedding of the new speaker is adjusted so that the neural network 110a is adapted to the characteristics of the specific individual. In other words, Figure 1 The purpose of the training process shown is to learn a prior. During adaptation, this prior is combined with new data to quickly adapt to the characteristics of the new speaker.

[0054] Typically, a distributed computing system with multiple computers is used to execute Figure 1 The training process shown. And as mentioned above, the adaptation process can be performed on hardware with much lower computational cost, such as a desktop computer, laptop computer, or mobile computing device. For convenience, the adaptation process will be discussed as being performed by a system of one or more computers.

[0055] The architecture 200 includes a visual speech recognition neural network 110b, for example, corresponding to the one already described above using reference Figure 1 A trained version of the visual speech recognition neural network 110a is trained using the process described above.

[0056] To adapt the model to a new individual speaker, the system uses adaptation data 220 of a set of videos 230 ("video clips") representing the new individual speaker speaking and corresponding transcriptions 240 of text spoken in each video. For example, each video 230 may be a video of the speaker captured while the speaker speaks text written on a text prompt or otherwise presented to the user on the user's device.

[0057] Typically, the adaptation data 220 used for the adaptation process can be several orders of magnitude smaller than the training data 130 used for the training process. In some implementations, the training data 130 includes multiple hours of video recording for each of a plurality of different individual speakers, while the adaptation process 220 can use less than ten minutes of video recording of a new individual speaker.

[0058] Furthermore, the adaptation process is typically much less computationally intensive than the training process. Thus, as indicated above, in some embodiments, the training process is performed in a data center with tens or hundreds or thousands of computers, while the adaptation process is performed on a mobile device or a single Internet-enabled device.

[0059] To begin the adaptation phase, the system may initialize a new embedding vector 210 for the new speaker. Typically, the new embedding vector 210 may be different from any embedding vector used during the training process.

[0060] For example, the system can initialize new embedding vector 210 randomly or using any available data that characterizes the new speaker. In particular, the system can initialize new embedding vector 210 randomly or from adaptation data 220 using one of the techniques described above for generating speaker embeddings in table 120. Even when one of the techniques described above for generating embeddings using characteristics of a speaker is used, since adaptation data 220 typically has less data for any given speaker than is available in the training data, the newly generated embedding vector 210 will typically provide less information about the speaker's speech than the speaker embedding vector used during training.

[0061] The adaptation process can be performed in a variety of ways. In particular, the system can use non-parametric techniques or parametric techniques.

[0062] Non-parametric techniques involve using adaptation data 220 to adapt new speaker embedding 210, and optionally model parameters of neural network 110b, or both.

[0063] In particular, when performing the non-parametric technique, at each iteration of the adaptation phase, the system processes one or more video clips 230 in the adaptation data 220 and the current embedding vector 210 for the new speaker using the neural network 110b to generate a predicted speech recognition output for each video clip 230.

[0064] The system then updates the embedding vector 210 by back-propagating the gradient of the loss—e.g., CTC loss—between the true value transcription 240 of the video clip 230 and the predicted speech recognition output of the video clip 230 through the neural network 110b to calculate the gradient relative to the embedding vector 210, and then updates the embedding vector 210 using an update rule—e.g., a stochastic gradient descent update rule, an Adam update rule, or an rmsProp update rule.

[0065] In some of these cases, the system keeps the values ​​of the model parameters of the neural network 110b fixed during the adaptation phase. In other of these cases, the system also updates the model parameters at each iteration of the adaptation phase, for example, by using a gradient-based technique to update the model parameters using the same loss used to update the embedding vector 210.

[0066] Alternatively, the system can use a parameterization technique that involves training an auxiliary network to predict embedding vectors for new speakers using a set of demonstration data, e.g., a set of videos different from the videos in the training data used to train the neural network 110b. The trained auxiliary neural network can then be used to predict embedding vectors 210 for new speakers given the speaker's adapted data 220.

[0067] Figure 3 is a flow chart of an example process 300 for generating and using an adaptive visual speech recognition model. As described above, the process includes three phases: training, adaptation, and inference.

[0068] Typically, the training phase is performed on a distributed computing system with multiple computers.

[0069] And, as noted above, the other two stages can be performed on hardware that is much less computationally expensive, such as a desktop computer, laptop computer, or mobile computing device.

[0070] For convenience, the example process 300 will be described as being performed by a system of one or more computers, but it should be understood that different steps of the process 300 may be performed by different computing devices having different hardware capabilities.

[0071] The system generates an adaptive visual speech recognition model (310) using training data of videos representing speech of multiple different individual speakers. Figure 1 As described, the system can generate different embedding vectors for multiple individual speakers. The system can then use training data to train parameter values ​​of a neural visual speech recognition model, the training data including text and video data representing portions of text spoken by multiple different individual speakers. Each embedding vector typically represents a corresponding feature of one of the multiple different individual speakers.

[0072] The system uses the adaptation data representing the video of speech spoken by the new individual speaker to adapt the adaptive visual speech recognition model for the new individual speaker (320). Figure 2 As described, the adaptation process uses video data representing the portion of the text spoken by the new individual speaker.

[0073] During the adaptation phase, the system can use the adaptation data to generate new embedding vectors for new speakers and optionally fine-tune model parameters of the trained visual speech recognition neural network.

[0074] After adaptation, the system performs an inference process to convert the video of the new speaker and the embedding of the new speaker into a transcription of the text spoken in the video (330). Typically, the system uses a visual speech recognition model adapted for the new individual speaker, which includes using the new embedding vector for the individual speaker determined during the adaptation phase and the new video as input. As described above, the system can generate a transcription as a sequence of words and, in some cases, punctuation marks by applying a decoder to the speech recognition output generated by the adapted visual speech recognition model. Performing inference can also include one or more of: displaying the transcription in a user interface, translating the transcription into another language, or providing audio data representing a verbalization of the transcription for playback on one or more audio devices.

[0075] Although the above description describes adaptive visual speech recognition, the described techniques can also be applied to generate an adaptive visual speech recognition model, where the input to the model is a video sequence of a speaker speaking and a corresponding audio sequence of the speaker speaking (although the two may not be aligned in time with each other) and an embedding of the speaker, and the output is a transcription of the text spoken in the audio-video pair. An example of an audio-visual speech recognition model that can be modified to accept embeddings as input and can be adapted as described above is described in Recurrent Neural Network Transducer for Audio-Visual Speech Recognition in arXiv preprint arXiv:1911.04890 (2019) by Makino et al., available at arxiv.org. When the input also includes audio data, speaker embeddings can also or instead be generated using the audio data, for example, by processing the audio using an audio embedding neural network that has been trained to generate embeddings that uniquely identify the speaker.

[0076] The proposed adaptive visual speech recognition model or audio visual speech recognition model is useful, for example, for hearing-impaired users to generate text representing what the new speaker said so that the user can read it. In another example, the user can be a new speaker, and the visual speech recognition model or audio visual speech recognition model can be used in a dictation system to generate text, or can be used in a control system to generate text commands implemented by another system, such as an electronic system or an electromechanical system. The adaptive visual speech recognition model can be implemented as performing steps 320 and / or 330 by a computer system that includes at least one camera for capturing video of the new speaker processed in steps 320 and / or 330. In the case of an audio visual speech recognition model, the camera can include a microphone for capturing an audio track accompanying the captured video.

[0077] Where the systems discussed herein utilize data that potentially includes personal information, the data may be processed in one or more ways, such as aggregation and anonymization, prior to storage or use of the data, such that such personal information cannot be determined from the data stored or used. Furthermore, the use of such information may be such that personally identifiable information cannot be determined from the output of a system that uses such information.

[0078] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0079] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0080] Computer programs, also referred to or described as programs, software, software applications, apps, modules, software modules, scripts or codes, may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may but need not correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple collaborative files, for example, files storing one or more modules, subroutines or code portions. A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0081] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform specific operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform specific operations or actions means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0082] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and it may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0083] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or computers.

[0084] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs or ASICs, or by a combination of special purpose logic circuits and one or more programmed computers.

[0085] A computer suitable for executing a computer program may be based on a general or special purpose microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into a special purpose logic circuit. Typically, a computer will also include one or more mass storage devices, or be operably coupled to receive data from one or more mass storage devices or transfer data to the one or more mass storage devices, or both, the one or more mass storage devices being used to store data, such as a disk, a magneto-optical disk, or an optical disk. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0086] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0087] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to a user; and a keyboard and a pointing device, such as a mouse or a trackball, through which a user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, sending a web page to a web browser on a user's device in response to a request received from a web browser. In addition, a computer may interact with a user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and in turn receiving a response message from the user.

[0088] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0089] Machine learning models can be implemented and deployed using a machine learning framework—for example, the TensorFlow framework.

[0090] Embodiments of the subject matter described in this specification may be implemented in a computing system including a back-end component such as a data server, or including a middleware component such as an application server, or including a front-end component such as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or including any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0091] A computing system may include a client and a server. The client and the server are typically remote from each other and typically interact via a communication network. The relationship of the client and the server is generated by means of computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server sends data, e.g., an HTML page, to a user device, e.g., for the purpose of displaying data to a user interacting with a device used as a client and receiving user input from the user. Data generated at the device, e.g., the result of a user interaction, may be received at the server from the user device.

[0092] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the combination may be deleted from the claimed combination in some cases, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0093] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0094] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for adaptive visual speech recognition, the method comprising: receiving a video including a plurality of video frames depicting a first speaker; obtaining a first embedding characterizing the first speaker, the first embedding being an embedding vector representing one or more characteristics of the first speaker and determined using adaptation data for the first speaker, the adaptation data comprising one or more video clips of the first speaker and a corresponding ground-truth transcription for each of the one or more video clips; as well as A first input comprising (i) the video and (ii) the first embedding is processed using a visual speech recognition neural network having a plurality of parameters, wherein the visual speech recognition neural network is configured to process the video and the first embedding according to trained values ​​of the parameters to generate a speech recognition output defining a sequence of one or more words spoken by the first speaker in the video.

2. The method according to claim 1, wherein: The visual speech recognition neural network is configured to: generating additional input channels from the first embedding; as well as The additional channel is combined with one or more frames in the video before processing the frames in the video to generate the speech recognition output.

3. The method according to claim 1, wherein: The visual speech recognition neural network comprises a plurality of hidden layers, and wherein the neural network is configured to, for at least one of the hidden layers: generating an additional hidden channel from the first embedding; as well as The hidden channels and the output are combined before providing the output of the hidden layer for processing by another hidden layer of the visual speech recognition neural network.

4. The method according to claim 1, further comprising: Obtaining pre-trained values ​​for model parameters that have been determined by training the visual speech recognition neural network on training data including training examples corresponding to a plurality of speakers different from the first speaker, wherein determining the first embedding comprises determining the first embedding using the pre-trained values ​​and the adaptation data.

5. The method according to claim 4, wherein: Determining the first embedding includes: Initializing the first embedding; and The first embedding is updated by repeatedly performing the following operations: processing each of the one or more video segments in the adaptation data and the first embedding using the visual speech recognition neural network according to the current value of the parameter to generate a corresponding speech recognition output for each of the one or more video segments; and The first embedding is updated to minimize a loss function that measures, for each of the one or more video segments, a corresponding error between a ground-truth transcription of the video segment and a corresponding speech recognition output for the video segment.

6. The method according to claim 5, wherein: Updating the first embedding to minimize a loss function that measures, for each of the one or more video segments, a corresponding error between a ground-truth transcription of the video segment and a corresponding speech recognition output for the video segment comprises: back-propagating the gradient of the loss function through the visual speech recognition neural network to determine the gradient of the loss function with respect to the first embedding; and The first embedding is updated using the gradient of the loss function with respect to the first embedding.

7. The method according to claim 5, wherein: The current value is equal to the pre-trained value and equal to the trained value, and wherein the model parameters are fixed when determining the first embedding.

8. The method according to claim 5, wherein: The operations also include: The current value of the parameter of the visual speech recognition neural network is updated based on a gradient of the loss function with respect to the parameter of the visual speech recognition neural network, and wherein the training value is equal to the current value after determining the first embedding.

9. The method according to any one of claims 1 to 8, further comprising: A decoder is applied to the speech recognition output for the video to generate a sequence of one or more words spoken by the first speaker in the video.

10. The method according to any one of claims 1 to 8, wherein: The speech recognition output includes, for each video frame of the plurality of video frames, a corresponding probability distribution over a vocabulary of text elements.

11. A system for adaptive visual speech recognition, the system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform the operations of the method according to any one of claims 1 to 10.

12. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 10.

Citation Information

Patent Citations

  • Visual speech recognition by phoneme prediction

    CN112041924A

  • Method, system, and computer-readable medium for recognizing speech using depth information

    CN112639964A

  • Automotive visual speech recognition

    US20210065712A1