Visual speech recognition based on connectionist temporal classification loss

Abstract

An electronic device and method for visual speech recognition based on connectionist temporal classification (CTC) loss are disclosed. The electronic device receives a video that includes a human speaker and generates a prediction that corresponds to lip movements of the human speaker. The prediction is generated based on applying a deep neural network (DNN) to the video, and the DNN is trained using a CTC loss function. Based on the prediction, the electronic device detects word boundaries in a sequence of characters that correspond to the lip movements, and divides the video into a sequence of video clips based on the detection. Each video clip corresponds to a word spoken by the human speaker. The electronic device generates a sequence of word predictions by processing the sequence of video clips, and generates a sentence or phrase based on the generated sequence of word predictions.

Description

[0001] Cross-references to related applications / merging by reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 689,270, filed March 8, 2021, with the U.S. Patent and Trademark Office, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 262049, filed October 4, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various embodiments of this disclosure relate to speech recognition and deep neural networks. More specifically, various embodiments of this disclosure relate to electronic devices and methods for visual speech recognition based on connectionist temporal classification loss. Background Technology

[0004] Advances in neural networks have led to the development of various techniques for speech recognition. For example, speech-to-text models exist that can process audio to generate subtitles. Subtitles may not always be flawless, especially when audio is recorded in noisy environments or when the speaker in the video has imperfect pronunciation. Another option for generating subtitles includes analyzing the lip movements of the speaker in the video. However, most conventional techniques for lip reading perform poorly in detecting word boundaries and rely on the audio portion of the video to extract word boundaries.

[0005] As illustrated in the remainder of this application and with reference to the accompanying drawings, other limitations and disadvantages of conventional and traditional methods will become apparent to those skilled in the art by comparing the described system with certain aspects of this disclosure. Summary of the Invention

[0006] Essentially as shown in at least one of the accompanying drawings and / or described in conjunction with at least one of the accompanying drawings and set forth more fully in the claims, an electronic device and method for visual speech recognition based on connectionist temporal classification loss are provided.

[0007] These and other features and advantages of this disclosure will become apparent from the following detailed description of the disclosure, taken in conjunction with the accompanying drawings, in which the same reference numerals refer to the same parts throughout. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating an exemplary network environment for visual speech recognition based on connectionist temporal classification loss, according to an embodiment of this disclosure.

[0009] Figure 2This is a block diagram illustrating an exemplary electronic device for visual speech recognition according to an embodiment of the present disclosure.

[0010] Figure 3 This is a diagram illustrating an exemplary processing pipeline for visual speech recognition according to an embodiment of the present disclosure.

[0011] Figure 4 This is a flowchart illustrating exemplary operation for visual speech recognition based on connectionist temporal classification loss according to an embodiment of the present disclosure. Detailed Implementation

[0012] Implementations of the following can be found in the disclosed electronic devices and methods for visual speech recognition based on connectionist temporal classification (CTC) loss. An exemplary aspect of this disclosure provides an electronic device that can provide a method for visual speech recognition based on CTC loss. The electronic device can be configured to receive a video including one or more human speakers and generate predictions corresponding to the lip movements of the one or more human speakers in the video. The predictions can be generated based on applying a deep neural network (DNN) to the video (frame-by-frame). The loss function supervising the training of the DNN can be a combination of two separate functions: a connectionist temporal classification (CTC) loss function and a cross-entropy loss function. The predictions can be fed into the CTC loss function to find the best (or optimal) path with the highest probability. Another loss function can be used during the training phase to compare the predictions (e.g., the prediction matrix) with the actual alignment. The two loss functions combined can improve the accuracy and convergence speed of the DNN. Unlike known CTC loss functions, the CTC function of this disclosure can configure the DNN during the training phase to predict word boundaries corresponding to video frames in addition to character predictions.

[0013] For each image frame of the video, the prediction comprises a set of probability values ​​corresponding to a set of class labels (e.g., characters including at least one of word characters (i.e., letters), non-word characters, whitespace, and pseudo-characters). This allows the disclosed electronic device to detect redundant (e.g., repeated) image frames in the video that may not be desirable, or blank image frames that may exist between each word. Based on the prediction, the electronic device can be configured to detect one or more word boundaries in a character sequence corresponding to lip movement. Each such word boundary can be defined by a word character at a first end of the character sequence and a non-word character or character interval at a second end of the character sequence. The prediction allows the disclosed electronic device to detect word boundaries based on CTC loss and lip movement analysis, thereby eliminating the need to use audio information associated with the received video. Traditionally, audio information has been used as prior knowledge in word boundary prediction because known lip-reading techniques perform poorly in word boundary detection. This disclosure eliminates the need to use audio information in word boundary prediction.

[0014] The electronic device can be configured to divide a video into a sequence of video clips based on detection. Each video clip in the sequence can correspond to a word spoken by one or more human speakers. The electronic device can also be configured to generate a word prediction sequence by processing the video clip sequence. Subsequently, the electronic device can be configured to generate sentences or phrases based on the generated word prediction sequence. The generation of sentences or phrases may, for example, include concatenating the word prediction sequence using character spacing or non-word characters included between neighboring word predictions in the word prediction sequence.

[0015] Figure 1 This is a block diagram illustrating an exemplary network environment for visual speech recognition based on connectionist temporal classification loss, according to an embodiment of this disclosure. (Refer to...) Figure 1 The figure illustrates a network environment 100. Network environment 100 may include an electronic device 102, a server 104, and a video source 106. Electronic device 102, server 104, and video source 106 may be communicatively coupled to each other via a communication network 108. Electronic device 102 may also include a display device 110. A deep neural network (DNN) 112 associated with electronic device 102 is also shown in network environment 100. The figure also shows a video 114 that may include one or more human speakers 114A as part of multiple image frames associated with a scene.

[0016] Electronic device 102 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to receive video 114 from video source 106 and implement visual speech recognition based on applying DNN 112 to each frame of video 114. In an exemplary embodiment, electronic device 102 may be a media player with display functionality, and display device 110 may be included in electronic device 102. Examples of such implementations of electronic device 102 may include (but are not limited to) televisions (TV), Internet Protocol TV (IPTV), smart TVs, smartphones, personal computers, laptops, tablets, wearable electronic devices, or any other display device capable of receiving, decoding, and playing content encapsulated in signals from cable television or satellite networks, over-the-air broadcasts, or the Internet. In another exemplary embodiment, electronic device 102 may be a media player capable of communicating with display device 110 via a wired or wireless connection. Examples of such implementations of electronic device 102 may include (but are not limited to) digital media players (DMPs), mini game consoles, TV tuners, Advanced Television Systems Committee (ATSC) 3.0 tuners, set-top boxes, over-the-top (OTT) players, digital media streamers, media extenders / modulators, digital media hubs, computer workstations, mainframe computers, handheld computers, smart appliances, plug-in devices, and / or any other computing devices with content streaming capabilities.

[0017] Server 104 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to store video 114 and can be used to train DNN 112 using CTC loss and cross-entropy loss functions. According to one embodiment, server 104 may be implemented as a cloud server and can perform operations via web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Other exemplary implementations of server 104 may include (but are not limited to) database servers, file servers, content servers, web servers, application servers, mainframe servers, or cloud computing servers.

[0018] In at least one embodiment, server 104 can be implemented as multiple distributed cloud-based resources using several techniques well known to those skilled in the art. Those skilled in the art will understand that the scope of this disclosure is not limited to implementing server 104 and electronic device 102 as two separate entities. In some embodiments, the functionality of server 104 can be wholly or at least partially incorporated into electronic device 102 without departing from the scope of this disclosure.

[0019] Video source 106 may include appropriate logic, circuitry, and interfaces configured to transmit video 114 to electronic device 102. Video 114 on video source 106 may include one or more human speakers 114A. In one embodiment, video source 106 may be implemented as a storage device storing video 114. Examples of such implementations of video source 106 may include (but are not limited to) pen drives, USB flash drives, hard disk drives (HDDs), solid-state drives (SSDs), and / or secure digital cards (SD cards). In another embodiment, video source 106 may be implemented as a media streaming server that can transmit media content to electronic device 102 via communication network 108. In yet another embodiment, video source 106 may be a TV tuner, such as an ATSC tuner, that can receive digital TV (DTV) signals from an over-the-air broadcast network and extract media content from the received DTV signals. Video source 106 may then transmit the extracted video 114 to electronic device 102.

[0020] exist Figure 1 In this disclosure, video source 106 and electronic device 102 are shown as two separate devices. However, this disclosure is not limited thereto, and in some embodiments, the functionality of video source 106 may be fully or at least partially incorporated into electronic device 102 without departing from the scope of this disclosure.

[0021] Display device 110 may include appropriate logic, circuitry, and reception that can be configured to display output generated by electronic device 102. For example, display device 110 may be used to display video 114 received from electronic device 102. Display device 110 may also be configured to display subtitles including sentences or phrases corresponding to the lip movements of human speakers(s) in video 114. Display device 110 may interface with or connect to electronic device 102 via an I / O port (such as a High Definition Multimedia Interface (HDMI) port) or a network interface. Alternatively, display device 110 may be an embedded component of electronic device 102.

[0022] In at least one embodiment, the display device 110 may be a touchscreen that allows a user to provide user input via the display device 110. The display device 110 may be implemented using several known technologies, such as (but not limited to) at least one of liquid crystal display (LCD) displays, foldable or rollable displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies, or other display devices. According to one embodiment, the display device 110 may refer to a display screen of a head-mounted device (HMD), a smart glasses device, a see-through display, a projection-based display, an electrochromic display, or a transparent display.

[0023] A DNN 112 can be referred to as a computational network or system of nodes (e.g., artificial neurons). In a deep learning implementation, the nodes of a deep learning model can be arranged in layers as defined in a neural network topology. These layers may include an input layer, one or more hidden layers, and an output layer. Each layer may include one or more nodes (or, for example, artificial neurons). The outputs of all nodes in the input layer can be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer can be coupled to the output of at least one node in the other layers of the model. The output of each hidden layer can be coupled to the input of at least one node in the other layers of the deep learning model. One or more nodes in the final layer can receive input from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer can be determined from hyperparameters, which can be set before or after training the DNN 112 on the training dataset.

[0024] Each node in a DNN 112 can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that can be tuned during model training. The parameter set may include, for example, weight parameters, regularization parameters, etc. Each node can compute its output using the mathematical function based on one or more inputs from nodes in other (or previous) layers of the deep learning model. All or some of the nodes in the deep learning model can correspond to the same or different mathematical functions.

[0025] A DNN 112 can be trained on a lip movement analysis (or lip reading) task using a video dataset as input to generate (multiple) predictions of lip movements of one or more human speakers in each sample video of the video dataset. During training, the DNN 112 can generate predictions indicating the characters / words corresponding to lip movements and the corresponding word boundaries between characters using a CTC loss function. Generally, CTC is a neural network input and associated scoring function used to train a recurrent neural network (RNN) against a time-series input. In this disclosure, the time-series input can represent human mouth movements (i.e., depicted in a video). The DNN 112 can predict word boundaries by processing video frames. The DNN 112 can also be trained based on a cross-entropy loss function. The trained DNN 112 can be used to compare the predictions with reference class labels that may include a baseline truth alignment.

[0026] The two loss functions (i.e., the CTC loss function and the cross-entropy loss function) can mutually improve convergence speed and accuracy during the training phase of the DNN 112. The detailed implementation of the cross-entropy loss function is likely known to those skilled in the art; therefore, for the sake of brevity, a detailed description of the cross-entropy loss function is omitted in this disclosure.

[0027] A DNN 112 can be defined by its hyperparameters and topology / architecture. For example, a DNN 112 can have the number of nodes (or neurons), activation functions (multiple), number of weights, cost function, regularization function, input size, learning rate, number of layers, and so on.

[0028] In training a DNN 112, one or more parameters of each node can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result, according to the loss function used for the DNN 112. This process can be repeated for the same or different inputs until the minimum of the loss function is obtained and the training error is minimized. Several methods for training are known in the art, such as gradient descent, statistical gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc.

[0029] In one embodiment, DNN 112 may include electronic data, which may be implemented, for example, as a software component executable on electronic device 102. DNN 112 may include code and routines configured to enable a computing device, such as electronic device 102, to perform one or more operations for visual speech recognition based on the CTC loss function. Additionally or alternatively, DNN 112 may be implemented using hardware, including (but not limited to) a processor, microprocessor (e.g., for implementing or controlling the implementation of one or more operations), field-programmable gate array (FPGA), coprocessor (e.g., an AI accelerator), or application-specific integrated circuit (ASIC). In some embodiments, DNN 112 may be implemented using a combination of both hardware and software.

[0030] In some embodiments, DNN 112 may be implemented based on a hybrid architecture of multiple deep neural networks (DNNs). Examples of DNN 112 may include neural network models such as (but not limited to) artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), CNN-RNN (CNN-RNN), R-CNN, fast R-CNN, faster R-CNN, YOLO (YOLO) networks, residual neural networks (Res-Net), feature pyramid networks (FPN), retina networks, single-shot detectors (SSD), natural language processing, and (in some cases, OCR) typically uses networks such as CNN-RNN, RNNs based on long short-term memory (LSTM) networks, LSTM+ANN, hybrid lip-reading (HLR-Net) models, and / or combinations thereof.

[0031] Communication network 108 may include a communication medium through which electronic device 102, server 104, and video source 106 can communicate with each other. Examples of communication network 108 may include (but are not limited to) the Internet, cloud networks, wireless local area networks (WLANs), Wi-Fi networks, personal area networks (PANs), local area networks (LANs), telephone lines (POTSs), and / or metropolitan area networks (MANs), such as long-term evolution (LTE) networks (e.g., fourth-generation or fifth-generation (5G) mobile networks, i.e., 5G new radio). Various devices in network environment 100 may be configured to connect to communication network 108 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include (but are not limited to) at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Li-Fi, 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, or Bluetooth (BT) communication protocols, or combinations thereof.

[0032] In operation, electronic device 102 may receive input, for example, to turn on electronic device 102 or to activate visual speech recognition mode. Based on said input, electronic device 102 may be configured to perform a set of operations to perform visual speech recognition, as described herein.

[0033] At any time, electronic device 102 may receive video 114 from video source 106. Video 114 may be any digital media that can be rendered, streamed, broadcast, and stored on any electronic device or storage device. Video 114 may include one or more human speakers 114A. For example, video 114 may be a film clip depicting a scene in which two or more actors are having a conversation. Examples of video 114 may include (but are not limited to) images (such as overlay graphics), animation (such as 2D / 3D animation or motion graphics), audio / video data, traditional television programs (provided via traditional broadcast, cable television, satellite, the Internet, or other means), pay-per-view programs, on-demand programs (such as in video-on-demand (VOD) systems), or Internet content (such as streaming media, downloadable media, webcasting, etc.). In one embodiment, the received video 114 may be pre-recorded video or live video.

[0034] Electronic device 102 can be configured to generate predictions corresponding to lip movements of one or more human speakers 114A in video 114. These predictions can be generated based on applying a DNN 112 to video 114. The DNN 112 can be a pre-trained network using a CTC loss function during the training phase. Details regarding the generation of the predictions are provided, for example, in... Figure 3 Provided in [the document]. According to one embodiment, for each video frame of video 114, the prediction may include a set of probability values ​​corresponding to a set of class labels. The set of class labels may include characters, which may include at least one of word characters (i.e., letters), non-word characters, whitespace, whitespace, and pseudo-characters (as defined in CTC loss).

[0035] Based on the prediction, electronic device 102 can detect one or more word boundaries in a character sequence corresponding to lip movement. Each of the detected one or more word boundaries can be defined by a word character at a first end of the character sequence and a non-word character (e.g., a separator, punctuation mark, etc.) or character spacing at a second end of the character sequence. Details regarding the detection of the one or more word boundaries are provided, for example, in... Figure 3 Provided by China.

[0036] Electronic device 102 can also be configured to segment video 114 into a sequence of video clips based on detection. Each video clip in the sequence of video clips can correspond to a word spoken by one or more human speakers 114A. After segmentation, electronic device 102 can be configured to generate a word prediction sequence by processing the sequence of video clips, as in... Figure 3 As described in [the text]. Based on the word prediction sequence, electronic device 102 can generate sentences or phrases. The generation of sentences or phrases may include concatenating the word prediction sequence using character intervals or non-word characters included between adjacent word predictions in the word prediction sequence. The generation of sentences or phrases, for example, in [the text]... Figure 3 As described in the text.

[0037] Figure 2 This is a block diagram illustrating an exemplary electronic device for visual speech recognition according to an embodiment of the present disclosure. Combined with... Figure 1 unit pair Figure 2 Please provide an explanation. (Refer to...) Figure 2 The diagram illustrates electronic device 102. Electronic device 102 may include circuitry 202, memory 204, a lip movement detector 206, input / output (I / O) devices 208, and a network interface 210. I / O devices 208 may include a display device 110. Memory 204 may include a digital neural network (DNN) 112. The network interface 210 allows electronic device 102 to be connected to server 104 and video source 106 via communication network 108.

[0038] Circuit 202 may include appropriate logic, circuitry, and / or interfaces that may be configured to execute program instructions associated with different operations to be performed by electronic device 102. Circuit 202 may include one or more processing units that may be implemented as an integrated processor or a cluster of processors that uniformly implement the functionality of the one or more processing units. Circuit 202 may be implemented based on several processor technologies known in the art. Examples of implementations of circuit 202 may include x86-based processors, graphics processing units (GPUs), reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, microcontrollers, central processing units (CPUs), and / or other control circuitry.

[0039] Memory 204 may include appropriate logic, circuitry, interfaces, and / or code, which may be configured to store one or more instructions to be executed by circuitry 202. Memory 204 may be configured to store DNN 112 and video 114. Memory 204 may be configured as a collection of class tags. Examples of implementations of memory 204 may include (but are not limited to) random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or secure digital card (SD card).

[0040] The lip movement detector 206 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to detect lip movements in video 114. According to one embodiment, the lip movement detector 206 may be configured to generate predictions corresponding to detected lip movements of one or more human speakers 114A in video 114. The lip movement detector 206 may be implemented as an AI model or may be implemented in an image recognition tool. The lip movement detector 206 may also be implemented based on several processor technologies known in the art. Examples of processor technologies may include (but are not limited to) a central processing unit (CPU), x86-based processors, reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, graphics processing units (GPUs), and other processors.

[0041] I / O device 208 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to receive input and provide output based on the received input. I / O device 208 may include various input and output devices that can be configured to communicate with circuitry 202. In one example, electronic device 102 may display the generated statements or phrases (via display device 110 associated with I / O device 208). Examples of I / O device 208 may include (but are not limited to) a touchscreen, keyboard, mouse, joystick, display device (e.g., display device 110), microphone, or speaker.

[0042] Network interface 210 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to facilitate communication between electronic device 102, server 104, and video source 106 via communication network 108. Network interface 210 may be implemented using various known technologies to support wired or wireless communication between electronic device 102 and communication network 108. Network interface 210 may include (but is not limited to) an antenna, radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, an encoder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or local buffer circuitry.

[0043] Network interface 210 can be configured to communicate wirelessly with a network, such as the Internet, intranet, wireless network, cellular telephone network, wireless local area network (LAN), or metropolitan area network (MAN). Wireless communication can be configured to use one or more of various communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Li-Fi, Wi-MAX, email protocols, instant messaging, and short message service (SMS). For example, in... Figure 3 and 4 The various operations of circuit 202 for visual speech recognition based on connectionist temporal classification loss are further described below.

[0044] Figure 3 This is an illustration of an exemplary processing pipeline for visual speech recognition according to embodiments of the present disclosure. (Combined with information from...) Figure 1 and Figure 2 Explained by unit Figure 3 . Reference Figure 3 The illustration shows an exemplary processing pipeline 300 illustrating exemplary operations for visual speech recognition. These exemplary operations can be performed by any computing system, such as... Figure 1 The electronic device 102 or by Figure 2 Circuit 202 is executed.

[0045] In operation, circuit 202 can be configured to receive video 114 from video source 106. Video 114 may include one or more human speakers 114A as part of a plurality of image frames corresponding to a scene. For example, video 114 may correspond to a television program in which one or more human speakers 114A may speak while performing in the scene.

[0046] In one embodiment, circuit 202 may be configured to extract image frames of video 114. Each image frame of video 114 may be associated with a frame number. The frame number may be any numeric value between 1 and the total number of image frames in video 114 (e.g., 1, 2, 3, 4, or any other numeric value). For example, if video 114 is high-definition (HD) video, the size of each corresponding image frame may be 1280 × 720 pixels. In one embodiment, video 114 may be divided into T × H × W × 3 video frames, where T may represent the frame number associated with each frame of video 114, H may represent the height of the frame (i.e., 720), W may represent the width of the frame (i.e., 1280), and 3 may represent the number of color channels per frame of video 114. For example, if video 114 is full HD video, the size of each frame of the video may be 1920 × 1080 pixels, which may be represented as 1 × 1080 × 1920 × 3 video frames.

[0047] At 302, lip movement analysis can be performed. Circuit 202 can be configured to generate a prediction 304 corresponding to the lip movements of one or more human speakers 114A in video 114. Prediction 304 can be generated based on applying DNN 112 to video 114. According to one embodiment, prediction 304 can include a T×N matrix corresponding to each frame of video 114, where T can represent the frame number associated with video 114 and N can represent all possible class labels. For example, if two words are to be detected (e.g., “Hel lo” and “Wor ld”), prediction 304 can correspond to a T×(10+1) or T×11 matrix, where 10 can represent the number of letter classes (5 for Hel lo and 5 for wor ld) and 1 can represent the whitespace class for the character space between Hel lo and wor ld. Here, the number of classes (11) should be equal to the number of letters (10) and the whitespace class (1).

[0048] According to one embodiment, the DNN 112 can receive a sequence of frames (included in video 114) as input and can detect one or more human speakers 114A in each frame of the frame sequence. Furthermore, the DNN 112 can track the position of the lips of the detected one or more human speakers 114A in each frame. Based on the position, the DNN 112 can extract lip movement information from each frame of the image sequence. In one embodiment, one or more image processing techniques can be used to analyze video 114A to detect lip movement and extract lip movement information.

[0049] DNN 112 can process lip movement information to generate prediction 304. For each frame of video 114, prediction 304 can include a set of probability values ​​corresponding to a set of class labels. For each class label, the probability value can indicate the confidence of DNN 112 (by a probability value between 0 and 1) in the prediction of the class label for a given pattern of lip movement. A higher probability value can represent a higher likelihood or confidence. The set of class labels can include soft or hard labels used to represent classes. Prediction 304 corresponding to lip movement can belong to one of such classes. By way of example and not limitation, the set of class labels can include characters or whitespace. The characters can include at least one of word characters (i.e., letters), non-word characters, whitespace, and / or pseudo-characters. Examples of non-word characters can include (but are not limited to) separators, punctuation marks, a set of special characters such as "*", "&", and a set of non-English alphabets such as "β" and "π".

[0050] In an exemplary scenario, if prediction 304 comprises a set of probability values ​​corresponding to a set of class labels (class A, class B, class C, and blank), then prediction 304 (in matrix form) is shown in Table 1 below:

[0051]

[0052]

[0053] Table 1: Forecast

[0054] For each frame of the video (11-12 and 26), prediction 304 is biased towards the whitespace class with a probability greater than 0.8. The whitespace class can indicate word boundaries in prediction 304.

[0055] Circuit 202 can be configured to detect one or more word boundaries in a character sequence corresponding to lip movements based on prediction. The one or more word boundaries may include the start and end points of each word associated with a dialogue between one or more human speakers 114A and / or other words pronounced or spoken by one or more human speakers 114A in video 114.

[0056] In one embodiment, each of the detected one or more word boundaries can be defined by a word character at a first end of the character sequence and a non-word character or character gap at a second end of the character sequence. For example, if the character sequence corresponding to lip movement is "how are you", then circuit 202 can be configured to detect two word boundaries to mark the beginning and end of each predicted word. Figure 3 In the example, word boundaries (such as word boundaries WB1 and WB2) are shown, and such boundaries can correspond to frames in video 114 that can be classified as blank frames in prediction 304.

[0057] Circuit 202 can be configured to divide video 114 into a sequence of video clips based on the detection. Each video clip in the sequence can correspond to a word spoken by one or more human speakers 114A. For example, a statement or phrase spoken by one or more human speakers 114A could be "how are you". Circuit 202 can be configured to divide video 114 into three video clips, for example... Figure 3 The video clips VC1, VC2, and VC3 are shown. VC1, VC2, and VC3 can correspond to spoken words such as "how," "are," and "you."

[0058] Circuit 202 can also be configured to generate a word prediction sequence by processing a sequence of video clips. Each word prediction in the word prediction sequence can be generated based on applying a DNN 112 to the corresponding video clip in the video clip sequence. Subsequently, based on the generated word prediction sequence, circuit 202 can be configured into sentences or phrases. The generation of sentences or phrases can include concatenating the word prediction sequence using character spacing or non-word characters included between neighboring word predictions in the word prediction sequence. For example, in Figure 3 The diagram illustrates the sequence of predicted words “how,” “are,” and “you.” Circuit 202 can be configured to concatenate the predicted word sequences to generate the phrase “how are you.” Punctuation or other formatting rules can be applied to the statement or phrase based on the facial expressions of one or more human speakers 114A and / or semantic rules associated with the language used to generate the statement or phrase. For example, “how are you” can be updated to “How are you?”.

[0059] Figure 4 This is a flowchart illustrating exemplary operation for visual speech recognition based on connectionist temporal classification loss according to an embodiment of the present disclosure. Figure 4 It is a combination of from Figure 1 , 2 The description is based on unit 3. (Refer to...) Figure 4 The flowchart 400 is shown therein. Flowchart 400 may include operations 402 to 414, and may be generated by... Figure 1 The electronic device 102 or by Figure 2 The circuit 202 is implemented. Flowchart 400 can begin at 402 and continue to 404.

[0060] At 404, video including one or more human speakers can be received. In one embodiment, circuit 202 can be configured to receive video 114A including one or more human speakers 114A, as in Figure 3 As described in [the text].

[0061] At 406, a prediction corresponding to the lip movements of one or more human speakers in the video can be generated. In one embodiment, circuit 202 can be configured to generate a prediction 304 corresponding to the lip movements of one or more human speakers 114A in video 114. Prediction 304 can be generated based on applying DNN 112 to video 114, and DNN 112 can be trained using the CTC loss function. In one example, for each image frame of video 114, prediction 304 can include a set of probability values ​​corresponding to a set of class labels. The set of class labels can include (but is not limited to) words or characters, which include at least one of word characters, non-word characters, whitespace, and pseudo-characters. The generation of prediction 304 is, for example, in... Figure 3 As described in the text.

[0062] At 408, one or more word boundaries in the character sequence corresponding to lip movement can be detected. In one embodiment, circuit 202 can be configured to detect one or more word boundaries in the character sequence corresponding to lip movement based on prediction 304. For example, each of the detected one or more word boundaries can be defined by a word character at a first end of the character sequence and a non-word character or character gap at a second end of the character sequence. The detection of one or more word boundaries is, for example, in... Figure 3 As described in the text.

[0063] At 410, the video can be segmented into a sequence of video clips, and the video clips in the sequence correspond to words spoken by one or more human speakers. In one embodiment, circuit 202 can be configured to segment video 114 into a sequence of video clips based on detection. Video segmentation is, for example, in... Figure 3 As described in the text.

[0064] At point 412, a word prediction sequence can be generated by processing the video clip sequence. In one embodiment, circuit 202 can be configured to generate a word prediction sequence by processing the video clip sequence. The generation of the word prediction sequence is, for example, in... Figure 3 As described in the text.

[0065] At 414, a sentence or phrase can be generated based on the generated word prediction sequence. In one embodiment, the circuitry can be configured to generate a sentence or phrase based on the detected word prediction sequence. The generation of the sentence or phrase may include (but is not limited to) concatenating the word prediction sequence using character spacing or non-word characters included between neighboring word predictions in the word prediction sequence. The generation of the sentence or phrase, for example, is... Figure 3 As described in the text. Control can continue until the end.

[0066] Although flowchart 400 is shown as discrete operations, such as 404, 406, 408, 410, 412, and 414, this disclosure is not limited thereto. Accordingly, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or removed depending on the implementation, without departing from the spirit of the disclosed embodiments.

[0067] Various embodiments of this disclosure may provide a non-transitory computer-readable medium and / or storage medium storing computer-executable instructions that can be executed by a machine and / or a computer to operate an electronic device (e.g., electronic device 102). The instructions may cause electronic device 102 to perform operations including retrieving video 114, wherein video 114 includes one or more human speakers 114A. The operations may further include generating a prediction 304 corresponding to lip movements of the one or more human speakers 114A in video 114. Prediction 304 may be generated based on applying a DNN 112 to video 114, and the DNN 112 may be trained using a connectionist temporal classification (CTC) loss function. The operations may further include detecting one or more word boundaries in a character sequence corresponding to lip movements based on prediction 304. The operations may further include segmenting the video into a sequence of video clips based on the detection. Each video clip in the sequence of video clips corresponds to a word spoken by one or more human speakers 114A. The operations may further include generating a sequence of predicted words by processing the sequence of video clips. The operation may also include generating sentences or phrases based on the generated word prediction sequence.

[0068] Exemplary aspects of this disclosure may provide an electronic device (e.g., circuit 202) including a circuit. Figure 1 The electronic device 102. Circuit 202 can be configured to receive a video 114 including one or more human speakers 114A. Circuit 202 can be configured to generate a prediction 304 corresponding to lip movements of one or more human speakers 114A in the video 114. Prediction 304 can be generated based on applying a DNN 112 to the video 114, and the DNN 112 can be trained using a connectionist temporal classification (CTC) loss function. Circuit 202 can be configured to detect one or more word boundaries in a character sequence corresponding to lip movements based on prediction 304. Circuit 202 can be configured to segment the video into a sequence of video clips based on the detection. Each video clip in the sequence of video clips corresponds to a word spoken by one or more human speakers 114A. Circuit 202 can be configured to generate a sequence of word predictions by processing the sequence of video clips. Circuit 202 can be configured to generate sentences or phrases based on the generated sequence of word predictions.

[0069] In one embodiment, each of the detected one or more word boundaries may be defined by a word character at a first end of the character sequence and a non-word character or character interval at a second end of the character sequence.

[0070] In one embodiment, for each image frame of the video, prediction 304 includes a set of probability values ​​corresponding to a set of class labels.

[0071] In one embodiment, the set of class tags includes words or characters, which include at least one of word characters, non-word characters, whitespace, and pseudo-characters.

[0072] In one embodiment, the DNN 112 can also be trained based on the cross-entropy loss function.

[0073] In one embodiment, each word prediction in the word prediction sequence can be generated by applying DNN 112 to the corresponding video clips in the video clip sequence.

[0074] In one embodiment, the generation of a sentence or phrase includes concatenating the word prediction sequence using character spacing or non-word characters included between neighboring word predictions in the word prediction sequence.

[0075] This disclosure can be implemented in hardware or a combination of hardware and software. It can be implemented in a centralized manner on at least one computer system or in a distributed manner, wherein different units can be distributed across several interconnected computer systems. Computer systems or other apparatus adapted to implement the methods described herein may be suitable. The combination of hardware and software can be a general-purpose computer system having computer programs that, when loaded and executed, can control the computer system to implement the methods described herein. This disclosure can be implemented in hardware including a portion of an integrated circuit that also implements other functions.

[0076] This disclosure can also be embedded in a computer program product that includes all features that allow the implementation of the methods described herein and, when loaded into a computer system, enables the implementation of those methods. In this context, a computer program means any expression of a set of instructions in any language, code, or notation intended to cause a system capable of information processing to directly implement a particular function or, after any one or both of the following: a) conversion to another language, code, or notation; or b) reproduction in a different material form.

[0077] While this disclosure has been described with reference to specific embodiments, those skilled in the art will understand that various changes and equivalent alternatives can be made without departing from the scope of this disclosure. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from its scope. Therefore, this disclosure should not be limited to the disclosed embodiments, but rather encompasses all embodiments falling within the scope of the appended claims.

Claims

1. An electronic device comprising: The circuit is configured as follows: Receive video including one or more human speakers; Generate predictions corresponding to the lip movements of one or more human speakers in a video. The prediction is generated based on the application of a deep neural network (DNN) to the video. The DNN is trained using a connectionist temporal classification (CTC) loss function. For each frame of the video, the prediction is biased towards the blank class with a probability greater than 0.

8. The blank class indicates word boundaries in the prediction, and The word boundaries correspond to video frames that can be classified as blank frames in the prediction. Based on the prediction, detect one or more word boundaries in the character sequence corresponding to lip movement; Based on the detection, the video is divided into a sequence of video clips, wherein each video clip in the sequence corresponds to a word spoken by the one or more human speakers; Word prediction sequences are generated by processing video clip sequences; and Sentences or phrases are generated based on the predicted sequence of words.

2. The electronic device according to claim 1, wherein, Each of the detected one or more word boundaries is defined by the word character at the first end of the character sequence and the non-word character or character interval at the second end of the character sequence.

3. The electronic device according to claim 1, wherein, For each image frame of the video, the prediction includes a set of probability values ​​corresponding to a set of class labels.

4. The electronic device according to claim 3, wherein, The set of class tags includes words or characters, and the words or characters include at least one of word characters, non-word characters, whitespace, and pseudo-characters.

5. The electronic device according to claim 1, wherein, It also uses the cross-entropy loss function to train the DNN.

6. The electronic device according to claim 1, wherein, Each word prediction in the word prediction sequence is generated by applying a DNN to the corresponding video clips in the video clip sequence.

7. The electronic device according to claim 1, wherein, The generation of the statement or phrase involves concatenating the word prediction sequence using character intervals or non-word characters included between neighboring word predictions in the word prediction sequence.

8. A method comprising: Receive video including one or more human speakers; Generate predictions corresponding to the lip movements of one or more human speakers in a video. The prediction is generated based on the application of a deep neural network (DNN) to the video. The DNN is trained using a connectionist temporal classification (CTC) loss function. For each frame of the video, the prediction is biased towards the blank class with a probability greater than 0.

8. The blank class indicates word boundaries in the prediction, and The word boundaries correspond to video frames that can be classified as blank frames in the prediction. Based on the prediction, detect one or more word boundaries in the character sequence corresponding to lip movement; Based on the detection, the video is divided into a sequence of video clips, wherein each video clip in the sequence corresponds to a word spoken by the one or more human speakers; Word prediction sequences are generated by processing video clip sequences; and Sentences or phrases are generated based on the predicted sequence of words.

9. The method according to claim 8, wherein, Each of the detected one or more word boundaries is defined by the word character at the first end of the character sequence and the non-word character or character interval at the second end of the character sequence.

10. The method according to claim 8, wherein, For each image frame of the video, the prediction includes a set of probability values ​​corresponding to a set of class labels.

11. The method according to claim 10, wherein, The set of class tags includes words or characters, which include one of word characters, non-word characters, whitespace, and pseudo-characters.

12. The method according to claim 8, wherein, It also uses the cross-entropy loss function to train the DNN.

13. The method according to claim 8, wherein, Each word prediction in the word prediction sequence is generated by applying a DNN to the corresponding video clips in the video clip sequence.

14. The method according to claim 8, wherein, The generation of the statement or phrase involves concatenating the word prediction sequence using character intervals or non-word characters included between neighboring word predictions in the word prediction sequence.

15. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform the following operations: Receive video including one or more human speakers; Generate predictions corresponding to the lip movements of one or more human speakers in a video. in, The prediction is generated based on the application of a deep neural network (DNN) to the video. The DNN is trained using a connectionist temporal classification (CTC) loss function. For each frame of the video, the prediction is biased towards the blank class with a probability greater than 0.

8. The blank class indicates word boundaries in the prediction, and The word boundaries correspond to video frames that can be classified as blank frames in the prediction. Based on the prediction, detect one or more word boundaries in the character sequence corresponding to lip movement; Based on the detection, the video is divided into a sequence of video clips, wherein each video clip in the sequence corresponds to a word spoken by the one or more human speakers; Word prediction sequences are generated by processing video clip sequences; as well as Sentences or phrases are generated based on the predicted sequence of words.

16. The non-transitory computer-readable medium according to claim 15, wherein, Each of the detected one or more word boundaries is defined by the word character at the first end of the character sequence and the non-word character or character interval at the second end of the character sequence.

17. The non-transitory computer-readable medium according to claim 15, wherein, For each image frame of the video, the prediction includes a set of probability values ​​corresponding to a set of class labels.

18. The non-transitory computer-readable medium according to claim 17, wherein, The set of class tags includes words or characters, which include one of word characters, non-word characters, whitespace, and pseudo-characters.

19. The non-transitory computer-readable medium according to claim 15, wherein, It also uses the cross-entropy loss function to train the DNN.

20. The non-transitory computer-readable medium according to claim 15, wherein, The generation of the statement or phrase involves concatenating the word prediction sequence using character intervals or non-word characters included between neighboring word predictions in the word prediction sequence.

Images