Automatically caption the audible portion of content on your computing device

Abstract

A computing device for automatically captioning content is described. Unlike other captioning systems that typically rely on information contained in an audio signal being sent to a speaker, the computing device generates captions directly from audio data being output from a content source. The computing device can analyze metadata to determine whether the audio data is suitable for captioning or whether the audio data is some other type of audio data. In response to identifying audio data for captioning, the computing device can quickly generate a description of audible sounds interpreted from the audio data. Regardless of the application source of the audible content and regardless of whether the computing device ever produced audio for the content, the computing device can visually provide a description of the audible content.

Description

Background Art

[0001] With the exception of professionally produced movies and television shows, most audible content is not captioned, and therefore, many users who are hard of hearing or otherwise unable to hear the content are unable to access the content. Some automatic captioning systems are available, but these systems can be expensive, complex, and difficult to maintain. Some automatic captioning systems require audio output to a speaker as input, and may not be well suited for providing captioning in a quiet environment. To avoid the cost and complexity involved in maintaining an automatic captioning system, some computing devices access cloud-based captioning services that require uploading content to a remote server for captioning, thereby potentially reducing the privacy and convenience of the user. Summary of the invention

[0002] A computing device for automatically captioning content as a local, system-level service is described. Unlike other captioning systems that typically rely on information contained in an audio signal being sent as input to a speaker, the computing device generates captions directly from audio data being output from a content source (e.g., an application). The computing device can analyze metadata (e.g., non-audio portions) tagged to the audio data to determine whether the audio data is suitable for captioning, or whether the audio data is some other type of audio data (e.g., a system sound effect).

[0003] In response to identifying audio data for subtitles, and in response to determining that automatic subtitles are enabled (e.g., via user input), the computing device uses a machine learning model to quickly generate a description of audible sounds interpreted from the audio data. Regardless of the application source of the audible content, and regardless of whether the computing device has ever generated audio (e.g., audible sound signals) for the content, the computing device can visually provide a description of the audible content (e.g., as a persistent user interface element). In this way, even when the audio data does not include embedded subtitles, even when the device does not use a server for subtitles, and even when the computing device is muted and does not output audible sounds, the computing device subtitles the system-level audio to describe the audible portion of the content.

[0004] By relying on the original audio data rather than the audio signal generated as input to the speaker, the machine learning model can generate subtitles that more accurately represent the sound from the content, as originally intended by the content source. In addition, by determining whether the audio data is suitable for subtitles before using the machine learning model, the computing device can avoid wasting resources by over-analyzing all the audio data that the content source is outputting, including the audio data that almost certainly does not contain any content for subtitles. This enables the computing device to perform more efficiently because a smaller and / or less complex machine learning model can be used. For example, the machine learning model can automatically recognize speech and analyze only the audio data including speech. In this way, the machine learning model can automatically recognize speech and automatically classify sounds locally from the computing device that generates the audio data, thereby enhancing privacy and offline convenience. Therefore, the computing device can automatically subtitle the system-level audio in a way that can at least increase the user's satisfaction with the automatic subtitle system. Users with medical needs can improve their quality of life by accessing a computing device 100 that automatically subtitles audio data in this way. Users who may have previously been unable to listen to or understand the audible part of the content can now enjoy the content like any other user without medical needs.

[0005] Throughout this disclosure, examples are described in which a computing device can analyze information (e.g., audio data). However, the computing device can use the information only after the computing device receives explicit permission from the user of the computing device to use the data. For example, in the case discussed below where a computing device analyzes audio data being output from an application executing at the computing device, an individual user can be provided with an opportunity to provide input to control whether a program or feature of the computing device can collect and use audio data, such as for automatic captioning. An individual user can continuously control which programs can use the audio data and which programs cannot use the audio data.

[0006] In addition, the collected information may be pre-processed in one or more ways to remove personally identifiable information before it is transmitted, stored, or otherwise used by the computing device and / or remote computing system. For example, before the example computing device shares subtitles generated from audio data with another device (e.g., to train a model executed at the other device), the example computing device may pre-process the audio data to ensure that any user-identifying information or device-identifying information embedded in the data is removed. Thus, the user may control whether information about the user and the user's device is collected, and if so, how the computing device and / or remote computing system uses the information.

[0007] In one example, a method for automatically captioning an audible portion of content on a computing device is described. The method includes obtaining, from an audio mixer of the computing device, audio data output from an application executed at the computing device, the audio data including non-metadata indicating the audible portion of the content and optionally including metadata, and optionally determining, based on the metadata of the audio data, whether the audio data is of a type suitable for automatic captioning. The method further includes: in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content, and outputting the description of the audible portion of the content for display when displaying a visual portion of the content.

[0008] In a different example, a method for automatically captioning an audible portion of content on a computing device is described. The method includes displaying, by the computing device, a graphical user interface of an application executing at the computing device, obtaining audio data output from the application while displaying the graphical user interface, the audio data including non-metadata indicating the audible portion of the content and optionally including metadata, and optionally determining whether the audio data is of a type suitable for automatic captioning based on the metadata of the audio data. In response to determining that the audio data is of a type suitable for captioning, the method further includes determining a description of the audible portion of the content, and outputting the description of the audible portion of the content as a persistent element in addition to the graphical user interface of the application for display when displaying the visual portion of the content in the graphical user interface of the application.

[0009] In yet another example, a method for automatically captioning an audible portion of content on a computing device is described. The method includes: receiving user input from an application executed at the computing device to automatically caption audio data; and in response to receiving the user input, obtaining audio data output from the application executed at the computing device, the audio data including the audible portion of the content. The method further includes optionally determining, based on metadata of the audio data, whether the audio data is of a type suitable for captioning, and in response to determining that the type of the audio data is suitable for captioning, determining a description of the audible portion of the content. The method further includes outputting the description of the audible portion of the content as a persistent element for display in addition to the visual portion of the content and in addition to a graphical user interface of the application.

[0010] In yet another example, a computing device comprising at least one processor configured to perform the method of any of the preceding examples is described.

[0011] In yet another example, a system including means for performing the method of any of the preceding examples is described.

[0012] In another example, a computer-readable storage medium comprising instructions that, when executed, configure a processor of a computing device to perform any of the methods described in any of the preceding examples is described.

[0013] The details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the description and drawings and from the claims. This summary is provided to introduce the subject matter further described in the detailed description and drawings. Therefore, this summary should not be considered to describe essential features, nor should it be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Details of one or more aspects of automatic subtitling are described below. The same reference numerals are used in different instances in the specification and drawings to indicate similar elements:

[0015] Figure 1 is a conceptual diagram illustrating a computing device configured to automatically caption audio data.

[0016] Figure 2 is another conceptual diagram illustrating a computing device configured to automatically caption audio data.

[0017] Figure 3 is a conceptual diagram illustrating a machine learning model of a computing device configured to automatically caption audio data.

[0018] Figure 4 is a flow diagram illustrating example operations of a computing device configured to automatically caption audio data.

[0019] FIG. 5A to FIG. 5G are each screen shots illustrating an example user interface of a computing device for automatically captioning audio data. DETAILED DESCRIPTION

[0020] With the exception of professionally produced movies and television shows, most audible content is not subtitled and, therefore, is inaccessible to many users who are hard of hearing or otherwise unable to hear the content. Manual and automatic subtitling systems exist, but have various disadvantages.

[0021] For example, some manual captioning systems caption audible content based on embedded captioning information contained in pre-recorded content. Other manual captioning systems caption content based on input from a human operator (e.g., in a live broadcast).

[0022] Some automatic subtitling systems are available, but these systems may be expensive, complex, and difficult to maintain. For example, an automatic subtitling system may require the audio output of the sound system as a source for subtitling. Subtitling systems may require the audio output to maintain a minimum sound level, making such systems less suitable for operation in quiet environments. In addition, the audio output may include more audible sounds (e.g., system sounds, sounds from other background applications) than sounds used only for subtitling, and therefore, subtitling may take longer and waste resources unnecessarily, and indiscriminately subtitling all sounds from all audio sources of the computing device.

[0023] Some automatic captioning systems are application-based and can be executed locally on a computing device. However, such systems may still require an Internet connection for captioning. For example, to avoid the cost and complexity involved in maintaining an automatic captioning system, a computing device may access a cloud-based captioning service that requires uploading content to a remote server for captioning, thereby potentially reducing the user's privacy and convenience.

[0024] Techniques for efficiently and distinctively implementing automatic captioning of audible content are described. The techniques do not require a network connection (e.g., to the Internet) and do not generate captions based on audible signals delivered to a sound system. Alternatively, an example computing device can automatically generate captions based on audio data collected by an audio mixer of the computing device. As used throughout, "audio data" refers to information that is sent to a system audio mixer or sound engine and is converted into an audio signal to generate sound as an output. Audio data is contrasted with an audio signal generated in response thereto.

[0025] The operating system or platform of the example computing device includes a system-level audio mixer that collects audio data being output from different content sources (e.g., applications) and mixes the audio data to produce an audio signal. The audio signal may be amplified by a sound system to produce audible sound. Unlike other subtitle systems that indiscriminately subtitle audio signals, the example computing device discriminately subtitles audible content by only subtitles audible content from audio data of a type suitable for subtitles.

[0026] The computing device can analyze metadata (e.g., non-audio portions of the audio data) to determine whether the audio data is suitable for subtitles or whether the audio data is some other type of audio data (e.g., system sound effects). Relying on metadata that is not present in the audio signal enables the computing device to quickly filter out audio data that is not used for subtitles, and therefore, enables the computing device to avoid wasting resources as other subtitle systems attempt to subtitle all content, even content that cannot or should not be subtitled.

[0027] In response to identifying the audio data for captioning, and in response to determining to enable automatic captioning (e.g., via user input), the computing device uses a machine learning model that has been trained using machine learning to quickly generate a description of audible sounds interpreted from the audio data. For example, the computing device can rely on an end-to-end recurrent neural network transducer automatic speech recognition model or other machine learning model to determine a transcription of spoken audio or identify specific noises from non-verbal audio, as well as possible sources of the specific noises.

[0028] Regardless of the source of the audible content, and regardless of whether the computing device ever generates audio (e.g., an audible sound signal) for the content, the computing device can visually provide a description of the audible content (e.g., as a persistent user interface element). In this manner, the computing device performs system-level captioning operations that provide a description of the audible portion of the content even if the content does not include embedded captions, and even if the computing device is muted and never outputs an audio signal.

[0029] By relying on audio data, rather than using an audio signal as input to a speaker, a machine learning model can generate subtitles that more accurately represent the sounds from the audible content as originally intended by the content source. Audio data from an application or other content source can often be a more accurate representation of the audible content because the audio data has not been altered since it was output by the content source. Other automatic captioning systems may rely on processed audio signals that were intended as input to a speaker, and therefore may introduce differences from what was intended by the content source.

[0030] Furthermore, by determining whether the audio data is suitable for captioning before assigning the task of obtaining a description to a machine learning model, the computing device avoids wasting resources over-analyzing all audio data being output by the content source, including portions of the audio data that almost certainly do not include any content for captioning. This enables the computing device to train and execute smaller and / or less complex machine learning models to perform automatic speech recognition and automatic sound classification techniques. There is no need to additionally train the model to ignore various types of audio data that may not be suitable for captioning. This further enables the model to be executed locally on the computing device that generated the audio data, thereby enhancing privacy and offline convenience.

[0031] Relying on more accurate data can improve the accuracy of the machine learning model, and relying only on relevant data can improve the efficiency of the machine learning model, which can further improve user satisfaction with the automatic captioning system. In addition, users with medical needs can experience an improvement in their quality of life by accessing a computing device 100 that automatically captions audio data in this manner.

[0032] In some cases, the description may include a transcription of the spoken content, and may even identify the name or description of the speaker. In some cases, the audible sounds include non-verbal content, such as animal sounds, environmental sounds, and other sounds that may or may not be relevant to the description. The description may even include an interpretation of some non-verbal sounds (e.g., a dog barking), including identification of the corresponding source (e.g., a dog barking).

[0033] Figure 1 1 is a conceptual diagram illustrating a computing device configured to automatically add subtitles to audio data. Computing device 100 can be any type of mobile or non-mobile computing device. As a mobile computing device, the computing device can be a mobile phone, a laptop computer, a wearable device (e.g., a watch, glasses, headphones, clothing), a tablet device, an automobile / vehicle device, a portable gaming device, an e-reader device, a remote control device, or other mobile computing device. As a non-mobile computing device, computing device 100 can represent a server, a network terminal device, a desktop computer, a television device, an entertainment set-top box device, a streaming media device, a desktop assistant device, a speaker device, a sound system, a non-portable gaming device, a business conference device, or other non-mobile computing devices.

[0034] The computing device 100 includes a user interface device 102, which includes a speaker component 104, a display component 106, and an input component 108. The computing device 100 also includes an operating system 110 and applications 112. These and other components of the computing device 100 are communicatively coupled in various ways, including through the use of wired and wireless buses and links. The computing device 100 may include, for example, an operating system 110 and applications 112. Figure 1 Additional or fewer components may be shown.

[0035] Operating system 110 generally controls the functionality of computing device 100, including user interface device 102 and other peripherals. Operating system 110 provides an execution environment for applications such as application 112. Operating system 110 may control task scheduling and other general functionality, and typically does so through a system-level user interface 114.

[0036] User interface device 102 manages input and output to system-level user interface 114 controlled by operating system 110. System-level user interface 114 includes a persistent graphical element 120 for displaying automatic subtitles. System-level user interface 114 also includes one or more subtitle control elements 122A and 122B, which, if selected, control whether operating system 110 automatically subtitles audio data, including whether user interface 114 displays persistent graphical element 120. In some cases, subtitle control elements 122A and 122B may include icons indicating whether computing device 100 is automatically subtitlesing audio data. System-level user interface 114 may display user interfaces for other applications executing at computing device 100, such as application user interface 116 for application 112.

[0037] The display assembly 106 and the speaker assembly 104 can be separated or integrated into a single assembly. The display assembly 106 can be made of any suitable display technology, including LED, OLED and LCD technology. The speaker assembly 104 (e.g., a single speaker or multiple speakers) is configured to receive an audio signal as input and convert the audio signal into audible sound. The input assembly 108 can be a microphone, a presence sensitive device, a touch screen, a mouse, a keyboard, or other types of components configured to receive user input.

[0038] The user interface device 102 may output information about the input detected by the input component 108, and the operating system 110 may correlate the detected input with an element of the user interface 114. In response to receiving input (e.g., a gesture) at the input component 108, the operating system 110 and / or the application 112 may receive information about the detected input from the user interface device 102. The operating system 110 or the application 112 may perform a function in response to the input. For example, the operating system 110 may determine that the input corresponds to a selection of any one of the subtitle control elements 122A or 122B, and in response, automatically subtitle the audio data (or terminate the automatic subtitle addition of the audio data).

[0039] Application user interface 116 includes media content 118, such as movies, animations, videos, or other content with embedded audio. Media content 118 can be any multimedia or social media content, including from a video platform, a streaming platform, from a broadcast, or from a podcast, from a video, from a video chat application, from a phone application, or from any other audio data source executed by or accessible from a computing device.

[0040] The application 112 can output the media content 118 to the operating system 110. Using the display component 106, the operating system 110 can cause the user interface device 102 to output an image or other visual representation of the media content 118 within the application user interface 116. The computing device 100 can also cause the user interface device 102 to forward audio signals generated from the media content 118 to the speaker component 104. In this way, the operating system 110 causes the speaker component 104 to output audio while the image or other visual is displayed at the display component 106.

[0041] In operation, the operating system 110 automatically captions the audible portion of the content. The computing device obtains audio data for captioning from an audio mixer or sound engine of the operating system 110. The audio data from the application 112 may include the audible portion of the media content 118.

[0042] For example, Figure 1 As shown, a user of computing device 100 can interact with application 112 by viewing media content 118 displayed on the screen of display component 106. The user may want to enable subtitles for media content 118, but media content 118 only includes unsubtitled audio data, and application 112 cannot generate subtitles and embed them into media content 118.

[0043] However, in response to receiving user input (e.g., a touch gesture at a location of input component 108 corresponding to a location of the screen of display component 106 where selectable control element 122B is displayed), computing device 100 may automatically caption the audible portion of media content 118 by processing the same audio data output from application 112 and received at the audio mixer.

[0044] The audio mixer is configured to collect all audio data output from various content sources including applications 112 executing in the execution environment of the operating system 110. The audio mixer provides an interface between the application sources that generate content and the speaker component 104 that creates sound from the content. The audio mixer manages the raw audio data byte stream, analyzes the audio data, and directs the audio signal to be output to the speaker component 104 or other suitable output component of the user interface device 102 to create sound.

[0045] The operating system 110 can rely on the metadata to focus automatic captioning on audible portions of content for captioning, rather than indiscriminately automatically captioning all audio data from the audio signal sent to the speaker assembly 104. In other words, the operating system 110 can identify "captionable" audio data based on the metadata and avoid captioning all audio data unless all audio data is deemed captionable.

[0046] Some examples of metadata include a usage indicator that specifies why the application source is outputting audio data. The audio mixer can use the usage indicator to control routing, focus, and volume decisions about the audio data. Metadata can include other flags that further specify how the sound should be played and information for audio / video synchronization.

[0047] The metadata may include a content type (e.g., music, movie, voice, sonification, unknown) that specifies what the application source is playing. The content type may express a general category of content (e.g., one type of movie pipeline service and another type of music streaming service). The audio mixer may optionally configure some audio post-processing and adjust other settings based on the type identifier. When the audio data is a soundtrack for accompanying a movie or television show, the type identifier may specify the movie type. When the content is a song (e.g., not necessarily a soundtrack), the type identifier may specify a music type identifier, and when the content is spoken audio (e.g., a news broadcast, video, or phone call), the type identifier may indicate a speech identifier. As another example, when the audio data is used for notifications, alarms, or sounds that accompany user actions (e.g., a beep or sound effect expressing a keystroke) or an event (e.g., a sound such as a reward received during a game), the content type may indicate a sonification.

[0048] In addition to the audio mixer's reliance on content type, the operating system 110 identifies audio data for subtitling based on a content type identifier that is optionally present in the metadata. For example, the operating system 110 automatically subtitles audio data of the following types: movie, music, or speech, but not audio data of some vocalization types or other audio data that is undefined or otherwise defined. In this way, using the content type, the operating system 110 can determine whether the audio data is likely to have audio that requires subtitling or whether the audio data is likely to indicate some other sound that is not intended for subtitling.

[0049] In response to determining that the type of audio data associated with media content 118 is for subtitles, operating system 110 determines a description of the audible portion of content 118. For example, operating system 110 can execute a machine learning model (e.g., an end-to-end recurrent neural network transducer automatic speech recognition model) trained to generate a description of audible content as subtitles. The machine learning model can be any type of model suitable for learning the description of a sound, and the description of the sound includes the transcription of spoken audio. However, since only the machine learning model needs to be trained to identify sounds and spoken audio from certain types of content, the machine learning model used by operating system 110 can be smaller in size and less complex. The machine learning model does not need to process all audio data sent to an audio mixer, but only needs to process some audio data that may include content for subtitles. This is in contrast to other subtitle systems that require remote processing facilities and remote upload of content, have privacy risks, and sacrifice convenience.

[0050] The operating system 110 receives the description from the machine learning model and displays the description to the user. The description may include a transcription of a spoken conversation or a song. The description may identify the context of the sound or identify the speaker, singer, or individual actor or performer. The description may include a description of the sound, such as "a dog barking" when audio of a dog is detected or "door closing" for audio of a door slamming.

[0051] Operating system 110 outputs a description of the audible portion of the content for display while the visual portion of media content 118 is displayed within application user interface 116. For example, operating system 110 can present persistent element 120 as an overlay of user interface 114 that appears on top of application user interface 116 along with other graphical elements of user interface 114. A user can manipulate persistent element 120 to move to different areas of the display screen or to enlarge or reduce the size of the element to display additional or fewer subtitles.

[0052] By relying on the original audio data, rather than the audio signal generated as speaker input, the machine learning model of the operating system 110 generates subtitles that more accurately represent the sound from the media content 118, as originally intended by the content source (e.g., application 112). In addition, by determining whether the audio data is used for subtitles before using the machine learning model, the operating system 110 avoids wasting resources to over-analyze all the audio data that the application 112 is outputting, including almost certainly not including any content for subtitles. This enables the computing device 100 to execute a more efficient, smaller and / or less complex machine learning model. In this way, the machine learning model can perform automatic speech recognition and automatic sound classification techniques locally from the computing device 100 that generates the audio data, thereby enhancing privacy and offline convenience. Therefore, the computing device 100 can automatically subtitle the system-level audio in a manner that can at least increase the user's satisfaction with the automatic subtitle system. Users with medical needs can improve their quality of life by accessing a computing device 100 that automatically subtitles audio data in this manner.

[0053] Figure 2 2 is another conceptual diagram illustrating a computing device 200 configured to automatically caption audio data. Computing device 200 is an example of computing device 100 with some additional details. Figure 2 As shown, computing device 200 may be a mobile phone 100-1, a laptop computer 100-2, a television / monitor 100-3, a desktop computer 100-4, a tablet device 100-5, a computerized watch 100-6, other wearable devices, or a computing system installed in a vehicle 100-7.

[0054] Apart from Figure 1 In addition to each component shown, computing device 200 includes one or more processors 202, computer-readable media 204, one or more sensors 210, one or more input / output (I / O) devices 212, and one or more communication devices 214. Computer-readable media 204 includes instructions that, when executed by processor 202, execute applications 112 and operating system 110.

[0055] Processor 202 may include any combination of one or more controllers, microcontrollers, processors, microprocessors, hardware processors, hardware processing units, digital signal processors, graphics processors, graphics processing units, etc. Processor 202 may be an integrated processor and memory subsystem (e.g., implemented as a SoC) that processes computer-executable instructions to control the operation of computing device 200.

[0056] When functioning in the physical operating environment, the sensor 210 obtains contextual information indicative of the physical operating environment of the computing device and / or characteristics of the computing device 200. Examples of the sensor 210 include a motion sensor, a temperature sensor, a position sensor, a proximity sensor, an ambient light sensor, a humidity sensor, a pressure sensor, etc. The operating system 110 may customize the operation of the computing device 200 based on the sensor information obtained by the sensor 210.

[0057] Input / output devices 212 provide connectivity to computing device 200 and other devices and peripherals, including data network interfaces that provide connections and / or communication links between devices, data networks (e.g., mesh networks, external networks, etc.), and other devices. Input / output devices 212 can be used to couple computing device 200 to any type of component, peripheral, and / or accessory device. Input / output devices 212 also include data input ports, via which any type of data, media content, and / or input can be received, such as user input to computing device 200, as well as any type of communication data, and audio, video, and / or image data received from any content and / or data source.

[0058] Communication device 214 enables wired and / or wireless communication of device data 506, such as any type of media content generated or received by applications such as application 112 executing on computing device 200. Communication device 214 may also include a transceiver for cellular telephone communications and / or network data communications.

[0059] The computer readable medium 204 is configured to provide persistent and non-persistent storage of executable instructions (e.g., firmware, recovery firmware, software, applications, modules, programs, functions, etc.) and data (e.g., user data, operational data) to the computing device 200 to support the execution of the executable instructions. Examples of computer readable medium 204 include volatile and non-volatile memory, fixed and removable media devices, and any suitable memory device or electronic data storage that maintains executable instructions and supporting data. The computer readable medium 204 may include various implementations of random access memory (RAM), read-only memory (ROM), flash memory, and other types of storage memory in various memory device configurations. The computer readable medium 204 does not include propagating signals. The computer readable medium 204 may be a solid state drive (SSD) or a hard disk drive (HDD). Figure 2 The computer-readable medium 204 in the example of EMBODIMENT includes the application 112 and the operating system 110 .

[0060] The operating system 110 of the computing device 200 includes an audio mixer 206 and a subtitle module 208. The audio mixer 206 and the subtitle module 208 may be implemented as dedicated hardware or software components of the operating system 110. In other examples, the audio mixer 206 or the subtitle module 208 may be implemented separately from the operating system 110, for example, as a system plug-in or other additional service installed locally on the operating system 110.

[0061] The audio mixer 206 is configured to merge audio data generated by applications executing within the operating environment provided by the operating system 110. The audio mixer 206 combines audio streams from applications, such as the application 112, and generates an audio output signal that, when combined and output from the speaker assembly 104, reproduces the sounds encoded in the audio streams. The audio mixer 206 may adjust the audio signals in other ways, such as controlling focus, intent, and volume.

[0062] The captioning module 208 is configured to automatically caption the audio data in raw form as received by the audio mixer 206 (e.g., as a byte stream). Rather than processing all mixed audio signals for captioning, the captioning module 208 identifies separate, pre-mixed audio data streams received at the audio mixer 206 that are suitable for captioning. For example, the captioning module 208 may automatically caption the audio data of a spoken audio type, but not the audio data of a notification or vocalization type such as system beeps and ringtones. The captioning module 208 may apply a filter to the byte stream received by the audio mixer 206 to identify audio data of a type suitable for captioning. The captioning module 208 uses a machine learning model to determine a description of the sound represented by the audio data identified for captioning.

[0063] Figure 3 312 is a conceptual diagram illustrating a machine learning model of a computing device configured to automatically caption audio data. Machine learning model 300 may be part of caption module 208. That is, caption model 208 may include machine learning model 300 to automatically convert encoded audio data into detailed descriptions or captions of audible content without having to convert the audio data into sound. As part of executing caption module 208, computing device 200 may execute machine learning model 300 to determine descriptions 318 of audible portions of content present in audio data 312. Machine learning model 300 may include one or more types of machine learning models that are combined into a single model that provides description 318 in response to audio data 312.

[0064] The machine learning model 300 is configured to perform inference; the machine learning model 300 is trained to receive audio data 312 as input and provide descriptions 318 (e.g., subtitles) of sounds determined by the machine learning model 300 from the audio data 312 as output data. In addition to the annotations (i.e., annotation data) contained in the audio data 312, the audio data 312 may include unannotated data that has not been annotated for subtitles. By performing inference using the machine learning model 300, the subtitle module 208 processes the audio data 312 locally and from within a secure enclave of the operating system 110 to ensure user privacy and security.

[0065] The machine learning model 300 can be or include one or more of various types of machine learning models. In addition, the machine learning techniques described herein for training the machine learning model 300 can be easily interchangeable and combinable. Although certain example techniques have been described, many other techniques exist and can be used in conjunction with various aspects of the present disclosure. The machine learning model 300 can perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0066] The machine learning model 300 can be trained using supervised learning techniques, for example, the machine learning model 300 can be trained based on a training data set that includes examples of descriptions inferred from corresponding examples of audio data. The machine learning model 300 can also be trained using unsupervised learning techniques.

[0067] The machine learning model 300 may be or include one or more artificial neural networks (a type of "neural network"). As a neural network, the machine learning model 300 may include groups of connected or non-fully connected nodes, referred to as neurons or perceptrons. As a neural network, the machine learning model 300 may be organized into one or more layers, and in some cases, may include multiple layers when configured as a "deep" network. As a deep network, the machine learning model 300 may include an input layer, an output layer, and one or more hidden layers located between the input layer and the output layer.

[0068] The machine learning model 300 may be or include one or more recurrent neural networks. For example, the machine learning model may be implemented as an end-to-end recurrent neural network transducer automatic speech recognition model. Exemplary recurrent neural networks include long short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous time recurrent neural networks, neural history compressors, echo state networks, Elman networks, Elman networks; recursive neural networks, Hopfield networks, fully recurrent networks, and sequence-to-sequence configurations.

[0069] At least some of the nodes of the recurrent neural network may form a cycle. When configured as a recurrent neural network, the machine learning model 300 is particularly useful for processing input data that is sequential in nature, such as audio data 312 that has an inherent sequential nature. The recurrent neural network may transfer or retain information from a previous portion of the input data sequence to a subsequent portion of the input data sequence by using a loop or directed recurrent node connection. For example, the machine learning model 300 may transfer or retain information from a previous portion of the audio data 312 to a subsequent portion of the audio data 312 by using a loop or directed recurrent node connection between the previous and subsequent portions.

[0070] By definition, audio data 312 is sequential and may include time series data (e.g., sound data versus time). As a recurrent neural network, machine learning model 300 may analyze audio data 312 over time to detect or predict spoken sounds and non-verbal but relevant sounds to generate captions for audio data 312. More specifically, sequential sounds from audio data 312 may indicate spoken words in a sentence (e.g., for natural language processing, speech detection, or processing).

[0071] The machine learning model 300 may be or include one or more convolutional neural networks. A convolutional neural network may include one or more convolutional layers that perform convolutions on input data using learned filters or kernels. Convolutional neural networks are known to be useful for diagnosing visual problems, such as when analyzing image input data such as still images or videos. However, convolutional neural networks may also be applied to natural language processing of sound data, such as when generating subtitles from sequential audio data 312.

[0072] The machine learning model 300 may be trained using machine learning to receive audio data 312 as input data and provide description 318 as output data in response. The input data may include different types, forms, or variations of the audio data. As an example, in various embodiments, the audio data 312 may include raw, pre-mixed audio byte stream data passed from an application to an audio mixer, and the audio data 312 may also include processed byte stream data.

[0073] In response to receiving the audio data 312, the machine learning model 300 can provide a description 318. The output data can include different types, forms, or variations of the output data. As an example, in various implementations, the output data can include a description of the audible content corresponding to the visual content being displayed.

[0074] The machine learning model 300 can be trained in an offline manner or in an online manner. In offline training (also known as batch learning), the machine learning model 300 model is trained on the entirety of a static training data set, while in online learning, the machine learning model 300 is continuously trained (or retrained) as new training data becomes available (e.g., when the machine learning model 300 is used to perform inference). For example, the machine learning model 300 can be initially trained to replicate subtitles that have been applied to audible content (e.g., movie subtitles). As the machine learning model 300 is used to infer a description of the audio data 312, the description and corresponding portions of the audio data 312 can be fed back to the machine learning model 300 as new training data to enable the machine learning model 300 to continuously improve the description. The user can be provided with the opportunity to provide input to the machine learning model 300 to mark a particular description as having an error. Signals that the description may be wrong can also be used to train the machine learning model 300 to improve future predictions.

[0075] As part of the captioning module 208, the machine learning model 300 can be part of the operating system 110 and can therefore be included in a secure enclave to securely and privately process audio data for automatic captioning. Applications that interact with the operating system 110 can interact with the machine learning model 300 to process the audio data 312 into descriptions 318. For example, the application 112 can communicate through the operating system 110 with the model 300 using an application programming interface (API) (e.g., a common public API across all applications).

[0076] Figure 4 is a flow diagram illustrating example operations 400 of a computing device configured to automatically caption audio data. Figure 1 and Figure 2 The computing device 100 or 200 may perform operation 400. Figure 4 The operations 400 may be performed in an order different from that shown in FIG. 4 , including additional or fewer operations. The operations 400 are described below in the context of the computing device 200 .

[0077] At 402, computing device 200 obtains consent to use personal data to perform automatic captioning. For example, computing device 200 may generate captions using audio data only after computing device 200 receives explicit permission from a user of computing device 200 to use the audio data.

[0078] At 404 , computing device 200 displays the graphical user interface of the application. For example, computing device 200 can direct display component 106 to render user interface 114 and display application user interface 116 .

[0079] At 404, the computing device 200 obtains audio data including an audible portion of the content from the application. For example, the application 112 can render and play the media content 118. To provide subtitles, for example, if the media content 118 does not include annotated subtitles, the subtitle module 208 can extract the audio data being output from the application 112 when the audio mixer 206 receives the audio data.

[0080] At 406, computing device 200 determines whether the audio data is suitable for subtitles. For example, subtitle module 208 can filter out audio data of a type that does not require subtitles, such as notification sounds and other types of sounds accompanying content. Subtitle module 208 optionally determines whether the audio data is for subtitles based on a type identifier present in metadata embedded in the audio data. Subtitle module 208 optionally determines whether the audio data is for subtitles based on metadata indicating annotations (pre-created subtitles) included in the audio data. In response to determining that the audio data already has subtitles, subtitle module 208 can abandon performing automatic subtitles and display the subtitles pre-populated within the description.

[0081] The operating system 110 and the audio mixer 206 can define the various types of audio data that the application can assign to the output. Like other applications executed at the computing device 200, the application 112 uses an API to output audio data to the audio mixer 206. The API may include parameters for assigning audio data types within the metadata, for example, to assist the audio mixer 206 in correctly mixing and assigning the audio output determined from the data. Instead of relying on the audio data type to guide the sound, the subtitle module 208 uses a type identifier that is optionally present in the metadata to quickly and easily determine whether to perform subtitles. Specifically, the subtitle module 208 can focus on (key on) the specific type of audio data, the specific format, the duration, or other qualities and characteristics of the audio data to determine whether the audio data can be subtitled. Some types of audio data that may be suitable for subtitles include audio data of movie types, audio data of music types, and audio data of spoken types. On the contrary, audio data of vocalization types and other undefined audio data types may not be suitable for subtitles because the sound is not meant to convey content, but events or user actions.

[0082] If the audio data is not used for captioning, then at 414 , the computing device 200 displays the visual portion of the content. For example, the computing device 200 can display the media content 118 at the display component 106 .

[0083] However, if at 408, the computing device 200 determines that the audio data is for subtitles, the computing device 200 still determines whether to automatically subtitle the data based on whether the user selects automatic subtitles. For example, at 414, in the case where the user has not enabled automatic subtitles in the user interface 114 or the settings menu of the operating system 110, the computing device 200 avoids automatically subtitles the audio data and instead displays the visual portion of the content without generating descriptive subtitles. When the user has enabled automatic subtitles by selecting an appropriate option from a settings menu, etc., at 410, the subtitle module 208 uses a machine learning model to determine a description of the audible portion of the content. For example, the subtitle module 208 can perform an end-to-end automatic speech recognition model based on a recurrent neural network that is trained to take raw audio data output from an application such as application 112, and recognize spoken words and non-verbal sounds from the audio data to ultimately transcribe the spoken words into written words, and convert the non-verbal sounds into written descriptions of the sounds to be displayed as subtitles.

[0084] The captioning module 208 can improve the accuracy of the transcription and written description of the sound in various ways, including by biasing the end-to-end automatic speech recognition model based on the context of the computing device 200. For example, the captioning module 208 can bias the model based on the visual portion of the content or other information displayed on the screen. For example, when generating a description of the audible portion of a presentation (e.g., a slide show) that includes an audible narration, the portion of the text or images contained in the slides of the presentation can be used to bias the end-to-end automatic speech recognition model to generate the description, thereby improving the accuracy of the description. The captioning model 208 can use other types of contextual information, such as location information and information about other applications executed on the computing device 200, to bias the machine learning model.

[0085] At 412, the computing device 200 displays the description. For example, the captioning module 208 can output an indication of the description being generated as the machine learning model 300 receives the raw audio stream from the application 112.

[0086] In some cases, computing device 200 may translate portions of the description before displaying the description. For example, subtitle module 208 may determine the user's preferred language from operating system 110 and optionally translate the audio content while generating the description, thereby writing the text of the description in the user's preferred language rather than the original language of the audio content. In this way, a user from Japan, for example, may view the description of the audible content in Japanese or another preferred language, even if the audio of the content was recorded in a different language, such as Chinese or English.

[0087] At 414, computing device 200 displays the visual portion of the content simultaneously with the description. Operating system 110 may format the description as a persistent graphical element that the user can move around on top of the application graphical user interface to read a description of the audible portion of media content 118 being output for display.

[0088] FIG. 5A to FIG. 5G Each of the screen shots of an example user interface of a computing device for automatically captioning audio data is illustrated. Figures 5A to 5G .

[0089] exist Figure 5A In the example of FIG. 1 , computing device 200 displays user interface 114 at display component 106 . User interface 114 is associated with operating system 110 and shows application user interface 116 controlled by application 112 . Within application user interface 116 , application 112 includes media content 118 .

[0090] Figure 5B is a screenshot of the user interface 114 in response to receiving user input to display the settings menu 502. Figure 5B In the example of , settings menu 502 is an audio settings menu for adjusting volume control, alarm settings 504, mute control, etc. In addition, below the settings menu, computing device 200 displays subtitle control element 122A. Computing device 200 determines user input to automatically add subtitles to audio data in response to detecting a user selection of subtitle control element 122A. For example, Figure 5C The subtitle control element 122B is shown having replaced the subtitle control element 122A in response to the selection.

[0091] In some cases, in response to receiving a selection of the captioning control element 122B, the computing device 200 automatically terminates captioning the audible portion of the content on the computing device 200. In this manner, the control elements 122A and 122B provide the user with the ability to quickly start and stop automatic captioning.

[0092] exist Figure 5D In response to the selection, the computing device automatically captions the audio data output from the application 112 using the machine learning model. The computing device 200 generates a description of the audible portion of the media content 118 based on the output from the captioning module 208 including the confidence associated with the output.

[0093] The operating system 110 can cause the display component 106 to display the description within a persistent element 506 that the operating system 110 includes within the user interface 114. In some cases, the operating system 110 can display the persistent element 506 while avoiding the use of the speaker component 104 (e.g., a speaker, a headphone jack, or other sound system of the computing device 200) to output the audible portion of the content. Unlike other captioning systems, the computing system 200 can caption the content without generating audible sounds. In other cases, the operating system 110 can provide assistive listening functionality and display the persistent element 506 while outputting the audible portion of the content using the speaker component 104 (e.g., a speaker, a headphone jack, or other sound system of the computing device 200).

[0094] exist Figure 5E In the example of FIG. 1 , computing device 200 moves persistent element 506 from a first location of user interface 114 to a second location of user interface 114, demonstrating how persistent element 506 is separate and distinct from media content 118. A user of computing device 200 may provide a gesture at a location of input component 108 corresponding to the first location of the user interface and drag the input to a different location of the input component corresponding to the second location of user interface 114. Computing device 200 may cause persistent element 506 to move with the gesture.

[0095] In this way, Figure 5E The example of 506 shows that in response to receiving user input associated with persistent element 506, computing device 200 can move the persistent element away from application user interface 116 and a first portion of user interface 114 to obscure application user interface 116 and a second, different portion of user interface 114. This can enable a user to multitask, for example, scrolling content displayed in user interface 116 while audio data is being subtitled and always displayed on top of user interface 114.

[0096] exist Fig. 5F In the example of , computing device 200 enlarges persistent element 506 from a first size to a second (larger or smaller) size, demonstrating how persistent element 506 can be customized. A user of computing device 200 can provide a gesture at a location of input component 108 corresponding to persistent element 506 to stretch the persistent element or shrink the persistent element. In some cases, in response to receiving user input associated with persistent element 506, computing device 200 can modify the size of persistent element 506 to display a previous or subsequent description generated from an audible portion of the content. In this way, computing device 200 can cause the size of persistent element 506 to change as the size of the user input (e.g., a gesture) changes, and as a result adjust how much description is included in persistent element 506 at a particular time.

[0097] exist Figure 5G In the example of , the description of the audible portion of the content being displayed in the persistent element 506 includes text that identifies the verbal or non-verbal sources for different parts of the audible portion of the content. For example, the persistent element 506 indicates that the machine learning model of the captioning module 208 has identified a loud roar that may be from a lion. Also included in the persistent element 506 with the text indicating the non-verbal audio is a transcription of the verbal audio from the audible portion of the content. For example, a transcription of a journalist's conversation in the media content 118 is displayed in and around the indication of the lion's roar to provide ease of following the captioning of the media content 118.

[0098] Typically, computing device 200 may include a description of the noise and an indication of the source of the noise as part of the description within persistent element 506. The noise may include animal noise from an animal source, ambient noise from an environmental source, etc. The machine learning model of captioning module 208 is trained to recognize verbal and non-verbal audio from the audio data and describe the audio with sufficient description to enable a user of computing device 200 to understand the audio content.

[0099] Clause 1. A method for automatically captioning an audible portion of content on a computing device, the method comprising: obtaining, from an audio mixer of the computing device, audio data output from an application executed at the computing device, the audio data comprising data indicating the audible portion of the content; determining from the audio data whether the audio data is of a type suitable for captioning; in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content; and, while displaying a visual portion of the content, outputting the description of the audible portion of the content for display.

[0100] Clause 2. A method according to clause 1, wherein the data indicating the audible portion of the content is non-metadata, and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitles includes determining from the metadata whether the audio data is of a type suitable for subtitles.

[0101] Clause 3. The method of any of Clauses 1 and 2, wherein the description of the audible portion of the content comprises a transcription of spoken audio from the audible portion of the content.

[0102] Clause 4. The method of any of Clauses 1-3, wherein the description of the audible portion of the content comprises a description of non-verbal audio from the audible portion of the content.

[0103] Clause 5. The method of Clause 4, wherein the non-verbal audio comprises noise from a particular source, and the description of the noise from the particular source comprises an indication of the particular source.

[0104] Clause 6. The method of clause 5, wherein: the noise comprises animal noise from an animal source, or the noise comprises environmental noise from a non-animal source.

[0105] Clause 7. A method according to any one of clauses 1-6, wherein determining the description of the audible portion of the content includes: executing a machine learning model by the computing device, the machine learning model being trained to determine descriptions from audio data to determine the description of the audible portion of the content.

[0106] Clause 8. A method according to clause 7, wherein the machine learning model comprises an end-to-end recurrent neural network transducer automatic speech recognition model.

[0107] Clause 9. The method of any of Clauses 1-8, wherein the data indicative of the audible portion of the content comprises unannotated data that has not been annotated for captioning.

[0108] Item 10. A method for automatically captioning an audible portion of content on a computing device, the method comprising: displaying, by the computing device, a graphical user interface of an application executed at the computing device; while the graphical user interface is displayed, obtaining audio data output from the application, the audio data comprising data indicating the audible portion of the content; determining from the audio data whether the audio data is of a type suitable for automatic captioning; in response to determining that the audio data is of a type suitable for automatic captioning, determining a description of the audible portion of the content; and while displaying a visual portion of the content in the graphical user interface of the application, outputting the description of the audible portion of the content as a persistent element for display in addition to the graphical user interface of the application.

[0109] Clause 11. The method of Clause 10, wherein the description comprises at least one of: a transcription of spoken audio extracted from the audible portion of the content or text indicating non-verbal audio extracted from the audible portion of the content.

[0110] Clause 12. A method according to any of clauses 10-11, wherein the data indicating the audible portion of the content is non-metadata and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitles includes determining from the metadata whether the audio data is of a type suitable for subtitles.

[0111] Clause 13. The method of any of Clauses 10-12, wherein the description includes text identifying a human source or a non-human source of different portions of the audible portion of the content.

[0112] Clause 14. The method of any of Clauses 10-13, further comprising: in response to receiving a user input associated with the persistent element, modifying a size of the persistent element to display a previous or subsequent description generated from the audible portion of the content.

[0113] Clause 15. A method according to any one of clauses 10-14, wherein the persistent element is output for display by obscuring a first portion of the graphical user interface of the application, the method further comprising: in response to receiving user input associated with the persistent element, moving the persistent element away from the first portion of the graphical user interface of the application to obscure a second portion of the graphical user interface of the application.

[0114] Clause 16. A method for automatically captioning an audible portion of content on a computing device, the method comprising: receiving user input from an application executing at the computing device for automatically captioning audio data; in response to receiving the user input, obtaining the audio data output from the application executing at the computing device, the audio data comprising data indicating the audible portion of the content; determining from the audio data whether the audio data is of a type suitable for captioning; in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content; and outputting the description of the audible portion of the content as a persistent element for display in addition to a visual portion of the content and in addition to a graphical user interface of the application.

[0115] Clause 17. A method according to Clause 16, wherein the data indicating the audible portion of the content is non-metadata and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitles includes determining from the metadata whether the audio data is of a type suitable for subtitles.

[0116] Clause 18. A method according to any one of Clause 16 or Clause 17, wherein receiving the user input for automatically adding subtitles to the audio data comprises: displaying a subtitle control element within an audio settings menu of the computing device; and receiving the user input for automatically adding subtitles to the audio data in response to detecting a user selection of the subtitle control element.

[0117] Clause 19. The method of Clause 18, further comprising automatically terminating captioning of the audible portion of the content on the computing device in response to receiving a subsequent selection of the captioning control element.

[0118] Clause 20. A method as described in any of Clauses 16-19, wherein outputting the description of the audible portion of the content includes outputting the description while avoiding using a speaker, headphone jack, or other sound system of the computing device to output the audible portion of the content.

[0119] Clause 21. A method as described in any of Clauses 16-19, wherein outputting the description of the audible portion of the content includes outputting the description while outputting the audible portion of the content using a speaker, a headphone jack, or other sound system of the computing device.

[0120] Clause 22. The method according to any one of clauses 1-21 further comprises: training a machine learning model to determine a description of the audio data, wherein determining the description comprises using the machine learning model to determine the description.

[0121] Clause 23. A method according to Clause 22, wherein training the machine learning model includes using previously subtitled content to configure the machine learning model to infer subtitles embedded in the previously subtitled content from audio of the previously subtitled content.

[0122] Clause 24. A method according to Clause 22, wherein training the machine learning model includes using the description as a training input to the machine learning model.

[0123] Clause 25. A computing device comprising at least one processor configured to perform any of the methods recited in clauses 1-24.

[0124] Clause 26. A system comprising means for performing any of the methods recited in clauses 1-24.

[0125] Clause 27. A computer-readable storage medium comprising instructions that, when executed, configure a processor of a computing device to perform any of the methods of clauses 1-24.

[0126] Although various preferred embodiments of the present disclosure have been described in the foregoing description and shown in the accompanying drawings, it should be clearly understood that the present disclosure is not limited thereto, but can be embodied in various ways to be practiced within the scope of the following claims. From the foregoing description, it will be apparent that various changes can be made without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A computer-implemented method comprising: obtaining, by a processor of a computing device from an audio mixer of the computing device, audio data output from an application executing at the computing device, the audio data including non-metadata indicative of an audible portion of content and metadata tagged to the audio data and including a type identifier specifying a type of content; determining, by the processor, whether the content type specified by the type identifier tagged to the metadata of the audio data is a type suitable for subtitling; In response to determining that the type of content is a type suitable for subtitling, determining, by the processor, a description of an audible portion of the content for subtitling; as well as The description of the audible portion of the content is output by the processor for display while the visual portion of the content is output for display.

2. The method according to claim 1, in, Outputting the description of the audible portion of the content includes: A subtitle control element is output by the processor for display, wherein the subtitle control element includes an icon indicating that the computing device is performing subtitles.

3. The method according to any one of claims 1 and 2, wherein: The description of the audible portion of the content includes a transcription of spoken audio from the audible portion of the content.

4. The method according to claim 1, wherein: The description of the audible portion of the content includes a description of non-verbal audio from the audible portion of the content.

5. The method according to claim 4, wherein: The non-verbal audio includes noise from a particular source, and the description of the noise from the particular source includes an indication of the particular source.

6. The method according to claim 5, wherein: The noise includes animal noise from an animal source, or The noise includes ambient noise from non-animal sources.

7. The method according to claim 1, wherein: Determining the description of the audible portion of the content includes: executing, by the processor of the computing device, a machine learning model trained to determine descriptions from the audio data to determine the description of the audible portion of the content.

8. The method according to claim 7, wherein: The machine learning model includes an end-to-end recurrent neural network transducer automatic speech recognition model.

9. The method according to claim 1, wherein: The non-metadata indicative of the audible portion of the content includes unannotated data that has not been annotated for captioning.

10. The method according to claim 1, wherein: The description includes text indicating non-verbal audio extracted from an audible portion of the content.

11. The method according to claim 1, wherein: The description includes text identifying a human source or a non-human source of different portions of the audible portion of the content.

12. The method according to claim 1, wherein: Outputting the description of the audible portion of the content includes outputting, by the processor, the description of the audible portion of the content as a persistent element for display in addition to the visual portion of the content and in addition to a graphical user interface of the application.

13. The method according to claim 12, further comprising: In response to receiving, by the processor, user input associated with the persistent element, a size of the persistent element is modified to display a previous or subsequent description generated from the audible portion of the content.

14. A computing device comprising at least one processor, the at least one processor being configured to execute the method of any one of claims 1-13.

15. A computer-readable storage medium comprising instructions that, when executed, configure a processor of a computing device to perform the method of any one of claims 1-13.

Images