Automatic captioning of audible portions of content on a computing device - Patents.com

Abstract

To provide a computing device and method for automatically captioning content.SOLUTION: A computing device 100 generates captions directly from audio data being output from content sources. The computing device may analyze metadata to determine whether the audio data is suitable for captioning or whether the audio data is some other type of audio data. Responsive to identifying audio data for captioning, the computing device further can quickly generate a description of audible sounds interpreted from the audio data, and can provide the descriptions of audible content visually no matter the application source of the audible content and regardless whether the computing device ever produces audio of the content.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] background Most audible content outside of professionally produced movies and television programs is not captioned, making the content less accessible to many users who are hearing impaired or otherwise unable to hear the content. Although some automatic captioning systems are available, these systems can be expensive, complex, and difficult to maintain. Some automatic captioning systems require audio output to a speaker as input, and such systems may not be suitable for providing captioning in quiet environments. To avoid the cost and complexity involved in maintaining automatic captioning systems, some computing devices access cloud-based captioning services that require content to be uploaded to a remote server for captioning, potentially reducing user privacy and convenience. Summary of the Invention [Means for solving the problem]

[0002] overview A computing device is described for automatically captioning content as a local, system-level service. Unlike other captioning systems that often rely on information contained in an audio signal sent to a speaker as input, the computing device generates captions directly from audio data output from a content source (e.g., an application). The computing device may analyze metadata tagged to the audio data (e.g., non-audio portions) to determine whether the audio data is suitable for captioning or whether the audio data is another type of audio data (e.g., system sound effects).

[0003] In response to identifying the audio data as suitable for captioning and in response to determining that automatic captioning is enabled (e.g., via user input), the computing device uses a machine learning model to rapidly generate a description of the audible sounds interpreted from the audio data. The computing device can provide a description of the audible content visually (e.g., as a persistent user interface element) regardless of the application source of the audible content and whether or not the computing device generates the audio (e.g., an audible sound signal) for the content. In this manner, the computing device captions system-level audio to describe the audible portions of the content even if the audio data does not include embedded captions, even if the device does not use a server for captioning, and even if the computing device is muted and not outputting audible sounds.

[0004] By relying on the original audio data rather than the generated audio signal as input for the speaker, the machine learning model can generate captions that more accurately represent the sounds from the content as originally intended by the content source. Additionally, by determining whether the audio data is suitable for captioning prior to using the machine learning model, the computing device can avoid wasting resources by over-analyzing all of the audio data output by the content source, including portions of the audio data that almost certainly do not contain anything suitable for captioning. This allows for a more accurate representation of the sounds from the content as originally intended by the content source. A smaller and / or less complex machine learning model can be used, thereby allowing the computing device to perform more efficiently. For example, the machine learning model can automatically recognize speech to analyze only the audio data that includes the speech. Thus, the machine learning model can automatically recognize speech and automatically classify sounds locally from the computing device generating the audio data, thereby improving privacy and offline convenience. Thus, the computing device can automatically caption system-level audio such that, at a minimum, user satisfaction with the automatic captioning system can be improved. Users with medical needs can experience improved quality of life by accessing a computing device 110 that automatically captions audio data in this manner. Users who may not have previously been able to hear or understand the audible portions of the content can now enjoy the content in the same way as other users who do not have medical needs.

[0005] Throughout this disclosure, examples are described in which a computing device may analyze information (e.g., audio data). However, the computing device may only use the information if it receives explicit permission from a user of the computing device to use the data. For example, in the situation described below in which the computing device analyzes audio data output from an application running on the computing device, an individual user may be given the opportunity to provide input to control whether a program or feature of the computing device can collect and utilize the audio data, for example for automatic captioning. An individual user may always control what programs can and cannot act on audio data.

[0006] The collected information may also be pre-processed in one or more ways before being transferred, stored, or used by the computing device and / or remote computing system, such that personally identifiable information is removed. For example, an exemplary computing device may pre-process the audio data before sharing captions generated from the audio data with another device (e.g., to train a model running on the other device) to ensure that any user-specific or device-specific information embedded in the data is removed. Thus, a user may control whether information about the user and the user's device is collected, and if so, how such information may be used by the computing device and / or remote computing system.

[0007] In one example, a method for automatically captioning an audible portion of content on a computing device is described, the method including obtaining audio data output from an application executing on the computing device from an audio mixer of the computing device, the audio data including non-metadata and optionally metadata indicative of the audible portion of the content, the method further including determining, optionally from the metadata of the audio data, whether the audio data is of a type suitable for automatic captioning, in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content, and outputting the description of the audible portion of the content for display while displaying the visual portion of the content.

[0008] In a different example, a method is described for automatically captioning audible portions of content on a computing device, the method comprising: a computing device providing a graphical user interface to an application running on the computing device; displaying the interface and obtaining audio data output from the application while displaying the graphical user interface, the audio data including non-metadata and optionally metadata indicative of an audible portion of the content, the method further including optionally determining from the metadata of the audio data whether the audio data is of a type suitable for automatic captioning. In response to determining that the audio data is of a type suitable for captioning, the method further includes determining a description of the audible portion of the content, and outputting the description of the audible portion of the content for display as a persistent element distinct from the graphical user interface of the application while displaying the visual portion of the content within the graphical user interface of the application.

[0009] In yet another example, a method for automatically captioning an audible portion of content on a computing device is described. The method includes receiving a user input for automatically captioning audio data from an application executing on the computing device, and in response to receiving the user input, obtaining audio data output from the application executing on the computing device, the audio data including the audible portion of the content. The method further includes determining, optionally from metadata of the audio data, whether the audio data is of a type suitable for captioning, and in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content. The method further includes outputting the description of the audible portion of the content for display as a persistent element distinct from the visual portion of the content and distinct from a graphical user interface of the application.

[0010] In a further example, a computing device is described that includes at least one processor configured to execute the method described in any of the above examples.

[0011] In a further example, a system is described that includes means for performing the method according to any of the above examples.

[0012] In another example, a computer-readable storage medium is described that includes instructions that, when executed, configure a processor of a computing device to perform any of the methods described in any of the above examples.

[0013] The details of one or more implementations are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the description and drawings, and from the claims. This Summary is provided to introduce the subject matter that is further described in the Detailed Description and Drawings. As such, this Summary should not be considered as describing essential features, nor should it be used to limit the scope of the claimed subject matter.

[0014] Details of one or more aspects of automatic captioning are described below. The use of the same reference numbers in various locations in the description and figures indicates like elements. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a conceptual diagram illustrating a computing device configured to automatically caption audio data. [Diagram 2] FIG. 2 is another conceptual diagram illustrating a computing device configured to automatically caption audio data. [Diagram 3] FIG. 1 is a conceptual diagram illustrating a machine learning model of a computing device configured to automatically caption audio data. [Figure 4] 1 is a flowchart illustrating an example of the operation of a computing device configured to automatically caption audio data. [Figure 5A]1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. [Figure 5B] 1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. [Figure 5C] 1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. [Figure 5D] 1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. [Figure 5E] 1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. [Figure 5F] 1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. [Figure 5G] 1 is a screenshot illustrating an example of a user interface of a computing device for automatically captioning audio data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Detailed Description Most audible content outside of professionally produced films and television programs is not captioned, making the content less accessible to many users who are hearing impaired or cannot hear the content for other reasons. Manual and automatic captioning systems exist, but suffer from a variety of shortcomings.

[0017] For example, some manual captioning systems caption audible content based on embedded captioning information contained in pre-recorded content, while others caption content based on input from a human operator (e.g., during a live broadcast).

[0018] Although some automatic captioning systems are available, these systems can be expensive, complex, and difficult to maintain. For example, an automatic captioning system may require audio output to a sound system as a source for captioning. The captioning system may require this audio output to maintain a minimum sound level, making such a system unsuitable for operation in quiet environments. Furthermore, the audio output may include more audible sounds (e.g., system sounds, sounds from other background applications) than just sounds for captioning, and thus captioning may be more time consuming and waste resources by unnecessarily and indiscriminately captioning all sounds from all audio sources of a computing device.

[0019] Some automatic captioning systems may be application-based and run locally on a computing device. However, such systems may still require an Internet connection for captioning. For example, to avoid the costs and complexities involved in maintaining an automatic captioning system, a computing device may access a cloud-based captioning service that requires uploading content to a remote server for captioning, potentially reducing user privacy and convenience.

[0020] To efficiently and differentially enable automatic captioning of audible content These techniques do not require a network connection (e.g., to the Internet) and do not generate captions from audible signals transmitted to a sound system. Instead, an exemplary computing device can automatically generate captions from audio data collected by an audio mixer of the computing device. As used throughout this specification, "audio data" refers to information that is converted into an audio signal sent to a system audio mixer or sound engine to generate sound as an output. Audio data is contrasted with the audio signal that is generated in response thereto.

[0021] The operating system or platform of the exemplary computing device includes a system-level audio mixer that collects audio data output from various content sources (e.g., applications) and mixes the audio data to generate an audio signal. The audio signal may be amplified by a sound system to generate an audible sound. Unlike other captioning systems that indiscriminately caption audio signals, the exemplary computing device differentially captions audible content by captioning only audible content from audio data of a type suitable for captioning.

[0022] The computing device may analyze metadata (e.g., non-audio portions of the audio data) to determine whether the audio data is suitable for captioning or whether the audio data is another type of audio data (e.g., system sound effects). By relying on metadata missing from the audio signal, the computing device can quickly filter out audio data that is not suitable for captioning, and thus the computing device can avoid wasting resources like other captioning systems that attempt to caption all content, even content that cannot be captioned or that should not be captioned.

[0023] In response to identifying audio data for captioning and in response to determining that automatic captioning is enabled (e.g., via user input), the computing device uses a machine learning model trained using machine learning to rapidly generate descriptions of audible sounds interpreted from the audio data. For example, the computing device may rely on an end-to-end recurrent neural network transducer automatic speech recognition model or other machine learning model to determine a transcription of spoken audio or to identify specific noises from non-spoken audio and possible sources of the specific noises.

[0024] The computing device can provide a description of the audible content visually (e.g., as a persistent user interface element) regardless of the source of the audible content and whether or not the computing device generates audio (e.g., an audible sound signal) for the content. In this manner, the computing device performs a system-level captioning operation that provides a description of the audible portions of the content, even if the content does not include embedded captions and even if the computing device is muted and does not output an audio signal.

[0025] By relying on audio data rather than the audio signal used as input to the speaker, the machine learning model generates captions that more accurately represent the sounds from the audible content as originally intended by the content source. Audio data from an application or other content source may more accurately represent the audible content because it is generally not altered since it is output from the content source. Other automatic captioning systems generate captions that more accurately represent the sounds from the audible content as originally intended by the content source. They may rely on a processed audio signal intended as input to the speaker, and therefore may introduce differences from that intended by the content source.

[0026] Furthermore, by determining whether the audio data is suitable for captioning prior to utilizing a machine learning model for explanation, the computing device avoids wasting resources by overanalyzing all of the audio data output by the content source, including portions of the audio data that almost certainly do not contain anything suitable for captioning. This allows the computing device to train and run smaller and / or less complex machine learning models to perform automatic speech recognition and automatic sound classification techniques, without the need for further training the models to ignore various types of audio data that may not be suitable for captioning. This further allows the models to be run locally on the computing device where the audio data is generated, improving privacy and offline usability.

[0027] By relying on more accurate data, the accuracy of the machine learning model can be improved, and by relying only on relevant data, the efficiency of the machine learning model can be improved, further improving user satisfaction with the automatic captioning system, and users with medical needs can experience improved quality of life by having access to a computing device 110 that automatically captions audio data in this manner.

[0028] The description may include a transcription of the spoken content, possibly even identifying the name or description of the speaker. In some cases, the audible sounds include non-spoken content, such as animal sounds, environmental sounds, and other sounds that may or may not be related to the description. The description may even include a description of some non-spoken sounds (e.g., a dog barking) including an identification of the corresponding source (e.g., a barking dog).

[0029] FIG. 1 is a conceptual diagram illustrating a computing device configured to automatically caption audio data. Computing device 100 may be any type of mobile or non-mobile computing device. As a mobile computing device, the computing device may be a mobile phone, a laptop computer, a wearable device (e.g., a watch, glasses, headphones, clothing), a tablet device, an automobile / vehicle device, a portable gaming device, an electronic reader device, a remote control device, or other mobile computing devices. As a non-mobile computing device, computing device 100 may be a server, a network terminal device, a desktop computer, a television device, an entertainment set-top device, a streaming media device, a tabletop assistant device, a speaker device, a sound system, a non-portable gaming device, a business conference equipment, or other non-mobile computing devices.

[0030] Computing device 100 includes a user interface device 102 that includes a speaker component 104, a display component 106, and an input component 108. Computing device 100 also includes an operating system 110 and applications 112. These and other components of computing device 100 are communicatively coupled in a variety of ways, including the use of wired and wireless buses and links. Computing device 100 may include more or fewer components than those shown in FIG.

[0031] Operating system 110 generally controls the functioning of computing device 100, including user interface device 102 and other peripherals. Operating system 110 provides an execution environment for applications, such as application 112. Operating system 110 may also control task scheduling and other general functions, and generally does so through a system-level user interface 114.

[0032] The user interface device 102 manages input and output to a system level user interface 114 controlled by the operating system 110. The system level user interface 114 includes a persistent graphical element 120 for displaying automatic captions. The system level user interface 114 further includes one or more caption control elements 122A and 122B that, when selected, control whether the operating system 110 automatically captions audio data, including whether the user interface 114 displays the persistent graphical element 120. In some cases, the caption control elements 122A and 122B may include an icon that indicates whether the computing device 100 is automatically captioning audio data. The system level user interface 114 may display user interfaces of other applications running on the computing device 100, such as an application user interface 116 for the application 112.

[0033] The display component 106 and the speaker component 104 may be separate or integrated as a single component. The display component 106 may be made of any suitable display technology, including LED, OLED and LCD technology. The speaker component 104 (e.g., a single speaker or multiple speakers) is configured to receive an audio signal as an input and convert the audio signal into audible sound. The input component 108 may be a microphone, a presence sensing device, a touch screen, a mouse, a keyboard, or any other type of component configured to receive user input.

[0034] The user interface device 102 may output information about the input detected by the input component 108, and the operating system 110 may correlate the detected input with an element of the user interface 114. In response to receiving an input (e.g., a gesture) at the input component 108, the operating system 110 and / or the application 112 may receive information about the detected input from the user interface device 102. The operating system 110 or the application 112 may perform a function in response to the input. For example, the operating system 110 may determine that the input corresponds to a selection of either the caption control element 122A or 122B, and in response, automatically caption (or cease automatically captioning) the audio data.

[0035] The application user interface 116 includes media content 118, such as movies, videos, videos, or other content with embedded audio. The media content 118 may be any multimedia or social media content, including from a video platform, a streaming platform, radio, podcasts, videos, video chat applications, telephony applications, or other sources of audio data running on or accessible from a computing device.

[0036] The applications 112 may output the media content 118 to the operating system 110. Using the display component 106, the operating system 110 can cause the user interface device 102 to output images or other visual representations of the media content 118 within the application user interface 116. The computing device 100 can further cause the user interface device 102 to forward audio signals generated from the media content 118 to the speaker component 104. In this manner, the operating system 110 can cause the speaker component 104 to output audio simultaneously with images or other video displayed on the display component 106.

[0037] During operation, the operating system 110 automatically captions the audible portions of the content. The computing device obtains the audio data used for captioning from an audio mixer or sound engine of the operating system 110. The audio data from the application 112 may include the audible portions of the media content 118.

[0038] 1, a user of computing device 100 may interact with application 112 by viewing media content 118 displayed on the screen of display component 106. The user may want to enable captioning for media content 118, but the media content 118 contains only uncaptioned audio data and there is no way for application 112 to generate and embed captions in the media content 118.

[0039] Nevertheless, in response to receiving user input (e.g., a touch gesture at a location on the input component 108 corresponding to the location on the screen of the display component 106 where the selectable control element 122B is displayed), the computing device 100 may automatically caption the audible portions of the media content 118 by processing the same audio data output from the application 112 and received at the audio mixer.

[0040] The audio mixer is configured to collect all audio data output from various content sources executing within the execution environment of the operating system 110, including applications 112. The audio mixer provides an interface between the application sources generating the content and the speaker component 104, which creates sound from the content. The audio mixer manages the raw audio data byte stream, analyzes the audio data, and directs audio signals to be output to the speaker component 104 or other suitable output component of the user interface device 102 to create sound.

[0041] Rather than indiscriminately captioning all audio data from the audio signal sent to speaker component 104, operating system 110 may rely on metadata to focus automatic captioning on audible portions of content that are suitable for captioning. In other words, operating system 110 identifies "captivable" audio data based on metadata and refrains from captioning all audio data unless all audio data is deemed captivable.

[0042] Some examples of metadata include why an application source is outputting audio data. The metadata includes a usage indicator that specifies how the audio data should be played. An audio mixer may use this usage indicator to control routing, focus and volume decisions regarding the audio data. The metadata may include other flags that further specify how the sound should be played and information for audio / video synchronization.

[0043] The metadata may include a content type that specifies what the application source is playing (e.g., music, movie, speech, sonification, unknown). The content type may represent a general category of content (e.g., one type of movie streaming service and another type of music streaming service). The audio mixer can selectively configure some audio post-processing and adjust other settings from the type identifier. The type identifier may specify a movie type if the audio data is for a soundtrack that accompanies a movie or television program. The type identifier may specify a music type identifier if the content is a song (e.g., not necessarily a soundtrack), and the type identifier may indicate a speech identifier if the content is spoken audio (e.g., a news broadcast, video, or phone call). As another example, the content type may indicate sonification if the audio data is for a notification, alert, or sound used to accompany a user action (e.g., a beep or sound effect representing a key click) or an event (e.g., a sound for achieving a bonus in a game, etc.).

[0044] In addition to the audio mixer relying on content type, operating system 110 may optionally identify audio data for captioning based on a content type identifier found in the metadata. For example, operating system 110 may automatically caption audio data of type movie, music, or speech, but not automatically caption audio data of type sonification, or other audio data that is undefined or otherwise defined. Thus, using the content type, operating system 110 may determine whether the audio data is likely to have sounds that require captioning, or whether the audio data is likely to represent other sounds that are not intended for captioning.

[0045] In response to determining that the type of audio data associated with the media content 118 is suitable for captioning, the operating system 110 determines a description of the audible portion of the content 118. For example, the operating system 110 may execute a machine learning model (e.g., an end-to-end recurrent neural network transducer automatic speech recognition model) trained to generate a description of the audible content as a caption. The machine learning model may be any type of model suitable for learning a description of sounds, including transcriptions for spoken speech. However, the machine learning model used by the operating system 110 will be small in size and not very complex because the machine learning model only needs to be trained to identify sounds and spoken speech from a particular type of content. The machine learning model does not need to process all audio data sent to the audio mixer, but rather only the particular audio data that is likely to contain content for captioning. This contrasts with other captioning systems that require remote processing facilities and remote uploading of content, which compromises privacy and sacrifices convenience.

[0046] The operating system 110 receives the explanation from the machine learning model and displays the explanation to the user. The explanation may include a transcription of the spoken conversation or song. The explanation may identify the context of the sound or may identify the speaker, singer, or individual actor or performer. The explanation may include a description of the sound, for example, "dog barking" if a dog sound is detected. For the sound of a door slamming, the audio may include "door closed."

[0047] While displaying the visual portions of the media content 118 within the application user interface 116, the operating system 110 outputs a description of the audible portions of the content for display. For example, the operating system 110 may display the persistent element 120 as an overlay of the user interface 114 that appears on top of the application user interface 116 and other graphical elements of the user interface 114. A user can manipulate the persistent element 120 to display more or less of the caption by moving it to different areas of the display screen or by enlarging or reducing the size of the element.

[0048] By relying on the original audio data rather than the generated audio signal as input for the speaker, the machine learning model of the operating system 110 generates captions that more accurately represent the sounds from the media content 118 as originally intended by the content source (e.g., application 112). Furthermore, by determining whether the audio data is suitable for captioning prior to using the machine learning model, the operating system 110 avoids wasting resources by over-analyzing all of the audio data output by the application 112, including audio data that almost certainly does not contain anything suitable for captioning. This allows the computing device 110 to run more efficient, smaller and / or less complex machine learning models. Thus, the machine learning model can perform automatic speech recognition and automatic sound classification techniques locally from the computing device 110 that generates the audio data, thereby improving privacy and offline convenience. Thus, the computing device 110 can automatically caption system-level audio such that, at a minimum, user satisfaction with the automatic captioning system can be improved. Users with medical needs can experience improved quality of life by having access to a computing device 110 that automatically captions audio data in this manner.

[0049] Figure 2 is another conceptual diagram illustrating a computing device 200 configured to automatically caption audio data. Computing device 200 is an example of computing device 100 with additional details. As shown in Figure 2, computing device 200 may be a mobile phone 100-1, a laptop computer 100-2, a television set / display 100-3, a desktop computer 100-4, a tablet device 100-5, a computerized wristwatch 100-6 or other wearable device, or a computing system installed in a vehicle 100-7.

[0050] 1, computing device 200 includes one or more processors 202, computer-readable media 204, one or more sensors 210, one or more input / output (I / O) devices 212, and one or more communications devices 214. Computer-readable media 212 includes instructions that, when executed by processor 102, perform applications 112 and operating system 110.

[0051] Processor 202 may include any combination of one or more controllers, microcontrollers, processors, microprocessors, hardware processors, hardware processing units, digital signal processors, graphics processors, graphics processing units, etc. Processor 202 may be an integrated processor or a processor that processes computer-executable instructions to control the operation of computing device 200. It may also be a memory subsystem (eg, implemented as an SoC).

[0052] The sensors 210 acquire contextual information indicative of the computing device's physical operating environment and / or characteristics of the computing device 200 while functioning in the physical operating environment. Examples of the sensors 210 include motion sensors, temperature sensors, position sensors, proximity sensors, ambient light sensors, humidity sensors, pressure sensors, etc. The operating system 110 may adapt the operation of the computing device 200 according to the sensor information acquired by the sensors 210.

[0053] The input / output devices 212 provide connectivity with the computing device 200 and other devices and peripherals, including data network interfaces that provide connections and / or communication links between devices, data networks (e.g., mesh networks, external networks, etc.), and other devices. The input / output devices 212 can be used to couple the computing device 200 to any type of component, peripheral, and / or attached device. The input / output devices 212 also include data input ports through which the computing device 200 can receive user input, as well as any type of communication data, and any type of data, media content, and / or input, such as audio, video, and / or image data received from any content and / or data source.

[0054] The communications device 214 enables wired and / or wireless communication of device data 506, such as any type of media content generated or received applications executing on the computing device 200, such as the applications 112. The communications device 214 may also include a transceiver for cellular phone communication and / or network data communication.

[0055] The computer readable medium 204 is configured to provide persistent and non-persistent storage of executable instructions (e.g., firmware, recovery firmware, software, applications, modules, programs, functions, etc.) and data (e.g., user data, operational data) to the computing device 200 to support execution of the executable instructions. Examples of the computer readable medium 204 include volatile and non-volatile memory, fixed and removable media devices, and any suitable memory device or electronic data storage that maintains executable instructions and supports data. The computer readable medium 204 may include various implementations of random access memory (RAM), read only memory (ROM), flash memory, and other types of storage memory in various memory device configurations. The computer readable medium 204 excludes propagating signals. The computer readable medium 204 may be a solid state drive (SSD) or a hard disk drive (HDD). The computer readable medium 204 in the example of FIG. 2 includes the application 112 and the operating system 110.

[0056] The operating system 110 of the computing device 200 includes an audio mixer 206 and a caption module 208. The audio mixer 206 and the caption module 208 may be implemented as dedicated hardware or software components of the operating system 110. In other examples, the audio mixer 206 or the caption module 208 may be implemented separately from the operating system 110, for example as a system plug-in or an additional add-on service installed locally to the operating system 110.

[0057] Audio mixer 206 is configured to integrate audio data generated by applications executing within the operating environment provided by operating system 110. Audio mixer 206 combines audio streams from applications, such as application 112, to generate an audio output signal that, when combined and output from speaker component 206, reproduces the sound encoded in the audio streams. Audio mixer 206 may also condition the audio signal in other ways, such as controlling focus, intent, and volume.

[0058] The caption module 208 is configured to automatically caption audio data in its raw form as it is received by the audio mixer 206 (e.g., as a byte stream). Rather than processing all post-mixed audio signals for captioning, the caption module 208 identifies individual pre-mixed streams of audio data received at the audio mixer 206 that are suitable for captioning. For example, the caption module 208 may automatically caption spoken voice type audio data rather than notification or sonification type audio data such as system beeps and bell sounds. The caption module 208 may apply filters to the byte stream received by the audio mixer 206 to identify types of audio data suitable for captioning. The caption module 208 uses machine learning models to determine descriptions of sounds represented by audio data identified for captioning.

[0059] FIG. 3 is a conceptual diagram illustrating a machine learning model of a computing device configured to automatically caption audio data. The machine learning model 300 may be part of the caption module 208. That is, the caption module 208 may include the machine learning model 300 to automatically convert coded audio data into detailed descriptions or captions of the audible content without necessarily converting the audio data into sound. The computing device 200 may execute the machine learning model 300 as part of the execution of the caption module 208 to determine descriptions 318 of audible portions of content found in the audio data 312. The machine learning model 300 may include one or more types of machine learning models that are combined into a single model that provides descriptions 318 in response to the audio data 312.

[0060] The machine learning model 300 is configured to perform inference, and the machine learning model 300 is trained to receive audio data 312 as input and provide as output data a description 318 (e.g., a caption) of a sound determined by the machine learning model 300 from the audio data 312. In addition to any annotations (i.e., annotated data) contained in the audio data 312, the audio data 312 may include unannotated data that was not annotated for captioning. By performing inference using the machine learning model 300, the caption module 208 processes the audio data 312 locally from within a secure enclave of the operating system 110 to ensure user privacy and security.

[0061] The machine learning model 300 may be or include one or more of various types of machine learning models. Also, the machine learning techniques described herein used to train the machine learning model 300 may be readily interchangeable and combinable. Although certain exemplary techniques have been described, many other techniques exist and may be used in conjunction with aspects of the present disclosure. The machine learning model 300 may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0062] The machine learning model 300 can be trained using supervised learning techniques, for example the machine learning model 300 can be trained based on a training dataset that includes examples of explanations inferred from corresponding examples of audio data. The machine learning model 300 can also be trained using unsupervised learning techniques.

[0063] The machine learning model 300 may be or include one or more artificial neural networks (a type of "neural network"). As a neural network, the machine learning model 300 may include a set of connected or not fully connected nodes called neurons or perceptrons. As a neural network, the machine learning model 300 may be organized into one or more layers, and in some cases may include multiple layers when configured as a "deep" network. As a deep network, the machine learning model 300 may include an input layer, an output layer, and one or more hidden layers positioned between the input layer and the output layer.

[0064] The machine learning model 300 may be or may include one or more recurrent neural networks. For example, the machine learning model may be implemented as an end-to-end recurrent neural network transducer automatic speech recognition model. Examples of recurrent neural networks include long-term short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, neural history compressors, echo state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks, fully recurrent networks, and sequence-to-sequence configurations.

[0065] At least some of the nodes of the recurrent neural network can form a cycle. When configured as a recurrent neural network, the machine learning model 300 may be particularly useful for processing input data of a sequential nature, such as the audio data 312, which has an inherent sequential nature. The recurrent neural network can pass or retain information from earlier portions of the input data sequence to later portions of the input data sequence by using recurrent or directed periodic node connections. For example, the machine learning model 300 can pass or retain information from earlier portions to later portions by using recurrent or directed periodic node connections between earlier portions of the audio data 312 and later portions of the audio data 312.

[0066] By definition, the audio data 312 is sequential and may include time series data (e.g., sound data versus time). As a recurrent neural network, the machine learning model 300 may analyze the audio data 312 over time to detect or predict spoken and non-spoken related sounds for generating captions for the audio data 312. More specifically, the sequential sounds from the audio data 312 may indicate spoken words in a sentence (e.g., for natural language processing, speech detection or processing).

[0067] The machine learning model 300 may be or may include one or more convolutional neural networks. A convolutional neural network may include one or more convolutional layers that perform convolutions on input data using learned filters or kernels. Convolutional neural networks are known to be useful in diagnosing vision problems, such as when analyzing image input data, such as still images or video. However, convolutional neural networks are also useful for natural language processing of audio data, such as when generating captions from sequential audio data 312. can be applied.

[0068] The machine learning model 300 can be trained using machine learning to receive audio data 312 as input data and, in response, provide explanations 318 as output data. The input data can include various types, forms, or variations of audio data. By way of example, in various implementations, the audio data 312 can include unprocessed pre-mixed audio byte stream data passed from an application to an audio mixer, and the audio data 312 can also include processed byte stream data.

[0069] In response to receiving the audio data 312, the machine learning model 300 can provide an explanation 318. The output data can include various types, forms, or variations of the output data. By way of example, in various implementations, the output data can include a description of the audible content that corresponds to the visual content being displayed.

[0070] The machine learning model 300 may be trained in an offline or online manner. In offline training (also known as batch learning), the machine learning model 300 is trained on an entire static training dataset, and in online learning, the machine learning model 300 is continuously trained (or retrained) as new training data becomes available (e.g., while performing inference using the machine learning model 300). For example, the machine learning model 300 may initially be trained to reproduce captions (e.g., movie captions) that have already been applied to audible content. As the machine learning model 300 is used to infer explanations for the audio data 312, the explanations and corresponding portions of the audio data 312 may be fed back to the machine learning model 300 as new training data to enable the machine learning model 300 to constantly improve the explanations. A user may be given the opportunity to provide input to the machine learning model 300 to flag certain explanations as being erroneous. Signals that an explanation may be erroneous may also be used in training the machine learning model 300 to improve future predictions.

[0071] As part of the caption module 208, the machine learning model 300 may be part of the operating system 110 and therefore may be included in a secure enclave to securely and privately process audio data for automatic captioning. Applications that interact with the operating system 110 may interact with the machine learning model 300 to process the audio data 312 into descriptions 318. For example, the applications 112 may communicate with the model 300 via the operating system 110 using an application programming interface (API) (e.g., a common public API across all applications).

[0072] Figure 4 is a flow chart illustrating example operations 400 of a computing device configured to automatically caption audio data. The computing devices 100 or 200 of Figures 1 and 2 may perform the operations 400. The operations 400 may include more or fewer operations and may be performed in a different order than that shown in Figure 4. The operations 400 are described below in the context of computing device 200.

[0073] At 402, computing device 200 obtains consent to perform automatic captioning using personal data. For example, computing device 200 may generate a caption using audio data only after computing device 200 receives explicit permission from a user of computing device 200 to use the audio data.

[0074] At 404, the computing device 200 displays a graphical user interface of the application. For example, the computing device 200 may instruct the display component 108 to display the user interface 114 to display the application user interface 116.

[0075] At 404, the computing device 200 obtains audio data from the application, including the audible portion of the content. For example, the application 112 may translate and play back the media content 118. To provide captioning, for example if the media content 118 does not include annotated captions, the caption module 208 may extract the audio data output from the application 112 as the audio data is received at the audio mixer 206.

[0076] At 406, the computing device 200 determines whether the audio data is suitable for captioning. For example, the caption module 208 may remove types of audio data that do not require captioning, such as notification sounds and other types of sounds that accompany the content. The caption module 208 determines whether the audio data is for captioning based on a type identifier found in metadata optionally embedded in the audio data. The caption module 208 determines whether the audio data is suitable for captioning based on metadata optionally indicating that annotations (pre-created captions) are included in the audio data. In response to determining that the audio data already has a caption, the caption module 208 may forgo performing automatic captioning and display the pre-added caption in the description.

[0077] The operating system 110 and audio mixer 206 may define various types of audio data that applications may assign to output. Like other applications running on the computing device 200, the applications 112 use the API to output audio data to the audio mixer 206. The API may include parameters for specifying the audio data type in the metadata, for example to assist the audio mixer 206 in properly mixing and distributing the audio output determined from the data. Instead of relying on the audio data type to guide the sound, the caption module 208 may use a type identifier, optionally found in the metadata, to quickly and easily determine whether to perform captioning. Specifically, the caption module 208 may specify a particular type of audio data, a particular format, duration, or other quality and characteristics of the audio data to determine whether the audio data can be captioned. Some types of audio data that may be suitable for captioning include movie-type audio data, music-type audio data, and spoken-type audio data. In contrast, sonification type audio data and other undefined audio data types would not be suitable for captioning because the sounds are intended to convey events or user actions, not content.

[0078] If the audio data is not suitable for captioning, then the computing device 200 displays the visual portion of the content at 414. For example, the computing device 200 may display the media content 118 on the display component 108.

[0079] However, if the computing device 200 determines at 406 that the audio data is suitable for captioning, the computing device 200 still determines whether to automatically caption the data based on whether automatic captioning was selected by the user. If the user has not enabled automatic captioning in the settings menu of the interface 114 or in the operating system 110, the computing device 200 refrains from automatically captioning the audio data and instead displays the visual portions of the content without generating a descriptive caption. If the user has enabled automatic captioning by selecting an appropriate option from a settings menu or the like, then at 410 the caption module 208 uses a machine learning model to determine descriptions for the audible portions of the content. For example, the caption module 208 may run an end-to-end automatic speech recognition model based on a recurrent neural network that is trained to take in raw audio data output from an application, such as the application 112, to identify spoken words and non-spoken sounds from the audio data, and ultimately transcribe the spoken words into written words and convert the non-spoken sounds into written descriptions of the sounds for display as captions.

[0080] The caption module 208 can improve the accuracy of the transcript and the written description of the sounds in various ways, including by biasing the end-to-end automatic speech recognition model based on the context of the computing device 200. For example, the caption module 208 may bias the model based on the visual portions of the content displayed on the screen or other information. For example, in generating a description of the audible portions of a presentation (e.g., a slide show) that includes an audible narration, the end-to-end automatic speech recognition model may be biased using portions of the text or images included in the slides of the presentation to generate the description, thereby improving the accuracy of the description. The caption module 208 may also bias the machine learning model using other types of contextual information, such as location information and information about other applications running on the computing device 200.

[0081] At 412, the computing device 200 displays the caption. For example, the caption module 208 may output a display of the caption generated when the machine learning model 300 receives the raw audio stream from the application 112.

[0082] The computing device 200 may, in some cases, translate portions of the description before displaying the description. For example, the caption module 208 may determine the user's preferred language from the operating system 110 and optionally translate the audio content while generating the description, so that the text of the description is written in the user's preferred language rather than the original language of the audio content. In this way, a user in, for example, Japan, can see a description of the audible content in Japanese or another preferred language, even if the audio for the content was recorded in a different language, such as Chinese or English.

[0083] At 414, the computing device 200 displays the visual portion of the content simultaneously with the description. The operating system 110 may format the description into a persistent graphical element, and the user can move around on the application user interface to read the description of the audible portion of the media content 118 that is output for display.

[0084] 5A-5G are each screenshots illustrating an example of a user interface of a computing device for automatically captioning audio data.

[0085] In the example of FIG. 5A, the computing device 200 includes a display component 108 displays a user interface 114. The user interface 114 displays an application user interface 116 that is associated with the operating system 110 and controlled by an application 112. The application 112 includes media content 118 within the application user interface 116.

[0086] FIGURE 5B is a screenshot of user interface 114 in response to receiving user input to display settings menu 502. In the example of FIGURE 5B, settings menu 502 is an audio settings menu for adjusting volume control, alert settings 504, mute control, etc. Computing device 200 also displays caption control element 122A below the settings menu. In response to detecting a user selection of caption control element 122A, computing device 200 determines a user input for automatically captioning audio data. For example, FIGURE 5C shows caption control element 122B replacing caption control element 122A in response to selection.

[0087] In some cases, in response to receiving a selection of caption control element 122B, computing device 200 automatically ceases captioning the audible portions of the content on computing device 200. In this manner, control elements 122A and 122B provide a user with the ability to quickly start and stop automatic captioning.

[0088] 5D , in response to the selection, the computing device uses the machine learning model to automatically caption the audio data output from the application 112. The computing device 200 generates a description of the audible portion of the media content 118 based on output from the caption module 208, including a confidence level associated with the output.

[0089] The operating system 110 may cause the display component 108 to display the description in a persistent element 506 that the operating system 110 includes in the user interface 114. In some cases, the operating system 110 may display the persistent element 506 while refraining from outputting an audible portion of the content using the speaker component 106 (e.g., a speaker, headphone jack, or other sound system of the computing device 200). Unlike other captioning systems, the computing device 200 may caption the content without generating audible sounds. In other cases, the operating system 110 may provide hearing assistance functionality to display the persistent element 506 while simultaneously outputting an audible portion of the content using the speaker component 106 (e.g., a speaker, headphone jack, or other sound system of the computing device 200).

[0090] 5E, the computing device 200 moves the persistent element 506 from a first location in the user interface 114 to a second location in the user interface 114, illustrating how the persistent element 506 is separate and distinct from the media content 118. A user of the computing device 200 may provide a gesture at a location in the input component 110 that corresponds to the first location in the user interface and drag the input to a different location in the input component that corresponds to the second location in the user interface 114. The computing device 200 may cause the persistent element 506 to move with the gesture.

[0091] 5E illustrates that in response to receiving a user input associated with persistent element 506, computing device 200 may move the persistent element to be distinct from application user interface 116 and a first portion of user interface 114 to obscure a second, distinct portion of application user interface 116 and user interface 114. This allows a user to multitask, e.g., scroll through content displayed within user interface 116, while audio data is captioned and always displayed on top of user interface 114.

[0092] In the example of FIG. 5F, the computing device 200 expands the persistent element 506 from a first size to a second (larger or smaller) size, illustrating how the persistent element 506 is customizable. A user of the computing device 200 can provide a gesture at a location of the input component 110 corresponding to the persistent element 506 to stretch the persistent element or shrink the persistent element. In some cases, in response to receiving a user input associated with the persistent element 506, the computing device 200 can resize the persistent element 506 to display previous or subsequent explanations generated from the audible portion of the content. In this manner, the computing device 200 can cause the persistent element 506 to resize in response to a size change in response to a user input (e.g., a gesture), thereby adjusting how much explanation is included in the persistent element 506 at a particular time.

[0093] In the example of FIG. 5G, the description of the audible portion of the content displayed within the persistent element 506 includes text that identifies the spoken or non-spoken origins of various parts of the audible portion of the content. For example, the persistent element 506 indicates that the machine learning model of the caption module 208 identified a loud roar, possibly from a lion. The persistent element 506 also includes text indicating that the non-spoken audio is a transcription of the spoken audio from the audible portion of the content. For example, to provide an easy-to-understand caption of the media content 118, a transcription of the reporter's lines in the media content 118 is displayed within or around the indication of the lion's roar.

[0094] In general, the computing device 200 may include a description of the noise and an indication of the source of the noise as part of the description in the persistent element 506. The noise may include animal noises from animals, ambient noises from environmental factors, etc. The machine learning models of the caption module 208 are trained to identify spoken and non-spoken sounds from the audio data and represent these sounds with a description sufficient for a user of the computing device 200 to understand the audible content.

[0095] Section 1 1. A method for automatically captioning an audible portion of content on a computing device, the method comprising: obtaining audio data output from an application running on the computing device from an audio mixer of the computing device, the audio data including data indicating the audible portion of the content; determining from the audio data whether the audio data is of a type suitable for captioning; in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content; and outputting the description of the audible portion of the content for display while displaying a visual portion of the content.

[0096] Section 2 2. The method of claim 1, wherein the data indicative of the audible portion of the content is non-metadata, the audio data further includes metadata, and determining whether the audio data is of a type suitable for captioning includes determining from the metadata whether the audio data is of a type suitable for captioning.

[0097] Section 3 3. The method of any one of claims 1 and 2, wherein the description of the audible portion of the content includes a transcription of spoken audio from the audible portion of the content.

[0098] Section 4 4. The method of any one of claims 1 to 3, wherein the description of the audible portion of the content includes a description of non-spoken audio from the audible portion of the content.

[0099] Section 5 5. The method of claim 4, wherein the non-speech audio includes noise from a particular source, and the description of the noise from the particular source includes an indication of the particular source.

[0100] Section 6 6. The method of claim 5, wherein the noise comprises animal noises originating from an animal source, or the noise comprises environmental noises originating from a non-animal source.

[0101] Section 7 7. The method of any one of claims 1 to 6, wherein determining the description of the audible portion of the content includes the computing device executing a machine learning model, the machine learning model being trained to determine a description from audio data to determine the description of the audible portion of the content.

[0102] Section 8 8. The method of claim 7, wherein the machine learning model comprises an end-to-end recurrent neural network transducer automatic speech recognition model.

[0103] Section 9 9. The method of any one of claims 1 to 8, wherein the data indicative of the audible portion of the content comprises unannotated data that is not annotated for captioning.

[0104] Section 10 1. A method for automatically captioning an audible portion of content on a computing device, the method comprising: displaying, by the computing device, a graphical user interface of an application executed on the computing device; and acquiring audio data output from the application while displaying the graphical user interface, the audio data including data indicative of the audible portion of the content, the method further comprising the steps of determining from the audio data whether the audio data is of a type suitable for automatic captioning; in response to determining that the audio data is of a type suitable for automatic captioning, determining a description of the audible portion of the content; and captioning visual portions of the content through the graphical user interface of the application. and outputting the description of the audible portion of the content for display as a persistent element distinct from the graphical user interface of the application while displaying the description within the face.

[0105] Section 11 11. The method of claim 10, wherein the description includes at least one of a transcription of spoken audio extracted from the audible portion of the content or text indicating non-spoken audio extracted from the audible portion of the content.

[0106] Section 12 12. A method according to any one of claims 10 to 11, wherein the data indicative of the audible portion of the content is non-metadata and the audio data further comprises metadata, and determining whether the audio data is of a type suitable for captioning comprises determining from the metadata whether the audio data is of a type suitable for captioning.

[0107] Section 13 13. The method of any one of clauses 10 to 12, wherein the description includes text identifying various portions of the audible portion of the content as being of human or non-human origin.

[0108] Section 14 14. The method of any one of clauses 10 to 13, further comprising the step of resizing the persistent element to display previous or future description generated from the audible portion of the content in response to receiving user input associated with the persistent element.

[0109] Section 15 15. The method of any one of clauses 10 to 14, wherein the persistent element is output for display by obscuring a first portion of the graphical user interface of the application, the method further comprising, in response to receiving user input associated with the persistent element, moving the persistent element to be distinct from the first portion of the graphical user interface of the application so as to obscure a second portion of the graphical user interface of the application.

[0110] Section 16 1. A method for automatically captioning an audible portion of content on a computing device, the method comprising: receiving user input for automatically captioning audio data from an application executing on the computing device; and in response to receiving the user input, obtaining audio data output from the application executing on the computing device, the audio data including data indicative of the audible portion of the content, the method further comprising: determining from the audio data whether the audio data is of a type suitable for captioning; in response to determining that the audio data is of a type suitable for captioning, determining a description of the audible portion of the content; and outputting the description of the audible portion of the content as a persistent element for display that is distinct from visual portions of the content and distinct from a graphical user interface of the application.

[0111] Section 17 17. The method of claim 16, wherein the data indicative of the audible portion of the content is non-metadata and the audio data further includes metadata, and determining whether the audio data is of a type suitable for captioning includes determining from the metadata whether the audio data is of a type suitable for captioning.

[0112] Section 18 18. The method of any one of claims 16 or 17, wherein receiving the user input for automatically captioning the audio data includes displaying a caption control element within an audio settings menu of the computing device, and receiving the user input for automatically captioning the audio data in response to detecting a user selection of the caption control element.

[0113] Section 19 20. The method of claim 18, further comprising the step of automatically ceasing captioning the audible portion of the content on the computing device in response to receiving a subsequent selection of the caption control element.

[0114] Section 20 20. The method of any one of clauses 16 to 19, wherein outputting the description of the audible portion of the content includes outputting the description while refraining from outputting the audible portion of the content using a speaker, headphone jack or other sound system of the computing device.

[0115] Section 21 20. The method of any one of clauses 16 to 19, wherein outputting the description of the audible portion of the content includes outputting the description while outputting the audible portion of the content using a speaker, headphone jack or other sound system of the computing device.

[0116] Section 22 22. The method of any one of claims 1 to 21, further comprising training a machine learning model to determine an explanation for the audio data, wherein determining the explanation comprises determining the explanation using the machine learning model.

[0117] Section 23 23. The method of claim 22, wherein training the machine learning model includes configuring the machine learning model to use previously captioned content to infer captions embedded in the previously captioned content from audio of the previously captioned content.

[0118] Section 24 23. The method of claim 22, wherein training the machine learning model comprises using the explanations as training inputs to the machine learning model.

[0119] Section 25 25. A computing device comprising at least one processor configured to perform any of the methods according to claims 1 to 24.

[0120] Section 26 25. A system comprising means for carrying out any of the methods according to claims 1 to 24.

[0121] Section 27 25. A computer readable storage medium comprising instructions that, when executed, configure a processor of a computing device to perform any of the methods recited in clauses 1 to 24.

[0122] While various preferred embodiments of the present disclosure have been described in the above description and illustrated in the drawings, it should be clearly understood that the present disclosure is not limited thereto and may be variously embodied as practiced within the scope of the appended claims. From the above description, it will be apparent that various changes may be made without departing from the spirit and scope of the present disclosure as defined by the following claims.

Claims

1. 1. A computer-implemented method comprising: A processor of a computing device obtaining audio data output from an application running on the computing device from an audio mixer of the computing device; the processor determining a content type associated with the audio data output from the application; the processor captioning a portion of the audio data output from the application based on the content type; and the processor outputs a caption of the captioned portion of the audio data output from the application for display on a user interface of the application.

2. 10. The method of claim 1, further comprising the step of: the processor outputting a captioning control element for display, the captioning control element including an icon indicating that the computing device is performing the captioning.

3. The method of claim 1 or 2, wherein the caption comprises a transcription of spoken audio from the audio data.

4. The method of claim 1 , wherein the caption comprises a description of non-spoken sounds from the audio data.

5. The method of claim 4 , wherein the non-spoken audio includes noise from a particular source, and the caption includes an indication of the particular source.

6. The method of claim 5 , wherein the noise comprises animal noises from an animal source, or the noise comprises environmental noises from a non-animal source.

7. The step of captioning includes the step of the processor of the computing device executing a machine learning model trained to determine the caption. The method of claim 1 , further comprising:

8. The method of claim 7 , wherein the machine learning model comprises an end-to-end recurrent neural network transducer automatic speech recognition model.

9. The method of claim 1 , wherein the caption comprises text indicative of non-spoken sounds extracted from the audio data.

10. The caption includes a description of non-spoken sounds from the audio data; The method of claim 1 , wherein the description includes text identifying various portions of the audio data as being of human or non-human origin.

11. The method of claim 1 , wherein the caption is displayed as a persistent element overlaid on the user interface of the application.

12. 12. The method of claim 11, further comprising the step of: the processor resizing the persistent element to display previous or subsequent explanations generated from the audio data in response to receiving user input associated with the persistent element.

13. The method of claim 1 , wherein the content type is selected from the group consisting of music, movie, speech, and sonification.

14. A computing device comprising at least one processor configured to perform the method of any one of claims 1 to 13.

15. A program for causing a processor to execute the method according to any one of claims 1 to 13.

Images