AUTOMATIC SUBTITLES OF AUDIBLE PARTS OF CONTENT ON A COMPUTER DEVICE

Abstract

Computer-implemented method, including: Erlangen (404), by a processor (202) of a computer device (200), by an audio mixer (206) of the computer device, of audio data output by an application running on the computer device, wherein the audio data includes data indicative of audible parts of content; Determine (406) by the processor, using the audio data, whether the audio data is of a type suitable for subtitling; in response to the determination that the audio data is of a type suitable for subtitling, the processor determines (410) a description of the audible parts of the content; and during the output of visual parts of the content for display, output (412), by the processor and for display, the description of the audible parts of the content.

Description

BACKGROUND

[0001] Aside from professionally produced films and television series, most audible content is not subtitled, making the content inaccessible to many users who have hearing difficulties or are otherwise unable to hear the content. Some automatic subtitling systems are available, but these systems are expensive, complex, and difficult to manage. Some automatic subtitling systems require audio output to a loudspeaker as input and may be less suitable for providing subtitling in quiet environments. To avoid the cost and complexity involved in managing an automatic subtitling system, some computing devices may rely on cloud-based subtitling services, which require content to be uploaded to remote servers for subtitling, potentially reducing user privacy and usability.US 2009 / 0 278 986 A1 relates to a subtitle production device that produces subtitles, a method for producing subtitles, and a program for producing subtitles. US 2007 / 0 118 364 A1 generally relates to the generation of subtitles and, more specifically, to a system and method for automatically generating subtitles using speech recognition. SUMMARY

[0002] A computer device for automatically captioning content is described as a local, system-level service. Unlike other captioning systems, which often rely on information contained in audio signals sent as inputs to speakers, the computer device generates captions directly from audio data output from content sources (e.g., applications). The computer device can analyze metadata (e.g., inaudible portions) associated with the audio data to determine whether the audio data is suitable for captioning or whether the audio data is a different type of audio data (e.g., system sound effects).

[0003] In response to identifying audio data for closed captioning and in response to determining that automatic closed captioning is enabled (e.g., via user input), the computing device utilizes a machine-learned model to quickly generate a description of audible sounds interpreted based on the audio data. The computing device may provide the descriptions of the audible content visually (e.g., as a persistent user interface element), regardless of the application source of the audible content and regardless of whether the computing device ever generates audio (e.g., audible sound signals) from the content.In this way, the computing device captions audio at the system level to describe audible portions of the content, even if the audio does not include embedded captions, even if the device does not use a captioning server, and even if the computing device is muted and not outputting audible sounds.

[0004] By relying on original audio data as opposed to audio signals generated as inputs for loudspeakers, the machine-learned model can generate subtitles that can more accurately reproduce the sounds from the content as originally intended by the content source. Furthermore, by determining whether audio data is suitable for subtitling before using the machine-learned model, the computing device avoids wasting resources by over-analyzing all audio data output from the content source, including portions of the audio data that almost certainly contain nothing to be subtitled. This allows the computing device to run more efficiently because a smaller and / or less complex machine-learned model can be used. For example, the machine-learned model can automatically detect speech and analyze only the audio data that contains speech.Therefore, the machine-learned model can automatically recognize speech and automatically classify sounds locally on the computing device generating the audio data, thereby improving privacy and offline convenience. The computing device can therefore automatically caption audio at the system level, in a manner that can, at a minimum, increase user satisfaction with automatic captioning systems. Users with special medical needs may experience an increase in their quality of life by accessing the computing device 110 that automatically captions audio data in this manner. A user who may previously have been unable to hear or understand the audible portions of content can now enjoy content just like any other user without special medical needs.

[0005] Throughout the disclosure, examples are described in which a computing device may analyze information (e.g., audio data). However, the computing device may only use the information after the computing device receives explicit permission from a user of the computing device to use the data. For example, in the situations discussed below where the computing device analyzes audio data output by applications executing on a computing device, individual users may be provided with the ability to provide input to control whether programs or features of the computing device can collect and use the audio data, for example, for automatic closed captioning. Individual users may have consistent control over what programs can and cannot do with the audio data.

[0006] Additionally, collected information may be pre-processed in one or more ways before being transmitted, stored, or otherwise used by the computing device and / or a remote computing system such that personal information is removed. For example, before a computing device shares closed captions generated from audio data with another device (e.g., to train a model running on the other device), the example computing device may pre-process the audio data to ensure that any user-identifying information or device-identifying information embedded in the data is removed. Thus, the user may have control over whether information about the user and the user device is collected, and how such information, if collected, may be used by the computing device and / or the remote computing system.

[0007] In one example, a method for automatically captioning audible portions of content on a computing device is described. The method includes obtaining, from an audio mixer of the computing device, audio data output by an application executing on the computing device, the audio data including non-metadata and optionally metadata indicative of the audible portions of the content, and determining, optionally based on the metadata of the audio data, whether the audio data is of a type suitable for automatic captioning. The method further includes, in response to determining that the audio data is of the type suitable for captioning, determining a description of the audible portions of the content, and, while displaying visual portions of the content, outputting the description of the audible portions of the content for display.

[0008] In another example, a method for automatically captioning audible portions of content on a computing device is described. The method includes displaying, by the computing device, a graphical user interface of an application executing on the computing device, obtaining audio data output by the application while displaying the graphical user interface, wherein the audio data includes non-metadata and, optionally, metadata indicative of the audible portions of the content, and determining, optionally based on the metadata of the audio data, whether the audio data is of a type suitable for automatic captioning.In response to determining that the audio data is of the type suitable for subtitling, the method further includes determining a description of the audible portions of the content, and, while displaying visual portions of the content in the graphical user interface of the application, outputting, as a persistent element separate from the graphical user interface of the application, the description of the audible portions of the content for display.

[0009] In yet another example, a method for automatically captioning audible portions of content on a computing device is described. The method includes receiving user input to automatically caption audio data from an application executing on the computing device, and, in response to receiving the user input, obtaining the audio data output from the application executing on the computing device, wherein the audio data comprises the audible portions of the content. The method further includes determining, optionally based on metadata of the audio data, whether the audio data is of a type suitable for captioning, and, in response to determining that the type of audio data is suitable for captioning, obtaining a description of the audible portions of the content.The method further includes outputting, for display, the description of the audible portions of the content as a persistent element separated from visual portions of the content and separated from a graphical user interface of the application.

[0010] In another example, a computing device is described that includes at least one processor configured to perform the methods according to any of the preceding examples.

[0011] In a further example, a system is described which includes means for carrying out the methods according to the preceding examples.

[0012] In another example, a computer-readable storage medium is described that includes instructions that, when executed, configure a processor of a computing device to perform any of the methods according to the preceding examples.

[0013] The details of one or more implementations are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the description and drawings, as well as from the claims. This Summary is provided to introduce the subject matter, which is further described in the Detailed Description and Drawings. Therefore, this Summary should not be considered to describe essential features, nor should it be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The following describes the details of one or more aspects of automatic subtitling. The use of the same reference numerals in various places in the description and figures indicates similar elements: Fig. 1 is a conceptual diagram illustrating a computing device configured to automatically caption audio data. Fig. 2 is another conceptual diagram illustrating a computing device configured to automatically subtitle audio data. Fig. 3 is a conceptual diagram illustrating a machine-learned model of a computing device configured to automatically subtitle audio data. Fig. 4 is a flowchart illustrating example operations of a computing device configured to automatically caption audio data. Fig. Figures 5A through 5H are screenshots illustrating an example user interface of a computing device that automatically captions audio data. DETAILED DESCRIPTION

[0015] Except for professionally produced films and television series, most audible content is not subtitled, making it inaccessible to many users who have hearing difficulties or are otherwise unable to hear the content. Manual and automatic subtitling systems exist, but they have various disadvantages.

[0016] For example, some manual captioning systems caption audible content based on embedded captioning information included in the pre-recorded content. Other manual captioning systems caption content based on input from a human operator (e.g., during a live broadcast).

[0017] Some automatic subtitling systems are available, but these systems are expensive, complex, and difficult to manage. For example, an automatic subtitling system may require audio output to a sound system as a source for subtitling. The subtitling system may require that the audio output maintain a minimum sound level, making such a system less suitable for operation in quiet environments. In addition, an audio output may include more audible sounds than just sounds for subtitling (e.g., system sounds, sounds from other background applications), and subtitling may therefore take longer and unnecessarily waste resources, indiscriminately subtitling all sounds from all audio sources on the computing device.

[0018] Some automatic captioning systems are application-based and can run locally on a computing device. However, such systems may still require an internet connection for captioning. To avoid the costs and expenses associated with managing an automatic captioning system, a computing device may rely on a cloud-based captioning service, which requires uploading content to remote servers for captioning, thereby reducing user privacy and usability.

[0019] Techniques are described that enable efficient and sophisticated automatic captioning of audible content. The techniques do not require a network connection (e.g., to the Internet) and do not generate captions from audible signals passed to a sound system. Rather, an example computing device can automatically generate captions from audio data collected by an audio mixer of the computing device. "Audio data," as used herein, refers to information sent to an audio mixer or sound engine of a system, which is converted into audio signals for generating sounds as output. The audio data contrasts with the audio signals generated in response.

[0020] An operating system or platform of the example computer system includes a system-level audio mixer that collects audio data output from various content sources (e.g., applications) and mixes the audio data to generate audio signals. The audio signals may be amplified by a sound system to produce audible sound. Unlike other captioning systems that indiscriminately caption audio signals, the example computer device discriminates audible content by captioning only audible content from audio data that is of a type suitable for captioning.

[0021] The computing device may analyze metadata (e.g., non-audio portions of audio data) to determine whether the audio data is suitable for closed captioning, or whether the audio data is a different type of audio data (e.g., system sound effects). Relying on metadata not found in audio data allows the computing device to quickly filter out audio data that is not intended for closed captioning and allows the computing device to avoid wasting resources like other computing systems that attempt to caption all content, even content that cannot or should not be captioned.

[0022] In response to identifying audio data for closed captioning, and in response to determining that automatic closed captioning is enabled (e.g., via user input), the computing device utilizes a machine-learned model trained using machine learning to rapidly generate a description of audible sounds interpreted from the audio data. For example, the computing device may utilize an automatic regressive end-to-end neural network transducer speech recognition model or another machine-learned model to determine a transcription of spoken audio or to identify specific sounds from non-spoken audio and likely sources for the specific sounds.

[0023] The computing device may provide descriptions of the audible content visually (e.g., as a persistent user interface element), regardless of the source of the audible content and regardless of whether the computing device ever generates audio (e.g., audible sound signals) from the content. In this way, the computing device performs system-level captioning operations that provide descriptions of audible content even if the content does not include embedded captions, and even if the computing device is muted and never outputs an audio signal.

[0024] By using audio data as opposed to audio signals used as inputs to speakers, the machine-learned model generates subtitles that more accurately represent the sounds from the audible content, as originally intended by the content source. Audio from an application or other content source can generally be a more accurate representation of audible content because the audio data is unaltered from the moment it is output from the content source. Other automatic subtitling systems may use a processed audio signal intended as input to a speaker, which can result in deviations from what the content source intended.

[0025] Furthermore, by determining whether audio data is suitable for subtitling before tasking the machine-learned model with a description, the computing device avoids wasting resources by overanalyzing all audio data output by the content source, including portions of the audio data that almost certainly contain nothing to be subtitled. This allows the computing device to train and run smaller and / or less complex machine-learned models to perform automatic speech recognition and automatic sound classification techniques. The model does not need to be further trained to ignore various types of audio data that are unlikely to be suitable for subtitling.This further allows the model to run locally on a computing device where the audio data is generated, thereby improving data protection and offline usability.

[0026] Using more accurate data can increase the accuracy of the machine-learned model, and using only relevant data can improve the efficiency of the machine-learned model, which can further increase user satisfaction with automatic captioning systems. In addition, users with special medical needs can experience an improvement in their quality of life by accessing the computing device 110 that automatically captions audio data in this manner.

[0027] The description may include a transcription of spoken content, in some cases even an identification of a name or a description of a speaker. In some cases, the audible sounds include non-spoken content, for example, animal noises, ambient sounds, and other sounds that may or may not be relevant to the description. The description may even include an explanation of some non-spoken sounds (e.g., a dog barking), including an identification of a corresponding source (e.g., a barking dog).

[0028] Fig. 1 is a conceptual diagram illustrating a computing device configured to automatically caption audio data. Computing device 100 may be any type of mobile or non-mobile computing device. As a mobile computing device, the computing device may be a mobile phone, a laptop computer, a wearable device (e.g., watch, glasses, headphones, clothing), a tablet device, an automotive / vehicle device, a handheld gaming device, an electronic reading device, a remote control device, or other mobile computing device. As a non-mobile computing device, computing device 100 may be a server, a network terminal device, a desktop computer, a television, an entertainment set-top device, a media streaming device, a tabletop assistive device, a speaker device, a sound system, a non-handheld gaming device, business conferencing equipment, or other non-mobile computing device.

[0029] Computing device 100 includes a user interface device 102 having a speaker component 104, a display component 106, and an input component 108. Computing device 100 also includes an operating system 110 and an application 112. These and other components of computing device 100 are communicatively connected in various ways, including through the use of internal wired and wireless buses and connections. Computing device 100 may include additional or fewer components than described in Fig. 1 shown.

[0030] Operating system 110 typically controls the functionality of computing device 100, including user interface device 102 and other peripherals. Operating system 110 provides an execution environment for applications, such as application 112. Operating system 110 may control task scheduling and other general functionality and generally does so through a system-level user interface 114.

[0031] The user interface device 102 manages inputs and outputs to the system-level user interface 114, which is controlled by the operating system 110. The system-level user interface 114 includes a persistent graphical element 120 for displaying automatic closed captions. The system-level user interface 114 further includes one or more closed caption controls 122A and 122B, which, when selected, control whether the operating system 110 automatically captions audio data, including whether or not the user interface 114 displays the persistent graphical element 120. In some cases, the closed caption controls 122A and 122B may include an icon indicative of whether or not the computing device 100 automatically captions the audio data. The system-level interface 114 may display user interfaces of other applications executing on the computing device 100, such asdisplay an application user interface 116 for the application 112.

[0032] The display component 106 and the speaker component 104 can be standalone or integrated as a single component. The display component 106 can be constructed from any suitable display technology, including LED, OLED, and LCD technologies. The speaker component 104 (e.g., a single speaker or multiple speakers) is configured to receive an audio signal as input and convert the audio signal into audible sound. The input component 108 can be a microphone, a presence-sensitive device, a touchscreen, a mouse, a keyboard, or another type of component configured to receive user input.

[0033] The user interface device 102 may output information about inputs recognized by the input component 108, and the operating system 110 may correlate the recognized inputs with elements of the user interface 114. In response to receiving an input at the input component 108 (e.g., a gesture), the operating system 110 and / or the application 112 may receive information from the user interface device 102 about the recognized input. The operating system 110 or the application 112 may perform a function in response to the input. For example, the operating system 110 may determine that the input corresponds to a selection of one of the two caption controls 122A or 122B and, in response, may automatically caption audio (or set automatic captioning).

[0034] The application user interface 116 includes media content 118, such as a movie, animation, video, or other content with embedded audio. The media content 118 may be any multimedia or social media content, including from video platforms, streaming platforms, radio or podcasts, videos, video chat applications, phone applications, or any other source of audio data executed by or accessible from the computing device.

[0035] The application 112 may output the media content 118 to the operating system 110. Using the display component 106, the operating system 110 may cause the user interface device 102 to output images or other visual representations of the media content 118 within the application user interface 116. The computing device 100 may further cause the user interface device 102 to route audio signals generated from the media content 118 to the speaker component 104. In this way, the operating system 110 causes the speaker component 104 to output audio concurrently with the images or other visual material displayed on the display component 106.

[0036] In operation, the operating system 110 automatically subtitles audible portions of content. The computing device receives audio data used for subtitling from an audio mixer or sound engine of the operating system 110. The audio data from the application 112 may include the audible portions of the media content 118.

[0037] For example, as in Fig. 1, a user of computing device 100 may interact with application 112 by viewing media content 118 displayed on a screen of display component 106. The user may want to enable closed captioning of media content 118, but media content 118 may only include uncaptured audio data, with no ability for application 112 to generate and embed closed captions into media content 118.

[0038] Nevertheless, in response to receiving user input (e.g., a touch gesture at a location on the input component 108 that corresponds to a location on the screen of the display component 106 where the selectable control 122B is displayed), the computing device 100 may automatically caption the audible portions of the media content 118 by processing the same audio data output from the application 112 and received at the audio mixer.

[0039] The audio mixer is configured to collect all audio data output from various content sources executing within the execution environment of the operating system 110, including the application 112. The audio mixer provides an interface between the application source that generates the content and the speaker component 104 that generates sounds based on the content. The audio mixer manages the raw audio data bytestream, analyzing audio data and routing output audio signals to the speaker component 104 or another suitable output component of the user interface device 102 to generate sounds.

[0040] Instead of indiscriminately captioning all audio data from the audio signals sent to speaker component 104, operating system 110 may use metadata to focus automatic captioning on audible portions of the content that are specifically destined for captioning. In other words, operating system 110 may identify "captionable" audio data based on metadata and refrain from captioning all audio data unless all of the audio data is considered captionable.

[0041] Some examples of metadata include a usage indicator that indicates why an application is outputting audio data. The audio mixer can use the usage indicator to guide decisions regarding routing, focus, and volume of the audio data. The metadata can include other flags that further specify how sounds should be played, as well as information for audio / video synchronization.

[0042] The metadata may include a content type that indicates what the application source is playing (e.g., music, film, speech, sonification, unknown). The content type may express a general category of content (e.g., one type of movie streaming service and another type of music streaming service). The audio mixer can selectively configure audio post-processing and adjust other settings based on the type identifier. The type identifier may indicate a type of film if the audio is for a soundtrack accompanying a film or television program. The type identifier may indicate a music type identifier if the content is a song (e.g., not necessarily a soundtrack), and the type identifier may indicate a speech type identifier if the content is spoken audio (e.g., a newscast, a video, or a phone call).As another example, the content type may indicate sonification if the audio data is for a notification, message, or sound used to accompany a user action (e.g., a beep or sound effect indicating a key press) or event (e.g., a sound for achieving a bonus during a game).

[0043] In addition to the audio mixer, which relies on the content type, the operating system 110 identifies audio data for subtitling based on the content type identifier, which is optionally included in the metadata. For example, the operating system 110 automatically subtitles audio data of one type: film, music, or speech, but not some sonification-type audio data or other audio data that is undefined or otherwise defined. In this way, the operating system 110 can determine, using the content type, whether the audio data is likely to contain audio that requires subtitling or whether the audio data is likely to be indicative of another sound that is not intended for subtitling.

[0044] In response to determining that the type of audio data associated with the media content 118 is for closed captioning, the operating system 110 determines a description of the audible portions of the content 118. For example, the operating system 110 may execute a machine-learned model (e.g., an automatic regressive end-to-end neural network transducer speech recognition model) trained to generate descriptions of audible content as closed captions. The machine-learned model may be any type of model suitable for learning descriptions of sounds, including transcriptions for spoken audio. However, the machine-learned model utilized by the operating system 110 may be smaller and less complex, as the machine-learned model only needs to be trained to identify sound and spoken audio from specific content types.The machine-learned model doesn't need to process all the audio data sent to the audio mixer, but only specific audio data that is likely to contain content for subtitling. In contrast, other subtitling systems require remote processing facilities and remote content upload, which poses a privacy risk and compromises convenience.

[0045] The operating system 110 receives the description from the machine-learned model and displays the description to the user. A description may include transcriptions of spoken dialogue or songs. The descriptions may identify a context for a sound or identify speakers, singers, or individual actors or performers. The descriptions may include sound descriptions, such as "a barking dog" when sounds of a dog are detected, or "door closing" for sounds of a slamming door.

[0046] While displaying visual portions of the media content 118 within the application user interface 116, the operating system 110 outputs the description of the audible portions of the content for display. For example, the operating system 110 may present the persistent element 120 as an overlay of the user interface 114, appearing on top of the application user interface 116, and other graphical elements of the user interface 114. A user may manipulate the persistent element 120 to move it to a different area of ​​a display screen or increase or decrease the size of the element to show additional or fewer subtitles.

[0047] By relying on original audio data rather than audio signals generated as inputs to speakers, the machine-learned model of the operating system 110 generates subtitles that more accurately represent the sounds from the media content 118 as originally intended by the content source (e.g., the application 112). Furthermore, by determining whether audio data is for subtitling before utilizing the machine-learned model, the operating system 110 avoids wasting resources by overanalyzing all audio data output by the application 112, including that which almost certainly contains nothing to subtitle. This allows the computing device 110 to execute a more efficient, smaller, and / or less complex machine-learned model.Therefore, the machine-learned model can perform automatic speech recognition and automatic sound classification techniques locally from the computing device 110 generating the audio data, thereby improving privacy and offline convenience. The computing device 110 can therefore automatically caption audio at the system level, in a manner that can at least increase user satisfaction with automatic captioning systems. Users with special medical needs can experience an increase in their quality of life by accessing the computing device 110, which automatically captions audio data in this manner.

[0048] Fig. Figure 2 is another conceptual diagram illustrating a computing device 200 configured to automatically caption audio data. Computing device 200 is an example of computing device 100 with some additional details. As shown in Fig. 2, the computing device 200 may be a mobile phone 100-1, a laptop computer 100-2, a television / display 100-3, a desktop computer 100-4, a tablet device 100-5, a computerized watch 100-6, or other portable device or a computing system installed in a vehicle 100-7.

[0049] In addition to each of the Fig. 1, the computing device 200 includes one or more processors 202, computer-readable media 204, one or more sensors 210, one or more input / output (I / O) devices 212, and one or more communication devices 214. The computer-readable media 212 includes instructions that, when executed by the processors 102, execute the application 112 and the operating system 110.

[0050] Processors 202 may include any combination of one or more controllers, microcontrollers, processors, microprocessors, hardware processors, hardware processing units, digital signal processors, graphics processors, graphics processing units, and the like. Processors 202 may be an integrated processor and memory subsystem (e.g., implemented as a SoC) that process computer-executable instructions to control operations of computing device 200.

[0051] Sensors 210 acquire contextual information indicative of a physical operating environment of the computing device and / or characteristics of the computing device 200 while operating in the physical operating environment. Examples of sensors 210 include motion sensors, temperature sensors, position sensors, proximity sensors, ambient light sensors, humidity sensors, pressure sensors, and the like. Operating system 110 may customize operations of the computing device 200 according to the sensor information acquired by sensors 210.

[0052] Input / output devices 212 may provide connectivity for computing device 200 and other devices and peripherals, including data network interfaces that provide connection and / or communication links between the device, data networks (e.g., a mesh network, external network, etc.), and other devices. Input / output devices 212 may be used to couple computing device 200 to any type of component, peripheral, and / or accessory device. Input / output devices 212 also include data input ports through which any type of data, media content, and / or input, such as user input to computing device 200, as well as any type of communication data, audio, video, and / or image data received from any content and / or data source, may be received.

[0053] The communication devices 214 enable wired and / or wireless communication of device data 506, such as any type of generated media content or received applications executing on the computing device 200, such as the application 112. The communication devices 214 may also include transceivers for cellular phone communication and / or for network data communication.

[0054] Computer-readable media 204 is configured to provide computing device 200 with persistent and / or non-persistent storage of executable instructions (e.g., firmware, recovery firmware, software, applications, modules, programs, functions, and the like) and data (e.g., user data, operational data) to support execution of the executable instructions. Examples of computer-readable media 204 include volatile memory and non-volatile memory, fixed and removable media devices, and any suitable storage devices or electronic data storage devices that manage executable instructions and hold data. Computer-readable media 204 may include various implementations of random access memory (RAM), read-only memory (ROM), flash memory, and other types of mass storage in various storage device configurations.The computer-readable media 204 excludes propagating signals. The computer-readable media 204 may be a solid-state drive (SSD) or a hard disk drive (HDD). The computer-readable media 204 in the example of FIG. Fig. 2 include the application 112 and the operating system 110.

[0055] The operating system 110 of the computing device 200 includes an audio mixer 206 and a closed captioning module 208. The audio mixer 206 and the closed captioning module 208 may be implemented as dedicated hardware or software components of the operating system 110. In other examples, the audio mixer 206 and the closed captioning module 208, respectively, may be implemented separately from the operating system 110, e.g., as a system plug-in or additional add-on service installed locally on the operating system 110.

[0056] Audio mixer 206 is configured to combine audio data generated by applications executing within the operating environment provided by operating system 110. Audio mixer 206 combines audio streams from applications, such as application 112, and generates audio output signals that represent the sounds encoded in the audio streams when combined and output from speaker component 206. Audio mixer 206 may adjust the audio signals in other ways, for example, to control focus, intensity, and sound volume.

[0057] The closed caption module 208 is configured to automatically subtitle audio data in raw form as received by the audio mixer 206 (e.g., as a byte stream). Instead of processing all post-mixed audio signals for closed captioning, the closed caption module 208 identifies individual, pre-mixed streams of audio data received at the audio mixer 206 that are suitable for closed captioning. For example, the closed caption module 208 may automatically subtitle spoken audio data, but not notification or sonification type audio data, such as system signal beeps and ringtones. The closed caption module 208 may apply a filter to the byte streams received from the audio mixer 206 to identify the audio data that is of a type suitable for closed captioning.The subtitling module 208 uses a machine-learned model to determine descriptions of sounds represented by audio data identified for subtitling.

[0058] Fig. 3 is a conceptual diagram illustrating a machine-learned model of a computing device configured to automatically caption audio data. The machine-learned model 300 may be part of the captioning module 208. That is, the captioning module 208 may include the machine-learned model 300 to convert encoded audio data into detailed descriptions or subtitles of the audible content, without necessarily converting the audio data to sound. The computing device 200 may execute the machine-learned model 300 as part of executing the captioning module 208 to determine the description 318 of the audible portions of the content contained in the audio data 312. The machine-learned model 300 may include one or more types of machine-learned models combined into a single model that provides the descriptions 318 in response to the audio data 312.

[0059] The machine-learned model 300 is configured to perform inference; the machine-learned model 300 is trained to receive audio data 312 as input and to provide descriptions 318 (e.g., subtitles) of sounds determined by the machine-learned model 300 from the audio data 312 as output data. In addition to any annotations (i.e., annotated data) included in the audio data 312, the audio data 312 may include unannotated data that has not been annotated for subtitling. By performing inference using the machine-learned model 300, the subtitling module 208 processes the audio data 312 locally and from within a secure enclave of the operating system 110 to ensure user privacy and security.

[0060] The machine-learned model 300 may be or include one or more different types of machine-learned models. Furthermore, the machine learning techniques described herein that are used to train the machine-learned model 300 may be easily interchangeable and combined. Although certain example techniques have been described, there are many others that may be used in connection with aspects of the present disclosure. The machine-learned model 300 may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other outputs.

[0061] The machine-learned model 300 can be trained using supervised learning techniques. For example, the machine-learned model 300 can be trained based on a training dataset containing examples of descriptions derived from corresponding examples of audio data. The machine-learned model 300 can also be trained using unsupervised learning techniques.

[0062] The machine-learned model 300 may be or include one or more artificial neural networks (a type of "neural network"). As a neural network, the machine-learned model 300 may include a group of connected or not fully connected nodes referred to as neurons or perceptrons. As a neural network, the machine-learned model 300 may be divided into one or more layers and, in some cases, may include multiple layers when configured as a "deep" network. As a deep network, the machine-learned model 300 may include an input layer, an output layer, and one or more hidden layers disposed between the input layer and the output layer.

[0063] The machine-learned model 300 may include one or more recurrent neural networks. For example, the machine-learned model may be implemented as an automatic recurrent end-to-end neural network reshaper speech recognition model. Examples of recurrent neural networks include long-term / short-term memory (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, history neural compressors, echo state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks, fully recurrent networks, and sequence-to-sequence configurations.

[0064] At least some of the nodes of a recurrent neural network may form a cycle. The machine-learned model 300, when configured as a recurrent neural network, may be particularly useful for processing input data of a sequential nature, such as audio data 312, which has an inherent sequential nature. A recurrent neural network may pass or retain information from a previous portion of an input data sequence to a subsequent portion of the input data sequence using recurrent or directed cyclic node connections. For example, the machine-learned model 300 may pass or retain information from a previous portion of the audio data 312 to a subsequent portion of the audio data 312 using recurrent or directed cyclic node connections between the previous and subsequent portions.

[0065] The audio data 312 is sequential by definition and may include time-series data (e.g., sound data versus time). The machine-learned model 300, as a recurrent neural network, may analyze the audio data 312 over time to detect or predict spoken sounds and non-spoken, but relevant, sounds for generating subtitles of the audio data 312. More specifically, the sequential sounds from the audio data 312 may indicate spoken words in a sentence (e.g., for natural language processing, speech recognition, or processing).

[0066] The machine-learned model 300 may be or include one or more convolutional neural networks. A convolutional neural network may include one or more convolutional layers that perform convolutions on input data using learned filters or kernels. Convolutional neural networks are known for their usefulness in diagnosing vision problems, such as analyzing image input data, such as still images or video. However, convolutional neural networks can also be applied to natural language processing of sound data, such as generating subtitles from sequential audio data 312.

[0067] The machine-learned model 300 may be trained using machine learning to receive the audio data 312 as input data and, in response, provide the descriptions 318 as output data. The input data may include different types, forms, or variations of audio data. For example, in various implementations, the audio data 312 may include raw, premixed audio bytestream data passed from an application to an audio mixer, and the audio data 312 may also include processed bytestream data.

[0068] In response to receiving the audio data 312, the machine-learned model 300 may provide the descriptions 318. The output data may include different types, forms, or variations of output data. For example, in various implementations, the output data may include descriptions of audible content corresponding to the displayed visual content.

[0069] The machine-learned model 300 can be trained offline or online. With offline training (also known as batch learning), the machine-learned model 300 is trained on the entirety of a static set of training data. With online learning, the machine-learned model 300 is continuously trained (or retrained) as new training data becomes available (e.g., while the machine-learned model 300 is being used to perform inference). For example, the machine-learned model 300 may initially be trained to replicate subtitles that have already been applied to audible content (e.g., movie subtitles).Since the machine-learned model 300 is used to infer descriptions of the audio data 312, the descriptions and the corresponding portions of the audio data 312 can be fed back to the machine-learned model 300 when new training data is available to enable the machine-learned model 300 to continuously improve descriptions. A user may be provided with the ability to provide input to the machine-learned model 300 to flag a particular description as erroneous. The signal that a description may be incorrect can also be used to train the machine-learned model 300 to improve future predictions.

[0070] As part of the closed captioning module 208, the machine-learned model 300 may be part of the operating system 110 and thus may be included in a secure enclave to securely and confidentially process audio data for automatic closed captioning. Applications interacting with the operating system 110 may interact with the machine-learned model 300 to process the audio data 312 into descriptions 318. For example, the application 112 may communicate with the model 300 via the operating system 110 using an application programming interface (API) (e.g., a public API common to all applications).

[0071] Fig. 4 is a flowchart illustrating exemplary operations 400 of a computing device configured to automatically caption audio data. The computing devices 100 or 200 of Fig. 1 and Fig. 2 can perform the operations 400. The operations 400 can be performed in a different order than in Fig. 4, including additional or fewer operations. The operations 400 are described below in the context of computing device 200.

[0072] At 402, computing device 200 receives permission to use personal data to perform automatic closed captioning. For example, computing device 200 may use audio data to generate closed captions only after computing device 200 receives explicit permission from a user of computing device 200 to use the audio data.

[0073] At 404, computing device 200 displays a graphical user interface of an application. For example, computing device 200 may instruct display component 108 to present user interface 114 and display application user interface 116.

[0074] At 404, computing device 200 receives audio data from the application that includes audible portions of the content. For example, application 112 may render and play media content 118. To provide closed captions, for example, when media content 118 does not include narrated captions, closed caption module 208 may extract the audio data output from application 112 while receiving the audio data at audio mixer 206.

[0075] At 406, computing device 200 determines whether the audio data is suitable for closed captioning. For example, closed captioning module 208 may filter out audio data that is of a type that does not require closed captioning, such as notification sounds and other types of sounds that accompany the content. Optionally, closed captioning module 208 determines whether the audio data is suitable for closed captioning based on the type identifier found in the metadata embedded in the audio data. Optionally, closed captioning module 208 determines whether the audio data is suitable for closed captioning based on the metadata indicating that narration (pre-generated closed captions) is included in the audio data. In response to determining that the audio data already has closed captions, closed captioning module 208 may forego performing automatic closed captioning and display the preset closed captions within a description.

[0076] The operating system 110 and the audio mixer 206 may define various types of audio data to which applications can assign outputs. Like other applications executing on the computing device 200, the application 112 utilizes an API to output audio data to the audio mixer 206. The API may include a parameter within the metadata to specify the audio data type, for example, to assist the audio mixer 206 in correctly mixing and distributing audio outputs determined from the data. Instead of relying on the audio data type to distribute sounds, the closed captioning module 208 utilizes the type identifier, optionally found in the metadata, to quickly and easily determine whether or not closed captioning is required.In particular, the subtitling module 208 may focus on certain types of audio data, specific formats, durations, or other features and characteristics of the audio data to determine whether the audio data can be subtitled. Some types of audio data that may be suitable for subtitling include film-type audio data, music-type audio data, and speech-type audio data. In contrast, sonification-type audio data and other undefined audio data types may be unsuitable for subtitling because the sounds are not intended to convey content, but rather an event or user action.

[0077] If the audio data is not suitable for closed captioning, the computing device 200 displays visual portions of the content at 414. For example, the computing device 200 may display the media content 118 on the display component 108.

[0078] However, if computing device 200 determines at 408 that the audio data is intended for closed captioning, computing device 200 still determines whether to automatically caption the data based on whether automatic captioning is selected by the user. For example, at 414, in cases where the user has not enabled automatic captioning in a settings menu of user interface 114 or operating system 110, computing device 200 refrains from automatically captioning the audio data and instead displays visual portions of the content without generating descriptive captions. If the user has enabled automatic captioning by selecting an appropriate option from a settings menu or the like at 410, captioning module 208 utilizes a machine-learned model to determine a description for the audible portions of the content.For example, the closed captioning module 208 may execute an automatic end-to-end speech recognition model based on a regressive neural network that is trained to utilize the raw audio data output from an application, such as the application 112, and identify spoken words and non-spoken sounds from the audio data to ultimately transcribe the spoken words into written words and convert the non-spoken sounds into written descriptions of the sounds for display as closed captions.

[0079] The closed captioning module 208 may improve the accuracy of transcripts and written descriptions of sounds in various ways, including by influencing the automatic end-to-end speech recognition model based on a context of the computing device 200. For example, the closed captioning module 208 may influence the model based on visual portions of the content or other information displayed on a screen. For example, when generating descriptions of audible portions of a presentation (e.g., a slideshow) that includes audible narration, the automatic end-to-end speech recognition model could be influenced using portions of text or images contained in slides of the presentation to generate the descriptions, which may improve the accuracy of the descriptions. The closed captioning module 208 may incorporate other types of contextual information, such asUse location information and information about other applications running on the computing device 200 to influence the machine-learned model.

[0080] At 412, computing device 200 displays the description. For example, caption module 208 may output an indication that a description is being generated when machine-learned model 300 receives the raw audio stream from application 112.

[0081] In some cases, computing device 200 may translate portions of the description before displaying the description. For example, captioning module 208 may use operating system 110 to determine a user's preferred language and, if appropriate, translate audio content while generating a description so that the text of the description is written in the user's preferred language rather than in a source language of the audio content. In this way, for example, a user from Japan may view descriptions of audible content in Japanese or another preferred language, even if the audio of the content was recorded in a different language, such as Chinese or English.

[0082] At 414, computing device 200 displays visual portions of the content concurrently with the description. Operating system 110 may format the description into a persistent graphical element that the user can hover around in the application interface to read a description of the audible portions of the media content 118 output for display.

[0083] Fig. Figures 5A through 5H are screenshots illustrating an example user interface of a computing device that automatically captions audio data. Fig. 5A through 5H are described sequentially and in the context of computing device 200.

[0084] In the example from Fig. 5A, computing device 200 displays user interface 114 on display component 108. User interface 114 is associated with operating system 110 and displays application user interface 116 controlled by application 112. Within application user interface 116, application 112 includes media content 118.

[0085] Fig. 5B is a screenshot of the user interface 114 in response to receiving user input to display a settings menu 502. In the example of Fig. 5B, the settings menu 502 is an audio settings menu for adjusting volume controls, alert settings 504, mute controls, etc. Additionally, the computing device 200 displays the closed caption control 122A below the settings menu. The computing device 200 detects a user input to automatically caption the audio data in response to detecting a user selection of the closed caption control 122A. Fig. For example, Figure 5C shows the subtitle control 122B having replaced the subtitle control 122A in response to the selection.

[0086] In some cases, in response to receiving a selection of the closed captioning control 122B, the computing device 200 automatically stops captioning audible portions of the content on the computing device 200. In this way, the controls 122A and 122B provide a user with the ability to quickly start and stop automatic captioning.

[0087] In Fig. 5D, the computing device automatically subtitles audio data output by the application 112 using a machine-learned model in response to the selection. The computing device 200 generates a description for audible portions of the media content 118 based on an output of the subtitle module 208, including a confidence level associated with the output.

[0088] The operating system 110 may cause the display component 108 to display the description within a persistent element 506 that the operating system 110 includes within the user interface 114. In some cases, the operating system 110 may display the persistent element 506 while simultaneously refraining from outputting the audible portions of the content using the speaker component 106 (e.g., a speaker, a headphone jack, or other sound system of the computing device 200). Unlike other closed captioning systems, the computing system 200 may caption content without generating audible sounds. In other cases, the operating system 110 may provide a hearing assistance feature and display the persistent element 506 while simultaneously outputting the audible portions of the content using the speaker component 106 (e.g., a speaker, a headphone jack, or other sound system of the computing device 200).

[0089] In the example of FIG. 5E, computing device 200 moves persistent element 506 from a first location of user interface 114 to a second location of user interface 114, demonstrating how persistent element 506 is separate and distinct from media content 118. A user of computing device 200 may provide gestures at a location of input component 110 corresponding to the first location of user interface and drag the input to another location of the input component corresponding to the second location of user interface 114. Computing device 200 may cause movement of persistent element 506 with the gesture.

[0090] In this way, the example from Fig. 5E illustrates that, in response to receiving user input associated with persistent element 506, computing device 200 may move the persistent element away from a first portion of application user interface 116 and user interface 114 to obscure a second, distinct portion of application user interface 116 and user interface 114. This may enable a user to multitask, e.g., scroll through content displayed within user interface 116 while captioning audio and consistently displaying it at the top of user interface 114.

[0091] In the example of FIG. 5F, computing device 200 enlarges persistent element 506 from a first size to a second (larger or smaller) size, illustrating how persistent element 506 can be customized. A user of computing device 200 can provide gestures at a location on input component 110 corresponding to persistent element 506 to stretch the persistent element or shrink the persistent element. In some cases, in response to receiving user input associated with persistent element 506, computing device 200 can resize persistent element 506 to display previous or subsequent descriptions generated from the audible portions of the content. In this way, computing device 200 can cause persistent element 506 to resize with changes in the size of user input (e.g.,gesture) and subsequently adjust how much of the description is contained in the persistent element 506 at a particular time.

[0092] In the example from Fig.5G includes the description of the audible portions of the content displayed in persistent element 506, text identifying a spoken or non-spoken source for different portions of the audible portions of the content. For example, persistent element 506 provides an indication that the machine-learned model of captioning module 208 has identified a loud roaring noise likely originating from a lion. Also included in persistent element 506 with text indicating non-spoken audio is a transcription of spoken audio from the audible portions of the content. For example, the transcription of the news anchor's dialogue in media content 118 is displayed in and around the reference to the lion's roar to provide easily understandable closed captioning of media content 118.

[0093] In general, computing device 200 may include descriptions of sounds and references to sources of the sounds as part of the description within persistent element 506. The sounds may include animal sounds from animal sources, ambient sounds from ambient sources, and the like. The machine-learned model of captioning module 208 is trained to identify spoken and non-spoken audio from audio data and to describe the audio with a description sufficient for a user of computing device 200 to understand the audible content.

[0094] Clause 1. A method for automatically subtitling audible portions of content on a computing device, the method comprising: obtaining, from an audio mixer of the computing device, audio data output by an application executing on the computing device, the audio data comprising data indicative of audible portions of the content; determining, from the audio data, whether the audio data is of a type suitable for subtitling; in response to determining that the audio data is of a type suitable for subtitling, determining a description of the audible portions of the content; and, while displaying visual portions of the content, outputting the description of the audible portions of the content for display.

[0095] Clause 2. The method of Clause 1, wherein the data indicative of the audible portions of the content is non-metadata, and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitling comprises determining, from the metadata, whether the audio data is of a type suitable for subtitling.

[0096] Clause 3. A method according to any one of clauses 1 or 2, wherein the description of the audible portions of the content comprises a transcription of spoken audio from the audible portions of the content.

[0097] Clause 4. The method of any of clauses 1-3, wherein the description of the audible portions of the content comprises a description of non-spoken audio from the audible portions of the content.

[0098] Clause 5. The method of Clause 4, wherein the non-spoken audio comprises a sound from a specific source and the description of the sound from the specific source comprises an indication of the specific source.

[0099] Clause 6. A method according to Clause 5, wherein: the noise comprises an animal noise from an animal source, or the noise comprises an ambient noise from a non-animal source.

[0100] Clause 7. The method of any of clauses 1-6, wherein determining the description of the audible portions of the content comprises executing, by the computing device, a machine-learned model trained to determine descriptions from audio data to determine the description of the audible portions of the content.

[0101] Clause 8. The method of Clause 7, wherein the machine-learned model comprises an automatic regressive end-to-end neural network transformer speech recognition model.

[0102] Clause 9. A method according to any of Clauses 1-8, wherein the data indicative of the audible parts of the content comprises unannotated data that has not been annotated for subtitling.

[0103] Clause 10. A method for automatically captioning audible portions of content on a computing device, the method comprising: displaying, by the computing device, a graphical user interface of an application executing on the computing device; obtaining audio data output by the application while displaying the graphical user interface, the audio data comprising data indicative of the audible portions of the content; determining, from the audio data, whether the audio data is of a type suitable for automatic captioning; in response to determining that the audio data is of a type suitable for automatic captioning, obtaining a description of the audible portions of the content;and, while displaying visual portions of the content in the graphical user interface of the application, outputting, as a persistent element separate from the graphical user interface of the application, the description of the audible portions of the content for display;

[0104] Clause 11. The method of Clause 10, wherein the description comprises at least one of: a transcription of spoken audio extracted from the audible portions of the content, or text indicative of non-spoken audio extracted from the audible portions of the content.

[0105] Clause 12. The method of any of clauses 10-11, wherein the data indicative of the audible portions of the content is non-metadata, and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitling comprises determining, from the metadata, whether the audio data is of a type suitable for subtitling.

[0106] Clause 13. A method according to any of clauses 10-12, wherein the description includes text identifying a human or non-human source for different portions of the audible parts of the content.

[0107] Clause 14. The method of any of clauses 10-13, further comprising: in response to receiving user input associated with the persistent element, resizing the persistent element to display previous or subsequent descriptions generated from the audible portions of the content.

[0108] Clause 15. The method of any of clauses 10-14, wherein the persistent element is output for display by obscuring a first portion of the application's graphical user interface, the method further comprising: in response to receiving user input associated with the persistent element, moving the persistent element away from the first portion of the application's graphical user interface to obscure a second portion of the application's graphical user interface.

[0109] Clause 16. A method for automatically subtitling audible portions of content on a computing device, the method comprising: receiving user input to automatically subtitle audio data from an application executing on the computing device; in response to receiving the user input, obtaining the audio data output from the application executing on the computing device, the audio data comprising data indicative of the audible portions of the content; determining, from the audio data, whether the audio data is of a type suitable for subtitling; in response to determining that the audio data is of a type suitable for subtitling, determining a description of the audible portions of the content;and outputting, as a persistent element separate from visual portions of the content and separate from a graphical user interface of the application, the description of the audible portions of the content for display;

[0110] Clause 17. The method of Clause 16, wherein the data indicative of the audible portions of the content is non-metadata, and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitling comprises determining, from the metadata, whether the audio data is of a type suitable for subtitling.

[0111] Clause 18. The method of any of clauses 16 or 17, wherein receiving user input to automatically caption the audio data comprises: displaying a caption control within an audio settings menu of the computing device; and receiving user input to automatically caption the audio data in response to detecting a user selection of the caption control.

[0112] Clause 19. The method of Clause 18, further comprising: automatically ceasing captioning of the audible portions of the content at the computing device in response to receiving a subsequent selection of the caption control.

[0113] Clause 20. The method of any of Clauses 16-19, wherein outputting the description of the audible portions of the content comprises outputting the description while omitting output of the audible portions of the content using a speaker, headphone jack, or other sound system of the computing device.

[0114] Clause 21. The method of any of Clauses 16-19, wherein outputting the description of the audible portions of the content comprises outputting the description while simultaneously outputting the audible portions of the content using a speaker, headphone jack, or other sound system of the computing device.

[0115] Clause 22. The method of any of clauses 1-21, further comprising: training a machine-learned model to determine descriptions of audio data, wherein determining the description comprises using the machine-learned model to determine the description.

[0116] Clause 23. The method of Clause 22, wherein training the machine-learned model comprises utilising previously subtitled content to train the machine-learned model to derive subtitles embedded in the previously subtitled content from audio of the previously subtitled content.

[0117] Clause 24. The method of Clause 22, wherein training the machine-learned model comprises using the description as a training input to the machine-learned model.

[0118] Clause 25. A computing device comprising at least one processor configured to perform any of the methods set out in Clauses 1-24.

[0119] Clause 26. System comprising means for performing any of the methods set out in Clauses 1-24.

[0120] Clause 27. Computer-readable storage medium containing instructions that, when executed, configure a processor of a computing device to perform any of the methods set out in Clauses 1-24.

[0121] While various preferred embodiments of the disclosure have been described in the foregoing specification and illustrated in the drawings, it should be clearly understood that this disclosure is not limited thereto, but may be variously embodied in practice within the scope of the following claims. From the foregoing description, it will be apparent that various changes may be made without departing from the spirit and scope of the disclosure as defined by the following claims.

Claims

[1] Computer-implemented method comprising: Obtaining (404), by a processor (202) of a computing device (200), from an audio mixer (206) of the computing device, audio data output by an application executing on the computing device, the audio data comprising data indicative of audible portions of content; Determining (406), by the processor using the audio data, whether the audio data is of a type suitable for subtitling; in response to determining that the audio data is of a type suitable for subtitling, determining (410), by the processor, a description of the audible portions of the content; and while outputting visual portions of the content for display, outputting (412), by the processor and for display, the description of the audible portions of the content. [2] The method of claim 1, wherein the data indicative of the audible portions of the content is non-metadata, and the audio data further includes metadata, wherein determining whether the audio data is of a type suitable for subtitling comprises determining, by the processor using the metadata, whether the audio data is of a type suitable for subtitling. [3] The method of any of claims 1 and 2, wherein the description of the audible portions of the content comprises a transcription of spoken audio from the audible portions of the content. [4] The method of any of claims 1-3, wherein the description of the audible portions of the content comprises a description of non-spoken audio from the audible portions of the content. [5] The method of claim 4, wherein the non-spoken audio comprises a sound from a particular source and the description of the sound from the particular source comprises an indication of the particular source. [6] The method of claim 5, wherein: the noise includes an animal noise from an animal source, or the noise includes ambient noise from a non-animal source. [7] The method of any of claims 1-6, wherein determining the description of the audible portions of the content comprises executing, by the processor of the computing device, a machine-learned model trained to determine descriptions from audio data to determine the description of the audible portions of the content. [8] The method of claim 7, wherein the machine-learned model comprises an automatic regressive end-to-end neural network transformer speech recognition model. [9] The method of any of claims 1-8, wherein the data indicative of the audible portions of the content comprises unannotated data that has not been annotated for subtitling. [10] The method of any of claims 1-9, wherein the description comprises text indicating non-spoken audio extracted from the audible portions of the content. [11] The method of any of claims 1-10, wherein the description comprises text identifying a human or non-human source for various portions of the audible portions of the content. [12] The method of any of claims 1-11, wherein outputting the description of the audible portions of the content comprises outputting, by the processor and for display, the description of the audible portions of the content as a persistent element (506) separate from visual portions of the content and separate from a graphical user interface of the application. [13] The method of claim 12, further comprising: in response to receiving, by the processor, user input associated with the persistent element, resizing the persistent element to display previous or subsequent descriptions generated from the audible portions of the content. [14] A computing device (200) comprising at least one processor (202) configured to perform the method of any of claims 1-13. [15] A computer-readable storage medium comprising instructions that, when executed, configure a processor of a computing device (100; 200) to perform the method of any of claims 1-13.

Images