System and method for identifying and providing information about semantic entities in an audio signal

Abstract

Systems and methods for determining semantic entities in an audio signal are provided. A method may include obtaining, by a computing device including one or more processors and one or more memory devices, an audio signal simultaneously heard by a user. The method may also include analyzing, by a machine learning model stored on the computing device, at least a portion of the audio signal in a background of the computing device to determine one or more semantic entities. The method may also include displaying the one or more semantic entities on a display screen of the computing device.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for identifying semantic entities in audio signals. More specifically, the present disclosure relates to systems and methods for utilizing machine learning operating in the background of a computing device to identify semantic entities in various audio signals heard by a user, display the semantic entities to the user, and provide the user with supplemental information about the selected semantic entities. Background Art

[0002] Mobile computing devices such as smartphones have provided the ability to listen to audio-based content on demand across a variety of platforms and applications. For example, people can listen to music and movies stored locally on their smartphones; stream movies, music, television programs, podcasts, and other content from a multitude of free and subscription-based services; access multimedia content available on the Internet; and so on. In addition, mobile computing devices can allow people to communicate verbally with each other in a variety of ways, such as phone calls, video calls, and audio / video messages sent via MMS. Some mobile computing devices can even be used to provide translation services, such as during a face-to-face conversation between speakers of two different languages.

[0003] Therefore, people can hear certain topics of interest while using their mobile computing devices in various environments and situations. However, although a person may be interested in a particular topic, often the person may not be able to interrupt the task they are performing to find additional information about the particular topic. Summary of the invention

[0004] Aspects and advantages of the present disclosure will be set forth in part in the following description, or may be obvious from the description, or may be learned through practice of embodiments of the present disclosure.

[0005] An example aspect of the present disclosure is directed to a method for identifying semantic entities within an audio signal. The method may include obtaining, by a computing device including one or more processors and one or more memory devices, an audio signal simultaneously heard by a user. The method may also include analyzing, by a machine learning model stored on the computing device, at least a portion of the audio signal in a background of the computing device to determine one or more semantic entities. The method may also include displaying the one or more semantic entities on a display screen of the computing device.

[0006] Another example aspect of the present disclosure is directed to a device for identifying semantic entities within an audio signal. The device may include one or more processors and a display screen. The one or more processors may be configured to obtain an audio signal. The one or more processors may also be configured to receive a user request to identify one or more semantic entities within a portion of the audio signal. The one or more processors may also be configured to analyze the portion of the audio signal by a machine learning model to determine one or more semantic entities within the portion. The one or more processors may also be configured to operate the display screen to display the one or more semantic entities.

[0007] Another example aspect of the present disclosure is directed to a system for identifying semantic entities within an audio signal. The system may include a computing device. The computing device may include one or more processors and a display screen. The system may also include a speaker device configured to play one or more audio signals for a user. The speaker device may also be configured to receive one or more user interactions from a user. The speaker device may be operable to receive a user interaction indicating a user request. In response to receiving the user interaction, the speaker device may also be operable to transmit the user request to the computing device. The computing device may be operable to receive the user request. In response to receiving the user request, the computing device may also be operable to analyze a first portion of a first audio signal with a machine learning model to identify one or more semantic entities. The computing device may also be operable to display one or more semantic entities on a display screen.

[0008] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects and advantages of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A full and enabling description of the present disclosure for one of ordinary skill in the art is set forth in the specification, which refers to the accompanying drawings, in which:

[0011] Figure 1 depicts a block diagram of an example system for identifying semantic entities within an audio signal according to example aspects of the present disclosure;

[0012] Figure 2 depicts a block diagram of an example machine learning speech recognition semantic entity recognizer model according to example aspects of the present disclosure;

[0013] Figure 3depicts a block diagram of an example machine learning language translation semantic entity recognizer model according to example aspects of the present disclosure;

[0014] Figure 4 Depicting example user interactions and user interfaces for displaying one or more semantic entities and one or more supplemental information options on a computing device;

[0015] Figure 5 An example user interface for displaying one or more semantic entities and one or more supplemental information options on a computing device is depicted;

[0016] Figure 6 An example user interface for displaying one or more semantic entities and one or more supplemental information options on a computing device is depicted;

[0017] Figure 7 An example user interface for displaying one or more semantic entities and one or more supplemental information options on a computing device is depicted;

[0018] Figure 8 An example user interface for displaying one or more semantic entities and one or more supplemental information options on a computing device is depicted;

[0019] Fig. 9 An example method for identifying semantic entities within an audio signal according to example aspects of the present disclosure is depicted;

[0020] Fig.10 An example method for identifying semantic entities within an audio signal according to example aspects of the present disclosure is depicted; and

[0021] Fig.11 An example method for identifying semantic entities within an audio signal according to example aspects of the present disclosure is depicted. DETAILED DESCRIPTION

[0022] In general, the present disclosure is directed to systems and methods that can operate in the background of a computing device to automatically identify and present semantic entities heard by a user in various audio signals and further provide the user with supplementary information about the semantic entities of interest. For example, the system and method of the present disclosure can be implemented on a computing device (such as a user's smart phone) in an always-on operating mode in the background of the computing device to identify semantic entities heard by the user from various audio signals, such as media files played on the computing device, face-to-face conversations, or other audio, telephone conversations, etc. that the user inadvertently hears in his / her environment. In addition, the semantic entities can be presented to the user in a clever way that allows the user to access supplementary information about the semantic entities (such as contextual supplementary information). For example, a person who listens to a podcast on his smart phone may be interested in learning more about a specific writer discussed in the podcast. Similarly, a person who talks to a tour guide who speaks a foreign language may be interested in learning more about the specific tourist attractions discussed by the tour guide. As another example, a person who is having a telephone conversation with a friend may be curious to see what is on the menu of a restaurant recommended by her friend. The systems and methods of the present disclosure can identify various semantic entities (e.g., people, places, locations, etc.) within an audio signal heard by a user, and display the semantic entities to the user via a user interface such as displayed on a computing device. In addition, the user can select a specific semantic entity, such as through a user interface of an application executing on the computing device, to access supplemental information about the selected semantic entity.

[0023] More specifically, the systems and methods of the present disclosure may allow a computing device to obtain an audio signal that a user hears at the same time. For example, an audio signal may include media played on a computing device, a communication signal (e.g., a phone call) transmitted to a computing device, ambient audio received by the computing device (such as a conversation between a user and a third person), and / or other audio signals. Then, a machine learning model stored on the computing device may analyze at least a portion of the audio signal to determine one or more semantic entities. For example, a speech recognition machine learning model may be trained to identify various people, places, things, dates / times, events, or other semantic entities in an audio signal including speech. The analysis of the audio signal may be performed in the background of the computing device. As used herein, when used with reference to analyzing an audio signal on a computing device, the phrase "in the background" means simultaneously with another task performed on the computing device or when the computing device is in an idle state. For example, the audio associated with a movie watched by a user on his / her smart phone may be analyzed while the movie is playing.

[0024] In addition, the computing device may then display one or more semantic entities, such as on a display screen of the computing device. For example, in various embodiments, one or more semantic entities may be displayed in a variety of ways, such as by displaying text, icons, pictures, etc. indicating the semantic entity, and may be displayed in a list format or via an application-specific user interface. In some embodiments, a user may select a semantic entity displayed on a display screen, and the computing device may display one or more supplementary information options associated with the selected semantic entity. For example, the supplementary information options may include database entries (e.g., webref entries, Wikipedia entries, etc.), search engine results (e.g., Google search results, etc.), and / or application interaction options (e.g., restaurant review applications, reservation applications, event ticket purchase applications, etc.), and may be selected based on the context of the audio signal and / or the context of the semantic entity. In this way, one or more semantic entities that may be potentially of interest in the audio signal may be automatically identified and displayed to the user, and when the user chooses to access such supplementary information, supplementary information about a particular semantic entity may be provided to the user.

[0025] According to example aspects of the present disclosure, a computing device (e.g., a smart phone) may include one or more processors and one or more memory devices, and may be configured to store one or more machine learning models. The computing device may also be configured to obtain various audio signals that a user hears simultaneously. For example, in some embodiments, the computing device may be configured to execute various applications with associated audio signals. In some embodiments, an audio signal may be played via an application executed on a computing device, such as in an Internet browser application window, a music application, a movie application, a phone call application, a voice-to-text dictation application, etc. For example, a computing device may be configured to play various media file formats, such as audio-visual files (.wmv, .avi, .mov, .mpeg, .flv, etc.), compressed or uncompressed audio files (.wav, .mp3, .wma, .aac, .m4a, etc.) and / or other types of media files. The term "obtaining" as used herein with respect to such media files includes accessing an audio signal associated with such a media file, such as when a media file is played on a computing device or transmitted to an associated peripheral device. In some embodiments, an audio signal may be obtained by transmitting the audio signal to a computing device, such as via one or more communication networks. For example, in some embodiments, a computing device (e.g., a smart phone) can be configured to receive various audio signals transmitted to the computing device, such as audio signals transmitted through a cellular phone network (e.g., CDMA, GSM, LTE, etc.), a wireless network (WLAN, Bluetooth, BLE, etc.), a radio network (e.g., AM, FM, etc.), or other communication networks. In some embodiments, such as when a user has authorized the computing device to capture ambient audio signals, the computing device can be configured to receive various ambient audio signals, such as human speech from a conversation via a microphone of the computing device. For example, a user's smart phone can be configured to be in an "always on" mode, where the microphone of the smart phone generates an audio signal based on the ambient audio that the user is simultaneously hearing, which can be analyzed in the background of the smart phone to identify semantic entities in the audio signal.

[0026] Thus, in some embodiments, in order to obtain the benefits of the techniques described herein, a user may be asked to allow audio signals to be collected and analyzed by his or her computing device. For example, in some embodiments, a user may be provided with an opportunity to control whether a program or feature collects these audio signals. If a user does not allow the collection and use of such audio signals, the user may not receive the benefits of the techniques described herein. Tools may also be provided for the user to revoke or modify consent. In addition, certain information or data may be processed in one or more ways before it is stored or used to protect user information. As an example, a computing device may temporarily store such audio signals in an audio buffer for analysis and discard the audio signals after analysis.

[0027] Then, at least a portion of the audio signal can be analyzed using a machine learning model stored on a computing device to determine one or more semantic entities. For example, in some embodiments, a portion of an audio file (such as a rolling audio buffer) can be input into a machine learning model trained to identify various semantic entities. In some embodiments, the machine learning model can be a speech recognition semantic entity recognizer model that is configured to recognize various semantic entities in human speech. In some embodiments, the machine learning model can be a language translation semantic entity recognizer model that is trained to recognize and / or translate various semantic entities in a foreign language. In some embodiments, the machine learning model can be a song recognition semantic entity recognizer model that is trained to recognize songs. The audio signal or a portion thereof can be input into the machine learning model, and one or more semantic entities can be received as an output of the machine learning model. In addition, the analysis of the audio signal can be performed in the background of the computing device, such as when the computing device is performing another task. For example, in an embodiment where the user has provided appropriate authorization, the audio signal associated with the phone call can be analyzed by a machine learning model on the user's smart phone to identify the semantic entities in the phone conversation when the phone conversation occurs.

[0028] Then, the one or more semantic entities can be displayed on a display screen of the computing device. For example, in some embodiments, the one or more semantic entities can be displayed in a user interface on a display screen. In some embodiments, the one or more semantic entities can be displayed in a "lock screen", such as when the computing device is in a locked or idle mode. In some embodiments, multiple semantic entities can be displayed, such as in a list in the order in which the semantic entities appear in the audio signal. In some embodiments, the one or more semantic entities can be displayed in a specific display area (field) of the user interface, such as in a specified area of ​​the user interface of an application running on the computing device. For example, during a phone call, a semantic entity (such as text or an icon indicating the semantic entity) can be displayed in a specific area of ​​the user interface for a phone call application executed by a smart phone.

[0029] In some embodiments, one or more semantic entities may be recorded and accessed by a user at a later time. For example, over the course of a day, a user may hear various audio signals, such as phone calls, songs, and / or face-to-face conversations, each of which may have an associated semantic entity that may be identified in the corresponding audio signal by the user's computing device. In some embodiments, the semantic entities identified in such audio signals may be recorded (e.g., data indicating the semantic entities may be stored locally on the computing device) for access at a later time. For example, in one embodiment, a user may access previously identified semantic entities via a "clipboard," which may be a user interface displayed on a computing device in which semantic entities that have been previously identified (such as over the course of a day) are displayed for the user's reference.

[0030] In some embodiments, the audio signal may be a streaming audio signal, such as an audio signal of an ongoing conversation and / or a playing media file. When the computing device obtains the streaming audio signal, the streaming audio signal or a portion thereof may be analyzed by a machine learning model on a rolling basis to identify multiple semantic entities. For example, multiple continuous portions of the audio signal may be analyzed to identify multiple semantic entities. Similarly, in some embodiments, multiple semantic entities may be identified in a single portion of the audio signal. In such a case, multiple semantic entities may be displayed on a display screen, such as when a semantic entity is identified and / or each corresponding semantic entity is displayed in the order in which multiple semantic entities are identified. For example, in some embodiments, an audio signal associated with a podcast played on a user's smart phone may be analyzed to identify multiple semantic entities, and each corresponding semantic entity may be displayed on the smart phone in response to being identified. In some embodiments, each corresponding semantic entity may be displayed for a predetermined time period (e.g., five seconds). In some embodiments, each corresponding semantic entity may be displayed in a specific area of ​​the user interface of the application, such as a "live tile", wherein each corresponding semantic entity (e.g., icon) is displayed in response to being identified. In some embodiments, the plurality of corresponding semantic entities may be displayed, such as in a list format. In some embodiments, a plurality of recently recognized semantic entities may be displayed, such as a scrolling list of three or four recently recognized semantic entities. In some embodiments, the plurality of semantic entities may be displayed without a request to display the plurality of semantic entities, such as on a lock screen of a user's smartphone.

[0031] In some implementations, one or more semantic entities may be displayed in response to receiving a user request. For example, in some implementations, a user may request a computing device to display one or more semantic entities by performing a user interaction with the computing device or an associated peripheral device.

[0032] For example, in some embodiments, a computing device may be communicatively coupled to an associated peripheral device. The associated peripheral device may be, for example, a speaker device, such as an earbud device coupled to the computing device via Bluetooth or other wireless connection. In some embodiments, a user may perform a user interaction indicating a user request with the associated peripheral device, and the associated peripheral device may transmit the user request to the computing device. For example, in some embodiments, the user interaction may be a "get" gesture on the earbud device, such as a pulldown gesture. In response to receiving the "get" gesture, the associated peripheral device (e.g., earbud device) may transmit the user request to the computing device. In response to receiving the user request, the computing device may then display one or more semantic entities on a display screen of the computing device.

[0033] In some embodiments, an associated peripheral device such as a speaker device (e.g., an earbud device) may also be configured to play one or more audio signals for a user. For example, a computing device may be operable to transmit an audio signal to a speaker device (such as via a Bluetooth connection), and upon receiving the audio signal, the speaker device may audibly play the audio signal for the user. When a user listens to an audio signal played via a speaker device, the user may perform a user interaction with the speaker device to request the computing device to display one or more semantic entities in the audio signal played by the speaker device. For example, according to an example aspect of the present disclosure, a user may listen to a podcast (or other media) by streaming the podcast from a user's smart phone to a communicatively coupled earbud device, and upon hearing one or more topics of interest that the user wants to learn more about, the user may perform a "get" gesture on the earbud device to request the smart phone to display one or more semantic entities determined from the audio signal played on the earbud device.

[0034] In some embodiments, the user request can be received through user interaction with the computing device. For example, in some embodiments, the display screen can be a touch-sensitive display screen that is configured to receive various user interactions by the user touching or performing various interactions with the display screen. For example, in some embodiments, the user can perform a "swipe" gesture, such as touching a first portion of the display screen and sliding his finger along the display screen to a second portion of the display screen, in order to request that one or more semantic entities be displayed on the display screen. For example, the computing device can be a user's smartphone that is in an idle or locked state while playing a media file, and the user can perform a "swipe" gesture on the display screen to request that one or more semantic entities be displayed. In response to receiving the user request, the computing device can display the one or more semantic entities.

[0035] Similarly, in some embodiments, a user request may be received by a user accessing an application on a computing device or executing an application on a computing device. For example, in some embodiments, one or more semantic entities may be recorded and accessed by a user at a later time. For example, in the course of a day, a user may hear various audio signals, such as phone calls, songs, and / or face-to-face conversations, each of which may have associated semantic entities that have been identified in corresponding audio signals by the user's computing device. In some embodiments, the semantic entities identified in such audio signals may be recorded (e.g., data indicating the semantic entities may be stored locally on a computing device) for access at a later time. For example, in one embodiment, a user may access previously identified semantic entities via a "clipboard" application, which may be a user interface displayed on a computing device, wherein semantic entities that have been identified previously (such as in the course of a day) are displayed for the user's reference. A user request may be, for example, an interaction with a computing device, wherein a user accesses or executes a "clipboard" application.

[0036] In some embodiments, the user request may be received by other suitable means, such as by voice command, button press, other user interface interaction, or other interaction indicating the user request. In addition, in various embodiments, the user request may be received by an associated peripheral device (such as an earbud device or other speaker device) and transmitted to the computing device.

[0037] In some embodiments, after receiving a user request, the computing device may determine the selected portion of the audio signal for analysis based at least in part on a predetermined time period before receiving a request from the user to display one or more semantic entities. For example, in some embodiments, the computing device may maintain a buffer in which the audio signal is temporarily stored when it is received (e.g., when playing a media file, when receiving a communication signal, or when an ambient audio signal is obtained via a microphone). After receiving a user request, the computing device may determine the selected portion of the audio signal for analysis based on a predetermined time period before receiving the request from the user. For example, a portion of the audio signal may be selected based on the time when the user request is received. In some embodiments, the portion of the audio signal may be a portion of the audio signal before the time when the user request is received. For example, an audio signal of 10-15 seconds before receiving the user request may be selected as the selected portion of the audio signal for analysis. In some embodiments, analysis of the audio signal may be performed in response to receiving a user request, such as by analyzing only the selected audio portion by a machine learning model to determine one or more semantic entities to be displayed on a display screen. In other embodiments, the entire audio signal may be analyzed, such as on a rolling or continuous basis, and in response to receiving a user request, semantic entities that have been identified within a selected audio portion may be displayed on a display screen.

[0038] In some embodiments, a user may select a selected semantic entity from one or more semantic entities displayed on a display screen of a computing device. For example, in some embodiments, a single semantic entity may be displayed at a time, and if the user performs a specific user interaction (such as touching the displayed semantic entity on a touch-sensitive display screen), the user may select the semantic entity. In other embodiments, multiple semantic entities may be displayed, such as in a list, and the user may select a specific semantic entity from the list. In various embodiments, the user selection may be received, for example, by a voice command, a button press, a user interface interaction, or other interaction indicating a user selection.

[0039] In response to receiving a user selection of the selected semantic entity, the computing device may determine one or more supplemental information options associated with the selected semantic entity. In addition, the computing device may display one or more supplemental information options associated with the selected semantic entity on a display screen of the computing device. In some embodiments, the one or more supplemental information options may include database entries (e.g., webref entries, Wikipedia entries, etc.), search engine results (e.g., Google search results, etc.), and / or application interaction options (e.g., restaurant review applications, reservation applications, event ticket purchase applications, calendar applications, etc.).

[0040] In some embodiments, one or more supplemental information options may be determined based at least in part on the context of the audio signal obtained by the computing device and / or the context of the selected semantic entity. The context of the audio signal obtained by the computing device may be, for example, the type of audio signal obtained by the computing device and / or the manner in which the audio signal is obtained. The context of the selected semantic entity may be, for example, a classification of the selected semantic entity, such as whether it is a person, a place (e.g., a restaurant, a venue, a tourist attraction, etc.), a thing (e.g., a product), an event, a date and / or a time, or other classification.

[0041] For example, for a media file (e.g., a podcast) played on a computing device, the supplemental information options associated with a selected semantic entity may be different than the selected semantic entity identified during a telephone conversation or ambient conversation between two or more individuals. For example, the supplemental information options provided for a particular restaurant mentioned in a podcast may include a search engine result (e.g., a Google result), a database entry (e.g., a Wikipedia entry), and / or a link to the restaurant's website, while the supplemental information options provided for a particular restaurant in the context of a telephone call or ambient conversation may include a link to a restaurant review website, an option to open a restaurant reservation application, an option to add a calendar entry for a particular date and time, etc.

[0042] Similarly, the supplemental information options associated with different types of selected semantic entities may be different. For example, the supplemental information options for a book mentioned in a podcast may include a link to the author's website, one or more options to purchase the book online, directions to a nearby library, etc., while the supplemental information options for a movie mentioned in the podcast may include a link to a movie trailer, an option to open a movie ticket purchase application, an option to access showtimes at a nearby theater, directions to a nearby theater, etc.

[0043] The system and method of the present disclosure can provide many technical effects and benefits. For example, the system and method provided herein can allow the semantic entities within the audio signal simultaneously heard by the user to be identified, either automatically or in response to a request from the user. In addition, by making full use of one or more machine learning models (e.g., neural networks), the system and method of the present disclosure can improve the computational efficiency and accuracy of semantic entity recognition. For example, a machine learning speech recognition semantic entity recognizer model or a machine learning language translation semantic entity recognizer model can be implemented on a computing device, which can be trained to recognize semantic entities in various audio signals. In this way, semantic entities in various languages ​​and various audio signals can be determined in an effective manner.

[0044] In addition, the system and method of the present disclosure can display one or more semantic entities identified in the audio signal to the user, thereby allowing the user to select a specific semantic entity of interest to the user. For example, one or more semantic entities identified in the audio signal can be displayed on a display screen of a computing device, where the user can select a specific semantic entity. After selecting a specific semantic entity, one or more supplementary information options associated with the selected semantic entity can be provided to the user. Moreover, the supplementary information provided to the user can be intelligently curated based on the context of the audio signal and / or the context of the semantic entity selected by the user. Therefore, the system and method of the present disclosure can allow the user to easily access various supplementary information about the semantic entities of interest to the user in an effective manner, thereby improving the user experience. In addition, by analyzing the audio signal in the background of the computing device, the semantic entity can be identified and cleverly presented to the user without interrupting any other tasks being performed by the computing device.

[0045] The system and method of the present disclosure also provide improvements to computing technology. In particular, a computing device can obtain an audio signal heard by a user at the same time, and analyze at least a portion of the audio signal in the background of the computing device through a machine learning model stored on the computing device to identify one or more semantic entities. In some embodiments, one or more semantic entities can be identified in response to a user request. One or more semantic entities can be displayed on a display screen of a computing device. In some embodiments, a user can select a semantic entity displayed on the display screen. The computing device can determine one or more supplementary information options associated with the selected semantic entity. In some embodiments, one or more supplementary information options can be determined at least in part based on the context of the audio signal obtained by the computing device or the context of the selected semantic entity. The computing device can display one or more supplementary information options on the display screen.

[0046] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0047] Figure 1 An example system for identifying semantic entities within an audio signal according to example aspects of the present disclosure is depicted. System 100 may include a computing device 102 (eg, a mobile computing device such as a smartphone), a server computing system 130, and a peripheral device 150 (eg, a speaker device).

[0048] The computing device 102 may include one or more processors 108 and a memory 110. The one or more processors 108 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operably connected. The memory 110 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. In some embodiments, the memory may include a temporary storage device for temporarily storing audio signals, such as an audio buffer. The memory 110 may store instructions 114 and data 112 that may be executed by the processor 108 to enable the user computing device 102 to perform operations.

[0049] The computing device 102 may also include one or more speakers 116. The one or more speakers 116 may, for example, be configured to audibly play audio signals (e.g., generate sound waves including sounds, speech, etc.) for a user to hear. For example, an audio signal associated with a media file played on the computing device 102 may be audibly played for a user by the one or more speakers 116. Similarly, an audio signal associated with a communication signal (e.g., a telephone call) received by the computing device 102 may be audibly played by the one or more speakers 116.

[0050] The computing device 102 may also include one or more display screens 118. The display screen 118 may be, for example, a display screen configured to display various information to a user. In some implementations, the one or more display screens 118 may be a touch-sensitive display screen capable of receiving user input.

[0051] The computing device 102 may include one or more user interfaces 120. A user may interact with the user computing device 102 using the user interface 120, such as to request display of a semantic entity or to request supplemental information about a particular semantic entity. The user interface 120 may be displayed on the display screen 118. Figure 4-8 An example user interface 120 according to example aspects of the present disclosure is discussed in further detail.

[0052] The computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen 118 or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). In some embodiments, a user may perform a "slide" gesture, such as touching a first portion of the touch-sensitive display screen 118 and sliding their finger along the display screen 118 to a second portion of the display screen, so as to request that one or more semantic entities be displayed on the display screen 118, as described herein. In some embodiments, the touch-sensitive component may be used to implement a virtual keyboard. Other example user input components 122 include one or more buttons, a conventional keyboard, or other means by which a user may provide user input. The user input component 122 may allow a user to provide user input, such as via the user interface 120 or in response to information displayed in the user interface 120.

[0053] The computing device 102 may also include one or more microphones 124. The one or more microphones 124 may be, for example, any type of audio sensor and associated signal processing components configured to generate an audio signal from ambient audio. For example, ambient audio (such as human speech) may be received by the one or more microphones 124, and the one or more microphones 124 may generate an audio signal based on the ambient audio.

[0054] According to another aspect of the present disclosure, the computing device 102 may also include one or more machine learning models 126. In some embodiments, the machine learning model 126 may be operable to analyze audio signals obtained by the computing device 102. For example, the computing device 102 may be configured to play various media files, and the associated audio signals may be analyzed by the one or more machine learning models 126 to identify semantic entities, as disclosed herein. In some embodiments, the one or more machine learning models 126 may be, for example, a neural network (e.g., a deep neural network) or other multi-layer nonlinear model that outputs a semantic entity (e.g., data describing the semantic entity) in response to the audio signal. Further reference will be made below to Figure 2 and 3 Let's discuss an example machine learning model 126 according to example aspects of the present disclosure.

[0055] The computing device 102 may also include a communication interface 128. The communication interface 128 may include any number of components to provide networked communications (e.g., transceivers, antennas, controllers, cards, etc.). In some implementations, the computing device 102 includes a first network interface operable to communicate using a short-range wireless protocol (such as, for example, Bluetooth and / or Bluetooth low energy), a second network interface operable to communicate using other wireless network protocols (such as, for example, Wi-Fi), and / or a third network interface operable to communicate via GSM, CDMA, AMPS, 1G, 2G, 3G, 4G, 5G, LTE, GPRS, and / or other wireless cellular networks.

[0056] Still refer to Figure 1 , the system 100 may also include a server computing system 130. The server computing system 130 may include one or more processors 132 and a memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 134 may store instructions 138 and data 136 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0057] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0058] In some implementations, the server computing system 130 may store or include one or more machine learning models. For example, the server computing system 130 may include one or more speech recognition semantic entity recognizer models 140, one or more song recognition semantic entity recognizer models 141, and / or one or more language translation semantic entity recognizer models 142.

[0059] For example, the speech recognition semantic entity recognizer model 140 can be trained to recognize semantic entities in an audio signal including speech. For example, an audio signal or a portion thereof can be analyzed by the speech recognition semantic entity recognizer model 140 to recognize semantic entities present in the audio signal. In various embodiments, the audio signal can be a voice associated with an application being executed by a computing device, such as a media played on a computing device (e.g., a podcast), a voice associated with a communication signal (e.g., a phone call) transmitted to or from a computing device, and / or an ambient audio (such as a conversation between two people) received by a computing device. In some embodiments, the audio signal can be analyzed by maintaining a copy of the audio signal (and / or data indicating the audio signal) in an audio buffer of the memory 110 of the computing device. At least a portion of the audio signal can be input into the speech recognition semantic entity recognizer model 140. A semantic entity (or multiple semantic entities) can then be received as an output of the speech recognition semantic entity recognizer model 140. In some embodiments, the audio signal and / or data indicating the audio signal maintained in the audio buffer can be discarded after analysis, thereby helping to maintain bystander and user privacy.

[0060] The song recognition semantic entity recognizer model 141 can be trained to recognize semantic entities in an audio signal including a song. In various embodiments, the audio signal can be a song associated with an application being executed by a computing device (such as a music streaming service application (e.g., Pandora, Spotify, etc.) operating on the computing device), a song associated with a communication signal transmitted to or from the computing device (e.g., a song transmitted via a phone call or video conference), and / or ambient audio received by the computing device (such as a song played at the location where the computing device is located). For example, the audio signal or a portion thereof can be analyzed by the song recognition semantic entity recognizer model 141 to recognize a song, such as a song title and an artist performing the song. In some embodiments, the audio signal can be analyzed by maintaining a copy of the audio signal (and / or data indicating the audio signal) in an audio buffer of the memory 110 of the computing device. At least a portion of the audio signal can be input into the song recognition semantic entity recognizer model 141. Semantic entities (such as song names and artists) can then be received as outputs of the song recognition semantic entity recognizer model 141.

[0061] The language translation semantic entity recognizer model 142 can be trained to recognize semantic entities in an audio signal including speech in a foreign language (i.e., a language other than the user's primary language). In various embodiments, the audio signal can be speech associated with an application being executed by a computing device, such as media played on the computing device (e.g., a foreign language podcast), speech associated with a communication signal transmitted to or from the computing device (e.g., a phone call), and / or ambient audio received by the computing device (such as a conversation between two people speaking different languages). In some embodiments, the language translation semantic entity recognizer model 142 can be trained to recognize semantic entities in a foreign language and output semantic equivalents in the primary language. In some embodiments, the audio signal can be analyzed by maintaining a copy of the audio signal (and / or data indicating the audio signal) in an audio buffer of the memory 110 of the computing device. At least a portion of the audio signal can be input into the language translation semantic entity recognizer model 142. The semantic entity (or multiple semantic entities) can then be received as an output of the language translation semantic entity recognizer model 142. In some implementations, the audio signal and / or data indicative of the audio signal maintained in the audio buffer may be discarded after analysis, thereby helping to maintain bystander and user privacy.

[0062] About Figure 2 and 3 Example machine learning models according to example aspects of the present disclosure, such as speech recognition semantic entity recognizer model 140, song recognition semantic entity recognizer model 141, and language translation semantic entity recognizer model 142, are discussed in more detail.

[0063] The server computing system 130 may include a model trainer 146 that trains one or more machine learning models 140-142 using various training or learning techniques such as, for example, back propagation of errors. In some embodiments, performing back propagation of errors may include performing truncated back propagation through time. The model trainer 146 may perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0064] In particular, model trainer 146 can train one or more machine learning models 140-142 based on a collection of training data 144. Training data 144 can include, for example, audio signals labeled with semantic entities. For example, a human reviewer can annotate various audio signals with semantic entity labels, which can be used as training data 144 for one or more machine learning models 140-142.

[0065] In some implementations, the server computing system 130 may implement a model trainer 146 to train a new model or update a version of an existing model on the additional training data 144. As an example, the model trainer 146 may train one or more machine learning models 140-142 using audio signals manually labeled with new semantic entities to provide outputs that include the new semantic entities.

[0066] The server computing system 130 may periodically provide the computing device 102 with one or more updated versions of one or more models 140-142 included in the machine learning model 126 stored on the computing device 102. The updated models 140-142 may be transmitted to the user computing device 102 via the network 180.

[0067] The model trainer 146 may include computer logic for providing the desired functionality. The model trainer 146 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some embodiments, the model trainer 146 includes a program file stored on a storage device, loaded into the memory 134, and executed by one or more processors 132. In other embodiments, the model trainer 146 includes one or more computer executable instruction sets stored in a tangible computer readable storage medium such as a RAM hard disk or an optical or magnetic medium.

[0068] In some embodiments, any process, operation, program, application, or instruction described as being stored at or executed by the server computing system 130 may instead be stored in whole or in part at or executed by the computing device 102, or vice versa. For example, the computing device 102 may include a model trainer 146 configured to train one or more machine learning models 126 stored locally on the computing device 102.

[0069] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications through network 180 may be carried via any type of wired and / or wireless connection using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0070] Still refer to Figure 1 , the system 100 may also include one or more peripheral devices 150. In some implementations, the peripheral device 150 may be an earbud device that may be communicatively coupled to the computing device 102.

[0071] The peripheral device 150 may include one or more user input components 152 configured to receive user input. (One or more) user input components 152 may be configured to receive user interactions indicating requests. For example, the user input component 152 may be a touch-sensitive component (e.g., a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to receive user interactions indicating requests, such as a "get" gesture (e.g., a pull-down motion), as described herein. Other example user input components 152 include one or more buttons, switches, or other means by which a user can provide user input. The user input component 152 may allow a user to provide user input, such as to request the display of one or more semantic entities.

[0072] Peripheral devices 150 may also include one or more speakers 154. One or more speakers 154 may, for example, be configured to audibly play audio signals (e.g., sounds, speech, etc.) for a user to hear. For example, an audio signal associated with a media file played on computing device 102 may be transmitted from computing device 102 (such as via one or more networks 180) and the audio signal may be audibly played for the user via one or more speakers 154. Similarly, an audio signal associated with a communication signal (e.g., a telephone call) received by computing device 102 may be audibly played by one or more speakers 154.

[0073] The peripheral device 150 may also include a communication interface 156. The communication interface 156 may include any number of components to provide networked communications (e.g., transceivers, antennas, controllers, cards, etc.). In some embodiments, the peripheral device 150 includes a first network interface operable to communicate using a short-range wireless protocol (such as, for example, Bluetooth and / or Bluetooth low energy), a second network interface operable to communicate using other wireless network protocols (such as, for example, Wi-Fi), and / or a third network interface operable to communicate via GSM, CDMA, AMPS, 1G, 2G, 3G, 4G, 5G, LTE, GPRS, and / or other wireless cellular networks.

[0074] According to example aspects of the present disclosure, computing device 102 may be configured to display semantic entities to a user. For example, computing device 102 may obtain audio signals that a user hears simultaneously. For example, an audio signal may include an audio signal associated with an application being executed by computing device 102, such as media played on computing device 102, a communication signal (e.g., a phone call) transmitted to computing device 102, an audio signal generated by microphone 124 when computing device 102 receives ambient audio (such as a conversation between a user and a third person), and / or other audio signals. Then, computing device 102 may input the audio signal or a portion thereof into (one or more) machine learning models 126 to identify semantic entities in the audio signal. Semantic entities may be, for example, people, places, things, dates / times, events, or other semantically different entities.

[0075] The analysis of the audio signal may be performed in the background of computing device 102, such as concurrently with another task being performed by computing device 102. For example, analysis of audio signals associated with media being played on computing device 102 may be performed by computing device 102 while the media is playing. In other words, analysis of the audio signal may be performed without interrupting the media playback or performing other tasks on computing device 102.

[0076] Additionally, the computing device 102 may then display the one or more semantic entities identified in the audio signal, such as on a display screen 118 of the computing device 102. For example, in various implementations, the one or more semantic entities may be displayed in various ways, such as by displaying text, icons, pictures, etc. indicating the semantic entities, and the one or more semantic entities may be displayed in a list format or via an application-specific user interface 120. Figure 4-8 An example user interface 120 according to example aspects of the present disclosure is discussed in further detail.

[0077] In some implementations, one or more semantic entities may be displayed in response to receiving a user request. For example, in some implementations, a user may request computing device 102 to display one or more semantic entities by performing a user interaction with computing device 102 (e.g., performing a “swipe” gesture or accessing a “clipboard” application) or a user interaction with an associated peripheral device 150 (e.g., performing a “get” gesture).

[0078] In some embodiments, after receiving a user request, the computing device 102 may determine the selected portion of the audio signal for analysis based at least in part on a predetermined time period before receiving a request from the user to identify one or more semantic entities. For example, in some embodiments, the computing device 102 may maintain a buffer in which the audio signal is temporarily stored when it is received (e.g., when a media file is played in an application, when a communication signal is received, or when an audio signal is generated by a microphone based on ambient audio). After receiving a user request, the computing device 102 may determine the selected portion of the audio signal for analysis based on a predetermined time period before receiving the request from the user. For example, a portion of the audio signal may be selected based on the time when the user request is received. In some embodiments, the portion of the audio signal may be a portion of the audio signal before the time when the user request is received. For example, an audio signal of 10-15 seconds before receiving the user request may be selected as the selected portion of the audio signal for analysis. In some embodiments, analysis of the audio signal may be performed in response to receiving a user request, such as by analyzing only the selected audio portion by the machine learning model 126 to determine one or more semantic entities to be displayed on the display screen 118. In other embodiments, the entire audio signal (or a portion thereof) may be analyzed in advance, such as on a rolling or continuous basis, and semantic entities that have been identified within a selected audio portion may be displayed on display screen 118 in response to receiving a user request.

[0079] In some embodiments, the user may select a selected semantic entity from one or more semantic entities displayed on the display screen 118 of the computing device 102. For example, in some embodiments, a single semantic entity may be displayed at a time, and if the user performs a particular user interaction (such as touching the displayed semantic entity on the touch-sensitive display screen 118), the user may select the semantic entity. In other embodiments, multiple semantic entities may be displayed, such as in a list, and the user may select a particular semantic entity from the list. In various embodiments, the user selection may be received, for example, by a voice command, a button press, a user interface interaction, or other interaction indicating a user selection.

[0080] In response to receiving the user selection of the selected semantic entity, the computing device 102 may determine one or more supplemental information options associated with the selected semantic entity. In addition, the computing device 102 may display the one or more supplemental information options associated with the selected semantic entity on the display screen 118 of the computing device 102. In some implementations, the one or more supplemental information options may include database entries (e.g., webref entries, Wikipedia entries, etc.), search engine results (e.g., Google search results, etc.), and / or application interaction options (e.g., restaurant review applications, reservation applications, event ticket purchase applications, calendar applications, etc.).

[0081] In some embodiments, one or more supplemental information options may be determined based at least in part on the context of the audio signal obtained by the computing device 102 and / or the context of the selected semantic entity. The context of the audio signal obtained by the computing device 102 may be, for example, the type of audio signal obtained by the computing device 102 and / or the manner in which the audio signal was obtained (e.g., a phone call, ambient audio, etc.). The context of the selected semantic entity may be, for example, a classification of the selected semantic entity, such as whether it is a person, a place (e.g., a restaurant, a venue, a tourist attraction, etc.), a thing (e.g., a product), a date and / or a time, or other classification. Figure 4-8 Example supplementary options determined based on the context of the audio signal or the context of the selected semantic entity are discussed in more detail.

[0082] Figure 2 A block diagram of an example machine learning model 200 according to example aspects of the present disclosure is depicted. In some implementations, the machine learning model 200 may be stored on a computing device such as a Figure 1 202). The machine learning model 200 may be a machine learning model configured to determine semantic entities 206 in an audio signal 204. In some implementations, the machine learning model 200 is trained to receive an audio signal 204, which may be an audio signal 204 that a user simultaneously hears. In some implementations, the machine learning model 200 may include a machine learning speech recognition semantic entity recognizer model 202 that may be operable to recognize semantic entities in speech and / or a song recognition semantic entity recognizer model 202 that may be operable to recognize semantic entities in songs. The speech / song recognition semantic entity recognizer model 202 may be used with, for example, Figure 1 , and corresponds to the speech recognition semantic entity recognizer model 140 and / or the song recognition semantic entity recognizer model 141 depicted in FIG. In some implementations, the speech / song recognition semantic entity recognizer model 202 may be a neural network (eg, a deep neural network).

[0083] For example, the audio signal 204 may be an audio signal associated with an application being executed by the computing device. For example, the application executed on the computing device may be an application configured to play various media files (such as compressed and / or uncompressed media files). The media file may include an associated audio signal 204, which may be audibly played by the computing device for the user to hear (e.g., as a sound wave emitted from a speaker). The computing device may input the simultaneously heard audio signal 204 into the machine learning model 200 and receive one or more semantic entities 206. In various embodiments, the audio signal 204 may first be converted, decoded, encoded, or otherwise processed to allow the audio signal 204 to be input into the machine learning model 200, such as data indicating the audio signal 204. Therefore, the audio signal 204 simultaneously heard by the user (e.g., as a sound wave) does not need to be input into the machine learning model 200 in the same format as the audio signal 204 heard by the user, but the audio signal 204 may be processed as needed to allow the audio signal 204 to be input into the machine learning model 200. Such processed audio signals 204 are intended to fall within the scope of the present disclosure.

[0084] In some implementations, the audio signal 204 may be an audio signal generated by a computing device based on ambient audio. For example, in some implementations, the ambient audio may be received by a microphone of the computing device, and the audio signal 204 may be generated by processing or converting the ambient audio into the audio signal 204. The audio signal 204 generated based on the ambient audio may be converted, encoded, decoded, or otherwise processed as needed.

[0085] In some implementations, the audio signal 204 may be an audio signal associated with a communication signal transmitted to or from a computing device. For example, in some implementations, a communication signal (e.g., a telephone call signal) may be received by a computing device, and the audio signal 204 associated with the communication signal may be input into the speech / song recognition semantic entity recognizer model 202 to identify semantic entities 206 in the audio signal 204 associated with the communication signal. The audio signal 204 associated with the communication signal may be converted, encoded, decoded, or otherwise processed as desired.

[0086] In some implementations, the audio signal 204 can be temporarily stored in an audio buffer, and one or more portions of the audio signal 204 can be input into the machine learning model 200. For example, in various implementations, the audio signal 204 can be analyzed as it is received, such as on a rolling or continuous basis. In some implementations, windows of the audio signal 204 can be analyzed. In some implementations, different portions (such as discrete portions) of the audio signal 204 can be analyzed, such as sequentially. After analyzing the audio signal 204, the audio signal 204 can be discarded.

[0087] One or more semantic entities 206 may be received as output of the machine learning model 200. For example, the semantic entities 206 identified by the machine learning model 200 may be semantically distinct entities corresponding to different semantic concepts, respectively. For example, each semantic entity may correspond to a particular person, place, thing, date / time, event, or other semantically distinct concepts.

[0088] For example, the semantic entities 206 may be received as data indicative of the semantic entities 206. Each semantic entity 206 may include or otherwise be represented by, for example, a text label, an icon, a picture, etc. that may allow the semantic entity 206 to be displayed.

[0089] Figure 3 A block diagram of an example machine learning model 300 according to example aspects of the present disclosure is depicted. In some implementations, the machine learning model 300 may be stored on a computing device such as Figure 1 102). The machine learning model 300 can be a machine learning model configured to determine semantic entities 306 in an audio signal 304. In some embodiments, the machine learning model 300 is trained to receive an audio signal 304, which can be an audio signal 304 that a user hears simultaneously. In some embodiments, the machine learning model 300 can include a machine learning language translation semantic entity recognizer model 302 that is operable to recognize semantic entities of foreign language speech. The language translation semantic entity recognizer model 302 can be used with, for example, Figure 1 In some implementations, the language translation semantic entity recognizer model 302 may be a neural network (eg, a deep neural network).

[0090] For example, the audio signal 304 may be an audio signal associated with an application being executed by the computing device. For example, the application executed on the computing device may be an application configured to play various media files (such as compressed and / or uncompressed media files). The media file may include an associated audio signal 304, which may be audibly played by the computing device for the user to hear (e.g., as a sound wave emitted from a speaker). The computing device may input the simultaneously heard audio signal 304 into the machine learning model 300 and receive one or more semantic entities 306. In various embodiments, the audio signal 304 may first be converted, decoded, encoded, or otherwise processed to allow the audio signal 304 to be input into the machine learning model 300, such as data indicating the audio signal 304. Therefore, the audio signal 304 simultaneously heard by the user (e.g., as a sound wave) does not need to be input into the machine learning model 300 in the same format as the audio signal 304 heard by the user, but the audio signal 304 may be processed as needed to allow the audio signal 304 to be input into the machine learning model 300. Such processed audio signals 304 are intended to fall within the scope of the present disclosure.

[0091] In some implementations, the audio signal 304 may be an audio signal generated by a computing device based on ambient audio. For example, in some implementations, the ambient audio may be received by a microphone of the computing device, and the audio signal 304 may be generated by processing or converting the ambient audio into the audio signal 304. The audio signal 304 generated based on the ambient audio may be converted, encoded, decoded, or otherwise processed as needed.

[0092] In some implementations, the audio signal 304 may be an audio signal associated with a communication signal transmitted to or from a computing device. For example, in some implementations, a communication signal (e.g., a telephone call signal) may be received by a computing device, and the audio signal 304 associated with the communication signal may be input into a language translation recognition semantic entity recognizer model 302 to recognize semantic entities 306 in the audio signal 304 associated with the communication signal. The audio signal 304 associated with the communication signal may be converted, encoded, decoded, or otherwise processed as desired.

[0093] In some implementations, the audio signal 304 can be temporarily stored in an audio buffer, and one or more portions of the audio signal 304 can be input into the machine learning model 300. For example, in various implementations, the audio signal 304 can be analyzed as it is received, such as on a rolling or continuous basis. In some implementations, a window of the audio signal 304 can be analyzed. In some implementations, different portions (such as discrete portions) of the audio signal 304 can be analyzed, such as sequentially. After analyzing the audio signal 304, the audio signal 304 can be discarded.

[0094] The language translation semantic entity recognizer model 302 may be trained to recognize semantic entities 306 from foreign language speech. For example, a user may have a primary language that the user speaks, but may talk to someone who speaks a foreign language (i.e., a language other than the user's primary language). For example, a user may have a face-to-face conversation with a person who speaks a foreign language while on vacation in a foreign country, or may make a phone call with a person who speaks a foreign language. The language translation semantic entity recognizer model 302 may be trained to translate a foreign language audio signal into a primary language, and recognize semantic entities 306 in the foreign language audio signal or the translated audio signal. For example, in some embodiments, the language translation semantic entity recognizer model 302 may be trained to first translate an audio signal 304 from a first language into a second language, and then recognize one or more semantic entities 306 in the second language. In some embodiments, the language translation semantic entity recognizer model 302 may be trained to recognize one or more semantic entities 306 in an audio signal 304 in a first language, but output the semantic entities 306 in a second language. The semantic entities 306 identified by the language translation semantic entity recognizer model 302 can be semantically different entities corresponding to different semantic concepts, respectively. For example, each semantic entity can correspond to a specific person, place, thing, date / time, event, or other semantically different concepts. In this way, the language translation semantic entity recognizer model 302 can recognize the semantic entities 306 in the audio signal 304 including foreign language speech.

[0095] For example, the semantic entities 306 may be received as data indicative of the semantic entities 306. Each semantic entity 306 may include or otherwise be represented by, for example, a text label, an icon, a picture, etc. that may allow the semantic entity 306 to be displayed.

[0096] Overall reference Figure 2 and Figure 3, the machine learning model 200 / 300 can operate in the background of a computing device such as computing device 102. For example, analysis of the audio signal 204 / 304 can be performed while the computing device 102 is simultaneously performing another task (e.g., playing media via an application) or while the computing device is in an idle state (e.g., when the user's smartphone is locked). In some embodiments, the machine learning model 200 / 300 can be implemented on a first thread of a processor having multiple threads, and concurrent tasks can be implemented on a second thread of the processor. In some embodiments, the machine learning model 200 / 300 can be implemented on a first processor of a computing device, and concurrent tasks can be implemented on a second processor of the computing device. In this way, the machine learning model 200 / 300 can operate in the background of a computing device.

[0097] Figure 4 Depicted are example user interactions and user interfaces for displaying one or more semantic entities and one or more supplemental information options on a computing device. The depicted user interactions and user interfaces are merely some example user interactions and user interfaces for displaying one or more semantic entities and one or more supplemental information options. Other suitable user interfaces may similarly be used. In some embodiments, Figure 4 The user interface depicted in may be displayed on a computing device 102, such as a smart phone, laptop, or other personal computing device.

[0098] In some embodiments, the computing device 402 may be configured to obtain an audio signal that the user hears simultaneously by obtaining an audio signal associated with an application being executed by the computing device 402, an audio signal associated with a communication signal transmitted to / from the computing device 402, or an audio signal generated based on ambient audio. For example, in some embodiments, the computing device 402 may be a smart phone or other computing device where media (such as a podcast) is being played on the computing device 402. The computing device 402 may be configured to obtain an audio signal associated with such a signal, such as a voice in a podcast. As described herein, the computing device 402 may also analyze the audio signal using a machine learning model (such as a speech recognition semantic entity recognizer model) to identify semantic entities in the audio signal.

[0099] Therefore, in order to obtain the benefits of the techniques described herein, the user may be asked to allow the collection and analysis of audio signals obtained by the user's computing device 402. For example, in some embodiments, the user may be provided with an opportunity to control whether a program or feature collects such audio signals. If the user does not allow the collection and use of such audio signals, the user may not receive the benefits of the techniques described herein. The user may also be provided with tools to revoke or modify consent. In addition, certain information or data may be processed in one or more ways before it is stored or used to protect user information. As an example, the computing device 402 may temporarily store such audio signals in an audio buffer for analysis and discard the audio signals after analysis.

[0100] For example, Figure 4 As depicted in , several semantic entities 404 discussed in the content of the podcast episode have been identified and displayed on the computing device 402. For example, the computing device 402 can be configured to display one or more semantic entities 404 in a user interface 406. The computing device 402 can be connected to Figure 1 The computing device 102 depicted in FIG. 1 corresponds to, and as Figure 4 As depicted in , user interface 406 may be user interface 406 displayed on display screen 408 of computing device 402. In some implementations, such as Figure 4 As shown in , user interface 406 can be displayed on a "lock screen" of computing device 402, such as a lock screen where computing device 402 is idle or an application is executed on computing device 402 but computing device 402 is password protected. In addition, as shown, multiple semantic entities 404 can be displayed on user interface 406. For example, as shown, four semantic entities 404A-D are displayed in user interface 406. In addition, as shown, in some embodiments, semantic entities 404A-D can be displayed as text. In other embodiments, semantic entities 404A-D can be displayed as icons, pictures, etc. In some embodiments, a first semantic entity 404A can be associated with a media file played in an application executed by computing device 402, such as the title of a podcast.

[0101] In some embodiments, semantic entities 404A-D can be displayed in the order in which they are identified. For example, semantic entities can be determined (i.e., identified) from an audio signal associated with an application being executed by computing device 402, such as a podcast that is simultaneously played for a user through a speaker of computing device 402. For example, a speech recognition semantic entity recognizer model can recognize semantic entities in speech included in an audio signal. The audio signal associated with a podcast can be analyzed while the podcast is audibly played for a user. It should be noted that when an audio signal is played for a user through an application executed by computing device 402, computing device 402 or its processor can obtain the audio signal at the same time. Then, as disclosed herein, the audio signal can be analyzed by a machine learning model in the background of computing device 402. In some embodiments, the analysis of the audio signal can require a period of time to perform analysis on the audio signal. For example, the audio signal can be analyzed in discrete parts, and the audio signal can only be analyzed until a specific part has been audibly played to the user (e.g., at the end of a five-second part). Similarly, the audio signal can be analyzed in a streaming manner, but the analysis can require processing time (e.g., 2-3 seconds). In some embodiments, after being identified, one or more semantic entities 404A-D can be displayed. Therefore, in some embodiments, although there may be a delay between audibly playing the audio signal for the user and displaying one or more semantic entities 404A-D on the display screen 408, the audio signal heard by the user can be obtained by the computing device 402 at the same time as it is heard by the user. In addition, once the semantic entity 404A-D has been determined, it can be displayed on the display screen 408 of the computing device 402, which can be substantially at the same time as the user hears the audio signal, or shortly after the audio signal is heard in the case of a delay.

[0102] In some embodiments, a semantic entity can be displayed for a specific length of time (e.g., five seconds). For example, each semantic entity can be displayed for a specific length of time, and when the specific length of time ends, it can be removed from the user interface 406. In some embodiments, multiple semantic entities 404A-D can be displayed in a specific area of ​​the user interface 406. For example, as shown in the figure, four recently determined semantic entities 404A-D can be displayed. When a new semantic entity is determined, the new semantic entity can be displayed by replacing the oldest semantic entity 404A-D. For example, semantic entity 404A can be removed from the user interface, semantic entities 404B-D can be moved upward in the user interface 406, and new semantic entity 404E (not shown) can be displayed in the user interface 406.

[0103] In some embodiments, the first semantic entity 404A may, for example, indicate that an audio signal is being played for a user. For example, the first semantic entity 404A may be the title of a podcast being played on the computing device 402. The first semantic entity 404A may be identified, for example, by the title of the podcast file, and / or may be a semantic entity 404 identified using a machine learning model, as disclosed herein. For example, a host may speak the name of a podcast in an audio signal, which may be identified using a speech recognition semantic entity recognizer model. Similarly, one or more semantic entities 404B-D may be, for example, one or more semantic entities discussed and identified in the actual content of the podcast (i.e., in an audio signal associated with the podcast).

[0104] The user interface 406 may include additional elements 412 displayed on the display screen 408. For example, as shown, the user interface 406 may include an area in which the time and date are displayed. In addition, the user interface 406 may include icons that may provide additional information. For example, as shown, a first icon may indicate that the user has unread emails, and a second icon may indicate that an application is being executed to play media, such as a podcast, on the device. Other icons that provide other information to the user may similarly be displayed in the user interface 406.

[0105] In some implementations, one or more semantic entities 404A-D may be displayed without a request. For example, one or more semantic entities 404A-D may be automatically displayed on display screen 408 without requiring user interaction with computing device 402. For example, in some implementations, a user may configure certain settings on computing device 402 such that when an application is executed on computing device 402, semantic entities 404A-D may be automatically displayed when they are determined from an audio signal that is simultaneously played for the user.

[0106] In some implementations, one or more semantic entities 404A-D may be displayed in response to receiving a request. For example, a user may request computing device 402 to display one or more semantic entities 404A-D, and one or more semantic entities 404A-D may be displayed in response to receiving the request.

[0107] For example, in some implementations, a user may perform user interactions with computing device 402. For example, Figure 4, a user may perform a "swipe" gesture 410 by touching a first portion of the touch-sensitive display screen 408 with one or more fingers and sliding the one or more fingers to a second portion of the touch-sensitive display screen 408. For example, a user may move his or her fingers in a downward motion across the touch-sensitive display screen 408. A user interaction, such as a "swipe" gesture 410, may be a request for the computing device 402 to display one or more semantic entities 404A-D on the display screen 408. Upon receiving the request, the computing device 402 may display the one or more semantic entities 404A-D, such as in an area of ​​the user interface 406. In various embodiments, other user interactions may similarly be used to provide a request, such as a button press, a verbal request, touching a particular area of ​​the user interface 406, navigating through one or more user interfaces, opening a particular application, and the like.

[0108] In some embodiments, the user may perform a task in response to an associated peripheral device (such as Figure 1 402) to receive the request through user interaction with the peripheral device 150 depicted in FIG. For example, in some embodiments, the computing device 402 can be communicatively coupled to the associated peripheral device 420. In some embodiments, the peripheral device 420 can be a speaker device that can be configured to audibly play an audio signal for the user. For example, an audio signal associated with an application executed on the computing device 402 can be transmitted from the computing device 402 to the peripheral device 420, and the speaker of the peripheral device 420 can audibly play the audio signal for the user. For example, in some embodiments, the peripheral device 420 can be an earbud device, such as an earbud device configured to wirelessly receive a signal from the computing device 402, such as via a Bluetooth connection. The earbud device can be a wearable device configured to be attached to or worn near the ear of the user. In various embodiments, the peripheral device 420 can be configured to communicate with the computing device 420 via any suitable wired or wireless connection.

[0109] For example, in some implementations, a user request may be received in response to a user performing a user interaction with an associated peripheral device 420. For example, the user interaction may be a "get" gesture 422, in which the user performs a pull-down motion on the peripheral device 420. Other user interactions with the associated peripheral device 420 may similarly be used to provide a request, such as a button press, etc. The peripheral device 420 may receive the request via the "get" gesture 422 (or other user interaction), and may transmit the request to the computing device 402, such as via a wired or wireless connection. The computing device 402 may then display one or more semantic entities 404A-D on the display screen 408 in response to receiving the request.

[0110] In some embodiments, the computing device 402 may determine a portion of the audio signal to be analyzed based at least in part on receiving a user request. For example, the computing device 402 may select a portion of the audio signal based on the time when the user request is received (such as when the user performs a "slide" or "get" gesture). For example, the computing device 402 may select a portion of the audio signal before the time when the user request is received. For example, the user may perform a user interaction (e.g., "slide", "get") to request the computing device 402 to identify one or more semantic entities 404A-D within a portion of the audio signal, and the computing device 402 may select a portion of the audio signal from a time period before receiving the user request. In other words, the computing device 402 may determine the selected portion of the audio signal for analysis based at least in part on a predetermined time period (e.g., 10 seconds) before receiving a request from the user to display one or more semantic entities 404A-D. Then, the computing device 402 may analyze the selected portion of the audio signal, identify one or more semantic entities 404A-D in the portion of the audio signal, and operate the display screen 408 to display the one or more semantic entities 404A-D.

[0111] According to example aspects of the present disclosure, a user can select one or more semantic entities 404A-D. For example, a user can select one of the semantic entities 404A-D displayed on a display screen 408 of a computing device 402. In some implementations, a user can select a semantic entity 404A-D directly from a user interface 406. In some implementations, a user can open an auxiliary user interface 430 to make a selection.

[0112] For example, Figure 4 , a user may touch a particular area of ​​user interface 406 (e.g., an area of ​​user interface 406 in which semantic entities 404A-D are displayed) to open auxiliary user interface 430. In some implementations, auxiliary user interface 430 may be a user interface of an application (such as a podcast application) being executed by computing device 402, and the audio signal obtained by computing device 402 may be an audio signal associated with the application.

[0113] For example, Figure 4 As shown in FIG. 4 , the auxiliary user interface 430 may include the semantic entities 404A-D, such as in an expanded format, wherein additional information about each semantic entity 404A-D is displayed. Figure 4As shown in FIG, the first semantic entity 404A “NPR Fresh Air: In Life and Fiction” may be displayed in the upper portion of the auxiliary user interface 430, along with an indication that the podcast is now playing in the podcast application, and the first semantic entity 404A is determined to be “just now.” In addition, the second to fourth semantic entities 404B-D may be displayed in the middle portion of the auxiliary user interface 430, and additional information may be similarly displayed for each semantic entity. For example, the second semantic entity 404B of “Tom Wolfe” may include the following information: Tom Wolfe is an American writer (e.g., a person), and the second semantic entity 404B is also determined to be “just now”, the third semantic entity 404C of “Electric Kool-Aid Acid Test” may include the following information: Electric Kool-Aid Acid Test is a book by Tom Wolfe (e.g., a specific book title by a specific author), and the third semantic entity 404C is determined to be “just now”, and the fourth semantic entity 404D of “Bonfire of the Vanities” may include the following information: Bonfire of the Vanities is a novel by Tom Wolfe (e.g., another specific book title by a specific author), and the fourth semantic entity 404D is determined to be “just now”.

[0114] The user may then select a particular semantic entity 404A-D, such as by touching the touch-sensitive display screen 408 in the area where the semantic entities 404A-D are displayed. Figure 4 , the user has selected the third semantic entity 404C of "Electric Kool-Aid Acid Test" as the selected semantic entity 442. Thus, in this manner, the selected semantic entity 442 may be one of the semantic entities 404A-D displayed on the display screen 408 of the computing device 402.

[0115] According to additional aspects of the present disclosure, computing device 402 may then determine one or more supplemental information options associated with the selected semantic entity 442. For example, Figure 4 As shown at the bottom 440 of the auxiliary user interface 430 in FIG. 4 , the selected semantic entity 442 can be displayed together with one or more supplemental information options 450A-E. In some embodiments, the supplemental information options 450A-E can be determined based at least in part on the context of the audio signal obtained by the computing device 402 or the context of the selected semantic entity 442.

[0116] In some embodiments, one or more supplemental information options 450 can be a database entry associated with the selected semantic entity 442 (e.g., a Wikipedia article, a webref entry, etc.), a search result associated with the selected semantic entity 442 (e.g., a Google search result), or one or more application interaction options associated with the selected semantic entity 442 (e.g., an option to execute an associated application, such as purchasing content or playing media).

[0117] In some implementations, one or more supplemental information options 450 may be determined based at least in part on the context of the audio signal obtained by the computing device 402 or the context of the selected semantic entity 442. For example, for the selected semantic entity 442 being a tangible thing that can be purchased, the one or more supplemental information options may include an option to purchase the selected semantic entity 442. Figure 4 , the first supplemental information option 450A is an option to perform a Google shopping search (e.g., access a subset of search results), the second supplemental information option 450B is an option to add the selected semantic entity to a shopping cart for later purchase, and the third supplemental information option 450C is an option to purchase the selected semantic entity 442 on the supplier's website or through the supplier's application (e.g., Amazon). The fourth supplemental information option 450D can be an option to access Internet-based search results. The fifth supplemental information option 450E can be an option to share the selected semantic entity 442 with others, such as via text messaging, email, or other applications.

[0118] In other embodiments, other supplemental information options may similarly be provided based on the context of the audio signal or the context of the selected semantic entity 442. For example, other supplemental information options for a book may include a link to a local library's website to search for the selected semantic entity, a link to a map of the local library, reviews of a particular book, etc. Similarly, if the user were to select the second semantic entity 404B for the author Tom Wolfe, the supplemental information options may include a link to the author's website, an entry for the author in a database (e.g., a Wikipedia article), a list of upcoming speaking events for the author, etc.

[0119] Figure 5 Depicted are example user interfaces for displaying one or more semantic entities and one or more supplemental information options on a computing device. The depicted user interfaces are merely some example user interfaces for displaying one or more semantic entities and one or more supplemental information options. Other suitable user interfaces may be similarly used. In some embodiments, Figure 5The user interface depicted in may be displayed on a computing device 102, such as a smart phone, laptop, or other personal computing device.

[0120] In some implementations, computing device 502 may be configured to obtain audio signals simultaneously heard by a user, audio signals associated with an application being executed by computing device 502, and / or audio signals associated with communication signals transmitted to / from computing device 502 by generating an audio signal based on ambient audio (such as by a microphone of computing device 502). Computing device 502 may be associated with Figure 1 502. For example, in some implementations, a microphone or other sound sensor of computing device 502 can be configured to receive ambient audio (such as sound waves from a conversation) and generate an audio signal based on the ambient audio. For example, a user who is conversing with someone in a foreign language can use their computing device 502 to translate speech from one language to another.

[0121] Thus, in some embodiments, in order to obtain the benefits of the techniques described herein, a user may be asked to allow the collection and analysis of audio signals obtained by his or her computing device 502. For example, in some embodiments, a user may be provided with an opportunity to control whether a program or feature collects such audio signals. If a user does not allow the collection and use of such audio signals, the user may not receive the benefits of the techniques described herein. Tools may also be provided for the user to revoke or modify consent. In addition, certain information or data may be processed in one or more ways before it is stored or used to protect user information. As an example, the computing device 502 may temporarily store such audio signals in an audio buffer for analysis and discard the audio signals after analysis.

[0122] In some implementations, computing device 502 can be configured to display user interface 506 on display screen 508. In some implementations, such as Figure 5 As shown in , user interface 506 can be displayed on a “lock screen” of computing device 502 , such as a lock screen where computing device 502 is idle or an application is executing on computing device 502 but computing device 502 is password protected.

[0123] In some implementations, upon receiving an audio signal (such as an audio signal generated by a microphone based on ambient audio), computing device 502 may determine that the audio signal includes foreign speech. Foreign speech in the audio signal may be translated. For example, computing device 502 may recognize that the audio signal includes speech in a language other than the user's primary language. In some implementations, a translation option 510 may be provided to the user to translate the foreign speech into the primary language. For example, Figure 5 As depicted in , a user can request translation of a foreign language by tapping the translation option 510, and identifying any semantic entities therein, to translate speech from a first language into speech in a second language, such as speech from a foreign language to the user's primary language. In other embodiments, foreign language speech can be automatically translated without request in an automatic translation mode.

[0124] like Figure 5 As depicted in , after tapping the translation option 510 or in the automatic translation mode, the foreign language speech can be translated from the first language to the second language. For example, the auxiliary user interface 530 can be displayed on the display screen 508 of the computing device 502. As shown in the upper portion of the user interface 530, the speech received in the foreign language 540 (i.e., the first language) can be transcribed in the language in which it is received. In addition, in the middle portion of the user interface 530, the translated speech 550 in the user's primary language (i.e., the second language) can be displayed.

[0125] In some implementations, one or more semantic entities 552 identified in the translated speech 550 may be indicated in the user interface 530. For example, Figure 5 As shown in , the semantic entity 552 identified in the translated speech 550 can be separated from the rest of the text by a box around the semantic entity 552. In other embodiments, the semantic entity 552 can be displayed in other suitable ways, such as different font sizes, font colors, underlining, italics, etc.

[0126] The user can select one or more semantic entities 552 displayed in the translated speech 550. For example, in the bottom 560 of the user interface 530, the selected semantic entity 562 can be displayed on the display screen 508 of the computing device 502.

[0127] According to additional aspects of the present disclosure, computing device 502 may then determine one or more supplemental information options associated with the selected semantic entity 562. For example, Figure 5 As shown in the bottom 560 of the auxiliary user interface 530 in FIG, the selected semantic entity 562 can be displayed together with one or more supplemental information options 570A-E. In some embodiments, the supplemental information options 570A-E can be determined based at least in part on the context of the audio signal obtained by the computing device 502 or the context of the selected semantic entity 562.

[0128] In some embodiments, one or more supplemental information options 570 can be a database entry associated with the selected semantic entity 562 (e.g., a Wikipedia article, a webref entry, etc.), a search result associated with the selected semantic entity 562 (e.g., a Google search result), or one or more application interaction options associated with the selected semantic entity 562 (e.g., an option to execute an associated application, such as opening a mapping application to display the location).

[0129] In some implementations, one or more supplemental information options 570 may be determined based at least in part on the context of the audio signal obtained by the computing device 502 or the context of the selected semantic entity 562. For example, Figure 5 , the selected semantic entity 562 is the Amazon Arena located in Manaus, Brazil. In addition, as shown, the first supplementary information option 570A is an option to perform a Google shopping search (e.g., access a subset of search results), which may be specific to events occurring at the Amazon Arena (the selected semantic entity 562), the second supplementary information option 570B is an option to access entries about the selected semantic entity 562 in a travel application (e.g., Lonely Planet), the third supplementary information option 570C is an option to save or store the selected semantic entity 562 (such as save to a list), the fourth supplementary information option 570D is an option to access Internet-based search results (e.g., Google results), and the fifth supplementary information option 570E may be an option to share the selected semantic entity 562 with others, such as via text messaging, email, or other applications.

[0130] Figure 6 Depicted are example user interfaces for displaying one or more semantic entities and one or more supplemental information options on a computing device. The depicted user interactions and user interfaces are merely some example user interfaces for displaying one or more semantic entities and one or more supplemental information options. Other suitable user interfaces may be similarly used. In some embodiments, Figure 6 The user interface depicted in FIG. 1 may be displayed on a computing device 102 , such as a smart phone, laptop, or other personal computing device.

[0131] In some implementations, computing device 602 may be configured to obtain the audio signals simultaneously heard by the user by obtaining an audio signal associated with an application being executed by computing device 602, an audio signal generated based on ambient audio, or an audio signal associated with a communication signal transmitted to or from computing device 602. For example, in some implementations, computing device 602 may be a smartphone or other computing device configured to allow the user to communicate with other people. For example, the communication signal may be a phone call, a video call, or other communication signal.

[0132] Therefore, in order to obtain the benefits of the techniques described herein, the user may be asked to allow the collection and analysis of audio signals obtained by the user's computing device 602. For example, in some embodiments, the user may be provided with an opportunity to control whether a program or feature collects such audio signals. If the user does not allow the collection and use of such audio signals, the user may not receive the benefits of the techniques described herein. The user may also be provided with tools to revoke or modify consent. In addition, certain information or data may be processed in one or more ways before it is stored or used to protect user information. As an example, the computing device 602 may temporarily store such audio signals in an audio buffer for analysis and discard the audio signals after analysis.

[0133] In some implementations, computing device 602 can be configured to display a user interface 606 on display screen 608, such as a user interface 606 for a phone call application. Figure 1 6. For example, the user interface 606 may include an icon 610 and a contact name 612 associated with a person with whom the user is communicating. Additional user interface 606 options may include, for example, a mute option 614 for muting a speaker, a keypad option 616 for accessing a keypad such as a virtual keypad, a speaker option 618 for placing a phone call on a speakerphone, an add call option 620 for initiating a conference call with other people, a hold option 622 for placing a call on hold, a video call option 624 for converting a phone call to a video call, and / or an end call option 626 for terminating the phone call. Other suitable user interfaces 606 may similarly be used, which may include other phone call options.

[0134] The computing device 602 can analyze an audio signal associated with a communication signal (e.g., a phone call) and can identify semantic entities in the audio signal. For example, a semantic entity can be identified in both the voice from the user and the voice from other people in a phone call. After the semantic entity is identified, the semantic entity can be displayed in the user interface 606. For example, during a phone conversation, one of the two parties in the conversation can ask the other party whether he / she wants to have dinner at a specific restaurant and / or on a specific date / time. As disclosed herein, the audio signal associated with the communication signal can be analyzed by one or more machine learning models, and semantic entities corresponding to the restaurant and / or date and time can be determined (i.e., identified). For example, as shown in the figure, a semantic entity 630 of "Dinner at Tallulah's" is displayed. In some embodiments, the semantic entity 630 can be displayed in a specific area of ​​the user interface 606, such as in a location such as Figure 6 . In some embodiments, a "glimmer" may be displayed indicating that a semantic recognition has been identified. The glimmer may be, for example, an animation that is also displayed in the user interface 606 to indicate that one or more semantic entities have been identified. In some embodiments, the semantic entity 630 may be displayed upon a user request, such as when the user taps the glimmer. The semantic entity 630 may then be displayed in an area in the user interface 606 instead of the glimmer. In some embodiments, the glimmer may be displayed first, and the semantic entity 630 may be displayed after the glimmer without a user request.

[0135] In some embodiments, a user can select the semantic entity 630 by, for example, tapping on the semantic entity 630. For example, a second user interface 640 can be displayed in a phone application executing on the computing device 602. The second user interface 640 can include, for example, various phone call options 614-624 as displayed in the user interface 606. The second user interface 640 can also include, for example, a selected semantic entity 642. The selected semantic entity 642 can include additional information, such as the name of the restaurant, the date of the proposed dinner, the city where the restaurant is located, the type of cuisine offered by the restaurant, an indication of the relative cost of having dinner at the restaurant, and / or other information. In addition, one or more supplemental information options 650 can also be displayed in the user interface 640. For example, Figure 6 As shown in , five supplemental information options 650A-E are displayed.

[0136] In some embodiments, one or more supplemental information options 650 can be a database entry associated with the selected semantic entity 642 (e.g., a Wikipedia article, a webref entry, etc.), a search result associated with the selected semantic entity 642 (e.g., a Google search result), or one or more application interaction options associated with the selected semantic entity 642 (e.g., an option to execute an associated application, such as creating a calendar entry or opening a mapping application to display the location).

[0137] In some implementations, one or more supplemental information options 650 may be determined based at least in part on the context of the audio signal obtained by the computing device 602 or the context of the selected semantic entity 642. Figure 6 , the selected semantic entity 642 is dinner Friday at Tallulah's Café (dinner Friday at Tallulah's Café). In addition, as shown in the figure, the first supplementary information option 650A is an option to create a calendar event (e.g., add an event to the user's calendar application for the date, time and / or location of the event), the second supplementary information option 650B is an option to view the location of the restaurant in a map application (e.g., Google Maps), the third supplementary information option 650C is an option to view reviews of the restaurant, such as via a restaurant review application or website, the fourth supplementary information option 650D is an option to access Internet-based search results (e.g., Google results), and the fifth supplementary information option 650E can be an option to share the selected semantic entity 642 with others, such as via text messages, emails, or other applications.

[0138] In other embodiments, other supplemental information options may be similarly provided based on the context of the audio signal or the context of the selected semantic entity 642. For example, other supplemental information options for restaurants may include a link to the restaurant's website, a link to a reservation application, a link to the restaurant's menu, etc. Similarly, supplemental information options for movies may include a link to a trailer for the movie, a link to a movie ticket purchase application, a link to a movie review website, a link to a list of showtimes for nearby theaters, an option to open a mapping application to obtain directions to a specific theater, etc.

[0139] Figure 7 Depicted are example user interfaces for displaying one or more semantic entities and one or more supplemental information options on a computing device. The depicted user interfaces are merely some example user interfaces for displaying one or more semantic entities and one or more supplemental information options. Other suitable user interfaces may be similarly used. In some embodiments, Figure 7The user interface depicted in FIG. 1 may be displayed on a computing device 102 , such as a smart phone, laptop, or other personal computing device.

[0140] In some embodiments, the computing device 702 may be configured to obtain an audio signal that the user hears at the same time by obtaining an audio signal associated with an application being executed by the computing device 702, an audio signal associated with a communication signal transmitted to or from the computing device 702, or an audio signal generated based on ambient audio. For example, in some embodiments, the computing device 702 may be a smart phone or other computing device, where media (such as a movie) is playing on the computing device 702, or an audio signal in the environment where the computing device 702 is located (such as a song played in an elevator where the user is riding), or an audio signal transmitted to / from the computing device (such as a song played via a video call). The computing device 702 may be configured to obtain such an audio signal. The computing device 702 may also analyze such an audio signal using a machine learning model (such as a song recognition semantic entity recognizer model) to identify semantic entities in the audio signal, as described herein.

[0141] Therefore, in order to obtain the benefits of the techniques described herein, the user may be asked to allow the collection and analysis of audio signals obtained by the user's computing device 702. For example, in some embodiments, the user may be provided with an opportunity to control whether a program or feature collects such audio signals. If the user does not allow the collection and use of such audio signals, the user may not receive the benefits of the techniques described herein. The user may also be provided with tools to revoke or modify consent. In addition, certain information or data may be processed in one or more ways before it is stored or used to protect user information. As an example, the computing device 702 may temporarily store such audio signals in an audio buffer for analysis and discard the audio signals after analysis.

[0142] In some implementations, computing device 702 may be configured to display one or more semantic entities 704 in user interface 706. Figure 1 The computing device 102 depicted in FIG. Figure 7 As depicted in , user interface 706 may be user interface 706 displayed on display screen 708 of computing device 702. In some implementations, such as Figure 7As shown in , user interface 706 can be displayed on a "lock screen" of computing device 702, such as a lock screen where computing device 702 is idle or an application is executed on computing device 702 but computing device 702 is password protected. In addition, as shown, in some embodiments, semantic entity 704 can be displayed as text (e.g., song title of an artist) and / or icon / picture (e.g., thumbnail of an artist).

[0143] For example, the identified semantic entity can be a song identified by a song identification semantic entity recognizer model, such as a song played in an application executed by computing device 702, a song transmitted to / from computing device 702, or a song overheard by computing device 702 based on ambient audio. After semantic entity 704 is identified, semantic entity 704 can be displayed in user interface 706. For example, Figure 7 As shown in , the semantic entity 704 has been identified and the semantic entity 704 has been displayed on the display screen 708. In some embodiments, the semantic entity 704 can be displayed in a specific area of ​​the user interface 706, such as in an area at the bottom of the user interface 706, such as Figure 7 . In some embodiments, a "glimmer" may be displayed indicating that a semantic recognition has been identified. The glimmer may be, for example, an animation that is also displayed in the user interface 706 to indicate that one or more semantic entities have been identified. In some embodiments, the semantic entity 704 may be displayed upon a user request, such as when the user taps the glimmer. The semantic entity 704 may then be displayed in a region in the user interface 706 instead of the glimmer. In some embodiments, the glimmer may be displayed first, and the semantic entity 704 may be displayed after the glimmer without a user request.

[0144] In some implementations, the user can select the semantic entity 704 by, for example, tapping the semantic entity 704. For example, after the user selects the semantic entity 704, the second user interface 740 can be displayed on the computing device 702. The second user interface 740 can also include, for example, additional information about the selected entity 704, such as in Figure 7 In the upper portion of the user interface 740 depicted in . For example, a picture of an artist may be displayed, and the song title, artist name, and duration since the selected semantic entity 704 was identified may all be displayed in the user interface 740 .

[0145] The user interface 740 may also include additional semantic entities previously recognized by the computing device 702. For example, in the middle portion of the user interface 740, previously recognized semantic entities may be displayed, such as the previously recognized song, artist, and duration since the semantic entity was recognized.

[0146] The selected semantic entity 742 may be displayed, for example, in the bottom portion of the user interface 740. The user interface 740 may also include one or more supplemental information options 750. For example, Figure 7 As shown in , five supplemental information options 750A-E are displayed.

[0147] In some embodiments, one or more supplemental information options 750 can be a database entry associated with the selected semantic entity 742 (e.g., a Wikipedia article, a webref entry, etc.), a search result associated with the selected semantic entity 742 (e.g., a Google search result), or one or more application interaction options associated with the selected semantic entity 742 (e.g., an option to execute an associated application, such as opening an application to play the selected semantic entity).

[0148] In some implementations, one or more supplemental information options 750 may be determined based at least in part on the context of the audio signal obtained by the computing device 702 or the context of the selected semantic entity 742. Figure 7 , the selected semantic entity 742 is a specific song by a specific artist. In addition, as shown, the first supplementary information option 750A is an option to play the song in a music application, the second supplementary information option 750B is an option to add the song to a playlist, the third supplementary information option 750C is an option to watch the music video of the song (such as in a video application), the fourth supplementary information option 750D is an option to access Internet-based search results (e.g., Google results), and the fifth supplementary information option 750E can be an option to share the selected semantic entity 742 with others, such as via text messages, emails, or other applications.

[0149] In other embodiments, other supplemental information options may similarly be provided based on the context of the audio signal or the context of the selected semantic entity 742. For example, other supplemental information options for songs and / or artists may be to view upcoming tour dates and locations for the artist, access album reviews, access the artist's website, access various music purchase options (such as options to purchase individual songs or entire albums from various suppliers and in various formats), etc.

[0150] Figure 8 An example user interface 800 for displaying one or more semantic entities on a computing device is depicted. Figure 8 The user interface 800 depicted in FIG. 1 is merely an example user interface for displaying one or more semantic entities. Other suitable user interfaces may be similarly used. In some embodiments, Figure 8The user interface depicted in FIG. 8 can be displayed on a computing device 802 (such as a smart phone, laptop computer, or other personal computing device) and can be used with Figure 1 The computing device 102 depicted corresponds to .

[0151] User interface 800 depicts an example implementation of a “clipboard”, which can be a user interface displayed on a computing device 802, in which semantic entities 804 that have been previously identified (such as during the course of a day) are displayed for the user's reference. For example, in some implementations, semantic entities identified in an audio signal can be recorded (e.g., data indicating the semantic entity can be stored locally on the computing device) to be accessed at a later time. For example, during the course of a day, the computing device 802 can identify multiple semantic entities 804, such as semantic entities 804 identified in a phone call or other communication, semantic entities 804 associated with a song identified by the computing device 802, semantic entities 804 identified in an application executed by the device (such as a podcast, music, or video played on the computing device 802), or other semantic entities identified by the computing device 802. Semantic entities (or data indicating them) can be stored locally for access at a later time. For example, a user can request to display multiple semantic entities 804 by, for example, accessing, executing, or otherwise implementing the “clipboard” user interface 800. The user interface 800 may organize the plurality of semantic entities 804 and display the plurality of semantic entities 804 for review by a user.

[0152] For example, the first zone 810 of the user interface 800 may display semantic entities 804A-C that have been identified in an application currently executing on the computing device 802 (such as a podcast playing on the computing device 802). The second zone 820 may display a semantic entity 804D (or multiple semantic entities), which may be one or more songs recently identified by the computing device 802. The third zone 830 may display a semantic entity 804E (or multiple semantic entities), which was identified in a first communication with a first person, and the fourth zone 840 may display semantic entities 804F-G (or a single semantic entity), which were identified in a second communication with a second person. The user interface 800 may also include a search zone 850, where a user can search for specific semantic entities. In some embodiments, a user can access additional information about the semantic entity 804 displayed in a particular zone 810 / 820 / 830 / 840. For example, a user can tap the second zone 820 to access a list of identified songs, where the second user interface (such as Figure 7 ) is displayed on computing device 802. In addition, other areas may be included in user interface 800.

[0153] Overall reference Figure 4-8 , illustrates the concept of determining one or more supplemental information options based at least in part on the context of an audio signal or a selected semantic entity obtained by a computing device. For example, the supplemental information options for a selected semantic entity may be determined based on the context of the audio signal (such as how the audio signal is received) or the context of the semantic entity (such as the classification of the selected semantic entity). For example, Figure 5 The supplemental information shown in Option 570A-E is different from Figure 4 Supplementary information options 450A-E are shown in the Figure 4 In , the selected semantic entity 442 is a tangible product available for purchase, so some of the supplemental information options 450 are directed to options for purchasing the related tangible product. Figure 5 , the selected semantic entity 562 is a location / venue, so several information options 570 are directed to options for purchasing tickets to the venue or learning more about the venue. Similarly, the context of the manner in which the audio signal is obtained can also be used to determine supplemental information options. For example, a semantic entity for a restaurant identified in a podcast can include different supplemental options (e.g., a link to the restaurant's website or a Wikipedia article for the chef) than the same semantic entity for the same restaurant identified in a phone call (e.g., a calendar event for a specific reservation at the restaurant).

[0154] Fig. 9 A flow chart of an example method (900) for identifying semantic entities within an audio signal according to an example aspect of the present disclosure is depicted. The method (900) may be performed by, for example, Figure 1 and Figure 4-8 In addition, for the purpose of illustration and discussion, Fig. 9 Steps are depicted as being performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the steps of any method discussed herein may be adapted, rearranged, expanded, omitted, or modified in various ways without departing from the scope of the present disclosure.

[0155] At (902), the method (900) may include obtaining an audio signal. The audio signal may be an audio signal that a user simultaneously hears. In some implementations, the audio signal may be an audio signal associated with an application being executed by the computing device, an audio signal generated by the computing device based on ambient audio, or an audio signal associated with a communication signal transmitted to or from the computing device.

[0156] At (904), method (900) may include receiving a request from a user to display one or more semantic entities. For example, in some embodiments, the request may be received in response to a user performing a user interaction with an associated peripheral device. In some embodiments, the associated peripheral device may be an earbud device communicatively coupled to the computing device. In some embodiments, the user interaction may be a "get" gesture performed on the earbud device. In some embodiments, the request may be a user interaction with the computing device. For example, in various embodiments, a user may perform a "swipe" gesture on a lock screen of a computing device, or access a "clipboard" user interface on a computing device. In some embodiments, the user request may be received in any suitable manner, such as by a voice command, a button press, etc.

[0157] At (906), method (900) may include analyzing at least a portion of the audio signal with a machine learning model to determine one or more semantic entities. The analysis of the audio signal may be performed in the background of the computing device. For example, the analysis may be performed simultaneously with another application being executed by the computing device, or when the computing device is in an idle state. In some embodiments, the machine learning model may be a speech recognition semantic entity recognizer model, a song recognition semantic entity recognizer model, or a language translation semantic entity recognizer model. In some embodiments, the machine learning model may be used to analyze discrete portions of the audio signal. In some embodiments, the machine learning model may analyze a rolling window of the audio signal. The computing device may determine (i.e., identify) one or more semantic entities by inputting the audio signal (or a portion thereof) into the machine learning model and receiving the one or more semantic entities as an output of the machine learning model.

[0158] In some embodiments, a request to display one or more semantic entities can be used to determine a selected portion of an audio signal for analysis. For example, in some embodiments, a computing device can select a portion of an audio signal based on the time at which the request is received. In addition, the portion of the audio signal can be a portion of the audio signal before the time at which the request is received. In other words, the computing device can determine the selected portion of the audio signal for analysis based at least in part on a predetermined time period before a request to display one or more semantic entities is received from a user. The machine learning model can then analyze the selected portion of the audio signal to determine one or more semantic entities in the selected portion.

[0159] At (908), the method (900) may include displaying one or more semantic entities on a display screen of the computing device. For example, various user interfaces (such as Figure 4-8The user interface depicted in FIG) may display one or more semantic entities, such as in a list or region of the user interface. In some implementations, one or more semantic entities may be displayed in response to receiving a request.

[0160] At (910), the method (900) may include receiving a user selection of a selected semantic entity. For example, a user may touch a touch-sensitive display screen of a computing device to select a particular semantic entity from a list of semantic entities. In some embodiments, the user selection of the selected semantic entity may be made when the user selects a single semantic entity displayed in a particular area of ​​a user interface. In some embodiments, the user selection may be received in any suitable manner, such as by a voice command, a button press, etc.

[0161] At (912), method (900) may include determining one or more supplemental information options associated with the selected semantic entity. For example, in some embodiments, the one or more supplemental information options may include a database entry associated with the selected semantic entity (e.g., a webref or Wikipedia entry), a search result associated with the selected semantic entity (e.g., a Google search result), or one or more application interaction options associated with the selected semantic entity (e.g., accessing a movie ticket purchasing application). In some embodiments, the one or more supplemental information options may be determined at least in part based on the context of an audio signal obtained by a computing device or the context of the selected semantic entity. For example, a classification or type of the selected semantic entity may be used to determine supplemental information options for the selected semantic entity. Similarly, the type of audio signal and / or the manner in which the audio signal is received may be used to determine one or more supplemental information options.

[0162] At (914), the method (900) may include displaying one or more supplemental information options associated with the selected semantic entity on a display screen. For example, in some embodiments, various user interfaces (such as Figure 4-8 The user interface depicted in ) can be used to display one or more supplemental information options.

[0163] Fig.10 A flow chart of an example method (1000) for identifying semantic entities within an audio signal according to an example aspect of the present disclosure is depicted. The method (1000) may be performed by, for example, Figure 1 and Figure 4-8 In addition, for the purpose of illustration and discussion, Fig.10Steps are depicted as being performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the steps of any method discussed herein may be adapted, rearranged, expanded, omitted, or modified in various ways without departing from the scope of the present disclosure.

[0164] At (1002), the method (1000) may include obtaining an audio signal. The audio signal may be an audio signal that a user simultaneously hears. In some implementations, the audio signal may be an audio signal associated with an application being executed by the computing device, an audio signal generated by the computing device based on ambient audio, or an audio signal associated with a communication signal transmitted to or from the computing device.

[0165] At (1004), method (1000) may include receiving a request from a user to identify one or more semantic entities within a portion of an audio signal. For example, in some embodiments, the request may be received in response to a user performing a user interaction with an associated peripheral device. In some embodiments, the associated peripheral device may be an earbud device communicatively coupled to a computing device. In some embodiments, the user interaction may be a "get" gesture performed on the earbud device. In some embodiments, the request may be a user interaction with a computing device. For example, in various embodiments, a user may perform a "swipe" gesture on a lock screen of a computing device, or access a "clipboard" user interface on a computing device. In some embodiments, the user request may be received in any suitable manner, such as by a voice command, a button press, etc.

[0166] In some embodiments, the request to identify one or more semantic entities can be used to determine the selected portion of the audio signal for analysis. For example, in some embodiments, the computing device can select a portion of the audio signal based on the time when the request is received. In addition, the portion of the audio signal can be a portion of the audio signal before the time when the request is received. In other words, the computing device can determine the selected portion of the audio signal for analysis based at least in part on a predetermined time period before the request is received from the user to display the one or more semantic entities.

[0167] At (1006), method (1000) may include analyzing a portion of the audio signal with a machine learning model to determine one or more semantic entities. The analysis of the audio signal may be performed in the background of the computing device. For example, the analysis may be performed simultaneously with another application being executed by the computing device, or when the computing device is in an idle state. In some embodiments, the machine learning model may be a speech recognition semantic entity recognizer model, a song recognition semantic entity recognizer model, or a language translation semantic entity recognizer model. In some embodiments, the machine learning model may be used to analyze discrete portions of the audio signal. In some embodiments, the machine learning model may analyze a rolling window of the audio signal. The computing device may determine (i.e., identify) the one or more semantic entities by inputting the portion of the audio signal into the machine learning model and receiving the one or more semantic entities as an output of the machine learning model.

[0168] At (1008), the method (1000) may include operating a display screen to display one or more semantic entities. For example, various user interfaces (such as Figure 4-8 The user interface depicted in ) can display one or more semantic entities, such as in a list or area of ​​the user interface.

[0169] Fig.11 A flow chart of an example method (1100) for identifying semantic entities within an audio signal according to an example aspect of the present disclosure is depicted. The method (1100) may be performed by, for example, Figure 1 and Figure 4-8 The computing device depicted in the computing device and the computing device such as Figure 1 and 4 The associated peripheral device may be, for example, a speaker device, and more specifically, in some embodiments, an earbud device. In addition, for purposes of illustration and discussion, Fig.11 Steps are depicted as being performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the steps of any method discussed herein may be adapted, rearranged, expanded, omitted, or modified in various ways without departing from the scope of the present disclosure.

[0170] At (1102), the method (1100) may include transmitting an audio signal from the computing device to the speaker device. For example, the audio signal may be associated with media being played on the computing device, such as via an application being executed on the computing device. The computing device may be communicatively coupled to the speaker device, such as via a Bluetooth connection, and an audio signal associated with the media being played on the computing device (or other audio signal as disclosed herein) may be transmitted from the computing device to the speaker device. At (1104), the speaker device may receive the audio signal from the computing device.

[0171] At (1106), the speaker device may play the audio signal. For example, the speaker device may audibly play the audio signal for the user.

[0172] At (1108), the speaker device may receive a user interaction indicative of a user request. For example, the user interaction may be a "get" gesture performed on the earbud device. In some embodiments, the user interaction indicative of a user request may be received in any suitable manner, such as by a voice command, a button press, etc.

[0173] At (1110), the speaker device may transmit the user request to the computing device. For example, the speaker device may transmit the user request via Bluetooth or other communication networks. At (1112), the computing device may receive the user request from the speaker device.

[0174] At (1114), the computing device may analyze at least a portion of the audio signal with a machine learning model to determine one or more semantic entities. The analysis of the audio signal may be performed in the background of the computing device. For example, the analysis may be performed simultaneously with another application being executed by the computing device, or when the computing device is in an idle state. In some embodiments, the machine learning model may be a speech recognition semantic entity recognizer model, a song recognition semantic entity recognizer model, or a language translation semantic entity recognizer model. In some embodiments, the machine learning model may be used to analyze discrete portions of the audio signal. In some embodiments, the machine learning model may analyze a rolling window of the audio signal. The computing device may determine (i.e., identify) one or more semantic entities by inputting the audio signal (or a portion thereof) into the machine learning model and receiving one or more semantic entities as output of the machine learning model.

[0175] In some embodiments, a request to display one or more semantic entities can be used to determine a selected portion of an audio signal for analysis. For example, in some embodiments, a computing device can select a portion of an audio signal based on the time when the request is received. In addition, the portion of the audio signal can be a portion of the audio signal before the time when the request is received. In other words, the computing device can determine the selected portion of the audio signal for analysis based at least in part on a predetermined time period before a request to display one or more semantic entities is received from a user. The machine learning model can then analyze the selected portion of the audio signal to determine one or more semantic entities in the selected portion. In some embodiments, the portion of the audio signal selected for analysis may include a portion of an audio signal played by a speaker device in a time period before receiving a user interaction.

[0176] In some implementations, the computing device may include a microphone configured to generate an audio signal based on ambient audio. The audio signal may be generated by the microphone, and a portion of the audio signal generated based on the ambient audio during a time period prior to receiving the user interaction may be analyzed by the computing user device. The user may similarly perform a user interaction indicating a user request as at (1108), which may be transmitted to the computing device as at (1110) and (1112). Then, the computing device may analyze at least a portion of the audio signal generated by the microphone, as at (1114).

[0177] At (1116), one or more semantic entities may be displayed on a computing device. For example, various user interfaces (such as Figure 4-8 The user interface depicted in ) can display one or more semantic entities, such as in a list or area of ​​the user interface.

[0178] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the server processes discussed herein can be implemented using a single server or multiple servers working in combination. Databases and applications can be implemented on a single system or distributed on multiple systems. Distributed components can operate sequentially or in parallel.

[0179] Although the subject matter has been described in detail with respect to specific example embodiments and methods thereof, it should be appreciated that those skilled in the art can easily generate replacements, variations, and equivalents to such embodiments after gaining an understanding of the foregoing. Thus, the scope of the present disclosure is by way of example rather than by way of limitation, and the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, as would be apparent to one of ordinary skill in the art.

[0180] Additionally, although the present disclosure is generally discussed with reference to computing devices such as smart phones, the present disclosure is equally applicable to other forms of computing devices including, for example, laptop computing devices, tablet computing devices, wearable computing devices, desktop computing devices, mobile computing devices, or other computing devices.

Claims

1. A method for identifying semantic entities within an audio signal, comprising: Obtaining, by a computing device including one or more processors and one or more memory devices, audio signals simultaneously heard by a user; Receiving, by a computing device, a request from a user to display one or more semantic entities; determining, by the computing device, the selected portion of the audio signal for analysis based at least in part on a predetermined period of time prior to receiving a request from a user to display the one or more semantic entities; analyzing, by a machine learning model stored on the computing device in a background of the computing device, the selected portion of the audio signal to determine the one or more semantic entities; as well as In response to receiving the request, the one or more semantic entities are displayed on a display screen of a computing device.

2. The method of claim 1, further comprising: receiving, by a computing device, a user selection of a selected semantic entity; The selected semantic entity includes a semantic entity selected by the user from among the one or more semantic entities displayed on the display screen of the computing device.

3. The method according to any one of the preceding claims, further comprising: determining, by a computing device, one or more supplemental information options associated with the selected semantic entity; as well as The one or more supplemental information options associated with the selected semantic entity are displayed on a display screen of the computing device.

4. The method of claim 1, wherein: The one or more supplemental information options are determined based at least in part on a context of the audio signal or a context of the selected semantic entity obtained by a computing device.

5. The method of claim 1, wherein: The one or more supplemental information options include one or more of: a database entry associated with the selected semantic entity, a search result associated with the selected semantic entity, or one or more application interaction options associated with the selected semantic entity.

6. The method of claim 1, wherein: The audio signals simultaneously heard by the user include at least one of: an audio signal associated with an application being executed by the computing device, an audio signal generated by the computing device based on ambient audio, or an audio signal associated with a communication signal transmitted to or from the computing device.

7. The method of claim 1, wherein: The request is received in response to a user performing a user interaction with an associated peripheral device.

8. The method of claim 1, wherein: The associated peripheral devices include an earbud device communicatively coupled to the computing device; and The user interaction includes an acquisition gesture performed on the earbud device.

9. The method of claim 1, wherein: Receiving the request includes receiving a user interaction with the computing device.

10. The method of claim 1, wherein: Displaying the one or more semantic entities on a display screen of the computing device includes displaying the one or more semantic entities in a user interface of an application being executed by the computing device.

11. The method of claim 1, wherein: The audio signals concurrently heard by the user include audio signals associated with an application being executed by the computing device.

12. The method of claim 1, wherein: The machine learning model includes at least one of: a speech recognition semantic entity recognizer model, a song recognition semantic entity recognizer model, or a language translation semantic entity recognizer model.

13. A device for identifying semantic entities in an audio signal, comprising one or more processors and a display screen, wherein the one or more processors are configured to: Obtaining audio signals heard by the user at the same time; receiving a request from a user to display one or more semantic entities; determining the selected portion of the audio signal for analysis based at least in part on a predetermined period of time prior to receiving a request from a user to display the one or more semantic entities; analyzing, by a machine learning model stored on the device, the selected portion of the audio signal in the background of the device to determine the one or more semantic entities; as well as In response to receiving the request, the one or more semantic entities are displayed on a display screen of the device.

14. The device of claim 13, wherein: The one or more processors are further configured to: determining one or more supplemental information options associated with the selected semantic entity; and One or more supplemental information options associated with the selected semantic entity are displayed on a display screen of the device.

15. The apparatus of claim 14, wherein: The one or more processors are further configured to: The one or more supplemental information options are determined based at least in part on a context of an audio signal obtained by the device or a context of a selected semantic entity.

16. A system for identifying semantic entities within an audio signal, comprising: a computing device including one or more processors and a display screen; as well as a speaker device configured to play one or more audio signals to a user, the speaker device further configured to receive one or more user interactions from the user; Wherein, the speaker device is operable to: receiving a user interaction from a user indicating a request to display one or more semantic entities; and In response to receiving the user interaction, transmitting a request from the user to the computing device; and wherein the computing device is operable to: Obtaining a first audio signal heard by a user simultaneously; receiving the request from a user; determining the selected portion of the first audio signal for analysis based at least in part on a predetermined period of time prior to receiving a request from a user to display the one or more semantic entities; analyzing the selected portion of the first audio signal in the background of the computing device by a machine learning model stored on the computing device to determine the one or more semantic entities; and In response to receiving the request from a user, the one or more semantic entities are displayed on a display screen of the computing device.

17. The system of claim 16, wherein: The computing device is further operable to: determining one or more supplemental information options associated with the selected semantic entity; and One or more supplemental information options associated with the selected semantic entity are displayed on a display screen of the computing device.

18. The system of claim 17, wherein: The one or more supplemental information options are determined based at least in part on a context of the first audio signal or a context of the selected semantic entity obtained by the computing device.

19. The system of claim 16, wherein: The computing device also includes a microphone configured to generate an audio signal based on the ambient audio; The first audio signal is generated by a microphone of the computing device.

Images