Implementations and methods for communicating with neural network semiconductors using a mobile device - Patents.com

Abstract

Systems and methods described herein include running a machine learning model on the received television broadcast content using an artificial intelligence system-on-chip (AI SoC), the machine learning model being configured to identify objects displayed in the received television broadcast content; displaying the identified objects for selection through a mobile application interface; and, upon selection through the mobile application interface of one or more objects and an overlay from the identified objects, modifying a display of the received television broadcast content to display the overlay.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002]

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 296,366, filed January 4, 2022, the contents of which are expressly incorporated by reference in their entirety into this specification for all purposes.

[0003]

[0002] The present disclosure is directed to mobile device applications, and more particularly, to mobile devices and applications thereof for interacting with neural network semiconductors. [Background technology]

[0004]

[0003] Today, there are many forms of consumer content. To first define the term, "consumer content" is the visual, audible, and verbal content that consumers obtain. As an example, television (TV) consumer content involves images, video, sound, and text. Delivery mechanisms for such consumer content include Ethernet, satellite, cable, and WiFi. Devices used to deliver content are TVs, mobile phones, automobile displays, security camera displays, personal computers (PCs), tablets, augmented reality / virtual reality (AR / VR) devices, and various Internet of Things (IoT) devices. Consumer content can also be divided into "real-time" content, such as live sporting events, and "prepared" content, such as movies and drama series. Today, both "real-time" and "prepared" consumer content are presented to consumers without further annotation or processing. Summary of the Invention

[0005]

[0004] Exemplary implementations described herein involve techniques for processing consumer content and combining appropriate cloud information found for relevant portions of the consumer content for presentation to the consumer. Such exemplary implementations may involve classifying and identifying people, objects, concepts, scenes, text, language, etc. in the consumer content, annotating the classified items in the content with corresponding information from the cloud, and presenting the annotated content to the consumer.

[0006]

[0005] Classification / identification is the process of processing images, videos, sounds, and language to identify people (who someone is), object classes (cars, ships, etc.), text / language meanings, any concept, or any scene. A good example of how this classification step can be achieved is various Artificial Intelligence (AI) models capable of classifying images, videos, and language. However, there can be other alternative methods such as traditional algorithms. The definition of cloud is any information residing on any server, any form of database, any computer memory, any storage device, or any consumer device.

[0007]

[0006] Aspects of the disclosure may include a method, the method including: using an artificial intelligence system-on-chip (AI SOC) to run a machine learning model on received television broadcast content, the machine learning model being configured to identify objects displayed in the received television broadcast content; displaying the identified objects for selection through a mobile application interface; and modifying a display of the received television broadcast content to display an overlay upon selection through the mobile application interface of one or more objects and an overlay from the identified objects.

[0008]

[0007] Aspects of the disclosure may involve a computer program having stored thereon instructions for performing a method, the instructions including receiving, from an artificial intelligence system-on-chip (AI SOC), identified objects displayed in received television content by a machine learning model, displaying the identified objects for selection through a mobile application interface, and transmitting instructions to modify a display of the received television broadcast content to display an overlay upon selection through the mobile application interface of one or more objects and an overlay from the identified objects. The computer instructions of the computer program may be stored in a non-transitory computer-readable medium and executed by one or more processors.

[0009]

[0008] Aspects of the present disclosure may include a system that may include means for running a machine learning model on received television broadcast content using an artificial intelligence system-on-chip (AI SOC), the machine learning model being configured to identify objects displayed in the received television broadcast content; means for displaying the identified objects for selection through a mobile application interface; and means for modifying a display of the received television broadcast content to display an overlay upon selection through the mobile application interface of one or more objects and an overlay from the identified objects.

[0010]

[0009] Aspects of the disclosure may include a device, such as a mobile device, that may include a processor configured to receive, from an artificial intelligence system-on-chip (AI SOC), identified objects displayed in received television content by a machine learning model, display the identified objects for selection through a mobile application interface, and, upon selection through the mobile application interface of one or more objects and overlays from the identified objects, send instructions to modify the display of the received television broadcast content to display the overlay. [Brief description of the drawings]

[0011] [Figure 1]

[0010] A diagram showing an example of how digital content is processed and supplemented with relevant information from the cloud, the Internet, systems, arbitrary databases, and people (e.g., as input from people's devices) according to an exemplary implementation. [Diagram 2]

[0011] A diagram illustrating the overall architecture of an AI Cloud TV SoC according to an exemplary implementation. [Figure 3A]

[0012] FIG. 1 illustrates examples of AI edge devices in various systems, according to an exemplary implementation. [Figure 3B]

[0012] A diagram showing examples of AI edge devices in various systems, according to an exemplary implementation. [Figure 3C]

[0012] A diagram showing examples of AI edge devices in various systems, according to an exemplary implementation. [Figure 3D]

[0012] A diagram showing examples of AI edge devices in various systems, according to an exemplary implementation. [Figure 4]

[0013] FIG. 1 illustrates an example control architecture for an AI SoC, according to an example implementation. [Diagram 5]

[0014] FIG. 1 illustrates an example communication tunnel between a mobile device and an AI SoC, according to an example implementation. [Figure 6A]

[0015] FIG. 1 illustrates an example of multiple users connecting to an AI SoC, according to an exemplary implementation. [Figure 6B]

[0016] FIG. 1 illustrates an example of connecting multiple users together over the Internet, according to an exemplary implementation. [Figure 7]

[0017] FIG. 1 illustrates an example use case of information overlay, according to an example implementation. [Figure 8]

[0017] FIG. 1 illustrates an exemplary use case of information overlay, according to an exemplary implementation. [Figure 9]

[0017] FIG. 1 illustrates an exemplary use case of information overlay, according to an exemplary implementation. [Figure 10]

[0017] FIG. 1 illustrates an exemplary use case of information overlay, according to an exemplary implementation. [Figure 11]

[0017] FIG. 1 illustrates an exemplary use case of information overlay, according to an exemplary implementation. [Figure 12]

[0017] FIG. 1 illustrates an exemplary use case of information overlay, according to an exemplary implementation. [Figure 13]

[0018] FIG. 1 illustrates an example use case of a social overlay, according to an example implementation. [Figure 14]

[0018] FIG. 1 illustrates an exemplary use case of a social overlay, according to an exemplary implementation. [Figure 15]

[0018] FIG. 1 illustrates an exemplary use case of a social overlay, according to an exemplary implementation. [Figure 16]

[0018] FIG. 1 illustrates an exemplary use case of a social overlay, according to an exemplary implementation. [Figure 17A]

[0019] 1A-1C are diagrams illustrating examples of display modes, according to an exemplary implementation. [Figure 17B]

[0019] FIG. 2 illustrates an example of a display mode, according to an exemplary implementation. [Figure 18]

[0020] 1 illustrates an example of a user interface of a mobile device application for managing overlays, according to an exemplary implementation. [Figure 19]

[0020] FIG. 1 illustrates an example of a user interface of a mobile device application for managing overlays, according to an exemplary implementation. [Figure 20]

[0020] FIG. 1 illustrates an example of a user interface of a mobile device application for managing overlays, according to an exemplary implementation. [Figure 21]

[0020] FIG. 1 illustrates an example of a user interface of a mobile device application for managing overlays, according to an exemplary implementation. [Figure 22]

[0020] FIG. 1 illustrates an example of a user interface of a mobile device application for managing overlays, according to an exemplary implementation. [Diagram 23]

[0021] FIG. 1 illustrates an example of a mobile device, according to an exemplary implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012]

[0022] The following detailed description provides diagrams and exemplary implementation details of the present application. Reference numbers and descriptions of redundant elements between figures are omitted for clarity. The terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term "automatically" may include fully automatic implementations or semi-automatic implementations with user or administrator control over certain aspects of the implementation, depending on the desired implementation of those skilled in the art practicing the implementation of the present application. Selections may be performed by a user through a user interface or other input means, or may be performed by a desired algorithm. The exemplary implementations described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be implemented by any means according to the desired implementation.

[0013]

[0023] FIG. 1 illustrates an example of how digital content is processed and supplemented with relevant information from the cloud, the Internet, systems, any databases, and people (e.g., input from people's devices) according to an exemplary implementation. Digital content 102 may be provided to an edge SoC device having an artificial intelligence processing element (AIPE) 104 for processing the digital content 102. The SoC 104 may be part of a network or may be an independent edge device (e.g., an Internet-enabled TV, etc.). The SoC 104 may receive the digital content 102 and process the digital content to detect or classify objects in the digital content 102. For example, the SoC 104 may process the digital content 102 to detect that the digital content 102 includes a basketball player, a basketball, and a basket. The SoC 104 may search and find information in the cloud / internet / systems / databases / people 106 related to the processed digital content, such as information about the basketball player. For example, the SoC 104 may detect or identify one or more players and the players' teams involved in a real-time sporting event. The cloud / internet / systems / databases / people 106 may contain relevant information about the player, and the SoC 104 may supplement the digital content 102 with the relevant information from the cloud / internet / systems / databases / people 106. The SoC 104 may then provide the digital content annotated with information from the cloud / internet / systems / databases / people 106 to the edge device 108 to display the digital content along with the supplemental information to the viewer. The viewer / consumer may have the option to display supplemental information along with the digital content, such as, but not limited to, personal information about the player, real-time statistics about the player, recent statistics from previous games, or season statistics over a period of time, or a player's biography, social media content about the player, e-commerce information related to the player.

[0014]

[0024] Artificial Intelligence Television (AI TV) is a TV that annotates TV content with cloud information and delivers the annotated content to consumers in real-time. Related art TVs are not capable of classifying TV content in real-time (e.g., at 60 frames per second). Current capabilities available for related art TVs involve delivering content to consumers, either by streaming content from the Internet (Smart TV) or receiving content through a set-top box, and receiving and processing user input, i.e., remote control input, voice input, or camera input.

[0015]

[0025] AI TV is a novel device that can process content using an AI TV System-on-Chip (SoC) with sufficient processing power to consume 60 frames per second and run the necessary classification and detection algorithms to classify and identify TV content in real time, find relevant information in the cloud, annotate the content with the found information and present it to the consumer. AI TV also has the ability to interact with the consumer to decide what to display, how to display it, and when to display the annotated information.

[0016]

[0026] Today's TVs have two types of system-on-chip (SoCs): TV SoC and Timing Control (TCON) SoC. The TV SoC is responsible for getting the content from remote control devices, microphones, or cameras through a High-Definition Multimedia Interface (HDMI) interface and user interface signals, over the Internet (usually through a Wi-Fi interface), or through a set-top box. The TV SoC then passes the image to the Timing Controller (TCON) SoC and the audio to the speaker. The TCON SoC then passes the image to a driver integrated circuit (IC) for enhancing the image quality and displaying the image on the screen. Some TVs combine the TV SoC and the TCON SoC into one TV SoC.

[0017]

[0027] Current TV SoCs and TCON SoCs do not have the processing power or functionality for AI TV, so a dedicated AI TV SoC is required to realize AI TV.

[0018]

[0028] 2 illustrates the overall architecture of an AI Cloud TV SoC according to an exemplary implementation. The AI ​​Cloud TV SoC 202 may be configured to process digital content. The AI ​​Cloud TV SoC 202 may include multiple elements utilized in processing digital content. For example, the AI ​​Cloud TV SoC 202 may include an input / pre-processing unit (IPU) 204, an AI processing unit (APU) 206, an Internet interface 208, a memory interface 210, an output processing unit (OPU) 212, and controller logic 214.

[0019]

[0029] The IPU 204 may receive digital content 220 as an input. The IPU 204 may prepare the digital content 220 for use by the AI ​​processing unit and the memory interface. For example, the IPU 204 may receive the digital content 220 as frames and audio data and prepares the frames and audio data to be processed by the APU. The IPU 204 provides the prepared digital content 220 to the APU 206. The APU 206 processes the digital content using various neural network models and other algorithms that it obtains from memory via the memory interface. For example, the memory interface 210 includes multiple neural network models and algorithms that can be utilized by the APU 206 to process the digital content.

[0020]

[0030] The memory interface 210 may receive neural network models and algorithms from the cloud / internet / system / database / people 216. The APU may retrieve one or more AI / neural network models from the memory interface. The APU 206 may process the preprocessed input digital content with one or more AI / neural network models. The internet interface 208 may search and find related supplemental information for the processed digital content and provide the related supplemental information to the memory interface 210. The memory interface 210 receives information from the cloud / internet / system / database / people 216 related to the processed digital content from the internet interface 208. The information from the cloud / internet / system / database / people 216 may be stored in the memory 218 and may also be provided to the OPU 212. The OPU 212 may utilize the information from the cloud / internet / system / database / people 216 to supplement the digital content and provide the supplemental information and the digital content to the consumer / viewer. The information from the internet may be stored in the memory 218 and may be accessible to the OPU. The OPU may access information stored in memory 218 through memory interface 210. Memory 218 may be internal or external memory. The OPU 212 prepares supplemental information and digital content 222 for display on a display device. The controller logic 214 may contain instructions for the operation of the IPU 204, APU 206, OPU 212, the Internet interface, and the memory interface 210.

[0021]

[0031] The above architecture may be utilized to process audio in the digital content 220. For example, the APU 206 may process the audio portion of the digital content, convert the audio to text, and process the audio content using natural language processing neural network models or algorithms. The Internet interface finds relevant information from the cloud / internet / systems / databases / people to create supplemental information, and the OPU prepares the supplemental information and digital content for presentation on the edge device in the same manner as described above for multiple frames.

[0022]

[0032] As shown, the AI ​​Cloud TV SoC receives input frames from the TV SoC and classifies the content using the AI ​​model processed in the AI ​​processing unit. It then connects to the cloud through a Wi-Fi interface to annotate the actual content / frame with any relevant information from the cloud, and then presents the annotated content to the viewer.

[0023]

[0033] The AI ​​TV SoC can be used inside a TV, a set-top box (STB), a streaming device, or a standalone device.

[0024]

[0034] 3A-3D show examples of AI edge devices in various systems according to exemplary implementations. FIG. 3A provides an example of an AI TV 302 comprising a TV SoC, an AI TV edge SoC, and a display panel in a fully integrated device. The AI ​​TV 302 includes an AI TV edge SoC that processes digital content and provides supplemental information to the digital content including associated data / information related to the digital content obtained from the cloud / internet / systems / databases / people for use by the AI ​​TV 302. FIG. 3B provides an example of an AI set-top box 304, an external device configured to be connected to a TV 306. The AI ​​set-top box 304 may be connected to the TV 306 via an HDMI connection, although other connections may be utilized to connect the AI ​​set-top box 304 and the TV 306. The AI ​​set-top box 304 comprises a set-top box (STB) SoC and an AI set-top box SoC. The AI ​​set-top box 304 receives digital content, processes the digital content, and provides supplemental information to the digital content as output, including associated data / information related to the digital content acquired from the cloud / internet / systems / databases / people. The supplemental information may be provided to the TV 306 via an HDMI connection along with the digital content. FIG. 3C provides an example of a streaming system device 308, an external device configured to be connected to the TV 310. The streaming system device 308 may be connected to the TV 310 via an HDMI connection, although other connections may be utilized to connect the streaming system device 308 and the TV 310. The streaming system device 308 comprises a streaming SoC and an AI streaming SoC. The streaming system device 308 receives digital content, processes the digital content, and provides supplemental information to the digital content as output, including associated data related to the digital content acquired from the cloud / internet / systems / databases / people.The supplemental information may be provided to the TV 310 via an HDMI connection along with the digital content. Figure 3D provides an example of an independent device, the AI ​​edge device 314. The AI ​​edge device 314 receives the digital content from the set-top box 312 via an HDMI connection, processes the digital content, and provides supplemental information to the digital content, including associated data related to the digital content acquired from the cloud / internet / systems / databases / people. The AI ​​edge device 314 provides the supplemental information to the TV 316 along with the digital content via an HDMI connection.

[0025]

[0035] Other implementations are possible, and the present disclosure is not specifically limited to the implementations described herein. The AI ​​SoC proposed herein can also be extended to other edge or server systems that can utilize such capabilities, including mobile devices, surveillance devices (e.g., cameras or other sensors connected to a central station or local user control system), personal computers, tablets or other user equipment, vehicles (e.g., Advanced Driver-assistance System (ADAS) systems, or systems utilizing Electronic Control Units (ECUs), Internet of Things edge devices (e.g., aggregators, gateways, routers), Augmented Reality / Virtual Reality (AR / VR) systems, smart home and other smart system implementations, etc., depending on the desired implementation. Controlling the AI ​​SoC

[0026]

[0036] FIG. 4 illustrates an example control architecture of an AI SoC according to an example implementation. There are many configurations and settings that a user can change, and a simple device such as a remote control cannot handle the complexity. A mobile device 402, such as a smartphone, a tablet with Wi-Fi capabilities, or any device connected to a local network 400 with a wired connection, is used to establish a communication channel between the user and an AI SoC 406, such as in an AI TV. Both the mobile device 402 and the AI ​​SoC 406 are connected to the same local network 400 through a network device 404, such as a router or switch, so that the devices can communicate with the AI ​​SoC through standard network protocols such as Transmission Control Protocol / Internet Protocol (TCP / IP).

[0027]

[0037] The mobile device 402 acts as a remote control for the AI ​​TV. Each user can download a mobile application (mobile app) and install it on the mobile device 402, which connects to the AI ​​SoC 406 on the same local network 400. First, the user can install the mobile application on the mobile device 402, such as a smartphone or tablet. Then, the mobile application searches for the AI ​​SoC(s) in the local network 400. Finally, the mobile application creates a communication tunnel (i.e., TCP / IP) to the AI ​​SoC 406.

[0028]

[0038] FIG. 5 illustrates an exemplary communication tunnel between a mobile device and an AI SoC according to an exemplary implementation. Once a communication tunnel is established between the mobile device and the AI ​​SoC (through a mobile application), information can flow between the mobile device (mobile application) and the AI ​​SoC. The mobile application requests data from the AI ​​SoC, and the AI ​​SoC returns the requested information to the mobile application. Multiple users using different mobile devices can be connected to the same AI SoC. Each mobile device (mobile application) is assigned to a different user. Each user can have a different set of controls / settings corresponding to their preferences. Multiple users connecting to one AI SoC

[0029]

[0039] FIG. 6A shows an example of multiple users connecting to an AI SoC, according to an exemplary implementation. User 1, User 2...User N are all connected to the AI ​​SoC. User 1, User 2...User N can send requests to the AI ​​SoC. The AI ​​SoC can send the requested information to the particular user. The AI ​​SoC can send notifications to all connected devices. Connecting users together

[0030]

[0040] FIG. 6B shows an example of connecting multiple users together via the Internet according to an exemplary implementation. All users within a local network are connected within the local network. Users outside the local network can also be connected via an Internet connection. Multiple local networks are connected through the Internet, so that all users are connected and can communicate with each other, thereby creating a virtual social community of AI SoC (AI TV / STB) users.

[0031]

[0041] All user configurations are controllable by the mobile application. The mobile application can control all configurable switches in the AI ​​SoC. Below are some example configurations that can be controlled by the mobile application:

[0032]

[0042] Channel Selection: Users can change the channel of their AI TV / STB through the functionality on the mobile application.

[0033]

[0043] AI Model Selection: Users can select which AI models to load into memory for processing by the AI ​​SoC.

[0034]

[0044] Display configuration: How information is displayed on TV screens and mobile screens etc.

[0035]

[0045] Selection of Classified Objects: Select classified objects, such as image, audio, and / or text objects, for highlighting or other purposes.

[0036]

[0046] Select information: Select the information displayed on the screen.

[0037]

[0047] Visual Effect Selection: Add or remove visual effects to the screen or live broadcast (for example, select basketball and add a fire effect during the broadcast of the basketball game).

[0038]

[0048] Selecting friends (eg, connected users): Add or remove selected friends with whom you want to exchange information on the TV or mobile display.

[0039]

[0049] Select an action: display information, show visual effects, chat / share information with other users (e.g. friends).

[0040]

[0050] Sending information to the AI ​​SoC, such as instructions to run the model

[0041]

[0051] Sending information to the AI ​​DB server: Information for searching new models, etc.

[0042]

[0052] Receiving information from the AI ​​SoC: results from executed models, etc.

[0043]

[0053] Receiving information from the AI ​​DB server: new models, additional metadata, etc.

[0044]

[0054] Through mobile apps, users can display various information and visual effects on the screen of their AI TV and / or the screen of their mobile device. Applications can be categorized into three types: information overlays, visual overlays, and social overlays.

[0045]

[0055] The information is about classified and identified people, objects, concepts, scenes, text, language within the consumer content processed by the AI ​​SoC. It comes from the AI ​​DB server and / or from the Internet (e.g., search results from the Internet according to the preferred implementation).

[0046]

[0056] The information overlay displays specific information about the classified object selected by the user. The information can be displayed on the screen of the AI ​​TV or mobile device. It can be any information about the classified object, sound / audio, and text.

[0047]

[0057] 7-12 show exemplary use cases of information overlays according to exemplary implementations. As shown in FIG. 7, information such as detailed statistics about each player in a sports match may be displayed on the screen. As shown in FIG. 8, information about actors and actresses may be displayed on the screen, and the mobile application can select the actor or actress to select, as well as the type of information to be displayed, such as news, trends, social media, etc., about the particular actor and / or actress. The user can view more information about news snippets from various sources (e.g., various news channels or internet sources), as shown in FIG. 9. The type of information is selected by the user on the mobile application. As shown in FIG. 10, information such as price, rating, and e-commerce site for a product classified by the AI ​​SoC may be displayed, and a link to the e-commerce site may be provided to the user.

[0048]

[0058] Visual overlays give users the ability to edit content on the fly. Various visual effects and / or videos can be superimposed on or near objects classified by the AI ​​SoC. The location of the visual overlay and the type of visual effect can be selected by the user on the mobile application. FIG. 11 shows an example of adding a visual overlay according to an exemplary implementation. In a sports game as shown in FIG. 11, visual effects such as a fireball or water splash can be superimposed on a basketball when a designated player shoots. A fireworks effect on the basket can also be created when a special play or event (i.e., a dunk shot) occurs by a designated player in a basketball game.

[0049]

[0059] In the example of Figure 12, the user may also superimpose the image onto the face of another character, depending on the desired implementation. For example, by using known AI models and techniques such as deepfake, the face of one character may be replaced with another face (e.g., another character, an animated icon, another person, etc.).

[0050]

[0060] An exemplary implementation may also utilize a social overlay, which gives users the ability to share "information overlays" and "visual overlays" with connected friends (other users). All users are connected together through the AI ​​SoC network, and groups of users (friends) who are willing to share more information may be formed, such as:

[0051]

[0061] 1. User preferences (e.g. AI model choice, favorite shows / channels, favorite characters / objects, etc.)

[0052]

[0062] 2. Send informational and visual overlays to friends

[0053]

[0063] 3. Receive informational and visual overlays from friends

[0054]

[0064] 4. Share text / voice messages within a group of friends or with individuals within a group of friends

[0055]

[0065] A group of users (friends) can also form a social group about a particular content and share information among the social group. This can create a virtual environment where users of the social group are watching the content side by side together (e.g., a virtual stadium, a virtual theater, etc.). A user can send an "information overlay" and / or a "visual overlay" to another friend(s) in the social group. The "information overlay" and / or the "visual overlay" can be displayed on the screens of multiple users who are connected as "friends". For example, a user can send a visual overlay to another user in the same social group, causing the visual overlay to be displayed on the display or mobile device of the other user.

[0056]

[0066] 13-16 show exemplary use cases of social overlays according to exemplary implementations. Users in a social group can exchange text (chat) and create informational and visual overlays on objects classified by the AI ​​SoC, as shown in FIG. 13. Friends can send text (as visual overlays) to other friends watching the same content, which can create a virtual environment as if multiple friends are watching in the same room, as shown in FIG. 13. A user can send text to another user in his / her friend group, which can be overlaid on any classified object, as shown in FIG. 14. As shown in FIG. 15, information gathering such as polls can be done among friends, simply asking for approval or disapproval or posting a simple question. As shown in FIG. 16, a user (or users) can chat with characters in a movie / show provided by an AI chatbot. Other examples of social overlays can also be utilized, and the present disclosure is not limited thereto. For example, depending on the desired implementation, a user may become a participant in a game show by entering answers, or a user may become a judge and cast votes in the show.

[0057]

[0067] 17A and 17B show examples of display modes according to an exemplary implementation. Multiple display modes are provided for information overlays, visual overlays, and social overlays. In one example shown in FIG. 17A, a "fixed mode" displays information in a fixed location, such as the top (or bottom, left, right) area of ​​the screen. In another example shown in FIG. 17B, an "attached mode" displays information near a classified object. The user can select a relative location from the object. Other display modes are possible, and the disclosure is not limited thereto. For example, the information may be displayed outside the content instead.

[0058]

[0068] 18-22 show examples of user interfaces of a mobile device application for managing overlays, according to an example implementation. In the example of FIG. 18, a user can use his mobile device to change channels on his television screen through a drop-down selection box.

[0059]

[0069] According to an exemplary implementation, various icons and menus may be provided by the user interface for selection to implement information overlays, visual overlays, social overlays, etc. For a given television program, detected people and objects from the AI ​​SoC may be provided for selection to select an overlay to be provided thereon or to provide other information, according to a desired implementation. In the example of FIG. 19, a person named "Stephen C." is selected as a subject of interest, as shown in screen 1900. A news icon may then be selected, as shown in 1901, to provide links to news articles or headlines as an information overlay. A friend or related person icon may then be selected, as shown in 1902, to provide relatives or known associates as an information overlay. A stats button may then be selected, as shown in 1903, to provide various statistics (e.g., sports stats) about the selected person as an information overlay. Other examples shown in FIG. 19 include compensation / budget statistics 1904, and nickname 1905. The information requested can be tailored according to a desired implementation and can be customizable (eg, based on the underlying television program, etc.), and this disclosure is not limited thereto.

[0060]

[0070] FIG. 20 illustrates an exemplary interface for providing a visual overlay on a television, according to an exemplary implementation. Specifically, after receiving a user selection ("Stephen C." and "ball") through the interface screen shown at 2000, a fireball is selected as the visual overlay to replace the ball with a fireball overlay when the ball is controlled by "Stephen C." during the basketball game. When the checkmark button is selected, the visual overlay is enabled and will be displayed during the broadcast of the television program, as shown at 2001. In this manner, the user can apply different visual overlays to each of the people and objects or a combination thereof. Visual overlays can be provided over people, objects, or when both are selected, over objects when the objects are being controlled by the selected person.

[0061]

[0071] 21 shows an example interface for providing a social overlay on another person's television, according to an example implementation. Through the mobile application interface, a user can select friends who are watching the same program to add a social overlay to, at 2101, and can also select the type of overlay to display on the friend's screen. For example, if the user wants to add an informational overlay, as shown at 2102, or a visual overlay, as shown at 2103, to a friend's screen, such an overlay can be displayed on the friend's screen, as shown at 2104.

[0062]

[0072] 22 shows an example interface for customizing the position and other aspects of the overlay, according to an example implementation. Settings related to the information overlays may be accessible through a user interface, as shown at 2201. Settings that can be adjusted related to the overlays may include changing the display mode for each type of overlay, as shown at 2202, enabling / disabling various overlays 2203, as well as being able to set the position of the overlay on an object (e.g., a person) as shown at 2204.

[0063]

[0073] FIG. 23 illustrates an example of a mobile device according to an exemplary implementation. The mobile device 2300 may include a camera 2301, a microphone 2302, a processor 2303, a memory 2304, a display 2305, an interface (I / F) 2306, and a direction sensor 2307. The camera 2301 may include any type of camera configured to record any form of video according to a desired implementation. The microphone 2302 may include any form of microphone configured to record any form of audio according to a desired implementation. The display 2305 may include a touch screen display configured to receive touch input and facilitate instructions to perform the functions described herein, or a regular display such as a liquid crystal display (LCD) or any other display according to a desired implementation. The I / F 2306 may include a network interface to facilitate connection of the mobile device 2300 to external elements such as servers and any other devices according to a desired implementation. The processor 2303 may be in the form of a hardware processor such as a central processing unit (CPU), or a combination of hardware and software units according to a desired implementation. The orientation sensor 2307 may include any form of gyroscope and / or accelerometer configured to measure any type of orientation measurement, such as tilt angle, orientation relative to x, y, z, acceleration (e.g., gravity), etc., according to a desired implementation. The orientation sensor measurements may also include a gravity vector measurement, indicative of the device's gravity vector, according to a desired implementation. The mobile device 2300 may be configured to receive input from a keyboard, mouse, stylus, or any other input device through the I / F 2306, according to a desired implementation.

[0064]

[0074] In an example implementation, an artificial intelligence system-on-chip (AI SoC) as shown in Figure 2 executes a machine learning model on the received television broadcast content, where the machine learning model is configured to identify objects displayed in the received television broadcast content. Thus, the processor 2303 may be configured to execute a method or instructions including displaying the identified objects for selection through a mobile application interface, as shown at 1900 in Figure 19, and modifying a display of the received television broadcast content to display an overlay upon selection of one or more objects and an overlay from the identified objects through the mobile application interface, as shown in Figures 20-22.

[0065]

[0075] The processor 2303 may be configured to execute the above-mentioned method or instructions, further including, where the overlay is an information overlay, as shown in Figures 17A, 17B and 19, searching for information related to the selected object or objects, and generating an overlay from the searched information.

[0066]

[0076] The processor 2303 may be configured to execute the above-described method or instructions, where, for the overlay to be a visual overlay, as shown in Figures 11 and 20, the step of modifying the display of the received television broadcast content to display the overlay includes displaying the visual overlay over the selected object or objects.

[0067]

[0077] The processor 2303 may be configured to execute the above method or instructions, wherein the step of modifying the display of the received television broadcast content to display an overlay as illustrated in and described with respect to Figures 11 and 20 includes, for a selection of one or more objects from the identified objects being a selection of a person and an object, displaying a visual overlay over the object when the object is associated with the person.

[0068]

[0078] The processor 2303 may be configured to execute the above-described method or instructions, further including the step of modifying the display of the received television broadcast content of the selected one or more users to display an overlay upon selection of the one or more users through the mobile application interface, as shown in FIG. 6B and FIG. 21.

[0069]

[0079] The processor 2303 may be configured to execute the above-described methods or instructions, further including retrieving information for display in the mobile application interface, regarding the selected object or objects, as shown in Figures 8 and 12.

[0070]

[0080] Depending on the desired implementation, the AI ​​SoC may be located in one of the television, the set-top box, or an edge device connected to the set-top box and the television, as shown in Figures 3A-3D. The processor 2303 may be configured to execute the methods or instructions described above, further including receiving a channel for obtaining received television content through a mobile application interface, as shown in Figure 18.

[0071]

[0081] The processor 2303 can be configured to execute the method or instructions described above, further including receiving a selection of the machine learning model through a mobile application interface, as described with respect to FIG. 6B, and the AI ​​SoC is configured to execute the selected machine learning model in response to the selection.

[0072]

[0082] The processor 2303 may be configured to execute the above-described method or instructions, further including the step of receiving, through a mobile application interface, a selection of a location on the selected one or more objects at which to provide the overlay, as shown in Figures 22 and 23, and wherein the step of modifying the display of the received television broadcast content to display the overlay includes providing the overlay at the selected location on the selected one or more objects.

[0073]

[0083] The processor 2303 may be configured to execute the above-described method or instructions, wherein the overlay includes a text message, and the step of modifying the display of the received television broadcast content to display the overlay, as shown in Figures 13 and 14, includes modifying the display of the multiple users to display the text message.

[0074]

[0084] The processor 2303 may be configured to execute the above method or instructions, where, as shown in FIG. 12 , the selection of the one or more objects is a first person having a first face and a second person having a second face, and the overlay includes an overlay of the second face over the first person and an overlay of the first face over the second person.

[0075]

[0085] The processor 2303 may be configured to execute the above-described method or instructions, further comprising, as shown in FIG. 16, when the selection of the one or more objects is a person, generating a chat application within the mobile application interface to facilitate chatting with the person.

[0076]

[0086] The processor 2303 may be configured to execute the method or instructions described above, further including receiving an instruction to initiate voting through a mobile application interface, as shown in FIG. 15, where the votes are provided to the mobile application interface of one or more users watching the received television content.

[0077]

[0087] The processor 2303 may be configured to execute the methods or instructions described above, where the overlay includes animation, as shown in FIG.

[0078]

[0088] The processor 2303 may be configured to execute the methods or instructions described above, where the overlay includes statistics related to the selected object or objects, as shown in FIG.

[0079]

[0089] Although the exemplary implementations described herein have been described with respect to mobile devices and televisions, other devices are possible and the disclosure is not limited thereto. Other devices (e.g., computers, laptops, tablets, etc.) can also run the applications described herein to interact with a set-top box or other device configured to display television or video broadcasts. Furthermore, the disclosure is not limited to television or video broadcasts, but may also apply to other streaming content, such as Internet streaming content, camera feeds from security cameras, peripheral devices such as another tablet, video tapes in a VCR, DVDs, or playback from other external media.

[0080]

[0090] Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In exemplary implementations, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0081]

[0091] As will be apparent from the description, unless specifically stated otherwise, throughout the description, descriptions utilizing terms such as "processing," "calculating," "computing," "determining," "displaying," and the like, are understood to include operations and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other information storage, transmission, or display device.

[0082]

[0092] An exemplary implementation may relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memories, random access memories, semiconductor devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. The computer-readable signal medium may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. The computer program may include a pure software implementation with instructions to perform the operations of a desired implementation.

[0083]

[0093] Various general-purpose systems may be used with the programs and modules in accordance with the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Additionally, the exemplary implementations are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the techniques of the exemplary implementations described herein. Instructions in the programming language may be executed by one or more processing devices (e.g., a central processing unit (CPU), processor, or controller).

[0084]

[0094] As known in the art, the above operations can be performed by hardware, software, or some combination of software and hardware. Various aspects of the exemplary implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which, when executed by a processor, cause the processor to perform a method for performing the implementation of the present application. Furthermore, some exemplary implementations of the present application may be performed exclusively in hardware, while other exemplary implementations may be performed exclusively in software. Furthermore, the various functions described may be performed in a single unit or distributed across multiple components in any number of ways. When performed by software, the method may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If necessary, the instructions may be stored on the medium in compressed and / or encrypted format.

[0085]

[0095] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary implementations be considered as examples only, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. running a machine learning model on the received television broadcast content using an artificial intelligence system-on-chip (AI SoC); the machine learning model configured to identify objects displayed in the received television broadcast content. displaying the identified objects for selection through a mobile application interface; responsive to selection, through the mobile application interface, of one or more objects from the identified objects and an overlay, modifying a display of the received television broadcast content to display the overlay; The method includes:

2. if the overlay is an information overlay, retrieving information related to the selected one or more objects; generating the overlay from the retrieved information; The method of claim 1 further comprising:

3. and, where the overlay is a visual overlay, the step of modifying the display of the received television broadcast content to display the overlay includes displaying the visual overlay over the selected one or more objects. The method of claim 1.

4. The step of modifying the display of the received television broadcast content to display the overlay comprises: and, where the selection of one or more objects from the identified objects is a selection of a person and an object, displaying the visual overlay on the object when the object is associated with the person. The method according to claim 3.

5. upon selection of one or more users through the mobile application interface, modifying the display of the received television broadcast content for the selected one or more users to display the overlay; The method of claim 1 further comprising:

6. retrieving information for display in the mobile application interface regarding the selected one or more objects; The method of claim 1 further comprising:

7. The AI ​​SoC is disposed in one of a television, a set-top box, or an edge device connected to a set-top box and a television, and the method includes receiving, through the mobile application interface, a channel for obtaining the received television content; The method of claim 1 further comprising:

8. receiving a selection of the machine learning model through the mobile application interface; Further comprising: the AI ​​SoC is configured to execute the selected machine learning model in response to the selection; The method of claim 1.

9. receiving, through the mobile application interface, a selection of a location on the selected object or objects at which to provide the overlay; Further comprising: modifying the display of the received television broadcast content to display the overlay includes providing the overlay at the selected location on the selected one or more objects. The method of claim 1.

10. the overlay includes a text message; and modifying the display of the received television broadcast content to display the overlay includes modifying a display of a plurality of users to display the text message. The method of claim 1.

11. wherein, for said selection of said one or more objects being a first person having a first face and a second person having a second face, said overlaying includes an overlay of said second face over said first person and an overlay of said first face over said second person. The method of claim 1.

12. where the selection of the one or more objects is a person, generating a chat application within the mobile application interface to facilitate a chat with the person; The method of claim 1 further comprising:

13. receiving, via the mobile application interface, an instruction to initiate voting; Further comprising: the votes are provided to a mobile application interface of one or more users viewing the received television content. The method of claim 1.

14. the overlay includes a moving image; The method of claim 1.

15. the overlay includes statistics related to the selected object or objects; The method of claim 1.

Images