Entity card including descriptive content related to entity from video

Abstract

A server system obtains a transcription of content from a video and applies machine learning resources to identify entities that are most likely to be searched by a user viewing the video based on the transcription of the content. The server system generates an entity card for each of the entities, each of the entity cards including descriptive content associated with a respective one of the entities. The server system provides a user interface to be displayed on respective displays of one or more user computing devices for playing the video on a first portion of the user interface. When the video is played and a first one of the entities is mentioned in the video, the user interface displays a first entity card on a second portion of the user interface, the first entity card including descriptive content related to the first entity.

Description

[0001] Priority declaration

[0002] This application is based on and claims priority to U.S. Application No. 18 / 091,837, filed on December 30, 2022, which claims priority to U.S. Provisional Application No. 63 / 341,674, filed on May 13, 2022, each of which is incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to providing entity cards for a user interface in association with a video displayed on a display of a user computing device. More specifically, the present disclosure relates to providing entity cards that aid in understanding the content of a video and include descriptive content related to entities (e.g., concepts, terms, topics, etc.) mentioned in the video. Background Art

[0004] When a user watches a video about, for example, a challenging or new topic, there may be keywords or concepts that the user is unfamiliar with but that are helpful in understanding the content of the video. For example, in a video about the Egyptian pyramids, the term "sarcophagus" may be an important concept that is widely discussed. However, a user who is not familiar with the term "sarcophagus" may not fully understand the content of the video. The user may pause the video and navigate to a search page to search for the term "sarcophagus". In some cases, the user may have difficulty spelling the term they want to search for and may not get accurate search results or may experience inconvenience in searching. In other cases, the user may stop watching the video after finding the content of the video too difficult to understand. Summary of the invention

[0005] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned through practice of example embodiments.

[0006] In one or more example embodiments, a computer-implemented method for a server system includes: obtaining a transcription of content from a video; applying machine learning resources to identify one or more entities that are most likely to be searched for by users viewing the video; generating one or more entity cards for each of one or more entities based on the transcription of the content, each of the one or more entity cards including descriptive content related to a corresponding entity of the one or more entities; and providing a user interface to be displayed on corresponding displays of one or more user computing devices, for: playing the video on a first portion of the user interface, and when the video is played and a first entity of the one or more entities is mentioned in the video, displaying a first entity card on a second portion of the user interface, the first entity card including descriptive content related to the first entity.

[0007] In some implementations, applying the machine learning resource to identify the one or more entities includes obtaining training data to train the machine learning resource based on observation data of users searching in response to only viewing the video.

[0008] In some implementations, applying machine learning resources to identify one or more entities includes identifying multiple candidate entities from a video by associating text from a transcription with a knowledge graph, and ranking the candidate entities to obtain the one or more entities based on one or more of: a relevance of each of the candidate entities to the subject matter of the video, a relevance of each of the candidate entities to one or more other candidate entities in a plurality of candidate entities, a number of times the candidate entity is mentioned in the video, and a number of videos in which the candidate entity appears in a corpus of videos stored in one or more databases.

[0009] In some implementations, applying the machine learning resource to identify the one or more entities includes evaluating user interactions with the user interface, and determining at least one adjustment to the machine learning resource based on the evaluation of the user interactions with the user interface.

[0010] In some implementations, the first entity is mentioned in the video at a first point in time in the video, and the first entity card is displayed on the second portion of the user interface at the first point in time.

[0011] In some implementations, one or more entities include a second entity and one or more entity cards include a second entity card, and the method further includes providing a user interface to be displayed on a corresponding display of one or more user computing devices, for: displaying a second entity card in a contracted form on a third portion of the user interface while continuing to play the video, the contracted form of the second entity card referencing a second entity to be mentioned in the video at a second time point after the first time point in the video; and when the second entity is mentioned in the video at the second time point, displaying a second entity card in a fully expanded form on the third portion of the user interface while continuing to play the video, the fully expanded form of the second entity card including descriptive content related to the second entity.

[0012] In some implementations, the one or more entities include a second entity and the one or more entity cards include a second entity card, and the method further includes providing a user interface to be displayed on a corresponding display of one or more user computing devices, for: when the second entity is mentioned in the video while the video is playing, displaying the second entity card on a second portion of the user interface while the video continues to play, the second entity card including descriptive content related to the second entity, wherein the second entity card is displayed on the second portion of the display by replacing the first entity card when the second entity is mentioned in the video.

[0013] In some implementations, the method also includes providing a user interface to be displayed on corresponding displays of one or more user computing devices, for: when the first entity is mentioned in the video while the video is playing, displaying a notification user interface element on a third portion of the user interface while the video continues to play, the notification user interface element indicating that additional information related to the first entity is available; and in response to the first entity being mentioned in the video while the video is playing and in response to receiving a selection of the notification user interface element, displaying the first entity card on the second portion of the user interface while the video continues to play.

[0014] In some implementations, the first entity card includes at least one of a text summary providing information related to the first entity or an image related to the first entity.

[0015] In one or more example embodiments, a computer-implemented method for a user computing device includes: receiving a video for playback in a user interface; providing the video for display on a first portion of the user interface displayed on a display of the user computing device; and when a first entity is mentioned in the video while playing the video: providing a first entity card for display on a second portion of the user interface while continuing to play the video, wherein the first entity card includes descriptive content related to the first entity, and the first entity card is generated in response to automatically identifying the first entity from a transcription of content of the video.

[0016] In some implementations, the first entity is mentioned in the video at a first point in time in the video, and a first entity card is provided for display on the second portion of the user interface at the first point in time.

[0017] In some implementations, the method includes providing a collapsed second entity card referencing a second entity mentioned in the video at a second time point after a first time point in the video for display on a third portion of the user interface, and when the second entity is mentioned in the video at the second time point, expanding the collapsed second entity card to fully display the second entity card on the third portion of the user interface while continuing to play the video, the second entity card including descriptive content related to the second entity.

[0018] In some implementations, the method includes, when a second entity is mentioned in a video while the video is playing, providing a second entity card for display on a second portion of the user interface while the video continues to play, the second entity card including descriptive content related to the second entity, wherein the second entity card is provided for display on the second portion of the display by replacing the first entity card when the second entity is mentioned in the video.

[0019] In some implementations, the method includes providing one or more entity search user interface elements for display on a user interface, the one or more entity search user interface elements being configured to perform a search related to the first entity when selected.

[0020] In some implementations, the method includes providing one or more search query user interface elements for display on the user interface, the one or more search query user interface elements being configured, when selected, to perform a search related to a subject of the video other than the first entity.

[0021] In some implementations, the method includes utilizing machine learning resources to identify a first entity and generate a first entity card.

[0022] In some implementations, the first entity is an entity among multiple entities mentioned in the video that is determined by the machine learning resource to be an entity among the multiple entities mentioned in the video that is most likely to be searched for by a user viewing the video.

[0023] In some implementations, the method includes, when a first entity is mentioned in a video while the video is playing, providing a notification user interface element for display on a third portion of the user interface while the video continues to play, the notification user interface element indicating that additional information related to the first entity is available; and in response to receiving a selection of the notification user interface element, providing a first entity card for display on a second portion of the user interface while the video continues to play.

[0024] In some implementations, the first entity card includes a text summary providing information related to the first entity and / or an image related to the first entity.

[0025] In one or more example embodiments, a user computing device includes: a display; one or more memories for storing instructions; and one or more processors for executing instructions stored in the one or more memories to: receive a video for playback in a user interface; provide the video for display on a first portion of the user interface displayed on the display; and when a first entity is mentioned in the video while playing the video: provide a first entity card for display on a second portion of the user interface while continuing to play the video, wherein the first entity card includes descriptive content related to the first entity, and the first entity card is generated in response to automatically identifying the first entity from a transcription of content of the video.

[0026] In one or more example embodiments, a server system includes: one or more memories for storing instructions; and one or more processors for executing instructions stored in the one or more memories, for: obtaining a transcription of content from a video; applying machine learning resources to identify one or more entities that are most likely to be searched for by users watching the video; generating one or more entity cards for each of one or more entities based on the transcription of the content, each of the one or more entity cards including descriptive content related to a corresponding entity of the one or more entities; and providing a user interface to be displayed on corresponding displays of one or more user computing devices, for: playing a video on a first portion of the user interface, and when the video is played and a first entity of the one or more entities is mentioned in the video, displaying a first entity card on a second portion of the user interface, the first entity card including descriptive content related to the first entity.

[0027] In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) is provided that stores instructions that can be executed by one or more processors of a user computing device and / or a server system. In some implementations, the computer-readable medium stores instructions that may include instructions to cause one or more processors to perform one or more operations of any of the methods described herein (e.g., operations of the server system and / or operations of the user computing device). The computer-readable medium may store additional instructions for performing other aspects of the server system and user computing device and corresponding methods of operation, as described herein.

[0028] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description, drawings and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] A detailed discussion of example embodiments directed to those of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0030] Figure 1 Depicted are example systems according to one or more example embodiments of the present disclosure;

[0031] Figure 2 Depicts an example block diagram of a user computing device and a server computing system according to one or more example embodiments of the present disclosure;

[0032] Figure 3An example system for generating an entity card for a user interface according to one or more example embodiments of the present disclosure is depicted;

[0033] FIG. 4A to FIG. 4C depicts an example user interface on which one or more entity cards are presented during display of a video according to one or more example embodiments of the present disclosure;

[0034] FIG. 5A to FIG. 5C depicts an example user interface on which an entity card is presented during display of a video according to one or more example embodiments of the present disclosure;

[0035] FIG. 6A to FIG. 6C depicts an example user interface on which a notification user interface element is presented for displaying one or more entity cards during display of a video in accordance with one or more example embodiments of the present disclosure;

[0036] Fig. 7A and Figure 7B depicts an example user interface on which a timeline is presented during display of a video for displaying one or more entity cards according to one or more example embodiments of the present disclosure; and

[0037] Figures 8 to 10 Each illustrates a flow chart of an example non-limiting computer-implemented method according to one or more example embodiments. DETAILED DESCRIPTION

[0038] Reference will now be made to embodiments of the present disclosure, one or more examples of which are illustrated in the accompanying drawings, wherein like reference numerals represent like elements. Each example is provided by way of explanation of the present disclosure and is not intended to limit the present disclosure. In fact, it will be apparent to those skilled in the art that various modifications and changes may be made to the present disclosure without departing from the scope or spirit of the present disclosure. For example, a feature illustrated or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to encompass modifications and changes within the scope of the appended claims and their equivalents.

[0039] The terms used herein are used to describe example embodiments and are not intended to limit and / or constrain the present disclosure. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are also intended to include the plural forms. In the present disclosure, terms such as "including", "having", "comprising" are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0040] It should be understood that although the terms first, second, third, etc. may be used in this article to describe various elements, these elements are not limited by these terms. On the contrary, these terms are used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, a first element may be referred to as a second element, and a second element may be referred to as a first element.

[0041] It will be understood that when an element is referred to as being “connected” to another element, the expression encompasses examples of being directly connected or coupled as well as being connected or coupled with one or more other elements interposed therebetween.

[0042] The term "and / or" includes a combination of multiple related listed items or any of the multiple related listed items. For example, the scope of the expression or phrase "A and / or B" includes the item "A", the item "B" and the combination of the items "A and B".

[0043] In addition, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A; (2) at least one of B; and (3) at least one of A and at least one of B. Likewise, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A; (2) at least one of B; (3) at least one of C; (4) at least one of A and at least one of B; (5) at least one of A and at least one of C; (6) at least one of B and at least one of C; and (7) at least one of A, at least one of B, and at least one of C.

[0044] According to an example embodiment, when a user is watching a video on a user interface of a display of a user computing device and an entity (e.g., a concept, a term, a theme, etc.) is mentioned in the video, an entity card is provided on the user interface, the entity card including information about the entity that may help the user understand the content of the video. For example, if the user is watching a video about the Egyptian pyramids, an entity card may be provided to the user interface to provide further information about the term "sarcophagus," such as a definition of the term, a photo of the sarcophagus, etc. The entity card is provided to the user interface while the video continues to play, so that the user does not need to navigate away from the application or webpage playing the video to learn more about the entity (e.g., the term "sarcophagus"). Thus, information about potentially difficult concepts or concepts that the user may want to learn more about may be presented to the user to help the user gain a quick understanding of the concept without leaving the application or webpage playing the video, and the user does not need to perform a separate search for the concept.

[0045] In some implementations, the entire entity card may be visible to the user on the user interface, or a portion of the entity card may be visible on the user interface and the user may be able to select the portion of the entity card to expand the entity card to also see hidden portions of the entity card to obtain further information about the entity.

[0046] In some implementations, the entity card can be displayed on the user interface at the same time as the entity is mentioned in the video. That is, the display of the entity card is synchronized with the time when the entity is mentioned in the video. The entity card can be displayed on the user interface every time the entity is mentioned in the video, only the first time the entity is mentioned in the video, or selectively displayed when the entity is mentioned multiple times in the video.

[0047] In some implementations, a user interface element separate from the entity card may be provided on the user interface to allow a user to perform a search on the entity. For example, if a user wishes to obtain further information about the entity beyond the information provided in the entity card, the user may select a user interface element that causes a search to be performed on the entity and a search results page may be displayed on a display of the user computing device.

[0048] In some implementations, one or more user interface elements corresponding to the corresponding suggested search queries may be provided on the user interface separately from the entity card. The one or more user interface elements allow the user to perform a search for information other than the entity itself, such as about other entities or other topics covered in the video. For example, if the user wishes to obtain further information about other entities or other topics covered in the video, the user may select a corresponding user interface element that causes a search to be performed on the corresponding entity or topic and a search results page may be displayed on the display of the user computing device.

[0049] In some implementations, there may be multiple concepts in the video that may be potentially difficult for the user to understand (or that the user is potentially interested in). According to an example embodiment, when a user watches a video and mentions multiple concepts, an entity card for each of the multiple concepts in the video may be displayed on the user interface. For example, a separate entity card is provided on the user interface for each concept, and the entity card includes information about the concept that may help the user fully understand the content of the video. The entity card is provided to the user interface while the video continues to play, so that the user does not need to navigate away from the application or web page that plays the video. Therefore, information about potentially difficult concepts or concepts that the user may want to know more about can be presented to the user to help the user obtain a quick understanding of the concept without leaving the application or web page that plays the video, and the user does not need to perform a separate search for each of these concepts.

[0050] In some implementations, all entity cards associated with a video may be visible to a user on a user interface while the video is playing, or only some of the entity cards associated with a video may be visible to a user on a user interface while the video is playing. For example, one or more of the entity cards may be fully expanded so that the user can view the entire contents of the entity card, while some or all of the remaining entity cards may be displayed on the user interface in a collapsed or hidden form. For example, when in a collapsed or hidden form, a user may view a portion of an entity card and the portion of the entity card may include some identification information (e.g., an identification of a corresponding entity) so that the user can understand the relevance of the entity card. For example, a user may be able to select the portion of the entity card to expand the entity card so as to also view the hidden portion of the entity card to obtain further information about the entity.

[0051] In some implementations, entity cards associated with a video may be visible to a user on a user interface as the video progresses, and the user may not view the entity cards until the corresponding entity is mentioned in the video. For example, when a first entity is mentioned in a video (e.g., at a first time point), a first entity card for the first entity may be displayed on the user interface at some time during the video. The first entity card may be displayed for a predetermined amount of time (e.g., a time sufficient for an average user to read or view the content contained in the first entity card) while the video continues to play, or until the next entity is mentioned in the video, at which time another entity card is provided on the user interface. For example, when a second entity is mentioned in a video (e.g., at a second time point), a second entity card for the second entity may be displayed on the user interface at some time during the video.

[0052] In some implementations, the second entity card can be displayed on the user interface by replacing the first entity card (ie, by occupying some or all of the space on the user interface previously occupied by the first entity card).

[0053] In some implementations, the second entity card may be in a collapsed or hidden form, and when the second entity is mentioned in the video at the second point in time, the second entity card may be expanded to fully display the second entity card on the user interface. If the first entity card was not already in a collapsed or hidden form before the second entity card was displayed, the first entity card may become in a collapsed or hidden form when the second entity card is expanded. In some implementations, the first entity card and the second entity card may each be fully displayed on the user interface.

[0054] In some implementations, when an entity is mentioned in a video while the video is playing, a notification user interface element is displayed on the user interface while the video continues to play. For example, the notification user interface element indicates that additional information related to the entity is available. In response to receiving a selection of the notification user interface element, an entity card is displayed on the user interface while the video continues to play. The notification user interface element may include an image (e.g., a thumbnail) of the entity to further let the user know that the notification user interface element is associated with the entity and that an entity card about the entity is available.

[0055] In some implementations, while the video is playing, a timeline may be displayed on the user interface, the timeline indicating one or more time points along the timeline where information about one or more entities is available via corresponding entity cards. As the video approaches or crosses each time point, a corresponding entity card is displayed on the user interface to provide information about the entity. For example, a user interface element may be provided that, when selected, allows entity cards to cycle through the user interface as the video progresses along the timeline.

[0056] According to example embodiments disclosed herein, for one or more entities mentioned in a video, one or more entity cards are provided to be displayed on a user interface of a display of a user computing device while the video is being played. For example, entities for which entity cards are provided may be identified from a video using a machine learning resource. For example, when a machine learning resource determines or predicts that an entity among a plurality of candidate entities mentioned in a video is likely to be searched by a user watching the video (e.g., has a confidence value greater than a threshold, has a probability of being searched greater than a threshold, is determined to be the most likely to be searched by a user watching the video compared to other entities mentioned in the video, etc.), the machine learning resource may identify the entity as an entity for which an entity card should be generated. For example, a machine learning resource may select a candidate entity as an entity for which an entity card should be generated based on one or more of the following: the relevance of each of the plurality of candidate entities to the subject of the video, the relevance of each of the plurality of candidate entities to one or more other candidate entities among the plurality of candidate entities, the number of times a candidate entity is mentioned in a video, and the number of videos in which the candidate entity appears in a video collection stored in one or more databases.

[0057] According to example embodiments disclosed herein, a server system includes one or more servers that provide video and physical cards for display on a user interface of a display of a user computing device.

[0058] According to the examples disclosed herein, entity cards are generated based on entities identified from a video. For example, a speech recognition program can be used to analyze a video to perform automatic speech recognition and obtain a transcription (text transcript) of the speech. The next operation may include associating some or all of the text from the transcription with a knowledge graph entity to obtain a set of knowledge graph entities associated with the video. Training data for machine learning resources can be obtained by identifying knowledge graph entities that also appear in search queries from real users who watch videos. Additional operations for identifying entities from videos that may be important for understanding video content and / or for identifying entities from videos that are likely to be searched by users include: determining the degree of relevance of an entity to other entities in the video, determining the extent to which an entity uses tf-idf (term frequency-inverse document frequency) signals throughout the entire video collection, and determining the degree of relevance of an entity to the subject of the video (e.g., using a significant term signal based on the query). Machine learning resources can be trained by applying weights to candidate entities (the more frequently a term is mentioned in a video, the higher weights can be assigned to entities, lower weights can be assigned to entities that are too broad and frequently appear in the entire video collection, and higher weights can be assigned to entities that are more relevant to the subject of the video, etc.). Machine learning resources can be applied to evaluate and rank candidate entities among the candidate entities identified in the video. For example, one or more of the highest ranked candidate entities can be selected as entities for which entity cards are to be generated.

[0059] To generate an entity card, information about the entity may be obtained from a variety of sources to populate the entity card with text and / or images that provide more information about the entity. For example, information about the entity may be obtained from one or more websites, an electronic service that provides an overview of a topic, etc.

[0060] In some implementations, entity cards can be limited to less than a predetermined length and / or size (e.g., less than 100 words). Entity cards can include information including one or more of a title (e.g., a title identifying the entity, such as the title "Barack Obama"), a subtitle (e.g., "Former President of the United States" in relation to a previous entity and title example Barack Obama), and attribute information (to provide attributes for the source of the information). For example, images can be limited to a thumbnail size, a specified resolution, etc.

[0061] According to the examples disclosed herein, the next operation includes rendering a user interface to be displayed on a display of a user computing device, the user interface including a video and one or more entity cards associated with the video, the one or more entity cards may appear on the user interface at various points during the video or may be displayed (completely or partially) throughout the video.

[0062] For example, machine learning resources may be updated or adjusted based on an evaluation of user interactions with a user interface. For example, if a user does not typically interact with a particular entity card during a video, it may be suggested that the entity is not a term or topic related to the content of the video that the user is interested in or does not understand. Therefore, the machine learning resources may be adjusted to reflect the user interactions (or lack of these user interactions) with the user interface. Similarly, if a user typically interacts with a particular entity card during a video, it may be suggested that the entity is a term or topic related to the content of the video that the user is interested in or does not understand. Therefore, the machine learning resources may be adjusted to reflect the user interactions with the user interface.

[0063] The system and method of the present disclosure provide many technical effects and benefits. In one example, the present disclosure provides a user with an easy understanding or more understanding of entities (e.g., terms, concepts, themes, etc.) associated with a video and a similarly easy identification of the content that the user may wish to consume in more detail. By providing such a user interface, the user can understand the entity faster without performing a separate search and without stopping the video, thereby improving the user experience. The user can also determine whether they are interested in learning more about the entity by searching for the entity after providing a brief overview (e.g., excerpt) and / or image of the entity via the entity card. In this way, the user can avoid searching for the entity, loading the search results, and reading various content items that may or may not be related to the entity, which are more expensive in terms of calculation than simply reading the information that has been presented, thereby saving the time, processing, memory, and network resources of the computing system (whether a server device, a client device, or both). Similarly, since the user will not be discouraged by the complexity of the content of the video, and the user will be more likely to watch the video in its entirety, the convenience and experience of the user are improved. Because the machine learning resource predicts entities that the user is likely to search for and automatically displays information about the entity during the video, the user is not discouraged by getting the wrong search results due to spelling errors, thereby also improving the convenience and experience of the user. Therefore, the user can avoid loading / viewing content from the search results page, which in turn saves processing, memory, and network resources of the computing system. Because the user avoids the inconvenience of switching between the application or web page playing the video and the search results page, but can continue to play the video without interruption while the entity card is accessible or presented to the user during the video, thereby also improving the convenience and experience of the user.

[0064] In some cases, systems of the type disclosed herein may learn a balance of type, viewpoint, source, and / or other preferred attributes of content items, such as based on different content types, different user groups, different contexts such as timing and location, etc., through one or more different machine learning techniques (e.g., by training a neural network or other machine learning model). For example, data describing actions taken by one or more users with respect to a user interface in various contextual scenarios (e.g., "clicked", "liked", etc.) may be stored and used as training data to train (e.g., via supervised training techniques) one or more machine learning models to generate predictions, after training, that facilitate providing content (e.g., entity cards) in the user interface that satisfy the corresponding preferences of one or more users. In this way, system performance is improved by reducing human intervention, providing less user searching, and further conserving processing, memory, and network resources of a computing system (whether a server device, a client device, or both).

[0065] Now referring to the accompanying drawings, Figure 1 is an example system according to one or more example embodiments of the present disclosure. Figure 1 An example of a system is illustrated that includes a user computing device 100, a server computing system 300, a video data repository 370, an entity data repository 380, an entity card data repository 390, and external content 400, each of which can communicate with each other via a network 200.

[0066] For example, the user computing device 100 may include any of a personal computer, a smart phone, a laptop computer, a tablet computer, etc.

[0067] For example, the network 200 may include any type of communication network, such as a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), etc. For example, wireless communication between elements of example embodiments may be performed via wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), Ultra-Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Radio Frequency (RF) signals, etc. For example, wired communication between elements of example embodiments may be performed via a pair of cables, coaxial cables, fiber optic cables, Ethernet cables, etc.

[0068] For example, server computing system 300 may include one server or a combination of servers (eg, web servers, application servers, etc.) that communicate with each other, for example, in a distributed manner.

[0069] In an example embodiment, the server computing system 300 may obtain data or information from one or more of the video data repository 370, the entity data repository 380, the entity card data repository 390, and the external content 400. The video data repository 370, the entity data repository 380, and the entity card data repository 390 may be provided integrally with the server computing system 300 (e.g., as part of the memory 320 of the server computing system 300), or may be provided separately (e.g., remotely). In addition, the video data repository 370, the entity data repository 380, and the entity card data repository 390 may be combined into a single data repository (database), or may be a plurality of respective data repositories. The data stored in one data repository (e.g., the entity data repository 380) may overlap with some data stored in another data repository (e.g., the entity card data repository 390). In some implementations, one data repository (e.g., the entity card data repository 390) may reference data stored in another data repository (e.g., the entity data repository 380).

[0070] The video data repository 370 may store videos and / or information about videos. For example, the video data repository 370 may store a collection of videos. Videos may be stored, grouped, or categorized in any manner. For example, videos may be stored by genre or category, by title, by date (e.g., creation date or last modification date, etc.), etc. Information about videos may include location information (e.g., a uniform resource locator (URL)) about where the video may be stored or accessed. Information about videos may include transcription information of the video. For example, the computing system may be configured to perform automatic speech recognition (e.g., using one or more speech recognition programs) on videos stored in the video data repository 370 or stored elsewhere to obtain a transcription (i.e., a text transcript) of the video, and the text transcript of the video may be stored in the video data repository 370.

[0071] The entity data repository 380 may store information about entities identified from a textual transcript of a video. The identification of entities within a video will be explained in more detail below. Entity cards may be created or generated for entities identified from a video and may be stored in the entity data repository 380. For example, entities for which entity cards are provided may be identified from a video using machine learning resources. For example, when a machine learning resource determines or predicts that one of a plurality of candidate entities mentioned in a video is likely to be searched for by a user viewing the video (e.g., having a confidence value greater than a threshold, a probability of being searched greater than a threshold, being determined to be the most likely to be searched for by a user viewing the video compared to other entities mentioned in the video, etc.), the machine learning resource may identify the entity as an entity for which an entity card should be generated. For example, a machine learning resource may select a candidate entity as an entity for which an entity card should be generated based on one or more of the following: the relevance of each of a plurality of candidate entities to the subject of the video, the relevance of each of a plurality of candidate entities to one or more other candidate entities in a plurality of candidate entities, the number of times a candidate entity is mentioned in a video, and the number of videos in which the candidate entity appears in a video collection stored in one or more databases (e.g., the video data repository 370). For example, entities that may be identified from a video titled “The Myth of Icarus and Daedalus” may include “Greek Mythology,” “Crete,” “Icarus,” and “Daedalus.” These entities may be stored in entity data store 380 and may be associated with the video titled “The Myth of Icarus and Daedalus” that may be stored in or referenced in video data store 370.

[0072] The entity card data repository 390 may store information about entity cards created or generated based on entities identified from the video. The creation or generation of entity cards will be explained in more detail below. For example, entity cards may be created or generated by obtaining information about entities from various sources (e.g., from external content 400) to populate the entity cards with text and / or images that provide more information about the entity. For example, information about the entity may be obtained from one or more websites, electronic services that provide overviews of topics, and the like. For example, entity cards stored in the entity card data repository 390 may be limited to less than a predetermined length and / or size (e.g., less than 100 words). Entity cards stored in the entity card data repository 390 may include information including one or more of a title (e.g., a title that identifies an entity, such as the title "Barack Obama"), a subtitle (e.g., "Former President of the United States" with respect to a previous entity and title example Barack Obama), and attribute information (to provide attributes for the source of the information). For example, an image forming part or all of an entity card may be limited to a thumbnail size, a specified resolution, and the like.

[0073] The external content 400 may be any form of external content, including news articles, web pages, video files, audio files, image files, written descriptions, ratings, game content, social media content, photos, commercial offers, transportation methods, weather conditions, or other suitable external content. The user computing device 100 and the server computing system 300 may access the external content 400 via the network 200. The external content 400 may be searched by the user computing device 100 and the server computing system 300 according to known search methods, and the search results may be ranked according to relevance, popularity, or other suitable attributes (including location-specific filtering or promotion).

[0074] refer to Figure 1 In one example embodiment, a user of a user computing device 100 may send a request to view a video to be provided via the server computing system 300. For example, the video may be obtained via a video sharing website, an online video hosting service, a streaming video service, or other video platform. For example, the user may view the video on the display 160 of the user computing device via the video application 130 or the web browser 140. In response to receiving the request, the server computing system 300 is configured to provide a user interface to be displayed on the display of the user computing device 100, for: playing the video on a first portion of the user interface, and when the video is played and a first entity of one or more entities is mentioned in the video, displaying a first entity card on a second portion of the user interface, the first entity card including descriptive content related to the first entity. As discussed in more detail below FIG. 4A to FIG. 7BAn example user interface is provided, illustrating displaying a video on a first portion of the user interface and displaying a first entity card on a second portion of the user interface. In some implementations, the server computing system 300 may store or retrieve a user interface including the video and the first entity card and provide the user interface in response to a request from a user computing device 100 to watch a video. In some implementations, the server computing system 300 may store or retrieve a video and a first entity card associated with the video, dynamically generate a user interface including the video and the first entity card in response to a request from a user computing device 100 to watch a video, and send the user interface to the user computing device 100. In some implementations, the server computing system 300 may store or retrieve a video, may store or retrieve one or more entities that have been identified from the video, dynamically generate one or more entity cards from the identified one or more entities, and dynamically generate a user interface including the video and one or more entity cards (e.g., including the first entity card) in response to a request from a user computing device 100 to watch a video. In some implementations, the server computing system 300 may store or retrieve a video, dynamically identify one or more entities from the video (e.g., after transcribing the video or obtaining a transcript of the video), dynamically generate one or more entity cards from the identified one or more entities, and dynamically generate a user interface including the video and one or more entity cards (e.g., including a first entity card) in response to a request from a user computing device 100 to view the video.

[0075] Reference now Figure 2 , an example block diagram of a user computing device and a server computing system according to one or more example embodiments of the present disclosure will now be described.

[0076] User computing device 100 may include one or more processors 110, one or more memory devices 120, video application 130, web browser 140, input device 150, and display 160. Server computing system 300 may include one or more processors 310, one or more memory devices 320, and user interface generator 330.

[0077] For example, the one or more processors 110, 310 may be any suitable processing device that may be included in the user computing device 100 or the server computing system 300. For example, such processors 110, 310 may include one or more of a processor, a processor core, a controller and an arithmetic logic unit, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an image processor, a microcomputer, a field programmable array, a programmable logic unit, an application specific integrated circuit (ASIC), a microprocessor, a microcontroller, etc. and combinations thereof, including any other device that can respond and execute instructions in a defined manner. The one or more processors 110, 310 may be a single processor or a plurality of processors that are operably connected (e.g., in parallel).

[0078] The memory 120, 320 may include one or more non-transitory computer-readable storage media, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM) and flash memory, USB drive; volatile memory devices, such as random access memory (RAM), hard disk, floppy disk, Blu-ray disk; or optical media, such as CD ROM disk and DVD; and combinations thereof. However, examples of the memory 120, 320 are not limited to the above description, and the memory 120, 320 may be implemented by other various devices and structures as will be understood by those skilled in the art.

[0079] For example, memory 120 may store instructions that, when executed, cause one or more processors 110 to: receive a video for playback in a user interface; provide the video for display on a first portion of the user interface displayed on a display; and when a first entity is mentioned in the video while the video is playing, provide a first entity card for display on a second portion of the user interface while the video continues to play, as described in an example according to the present disclosure. For example, memory 320 may store instructions that, when executed, cause one or more processors 310 to provide a user interface to be displayed on a respective display of one or more user computing devices, for: playing a video on a first portion of the user interface; and when a video is played and a first entity of one or more entities is mentioned in the video, displaying a first entity card on a second portion of the user interface, as described in an example according to the present disclosure.

[0080] The memory 120 may also include data 122 and instructions 124 that may be retrieved, manipulated, created, or stored by the one or more processors 110. In some example embodiments, such data may be accessed and used as input to: receive a video for playback in a user interface; provide the video for display on a first portion of the user interface displayed on a display; and when a first entity is mentioned in the video while the video is playing, provide a first entity card for display on a second portion of the user interface while the video continues to play, as described in accordance with examples of the present disclosure. The memory 320 may also include data 322 and instructions 324 that may be retrieved, manipulated, created, or stored by the one or more processors 310. In some example embodiments, such data may be accessed and used as input to provide a user interface to be displayed on a respective display of one or more user computing devices for: playing a video on a first portion of the user interface; and when a video is played and a first entity of one or more entities is mentioned in the video, displaying a first entity card on a second portion of the user interface, as described in accordance with examples of the present disclosure.

[0081] exist Figure 2 , the user computing device 100 includes a video application 130, which may also be referred to as a video player or video streaming application. The video application 130 enables a user of the user computing device 100 to view videos provided through the application and displayed on a user interface of the display 160. For example, the video may be provided to the video application 130 via the server computing system 300.

[0082] exist Figure 2 , the user computing device 100 includes a web browser 140, which may also be referred to as an Internet browser or simply a browser. The web browser 140 may be any browser for accessing websites or web pages (e.g., via the World Wide Web). A user of the user computing device 100 may provide input (e.g., a URL) to the web browser 140 to obtain content (e.g., a video) and display the content on a display 160 of the user computing device. For example, a rendering engine of the web browser 140 may display the content on a user interface (e.g., a graphical user interface). For example, a video may be provided or obtained using the web browser 140 via a video sharing website, an online video hosting service, a streaming video service, or other video platform. For example, the video may be provided to the web browser 140 via a server computing system 300.

[0083] exist Figure 2In the embodiment of the present invention, the user computing device 100 includes an input device 150 configured to receive input from a user and may include, for example, one or more of the following: a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, a button, a switch, an electronic pen or stylus, a gesture recognition sensor (e.g., for recognizing a user's gestures, including movement of a body part), an input sound device or a voice recognition sensor (e.g., a microphone for receiving voice commands), a trackball, a remote controller, a portable (e.g., a cellular or smart) phone, a tablet PC, a pedal or foot switch, a virtual reality device, etc. The input device 150 may also include a tactile device for providing tactile feedback to the user. For example, the input device 150 may also be embodied by a touch-sensitive display with touch screen capabilities. The user of the user computing device 100 may use the input device 150 to provide input for requesting to watch a video, provide input for selecting a user interface element displayed on the user interface, enter a search query, etc.

[0084] exist Figure 2 In the embodiment of the present invention, the user computing device 100 includes a display 160 that displays information viewable by a user, such as on a user interface (e.g., a graphical user interface). For example, the display 160 may be a non-touch-sensitive display or a touch-sensitive display. For example, the display 160 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED), a flexible display, a 3D display, a plasma display panel (PDP), a cathode ray tube (CRT) display, etc. However, the present disclosure is not limited to these example displays and may include other types of displays.

[0085] According to the example embodiments described herein, the server computing system 300 may include one or more processors 310 and memory 320 previously discussed above. The server computing system 300 may also include a user interface generator 330. For example, the user interface generator 330 may include a video provider 332, an entity card provider 334, and a search query provider 336. The user interface generator 330 may generate a user interface for display on the display 160 of the user computing device 100. The user interface may include various portions for displaying various content. For example, the user interface may display a video on a first portion of the user interface, display an entity card on a second portion of the user interface, and display various user interface elements on other portions of the user interface (e.g., displaying suggested search queries on a third portion of the user interface).

[0086] The video provider 332 may include information or content (e.g., video content) that may be used to render a user interface so that the video can be displayed and played back at the user computer device 100. In some implementations, the video provider 332 may be configured to retrieve the video from the video data repository 370 (e.g., in response to a request for the video from the user computing device 100).

[0087] The entity card provider 334 may include information or content (e.g., an entity card including a text summary and / or an image) that can be used to render a user interface, such that the entity card can be displayed on the user interface with the video at the user computer device 100. In some implementations, the entity card provider 334 may be configured to retrieve the entity card associated with the video from the entity card data store 390.

[0088] The search query provider 336 may include information or content (e.g., suggested search queries) that may be used to render a user interface, such that the search query may be displayed on the user interface at the user computer device 100 along with at least one of the video or the entity card. In some implementations, the search query provider 336 may generate one or more suggested search queries based on entities identified for the video and / or based on the content of the entity card associated with the video. For example, the search query provider 336 may generate suggested search queries based on previous user searches in response to viewing the video, based on previous user searches about the subject of the video, and the like.

[0089] will be in view of Figures 3 to 7B The following instructions shown in Figures 8 to 10 Additional aspects of the user computing device 100 and the server computing system 300 are discussed with reference to flowcharts of FIG.

[0090] Figure 3 An example system 3000 for generating entity cards for a user interface according to one or more example embodiments of the present disclosure is depicted. The example system 3000 includes a video 3010, an entity generator 3020, external content 400, an entity card generator 3030, a user interface entity card renderer 3040, and a user interface generator 3050. Each of the video 3010, the entity generator 3020, the entity card generator 3030, the user interface entity card renderer 3040, and the user interface generator 3050 can be part of a server computing system 300. In some implementations, the video 3010, the entity generator 3020, the entity card generator 3030, the user interface entity card renderer 3040, and the user interface generator 3050 can be part of a server computing system 300 (e.g., as part of a single server or distributed among multiple servers and / or data repositories).

[0091] refer to Figure 3, entity cards may be generated based on entities identified from a video. For example, a video 3010 may be provided to an entity generator 3020 for identifying one or more entities from the video 3010. For example, the video 3010 may be obtained from or stored in a video data repository 370. For example, the entity generator 3020 may include a video transcriber 3022, a signal annotator 3024, and a machine learning resource 3026. In some implementations, a video provider 332 may provide the video 3010 to the entity generator 3020, or a third-party application or service provider may provide the video 3010.

[0092] The video transcriber 3022 is configured to transcribe the video 3010. For example, the video transcriber 3022 may include a speech recognition program that analyzes the video 3010 and performs automatic speech recognition to obtain a transcription (e.g., a text transcript) of the speech from the video 3010. In some implementations, the transcription of the video 3010 may be obtained from the video data repository 370 or from a third-party application or service provider that generates a transcription of the video, such as through automatic speech recognition, and the entity generator 3020 may not include the video transcriber 3022.

[0093] The signal annotator 3024 is configured to associate some or all of the transcribed text obtained from the video transcriber 3022 or the video data repository 370 with knowledge graph entities to obtain a set of knowledge graph entities associated with the video 3010. A knowledge graph may generally refer to a description of interlinkages and / or interrelationships between objects, events, situations, or concepts, for example in the form of a graphical structured data model.

[0094] Machine learning resource 3026 is configured to predict which entities from video 3010 (e.g., which knowledge graph entities from video 3010) are most likely to be mentioned in the video or searched for from multiple entities identified in the video by a user watching video 3010 (e.g., have a confidence value greater than a threshold that a user is likely to search for the entity, have a probability of being searched for greater than a threshold, are determined to be most likely to be searched for by a user watching the video compared to other entities mentioned in the video, etc.). Training data for machine learning resource 3026 can be obtained by identifying or matching knowledge graph entities that also appear in search queries from real users watching video 3010.

[0095] The machine learning resources 3026 may be configured to identify entities for which entity cards are to be generated from the video 3010 by determining how relevant the entity is to other entities in the video 3010, determining how widespread the entity is across a corpus of videos (e.g., stored in the video data repository 370) using a tf-idf (term frequency-inverse document frequency) signal, and determining how relevant the entity is to the subject matter of the video (e.g., using a query-based salient term signal). For example, the machine learning resources 3026 may select candidate entities as entities for which entity cards are to be generated based on one or more of the following: relevance of each of a plurality of candidate entities to the subject matter of the video, relevance of each of a plurality of candidate entities to one or more other candidate entities in a plurality of candidate entities, the number of times the candidate entity is mentioned in the video, the number of videos in which the candidate entity appears in a corpus of videos stored in one or more databases (e.g., the video data repository 370), and the like.

[0096] The machine learning resource 3026 may be trained by applying weights to candidate entities (e.g., a higher weight may be assigned to an entity the more frequently a term is mentioned in the video 3010, a lower weight may be assigned to an entity that is too broad and frequently appears in the entire video, a higher weight may be assigned to an entity that is more relevant to the subject of the video, etc.). The machine learning resource 3026 may be configured to evaluate and rank candidate entities among the candidate entities identified in the video 3010. For example, the entity generator 3020 may be configured to select one or more of the highest-ranked candidate entities identified by the machine learning resource 3026 as entities for which entity cards are to be generated. For example, an entity among a plurality of candidate entities mentioned in or identified from the video 3010 may be identified by the machine learning resource 3026 as an entity for which an entity card should be generated when the machine learning resource 3026 determines or predicts that the entity is likely to be searched for by a user viewing the video 3010.

[0097] For example, the machine learning resource 3026 may be updated or adjusted based on an evaluation of user interactions with a user interface provided to the user computing device 100. For example, if a user does not typically interact with a particular entity card while watching a video 3010, it may be suggested that the entity is not a term or topic related to the content of the video 3010 that the user is interested in or does not understand. Therefore, the machine learning resource 3026 may be adjusted to reflect the user interactions with the user interface (or the lack of such user interactions). Similarly, if a user typically interacts with a particular entity card while watching a video 3010, it may be suggested that the entity is a term or topic related to the content of the video 3010 that the user is interested in or does not understand. Therefore, the machine learning resource 3026 may be adjusted to reflect the user interactions with the user interface. For example, the machine learning resource 3026 may regenerate entities associated with the video 3010 according to a preset schedule (e.g., every two weeks), which may result in identifying different entities compared to previous entities identified by the entity generator 3020. Therefore, after updating or adjusting the machine learning resource 3026, different entity cards may be displayed in association with the video 3010. The machine learning resource 3026 can also regenerate entities associated with the video 3010 based on a user input request to the entity generator 3020.

[0098] The entity generator 3020 may output one or more entities that are determined or predicted by the machine learning resource 3026 and that may be searched by the user watching the video to the entity card generator 3030. In response to receiving the one or more entities from the entity generator 30202, the entity card generator 3030 may be configured to generate an entity card for each of the one or more entities. For example, the entity card generator 3030 may be configured to obtain information about the entity from various sources based on the identity of the entity and the associated metadata of the entity to populate the entity card with text and / or images that provide more information about the entity. For example, the entity card generator 3030 may be configured to create or generate an entity card by obtaining information about the entity from various sources (e.g., from the external content 400) to populate the entity card with text and / or images that provide more information about the entity. For example, the information about the entity may be obtained from one or more websites, an electronic service that provides an overview of a topic, and the like. For example, the entity card generator 3030 may store the generated entity card in the entity card data repository 390. For example, the entity card may be limited to less than a predetermined length and / or size (e.g., less than 100 words). For example, an entity card may include information including one or more of a title (e.g., a title identifying an entity, such as the title "Barack Obama"), a subtitle (e.g., "Former President of the United States" with respect to a previous entity and title example Barack Obama), and attribute information (to provide attributes for the source of the information). For example, an image forming part or all of an entity card may be limited to a thumbnail size, a specified resolution, etc. In some implementations, the entity card generator 3030 may correspond to the entity card provider 334.

[0099] The user interface entity card renderer 3040 may be configured to render an entity card to be provided for at least a portion of a user interface to be provided for display on the display 160 of the user computing device 100 .

[0100] The user interface generator 3050, which may correspond to the user interface generator 330, may be configured to combine the video 3010 and the rendered entity card to generate a user interface to be provided for display on the display 160 of the user computing device 100. In some implementations, the rendering of the entity card, or at least the rendering of the user interface including the video and the entity card, may be performed at the user computing device 100. In some implementations, the rendering of the entity card, or at least the rendering of the user interface including the video and the entity card, may be performed at the server computing system 300.

[0101] FIG. 4A to FIG. 4C An example user interface is depicted on which one or more entity cards are presented during display of a video according to one or more example embodiments of the present disclosure. Figure 4A, shows an example user interface 4000 as displayed on the display 160 of the user computing device 100. The user interface 4000 includes a section in which a video 4010 is being played on a first portion 4012 of the user interface 4000. The user interface 4000 also includes a section entitled "In This Video" 4020 displayed on a second portion 4022 of the user interface 4000 that summarizes the content of the video 4010 at different points in time during the video 4010. The user interface 4000 also includes a section entitled "Related Topics" 4030 that includes an entity card 4040 displayed on a third portion 4032 of the user interface 4000. In this example, the entity that has been identified from the video 4010 is "King Tutankhamun," and the entity card 4040 includes descriptive content related to King Tutankhamun, including an image of the pharaoh and a text summary about the pharaoh. For example, entities may be identified by applying machine learning resources to identify one or more entities that are most likely to be searched for by users viewing video 4010 based on a transcription from the content of video 4010.

[0102] In some implementations, the user of the user computing device 100 can scroll down the user interface 4000 to view the content displayed in the entity card 4040 in the third portion 4032. Figure 4B , shows an example user interface 4000 as displayed on the display 160 of the user computing device 100. For example, Figure 4B As shown, in response to the user scrolling down to view the entity card 4040, the video 4010 may be maintained (anchored) at an upper portion of the user interface 4000' on the first portion 4012' of the user interface 4000'. The user interface 4000' also includes a section titled "Related Topics" that includes the entity card 4040 displayed on the second portion 4022' of the user interface 4000'. In this example, the entity card 4040 includes a title 4042 (King Tutankhamun) of the entity card 4040, a subtitle 4044 (Pharaoh), descriptive content 4046 (a textual summary and a thumbnail), and attributes 4048 that reference the source of the descriptive content 4046.

[0103] For example, in other portions of user interface 4000', additional user interface elements and entity cards may be provided. For example, suggested search queries 4060 may be provided on user interface 4000'. Here, suggested search queries 4060 correspond to entity search user interface elements that, when selected, are configured to perform a search related to an entity.

[0104] For example, in the third portion 4032' of the user interface 4000', a second entity card 4070 (Valley of the Kings) is also displayed. Here, in contrast to the entity card 4040 which is in an expanded form, the second entity card 4070 is displayed in a collapsed or folded form. The second entity card 4070 displayed in a collapsed or folded form includes sufficient identifying information (e.g., a title of the entity card such as "Valley of the Kings" and thumbnails related to the Valley of the Kings) so that the user understands what topic, concept, or theme the second entity card 4070 relates to. For example, the user can expand the second entity card 4070 by selecting the user interface element 4080 to obtain a more complete description of the second entity (Valley of the Kings), and the user can collapse the entity card 4040 by selecting the user interface element 4050.

[0105] For example, in the third portion 4032' of the user interface 4000', additional suggested search queries 4090 (e.g., "Howard Carter" and "mummification process") are also displayed. Here, the suggested search queries 4090 correspond to search query user interface elements that, when selected, are configured to perform a search related to topics of the video 4010 other than the first entity or the second entity (e.g., topics about the video other than any entity identified from the video 4010).

[0106] In an example embodiment of the present disclosure, video 4010 continues to play while the user views entity card 4040 and / or second entity card 4070. Therefore, viewing of video 4010 is not interrupted when the user wishes to learn more about the entity identified in video 4010. The user may obtain sufficient information about the entity from the entity card presented on user interface 4000'.

[0107] In some implementations, an entity may be mentioned in video 4010 at a first point in time in video 4010, and entity card 4040 may be provided for display on a portion of the user interface (e.g., third portion 4032 of user interface 4000 or second portion 4022′ of user interface 4000′) at the first point in time. Thus, entity card 4040 may be provided for display at a time synchronized with the discussion of the entity in video 4010.

[0108] like Figure 4BAs shown, second entity card 4070 is displayed in a collapsed form on third portion 4032 of user interface 4000'. Second entity card 4070 may identify or reference a second entity (Valley of the Kings) to be mentioned in video 4010 at a second time point after the first time point in video 4010. In some implementations, when the second entity is mentioned in video 4010 at the second time point, collapsed second entity card 4070 may be automatically expanded to fully display second entity card 4070 on third portion 4032' of user interface 4000' while video 4010 continues to play, and second entity card 4070 includes descriptive content related to the second entity. In some implementations, when a second entity is mentioned in video 4010 at a second point in time, the collapsed second entity card 4070 may be automatically expanded to fully display the second entity card 4070 on the second portion 4022' of the user interface 4000' while continuing to play the video 4010, the second entity card 4070 replacing the entity card 4040 on the user interface 4000' when the second entity is mentioned in the video 4010 and including descriptive content related to the second entity.

[0109] refer to Figure 4C , shows an example user interface 4000 as displayed on the display 160 of the user computing device 100. The user interface 4000" displays a search results page 4092 obtained in response to a user selecting a suggested search query 4060 provided on the user interface 4000". The suggested search query 4060 corresponds to an entity search user interface element that, when selected, is configured to perform a search related to an entity. Figure 4C In the example of , the entity search is for the entity King Tutankhamun and the search box 4092 is automatically populated with the entity, thereby saving the user from having to spell the entity, which may have been misspelled by the user, resulting in errors in the search results and frustration on the part of the user, as well as wasting computing resources on inefficient or erroneous searches.

[0110] As mentioned above about FIG. 4A to FIG. 4CAs discussed in the example of , in some implementations, all entity cards associated with a video (e.g., entity card 4040 and second entity card 4070) may be visible to a user on a user interface while the video is playing, or only some of the entity cards associated with a video may be visible to a user on a user interface while the video is playing. For example, one or more of the entity cards (e.g., entity card 4040 and / or second entity card 4070) may be fully expanded so that the user can view the entire contents of the entity card, while some or all of the remaining entity cards may be displayed on the user interface in a collapsed or folded form. For example, when in a collapsed or folded form, a user may view a portion of the entity card and the portion of the entity card may include some identification information (e.g., an identification of the corresponding entity) so that the user can understand the relevance of the entity card. For example, a user may be able to select a user interface element or collapse some portion of the visible portion of the entity card to expand the entity card so as to also view a hidden portion of the entity card to obtain further information about the entity. In some implementations, when the corresponding entity is mentioned in the video, the entity card may automatically expand to fully display the entity card on the user interface. To save space on the user interface, if the previously displayed entity card was not already in a contracted or collapsed form before the second entity card was displayed, the previously displayed entity card may become in a contracted or collapsed form when the second entity card is expanded. In some implementations, all entity cards may each be fully displayed on the user interface throughout the video.

[0111] FIG. 5A to FIG. 5C Depicted is an example user interface on which entity cards are presented during display of a video in accordance with one or more example embodiments of the present disclosure. Figure 5A , shows an example user interface 5000 as displayed on the display 160 of the user computing device 100. The user interface 5000 includes a section in which a video 5010 is being played on a first portion 5012 of the user interface 5000. For example, a transcript of the audio portion of the video (i.e., closed captions) 5014 may be provided for display in the video 5010 displayed on the user interface 5000. The user interface 5000 also includes a section entitled "Topics to Explore" 5020 displayed on a second portion 5022 of the user interface 5000, the section including one or more entity cards to be displayed at different points in time during the video 5010. For example, the second portion 5022 may include a first sub-portion 5032 including the first entity card 5030 and a second sub-portion 5052 including at least a portion of the second entity card 5050.

[0112] exist Figure 5AIn the example of , the first entity that has been identified from the video 5010 is "Greek Mythology", and the first entity card 5030 includes descriptive content related to Greek mythology, including images related to Greek mythology and a textual overview about Greek mythology. For example, the first entity card 5030 may also include information about the time point in the video 5010 at which the first entity is being discussed (e.g., 8 seconds into the video 5010). For example, the second entity that has been identified from the video 5010 is "Crete", and the second entity card 5050 includes descriptive content related to Crete, including images related to Crete and a textual overview about Crete. For example, the second entity card 5050 may also include information about the time point in the video 5010 at which the second entity is being discussed (e.g., 10 seconds into the video 5010).

[0113] For example, the first sub-portion 5032 may include one or more user interface elements. For example, a suggested search query 5040 may be provided on the first sub-portion 5032. Here, the suggested search query 5040 corresponds to an entity search user interface element that, when selected, is configured to perform a search related to an entity (e.g., a search for the first entity "Greek mythology").

[0114] like Figure 5A As shown, second entity card 5050 is only partially displayed on user interface 5000 because the topic or concept regarding the second entity has not yet been discussed in video 5010. For example, as each entity is discussed while the video continues to play, the entity cards in second portion 5020 may rotate, for example, in a carousel. Thus, video 5010 continues to play while the user views first entity card 5030 and second entity card 5050. Therefore, if the user wishes to learn more about the entity identified from video 5010 and obtains enough information about the entity from the entity card presented on user interface 5000 (or subsequent user interfaces 5000', 5000", etc.) to understand the content of video 5010, viewing of video 5010 is not interrupted and there is no need to perform a separate search and / or stop video 5010.

[0115] refer to Figure 5B, shows an example user interface 5000 as displayed on the display 160 of the user computing device 100. The user interface 5000' includes a section in which a video 5010 is being played on a first portion 5012 of the user interface 5000'. The user interface 5000' also includes a section entitled "Topics to Explore" 5020 displayed on a second portion 5022 of the user interface 5000', which includes one or more entity cards to be displayed at different points in time during the video 5010. For example, the second portion 5022 may include a first sub-portion 5032' including at least a portion of the first entity card 5030, a second sub-portion 5052' including the second entity card 5050, and a third sub-portion 5062' including at least a portion of the third entity card 5060.

[0116] exist Figure 5B In the example of , second entity card 5050 includes descriptive content related to Crete, including images related to Crete and a textual summary about Crete. For example, second entity card 5050 may also include information about the time point in video 5010 when the second entity is being discussed (e.g., 10 seconds into video 5010). For example, the third entity identified from video 5010 is "Ikaros", and third entity card 5060 includes descriptive content related to Ikaros, including images related to Ikaros and a textual summary about Ikaros. For example, third entity card 5060 may also include information about the time point in video 5010 when the third entity is being discussed (e.g., Figure 5C Information shown at 13 seconds into video 5010).

[0117] For example, the second sub-portion 5052' may include one or more user interface elements. For example, a suggested search query 5040' may be provided on the second sub-portion 5052'. Here, the suggested search query 5040' corresponds to an entity search user interface element, which, when selected, is configured to perform a search related to the entity (e.g., a search for the second entity "Crete").

[0118] like Figure 5BAs shown, first entity card 5030 and third entity card 5060 are only partially displayed on user interface 5000' because the topic or concept about the first entity has been discussed in video 5010, and the topic or concept about the third entity has not yet been discussed in video 5010. For example, as each entity is discussed while the video continues to play, the entity cards in second portion 5020 can rotate, for example, in a carousel. Therefore, while the user views first entity card 5030, second entity card 5050, and third entity card 5060, video 5010 continues to play. Therefore, if the user wishes to learn more about the entity identified from video 5010 and obtains sufficient information about the entity from the entity cards presented on user interface 5000, 5000', etc. to understand the content of video 5010, viewing of video 5010 is not interrupted and there is no need to perform a separate search and / or stop video 5010.

[0119] refer to Figure 5C , shows an example user interface 5000" as displayed on the display 160 of the user computing device 100. The user interface 5000" includes a segment in which a video 5010 is being played on a first portion 5012 of the user interface 5000". The user interface 5000" also includes a segment titled "Topics to Explore" 5020 displayed on a second portion 5022 of the user interface 5000", which includes one or more entity cards to be displayed at different points in time during the video 5010. For example, the second portion 5022 may include a first sub-portion 5052", which includes at least a portion of the second entity card 5050, a second sub-portion 5062", and a third sub-portion 5072", the first sub-portion including at least a portion of the second entity card 5050, the second sub-portion including the third entity card 5060, and the third sub-portion including at least a portion of the fourth entity card 5070.

[0120] exist Figure 5C In the example of , third entity card 5060 includes descriptive content related to Icarus, including images related to Icarus and a textual summary about Icarus. For example, third entity card 5060 may also include information about the time point in video 5010 at which the third entity is being discussed (e.g., 13 seconds into video 5010). For example, a fourth entity identified from video 5010 may include "Daedalus", and fourth entity card 5070 includes descriptive content related to Daedalus, including images related to Daedalus and a textual summary about Daedalus. For example, fourth entity card 5070 may also include information about the time point in video 5010 at which the fourth entity is being discussed (e.g., 20 seconds into video 5010).

[0121] For example, the second sub-portion 5062" may include one or more user interface elements. For example, a suggested search query 5040" may be provided on the second sub-portion 5062". Here, the suggested search query 5040" corresponds to an entity search user interface element, which, when selected, is configured to perform a search related to the entity (e.g., a search for the second entity "Icarus").

[0122] like Figure 5C As shown, second entity card 5050 and fourth entity card 5070 are only partially displayed on user interface 5000″ because the topic or concept regarding the second entity has been discussed in video 5010, and the topic or concept regarding the fourth entity has not yet been discussed in video 5010. For example, as each entity is discussed while the video continues to play, the entity cards in second portion 5020 may rotate, for example, in a carousel manner. Thus, video 5010 continues to play while the user views first entity card 5030, second entity card 5050, third entity card 5060, fourth entity card 5070, and so on. Therefore, if the user wishes to learn more about the entities identified in video 5010 and obtains sufficient information about the entities from the entity cards presented on user interfaces 5000, 5000', 5000″, etc. to understand the content of video 5010, viewing of video 5010 is not interrupted and there is no need to perform a separate search and / or stop video 5010.

[0123] As mentioned above about FIG. 5A to FIG. 5C As discussed in the example of , in some implementations, as the video progresses, entity cards associated with the video (e.g., entity cards 5030, 5050, 5060, 5070) may be visible to the user on the user interface, and the user may not view the entity cards in their entirety until the corresponding entities are mentioned in the video. For example, when a first entity is mentioned in the video (e.g., at a first time point), a first entity card for the first entity may be displayed on the user interface at some time during the video. The first entity card may be displayed for a predetermined amount of time (e.g., a time sufficient for an average user to read or view the content contained in the first entity card) while the video continues to play, or for a certain time until the next entity is mentioned in the video, at which time another entity card is provided on the user interface. For example, when a second entity is mentioned in the video (e.g., at a second time point), a second entity card for the second entity may be displayed on the user interface at some time during the video. In some implementations, the second entity card may be displayed on the user interface by replacing the first entity card (i.e., by occupying some or all of the space on the user interface previously occupied by the first entity card).

[0124] FIG. 6A to FIG. 6CAn example user interface is depicted on which a notification user interface element is presented during display of a video for displaying one or more entity cards in accordance with one or more example embodiments of the present disclosure. Fig. 6A , shows an example user interface 6000 as displayed on the display 160 of the user computing device 100. The user interface 6000 includes a section in which a video 6010 is playing on a first portion 6012 of the user interface 6000. The user interface 6000 also includes a section entitled “Related Searches” 6020 displayed on a second portion 6022 of the user interface 6000 that includes one or more suggested search queries 6030 related to the video 6010.

[0125] refer to Figure 6B , shows an example user interface 6000' as displayed on the display 160 of the user computing device 100. The user interface 6000' has Fig. 6A 6000 ′, except that notification user interface element 6040 is displayed in response to an entity mentioned in video 6010 (an entity for which an entity card exists). For example, when an entity is mentioned in video 6010 while video 6010 is playing, notification user interface element 6040 is displayed on user interface 6000 ′ while video 6010 continues to play. For example, notification user interface element 6040 indicates that additional information related to the entity is available. In response to receiving a selection of notification user interface element 6040, the entity card is displayed on the user interface while video 6010 continues to play. Notification user interface element 6040 may include some identification information of the entity (e.g., an identification and / or image such as a thumbnail of the corresponding entity) to further let the user know that notification user interface element 6040 is associated with the entity and enable the user to understand the relevance of the available entity cards.

[0126] refer to Figure 6C , shows an example user interface 6000' as displayed on the display 160 of the user computing device 100. For example, the user interface 6000' may be displayed in response to a user selecting the notification user interface element 6040 as displayed on the user interface 6000'. The user interface 6000' includes a segment in which a video 6010 is playing on a first portion 6012 of the user interface 6000. The user interface 6000 also includes a segment entitled "Related Searches" 6020 displayed on a second portion 6022 of the user interface 6000", however, the second portion 6022 is obscured by a new segment entitled "Mentioned Topics" 6050 on the second portion 6022 that is overlaid (e.g., as a pop-up window).

[0127] The section titled "Topics Mentioned" 6050 includes an entity card 6060 overlaid on the second portion 6022. In this example, the entity card 6060 includes a title 6062 of the entity card 6060 (King Tutankhamun), a subtitle 6064 (Pharaoh), descriptive content 6066 (a text summary and a thumbnail image), and attributes 6068 that reference the source of the descriptive content 6066.

[0128] For example, in other portions of the section titled "Topics Mentioned" 6050, additional user interface elements and entity cards may be provided. For example, suggested search queries 6070 may be provided. Here, suggested search queries 6070 correspond to an entity search user interface element that, when selected, is configured to perform a search related to the entity. For example, at least a portion of a second entity card 6080 is also provided. Second entity card 6080 may be related to the next entity to be discussed during the video.

[0129] In some implementations, notification user interface element 6040 can be displayed on user interface 6000' at the same time that an entity is mentioned in video 6010. In some implementations, notification user interface element 6040 can be displayed on user interface 6000' throughout video 6010, and selection of notification user interface element 6040 can cause a section titled "Topics Mentioned" 6050 to be displayed. For example, section titled "Topics Mentioned" 6050 is overlaid (e.g., as a pop-up window) on second portion 6022 and can remain open until closed (e.g., via user interface element 6090). For example, when an entity associated with second entity card 6080 is discussed in video 6010, second entity card 6080 can be fully displayed (e.g., by replacing entity card 6060). That is, display of entity cards can be synchronized with the time at which the associated entity is mentioned in the video. For example, entity cards can be displayed on the user interface each time the associated entity is mentioned in the video, displayed only the first time the entity is mentioned in the video, or selectively displayed when the entity is mentioned multiple times in the video.

[0130] As mentioned above about FIG. 6A to FIG. 6C As discussed in the example of , in some implementations, a notification user interface element can be used to indicate the availability of an entity card associated with a video while the video is playing, such as when the associated entity is discussed during the video. Thus, a user can have the option of viewing the entity card while watching the video by deciding to select the notification user interface element.

[0131] Fig. 7A and Figure 7B An example user interface is depicted on which a timeline is presented during display of a video for displaying one or more entity cards according to one or more example embodiments of the present disclosure.

[0132] refer to Fig. 7A , shows an example user interface 7000 as displayed on the display 160 of the user computing device 100. For example, the user interface 7000 includes a segment in which a video 7010 is playing on a first portion 7012 of the user interface 7000. The user interface 7000 also includes a segment entitled "Related Searches" 7020 displayed on a second portion 7022 of the user interface 7000 that includes one or more suggested search queries 7030 related to the video 7010. The user interface 7000 also includes a persistent timeline segment 7042 overlaid (e.g., as a pop-up window) on the second portion 7022 to obscure at least a portion of the second portion 7022.

[0133] Persistent timeline section 7042 includes persistent timeline 7040, at least a portion of first entity card 7050, and at least a portion of second entity card 7060. Persistent timeline 7040 may display a timeline of video 7010 and include one or more points 7044 indicating when entities will be discussed during video 7010. For example, the next entity to be discussed (e.g., Howard Carter) may be indicated by at least a portion of second entity card 7060 shown in persistent timeline section 7042 and including an image of the entity.

[0134] The first physical card 7050 and / or the second physical card 7060 may be optional, for example, so that the physical card Figure 7B Expand as shown.

[0135] refer to Figure 7B , the user interface 7000' may be displayed in response to a user selecting the first entity card 7050 as displayed on the user interface 7000, or in some implementations, the user interface 7000' may be displayed at a point in time during the video 7010 when the entity (King Tutankhamun) is being discussed. Figure 7B , the user interface 7000' includes a video 7010 displayed on a first portion 7012' of the user interface 7000'. The user interface 7000' also includes a section titled "Topics to Explore" that includes a first entity card 7050 and at least a portion of a second entity card 7060 displayed on a second portion 7022' of the user interface 7000'. The user interface 7000' also includes a section titled "Related Searches" 7020 displayed on a third portion 7032' of the user interface 7000' that includes one or more suggested search queries 7030 related to the video 7010.

[0136] like Figure 7BAs shown, the first entity card 7050 is expanded to include descriptive content about the entity, and similar to the examples discussed previously, includes a title (King Tutankhamun), a subtitle (Ancient Egyptian King), descriptive content (a textual overview and a thumbnail), and attributes referencing the source of the descriptive content. The second entity card 7060 may be at least partially displayed in the second portion 7022' of the user interface 7000' next to the first entity card 7050. The user interface element 7080 may be included in the user interface 7000' as an optional element that, when selected, allows the entity cards associated with the video 7010 to cycle through as the video plays, for example, in a carousel.

[0137] In some implementations, the section titled "Topics to Explore" may also include additional user interface elements. For example, suggested search queries 7070 may be provided. Here, suggested search queries 7070 correspond to an entity search user interface element that, when selected, is configured to perform a search related to an entity.

[0138] As mentioned above about FIG. 7A to FIG. 7B As discussed in the example of , in some implementations, a persistent timeline segment can be displayed while a video is playing to indicate to the user the available entity cards in the video. The entity card related to the entity currently being discussed can be centered (i.e., highlighted) in the persistent timeline segment, while the next entity card for the next entity to be discussed during the video can also be provided. For example, a user can select an entity card as displayed in the persistent timeline segment (e.g., in a collapsed or folded form) to expand the entity card to obtain further information about the entity. In some implementations, when the corresponding entity is mentioned in the video, the entity card can automatically expand to fully display the entity card on the user interface.

[0139] Figures 8 to 10 A flow chart illustrating an example non-limiting computer-implemented method according to one or more example embodiments of the present disclosure is illustrated.

[0140] refer to Figure 8 , method 800 includes operation 810 of a user computing device (e.g., user computing device 100) displaying a video on a first portion of a user interface displayed on a display 160 of the user computing device 100. At operation 820, when a first entity is mentioned in the video while the video is playing, the user interface displays a first entity card on a second portion of the user interface while the video continues to play, the first entity card including descriptive content related to the first entity. For example, the first entity card is generated in response to automatically identifying the first entity from a transcription of the content of the video.

[0141] refer to Fig. 9, method 900 includes operation 910 of a server computing system (e.g., server computing system 300) obtaining a transcription of content from a video (e.g., via automatic speech recognition). At operation 920, the method includes applying machine learning resources to identify one or more entities that are most likely to be searched for by users viewing the video based on the transcription of the content. For example, in a video about ancient Egypt, the identified entities may include "Valley of the Kings," "King Tutankhamun," and "Sarcophagus." At operation 930, the method includes generating one or more entity cards for each of one or more entities, each of the one or more entity cards including descriptive content related to a corresponding entity of the one or more entities. At operation 940, the method includes providing (or generating) a user interface to be displayed on a corresponding display of one or more user computing devices, for: playing a video on a first portion of the user interface, and when the video is played and a first entity of the one or more entities is mentioned in the video, displaying a first entity card on a second portion of the user interface, the first entity card including descriptive content related to the first entity.

[0142] refer to Fig.10, method 1000 includes an operation 1010 in which a server computing system (e.g., server computing system 300) obtains a transcription of content from a video (e.g., via automatic speech recognition). At operation 1020, the method includes associating text from the transcription with a knowledge graph to obtain a set of knowledge graph entities. At operation 1030, the method includes obtaining training data to build a machine learning model for a machine learning resource by identifying (or matching) knowledge graph entities with entities that appear in real search queries of users watching the video. At operation 1040, the method includes weighting entities based on the relevance of the entity to other entities in the video, the breadth of the entity, the relevance of the entity to the subject of the video, etc. For example, a machine learning resource can be trained by applying weights to candidate entities (the more frequently a term is mentioned in a video, the higher weight can be assigned to the entity, the lower weight can be assigned to entities that are too broad and frequently appear in the entire video, the higher weight can be assigned to entities that are more relevant to the subject of the video, etc.). At operation 1050, the method includes applying a machine learning resource to evaluate candidate entities among the candidate entities identified in the video and ranking these candidate entities. At operation 1060, the method includes the machine learning resource predicting one or more entities for which to generate entity cards for the video. For example, the machine learning resource may select a predetermined number of entities (e.g., three or four) that are the highest ranked candidate entities as entities for which to generate entity cards. For example, the machine learning resource may select those entities that are predicted (e.g., have a specified confidence level, have a probability of being searched above a threshold level, etc.) to be searched by the user. The identified entities may then be provided to an entity card generator for generating entity cards and / or the identified entities may be stored in a database (e.g., entity data repository 380).

[0143] Terms such as "module", "unit", "provider" and "generator" may be used herein in association with various features of the present disclosure. Such terms may refer to, but are not limited to, software or hardware components or devices that perform certain tasks, such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs). A module or unit may be configured to reside on an addressable storage medium and configured to execute on one or more processors. Thus, by way of example, a module or unit may include components (such as software components, object-oriented software components, class components, and task components), processes, functions, properties, programs, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and variables. The functionality provided in components and modules / units may be combined into fewer components and modules / units or further divided into additional components and modules.

[0144] Aspects of the above-described example embodiments may be recorded in a computer-readable medium (e.g., a non-transitory computer-readable medium) including program instructions for implementing various operations embodied by a computer. The medium may also include data files, data structures, etc., alone or in combination with program instructions. Examples of non-transitory computer-readable media include: magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD ROMs, Blu-ray discs, and DVDs; magneto-optical media, such as optical discs; and other hardware devices specifically configured to store and execute program instructions, such as semiconductor memories, read-only memories (ROMs), random access memories (RAMs), flash memories, USB memories, etc. Examples of program instructions include both machine codes (such as those generated by a compiler) and files containing higher-level codes that can be executed by a computer using an interpreter. Program instructions may be executed by one or more processors. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, and vice versa. In addition, non-transitory computer-readable storage media may be distributed between computer systems connected via a network, and computer-readable codes or program instructions may be stored and executed in a decentralized manner. Furthermore, the non-transitory computer-readable storage medium may also be embodied in at least one application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0145] Each block of the flow chart may represent a unit, module, segment or portion of code, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur out of order. For example, two blocks shown in succession may actually be executed substantially concurrently (simultaneously), or the blocks may sometimes be executed in reverse order, depending on the functionality involved.

[0146] Although the present disclosure has been described with respect to various example embodiments, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art may easily generate changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the disclosed subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, changes, and equivalents.

Claims

1. A computer-implemented method for a server system, the computer-implemented method comprising: obtain transcriptions of content from videos; applying machine learning resources to identify one or more entities most likely to be searched for by a user viewing the video based on the transcription of the content; generating one or more entity cards for each of the one or more entities, each of the one or more entity cards comprising descriptive content related to a respective entity of the one or more entities; and Providing a user interface to be displayed on respective displays of one or more user computing devices for: playing the video on the first portion of the user interface, and When the video is played and a first entity of the one or more entities is mentioned in the video, a first entity card is displayed on the second portion of the user interface, the first entity card including descriptive content related to the first entity.

2. The computer-implemented method of claim 1 , wherein applying the machine learning resource to identify the one or more entities comprises: Training data is obtained to train the machine learning resource based on observation data of users searching in response to only viewing the video.

3. The computer-implemented method of claim 2, wherein applying the machine learning resources to identify the one or more entities further comprises: identifying a plurality of candidate entities from the video by associating text from the transcription with a knowledge graph, and The candidate entities are ranked to obtain the one or more entities based on one or more of the following: the relevance of each of the candidate entities to the subject matter of the video, a relevance of each of the candidate entities to one or more other candidate entities of the plurality of candidate entities, the number of times the candidate entity is mentioned in the video, and The number of videos in which the candidate entity appears in a corpus of videos stored in one or more databases.

4. The computer-implemented method of claim 3, wherein applying the machine learning resources to identify the one or more entities further comprises: evaluating user interactions with the user interface, and At least one adjustment to the machine learning resources is determined based on the evaluation of the user interaction with the user interface.

5. The computer-implemented method of claim 1 , wherein the first entity is mentioned in the video at a first point in time in the video, and The first entity card is displayed on the second portion of the user interface at the first point in time.

6. The computer-implemented method of claim 5, wherein the one or more entities include a second entity and the one or more entity cards include a second entity card, and The method further includes providing the user interface to be displayed on the respective displays of the one or more user computing devices for: displaying the second entity card in a contracted form on a third portion of the user interface while continuing to play the video, the contracted second entity card referencing the second entity to be mentioned in the video at a second time point after the first time point in the video, and When the second entity is mentioned in the video at the second time point, a fully expanded form of the second entity card is displayed on the third portion of the user interface while the video continues to play, and the fully expanded form of the second entity card includes descriptive content related to the second entity.

7. The computer-implemented method of claim 1, wherein the one or more entities include a second entity and the one or more entity cards include a second entity card, and The method further includes providing the user interface to be displayed on the respective displays of the one or more user computing devices for: When the second entity is mentioned in the video while playing the video, the second entity card is displayed on the second portion of the user interface while continuing to play the video, the second entity card including descriptive content related to the second entity, wherein the second entity card is displayed on the second portion of the display by replacing the first entity card when the second entity is mentioned in the video.

8. The computer-implemented method of claim 1 , further comprising providing the user interface to be displayed on the respective displays of the one or more user computing devices for: When the first entity is mentioned in the video while the video is playing, displaying a notification user interface element on a third portion of the user interface while the video continues to play, the notification user interface element indicating that additional information related to the first entity is available, and In response to the first entity being mentioned in the video while the video is playing and in response to receiving a selection of the notification user interface element, displaying the first entity card on the second portion of the user interface while continuing to play the video.

9. The computer-implemented method of claim 1, wherein the first entity card includes at least one of a text summary providing information related to the first entity or an image related to the first entity.

10. A computer-implemented method for use with a user computing device, the computer-implemented method comprising: receiving video for playback in a user interface; providing the video for display on a first portion of the user interface displayed on a display of the user computing device; as well as When the first entity is mentioned in the video while the video is playing: providing a first physical card for display on a second portion of the user interface while the video continues to play, in The first entity card includes descriptive content related to the first entity, and The first entity card is generated in response to automatically identifying the first entity from a transcription of content of the video.

11. The computer-implemented method of claim 10, wherein the first entity is mentioned in the video at a first point in time in the video, and The first entity card is provided for display on the second portion of the user interface at the first point in time.

12. The computer-implemented method of claim 11, further comprising providing, for display on a third portion of the user interface, a second entity card referencing a contraction of a second entity to be mentioned in the video at a second point in time after the first point in time in the video, and When the second entity is mentioned in the video at the second time point, the collapsed second entity card is expanded to fully display the second entity card on the third portion of the user interface while continuing to play the video, wherein the second entity card includes descriptive content related to the second entity.

13. The computer-implemented method of claim 10, further comprising: When a second entity is mentioned in the video while the video is playing, providing a second entity card for display on the second portion of the user interface while the video continues to play, the second entity card including descriptive content related to the second entity, The second entity card is provided for display on the second portion of the user interface by replacing the first entity card when the second entity is mentioned in the video.

14. The computer-implemented method of claim 10, further comprising providing one or more entity search user interface elements for display on the user interface, the one or more entity search user interface elements being configured to perform a search related to the first entity when selected.

15. The computer-implemented method of claim 14, further comprising providing one or more search query user interface elements for display on the user interface, the one or more search query user interface elements being configured, when selected, to perform a search related to topics of the video other than the first entity.

16. The computer-implemented method of claim 10, further comprising utilizing machine learning resources to identify the first entity and generate the first entity card.

17. A computer-implemented method as described in claim 16, wherein the first entity is an entity among multiple entities mentioned in the video, and the entity is determined by the machine learning resource to be the entity among the multiple entities mentioned in the video that is most likely to be searched for by a user watching the video.

18. The computer-implemented method of claim 10, further comprising: When the first entity is mentioned in the video while the video is playing, providing a notification user interface element for display on a third portion of the user interface while the video continues to play, the notification user interface element indicating that additional information related to the first entity is available, and In response to receiving a selection of the notification user interface element, the first entity card is provided for display on the second portion of the user interface while the video continues to play.

19. The computer-implemented method of claim 10, wherein the first entity card includes a text summary providing information related to the first entity and / or an image related to the first entity.

20. A user computing device, the user computing device comprising: monitor; one or more memories for storing instructions; as well as one or more processors configured to execute the instructions stored in the one or more memories to: Receive video for playback in the user interface, providing the video for display on a first portion of the user interface displayed on the display, and When the first entity is mentioned in the video while the video is playing: providing a first physical card for display on a second portion of the user interface while the video continues to play, in The first entity card includes descriptive content related to the first entity, and The first entity card is generated in response to automatically identifying the first entity from a transcription of content of the video.

Images