Scene search for multimedia content

Abstract

This application relates to techniques for performing contextual searches on multimedia content. An example method includes: extracting entities associated with the multimedia content, where the entities include values representing one or more objects represented in the multimedia content; generating one or more query rewrite candidates based on the extracted entities and one or more terms in a query related to the multimedia content; providing the one or more query rewrite candidates to a search engine; scoring the one or more query rewrite candidates; ranking the scored one or more query rewrite candidates based on their respective scores; rewriting the query related to the multimedia content based on a query rewrite candidate at a specific rank; and in response to the query related to the multimedia content, providing a set of results from the search engine based on the rewritten query for display.

Description

[0001] Division of Cases Instructions

[0002] This application is a divisional application of Chinese Patent Application No. 201580034359.3 with an application date of May 18, 2015. Technical Field

[0003] The present disclosure generally relates to multimedia content and, in particular, to contextual search of such content. Background Art

[0004] Users are increasingly consuming multimedia content available on the web, such as streaming video, and may pose questions related to the multimedia content. Summary of the Invention

[0005] The disclosed subject matter relates to contextual search of multimedia content.

[0006] In some innovative implementations, the disclosed subject matter may be implemented in a method. The method includes: extracting entities associated with multimedia content, where the entities include values representing one or more objects represented in the multimedia content; generating one or more query rewrite candidates based on the extracted entities and one or more terms in a query related to the multimedia content, where generation is performed when a query related to the multimedia content is received from a user; providing the one or more query rewrite candidates to a search engine; scoring the one or more query rewrite candidates based on characteristics of a corresponding result set obtained by the providing; ranking the one or more scored query rewrite candidates based on their respective scores; rewriting a query related to the multimedia content based on a query rewrite candidate at a specific rank; and providing, in response to a query related to the multimedia content, a result set from the search engine based on the rewritten query for display.

[0007] In some innovative implementations, the disclosed subject matter can be implemented in a machine-readable medium. The machine-readable medium includes instructions that, when executed by a processor, cause the processor to perform operations including: receiving a query related to multimedia content; identifying one or more terms in the query; generating one or more query rewrite candidates based on entities associated with the multimedia content and the one or more terms in the query, where the entities include values representing one or more objects represented in the multimedia content; providing the one or more query rewrite candidates to a search engine; scoring the one or more query rewrite candidates based on characteristics of the corresponding result sets obtained through the providing; ranking the scored one or more query rewrite candidates based on the respective scores of the scored one or more query rewrite candidates; rewriting the query related to the multimedia content based on a query rewrite candidate at a particular rank; and in response to the query related to the multimedia content, providing a result set from the search engine based on the rewritten query for display.

[0008] In some innovative implementations, the disclosed subject matter can be implemented in a system. The system includes: a memory that includes instructions; and a processor configured to execute the instructions to receive a query related to streaming multimedia content, where the query includes one or more terms; generate one or more query rewrite candidates based on entities associated with the multimedia content and the one or more terms in the query, where the entities include values representing one or more objects represented in the multimedia content; provide the one or more query rewrite candidates to a search engine; score the one or more query rewrite candidates based on characteristics of the corresponding result sets obtained through the providing; rank the scored one or more query rewrite candidates based on the respective scores of the scored one or more query rewrite candidates; rewrite the query related to the multimedia content based on a query rewrite candidate at a particular rank; and provide a result set from the search engine based on the rewritten query without interrupting consumption of the multimedia content.

[0009] It is to be understood that for those skilled in the art, other configurations of the subject technology will become apparent from the following detailed description, in which various configurations of the subject technology are shown and described by way of illustration. As will be realized, the subject technology is capable of having other and different configurations and many details thereof can be modified in various other aspects, all of which do not depart from the scope of the subject technology. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The novel features of the subject technology are set forth in the appended claims. However, for illustrative purposes, some configurations of the subject technology are set forth in the accompanying drawings summarized below.

[0011] Figure 1 is a diagram of an example apparatus and network environment suitable for practicing some implementations of the subject technology.

[0012] Figure 2A is a use of Figure 1 example process of an implementation of the subject technology practiced by an example apparatus.

[0013] Figure 2B is a more detailed illustration of a use of Figure 1 example process of an implementation of the subject technology practiced by an example apparatus Figure 2A of the boxes.

[0014] Figure 3 illustrates another example process of an implementation of the subject technology practiced by an example apparatus using Figure 1

[0015] Figure 4 illustrates yet another example process of an implementation of the subject technology practiced by an example apparatus using Figure 1 Detailed Description

[0016] The following detailed description is intended to describe various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology may be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The subject technology is not limited to the specific details set forth herein and may be practiced without these specific details.

[0017] ​​When viewing a particular video, a user poses a query related to an object that appears in the video. For example, while viewing a video, when a person appears in the video, the user may want to know the name of the person who appears in the video. In another example, the user may want to identify the model of a car that appears in the video. To attempt to get an answer, the user would need to type the title associated with the video and additional context into a web-based search engine interface. The user would then need to further search through the results obtained from the search engine to identify a web page that includes an answer to the user's question about the video. This process is time-consuming, diverts the user's attention from the video, and thus degrades the user experience. The disclosed implementation modifies (e.g., rewrites) the user's query such that the modified (or rewritten) query automatically includes the context of the content that the query pertains to and also provides relevant results for the query while the user is consuming the content. In some implementations, entities associated with multimedia content (e.g., video content) may be extracted. These entities may include values that characterize one or more objects (e.g., people, items, etc.) presented in the multimedia content. One or more query rewrite candidates may be generated based on the extracted entities and one or more terms in the query related to the multimedia content. The generation may be performed when a query related to the multimedia content is received from the user. For example, the user may pose a query via a search box while viewing streamed video content. The user may also pose a query via voice. The query may be related to a person or an item that appears in the video content. By way of non-limiting example, the user may pose a query asking "Who is this person?" while viewing video content without providing any additional context. Further examples of user queries are discussed below.

[0018] Query rewrite candidates may be provided to a search engine. The query rewrite candidates may be scored based on one or more of the characteristics of the respective result sets obtained by providing the query rewrite candidates to the search engine, previous search queries associated with the search engine, or the time associated with the query. One or more query rewrite candidates may be ranked based on the respective scores of the scored query rewrite candidates. A particular ranked (e.g., highest ranked) query rewrite candidate may be used to rewrite the query. For example, the query may be replaced with the particular ranked query rewrite candidate. In response to a query related to multimedia content, a result set based on the rewritten query is provided (e.g., displayed) to the user.

[0019] Results can be provided to a user in real time while the user is consuming multimedia content. The results can be provided without interrupting the provision of the multimedia content to the user. Accordingly, the user does not need to perform a search on a separate search engine and then further search through the results obtained from the search engine to identify web pages that include an answer to the user's query regarding the multimedia content. Accordingly, the user's attention does not need to shift away from the multimedia content. This enhances the user experience.

[0020] Some aspects of the subject technology include storing information about queries provided to a search engine. The user has the option of preventing this information from being stored. The user may also be provided with the opportunity to control programs or features that collect or share user information (e.g., information about user search queries, user preferences, etc.). Accordingly, the user can control how information about the user is collected and how the server uses that information.

[0021] Figure 1 FIG. 7 is a diagram illustrating an example configuration for performing a contextual search on multimedia content according to some implementations of the subject technology. Server 180 includes a processor 112, a memory 120, a storage 126, a bus 124, an input / output module 128, an input device 116, an output device 114, and a communication module 118. Memory 120 includes an entity extractor 132, a candidate generator 134, a scoring and ranking engine 138, a query rewrite engine 140, a result manager 142, and a query rewrite candidate 144. Server 180 may additionally include an output device (e.g., a touchscreen display, a non-touchscreen display), an input device for receiving user input (e.g., a keyboard, a touchscreen, or a mouse). In some implementations, server 180 includes one or more modules that facilitate user interaction via a browser or a dedicated application executed on a client computing device 190 or process data stored in a social data repository. Server 180 may be implemented as a single machine with a single processor, a multi-processor machine, or a server farm that includes multiple machines with multiple processors. Communication module 118 may enable server 180 to send and receive data via network 170 with search engine 110, query repository 192, social data server 194, multimedia server 196, and client computing device 190.

[0022] In some implementations, server 180 may be associated with search engine 110 and may send and receive data with search engine 110 via network 150. Search engine 110 may be a web search engine. A web search engine is a software system designed to search for information on the World Wide Web. Search results may be presented as multiple lines of results, which are often referred to as a search engine results page (SERP). The information may include web pages, images, information, and other types of files. Search engine 110 may maintain real-time information by using a web crawler. As an example operation, not intended to limit the embodiments, search engine 110 may store information about many web pages that search engine 110 may retrieve from the markup language defining the web pages. These pages are retrieved by a web crawler (sometimes also referred to as a "spider").

[0023] Subsequently, search engine 110 may analyze the content of each page to determine how the web page should be indexed (e.g., words may be extracted from the title, page content, headings, or specific fields known as meta tags). Data about web pages and other web content may be stored in an index database for subsequent query use. Queries may be received from server 180. The index helps search engine 110 find information relevant to the query. When search engine 110 (typically, using keywords) receives a query, search engine 110 may examine the index and provide a list of the best-matching web pages and web content, often along with a short summary that includes the content title and sometimes a portion of the content text. In some implementations, query repository 192 may store one or more queries that search engine 110 receives over time. Queries may be received from a user or other web services accessing search engine 110.

[0024] In some implementations, multimedia server 196 may include data that includes, but is not limited to, video, audio, text, images, or any combination thereof. The data stored in multimedia server 196 may be provided for display in web pages and any other web content areas. For example, the video stored in multimedia server 196 may be streamed to a web page for display to user 176 via client computing device 190. In another example, the audio stored in multimedia server 196 may be streamed to a web page for playback to user 176 via client computing device 190.

[0025] In some implementations, server 180 may be associated with a social data server 194 that includes social data and may send and receive data with the social data server 194 via network 150. In some implementations, the social data server 194 may store social content items (e.g., posted content items) associated with a social networking service. The social data server 194 may also store data related to user accounts and content items associated with user accounts. For example, the social data server 194 may include data indicating content items that have been viewed, shared, commented on, liked, or disliked by user accounts associated with a user. The social data server 194 may store a social connection data structure indicating social connections between user accounts associated with the social networking service. The social data server 194 may also store user comments (or annotations) made on multimedia content consumed by a user and stored in the multimedia server 196.

[0026] In some implementations, the search engine 110, the multimedia server 196, the social data server 194, and the server 180 may communicate with each other via the network 150 and with the client computing device 190. The network 150 may include the Internet, an intranet, a local area network, a wide area network, a wired network, a wireless network, or a virtual private network (VPN). Although only one search engine 110, multimedia server 196, social data server 194, server 180, and client computing device 190 are shown, the subject technology may be implemented in conjunction with any number of search engines 110, multimedia servers 196, social data servers 194, servers 180, and client computing devices 190. In some non-limiting implementations, a single computing device may implement Figure 1 the functions of the search engine 110, multimedia server 196, social data server 194, server 180, and other components shown.

[0027] The client computing device 190 may be a portable computer, a desktop computer, a mobile phone, a personal digital assistant (PDA), a tablet computer, a netbook, a television with a processor built therein or coupled thereto, a physical machine, or a virtual machine. The client computing device 190 may each include one or more of a keyboard, a mouse, a display, or a touch screen. The client computing device 190 may each include a browser configured to display web pages. For example, the browser may display a web page including multimedia content from the multimedia server 196. Alternatively, the client computing device 190 may include a dedicated application (e.g., a mobile phone or tablet computer application) for accessing multimedia content.

[0028] As discussed above, the memory 120 of the server 180 may include an entity extractor 132, a candidate generator 134, a scoring and ranking engine 138, a query rewrite engine 140, a result manager 142, and a query rewrite candidate 144. In some implementations, the entity extractor 132 extracts entities associated with the multimedia content. The entity may include values representing one or more objects represented in the multimedia content. For example, the entity may include the names of people, places, objects that appear in the video, etc. Entities may be extracted from metadata associated with the multimedia content (e.g., file name, media nature, etc.). Entities may also be extracted from other data associated with the multimedia content. The other data may include, for example, user comments and annotations related to the multimedia content. The multimedia content may include a streaming video that the user 176 is watching. The multimedia content may also include content stored in the multimedia server 196. The multimedia content may be content currently displayed at the computing device 190.

[0029] In some implementations, the entity extractor 132 extracts entities that appear in the multimedia content at a specific time (or time range) in the multimedia content. For example, if the multimedia content is a video that the user 176 is watching and the user is at the ten-minute mark (or ten-minute time range) in the video, the entity extractor 132 may extract entities that appear at the ten-minute mark or in the time range including the ten-minute mark in the video. In some implementations, the entity extractor 132 extracts entities that appear in the content when the user 176 provides a query via an interface at the client computing device 190. For example, if the user 176 provides a query at the fifteen-minute mark in the video, the entity extractor 132 may extract entities that appear at the fifteen-minute mark or in the time range including the fifteen-minute mark in the video. The time range may be a predetermined (or preset) time range (e.g., a two-minute range, a twenty-second range, etc.). In some implementations, the entity extractor 132 extracts a timestamp associated with the entity, which may indicate when the entity appears in the multimedia content.

[0030] In some implementations, the entity extractor 132 may transform all of the metadata (or a subset of the metadata) of the multimedia content. The entity extractor 132 may read user comments (e.g., comment text) and annotations associated with the multimedia content and extract entities associated with this user-generated data. For example, the entity extractor 132 may read comments and annotations associated with the multimedia content and stored in the social data server 194. In some implementations, the entity extractor 132 marks word sequences in the text (e.g., text or metadata or comments) that are names of items such as people, company names, objects, and any other human and non-human representations that appear in the multimedia content.

[0031] In some implementations, the entity extractor 132 can locate text and classify the text into predefined categories, such as, names of persons, organizations, objects, locations, time expressions, quantities, monetary values, percentages, etc.). For example, when the entity extractor 132 parses a text box such as "Jim bought a red SUV in 2007 (Jim bought a red SUV in 2007)", the entity extractor 132 can generate an annotated text box highlighting the entity names [Jim] Person bought a [red] color [SUV] Vehicle in

[2007] Time. For example, the output of the entity extractor 132 can be represented as P = {p_1, p_2,...}, where p_1, p_2, etc. represent the extracted entities. Referring to the above example, [Jim] Person can be the entity p_1, [red] color can be another entity p_2, and so on. The candidate generator can use the entities extracted by the entity extractor 132 to generate query rewrite candidates 144.

[0032] In some implementations, the candidate generator 134 generates one or more query rewrite candidates 144 based on the extracted entities and one or more terms in the query provided by the user 176. The query provided by the user 176 can be related to multimedia content that the user can consume via the client computing device 190. In some implementations, when a query related to multimedia content is received from the user 176, the candidate generator 134 can perform the generation of the query rewrite candidates 144. In some cases, the query may not include any context. For example, when watching a video, the user may want to know the name of the person who appears in the video when a person appears in the video. In this example, the user 176 can provide the query "Who is this person?". When a person appears in the video, the user 176 can provide a query. In another example, the user may want to identify the model of the car that appears in the video. In this example, the user 176 can provide the query "Which car is this? (What car is this?)" or "Which car? (What car?)". The query can also include queries such as "What is the price of this car? (How much is this car?)", "Where is this car made? (Where is this car produced?)", "who is the person driving this car? (Who is the person driving this car?)", "Show me other videos that have this car (Show me other videos that have this car)". In these queries, the user does not provide context, such as, the name of the video, the characteristics of the car or the person, or any other context.

[0033] Queries provided by user 176 may include one or more query terms (e.g., which, car, made, color, etc.). In some implementations, candidate generator 134 may define a function C(p,q)->{(p,q,r_l),..,(p,q,r_{N_p})} that takes the input pair (p,q) for each element in the set of extracted entities P = {p_1,p_2,...} and the user query q and generates N_p rewritten candidates. A non-limiting example of this generation is the concatenation of p and q that can generate N_p rewrites. Using alternative information or natural language grammar, candidate generator 134 may generate additional query rewrite candidates 144. The alternative information may include queries stored in query repository 192. For example, candidate generator 134 may consider queries that include terms similar to p (e.g., p_1) and q that were previously provided to search engine 110 and stored in query repository 192. In some implementations, candidate generator 134 may concatenate different input pairs p and q to generate an output set Q = {C(p_l,q)+C(p_2,q)...}, where Q represents the set of query rewrite candidates 144.

[0034] In some implementations, query rewrite candidates 144 may be based on different combinations of query terms and entities extracted by entity extractor 132. For example, when watching a video, the user may want to know the name of the car that appears in the video when a person appears in the video. In this example, user 176 may provide the query "Who is the manufacturer of this car?". The query terms in this query are "Who, is, the, manufacturer, of, this, car". Entity extractor 132 may extract the entities that appear in the video when the query is made as "CarC, Red, Italy", where CarC is the name associated with the car, Red is the color of the car, and Italy is the country of origin. Candidate generator 134 may generate one or more query rewrite candidates 144 that include "Who is the manufacturer of CarC", "Who is CarC manufacturer", "CarC manufacturer who is". In this way, candidate generator 134 generates query rewrite candidates 144 to automatically include the context in the query provided by user 176, where the context is provided by the entities extracted by entity extractor 132.

[0035] In another example, when watching a video, a user may want to know the name of a movie in which a person is involved. In this example, the user 176 may provide a query "Which films has he acted in?" when the person appears in the video. The query terms in the query are "Which, films, has, he, acted, in". The entity extractor 132 may extract entities that appear in the video at the time of the query as "John Doe, Deep Space", where "John Doe" is a name associated with the person and Deep Space is the name of the movie. The candidate generator 134 generates one or more query rewrite candidates 144 including "Films acted John Doe Deep Space", "Acted JohnDoe Deep Space", "Films John Doe", "Which films has John Doe acted in", etc. In this manner, query rewrite candidates 144 may be based on different combinations of query terms and entities extracted by entity extractor 132 .

[0036] It should be understood that the disclosed embodiments are not limited to the combination (or concatenation) of query terms and the extracted entities. In some implementations, the candidate generator 134 generates query rewrite candidates based on other heuristics. The other heuristics may include queries that are similar to one or more of the generated query candidates (or include similar terms). Queries similar to one or more of the generated query candidates can be determined according to the query repository 192. For example, if "Films acted John Doe DeepSpace" is a candidate query, the candidate generator 134 can read the query repository 192 and identify another query, such as "Which films has John Doe acted in", which is similar to the candidate query (e.g., includes one or more similar terms). Thus, for example, based on the initial candidate query "Films acted John Doe Deep Space", "Which films has John Doe acted in" can be determined as another candidate query. Therefore, for the query "Which films has he acted in?" provided by the user 176, the query rewrite candidates 144 can be "Films acted John Doe Deep Space" and "Which films has John Doe acted in".

[0037] In some implementations, the scoring and ranking engine 138 can score one or more of the extracted entities based on the time associated with the query initially provided by the user 176 and the timestamps and nature of the extracted entities. For example, the scoring and ranking engine 138 can examine the timestamp information that can be associated with one or more of the extracted entities. If the user 176 annotates the entity p_i at time T when providing the query in the video, the entity p_i can receive a positive score from the scoring and ranking engine 138 and if p_i is not mentioned at time T or near time T, the entity p_i can receive a penalty. For example, consider that the user provides the query "When is the next game?" at the tenth minute into the streaming video. If, for example, the entities "athlete" or "sports teams" are annotated at the tenth minute or substantially at the tenth minute (or between 9 - 12 minutes), the scoring and ranking engine 138 can assign a higher score to the entities "athlete" or "sports teams" relative to other entities. Subsequently, the candidate generator 134 can use the extracted entities associated with the higher score relative to other extracted entities to generate the query rewrite candidates 144.

[0038] In some implementations, the scoring and ranking engine 138 may score one or more of the extracted entities based on the co-occurrence of n-grams that may be stored in the query repository 192. An n-gram may be a contiguous sequence of n terms in a given sequence from text or speech. In other words, for example, an n-gram may be considered a combination of terms. These terms may be, for example, phonemes, syllables, letters, words, or base pairs. N-grams are typically collected from a text or speech corpus (e.g., the query repository 192). An n-gram of size 1 is called a "unigram"; of size 2 is a "bigram" (or, less commonly, a "digram"); of size 3 is a "trigram". Larger sizes are sometimes denoted by the value of n (e.g., "four-gram", "five-gram", etc.).

[0039] In some implementations, the scoring and ranking engine 138 may read the query repository 192 to identify entity types that are likely for a given n-gram of the query q. For each entity p_i extracted by the entity extractor 132, if the extracted entity is one of the most likely entities based on examining the n-grams in the query repository 192, the scoring and ranking engine 138 may assign a positive score to the extracted entity and otherwise a penalty score. For example, for the query "When is the next game", the scoring and ranking engine 138 will increase the score of entities p_i that are of type "sport teams" and "athlete". The query rewrite candidates 144 may then be generated by the candidate generator 134 using the extracted entities associated with a higher score relative to other extracted entities.

[0040] In this way, based on heuristics such as previous queries or query logs, the time of providing the query and annotating the entities, and the co-occurrence of n-grams, the candidate generator 134 may generate one or more query rewrite candidates 144. The query rewrite candidates 144 determined by the candidate generator 134 may be scored and ranked by the scoring and ranking engine 138.

[0041] In some implementations, the scoring and ranking engine 138 may provide one or more query rewrite candidates 144 to the search engine 110. Subsequently, the scoring and ranking engine 138 may score the one or more query rewrite candidates 144 based on the characteristics of the corresponding result sets obtained by providing the query rewrite candidates 144 to the search engine 110. In some non-limiting implementations, the scoring and ranking engine 138 may also score the query rewrite candidates 144 based on previous search queries associated with the search engine 110 (e.g., queries stored in the query repository 192) or the time associated with the query initially provided by the user 176.

[0042] In some implementations, the result quality can be a characteristic of the result set obtained by providing query rewrite candidates 144 to the search engine 110. The result quality of the result set can be based on the number of results retrieved from the search engine using the query rewrite candidates as a basis or the search results. For example, if a result set has more results than other result sets, the quality of that result set can be determined to be higher relative to another result set. The quality can also be based on the diversity of the search results. For example, if a result set has results related to audio, video, and text while other result sets may have results related only to text, the quality of that result set can be determined to be higher relative to another result set. The quality can also be based on the relevance of the search results based on the query rewrite candidates. For example, if a result set has results that include terms similar to or the same as the query terms, the quality of that result set can be determined to be higher relative to another result set.

[0043] The quality of the result set can be determined based on any other property or characteristic of the result set. In some implementations, given a set of query rewrite candidates "Q", the scoring and ranking engine 138 defines a function S(p, q, r) -> R_+ that assigns a rewrite score given inputs p and q. For example, the number of high-quality search results obtained by providing a query rewrite candidate "r" to the search engine 110 can be the quality score. These examples are merely illustrative and are not intended to limit the disclosed implementations.

[0044] In some implementations, the scoring and ranking engine 138 ranks the query rewrite candidates 144 based on the respective scores of the scored one or more query rewrite candidates 144. For example, a query rewrite candidate associated with a higher number of results can have a higher rank compared to another query rewrite candidate that has a lower number of results. Referring to the example pointed out above, the query rewrite candidate "Which is CarC manufacturer" can obtain 100 results from the search engine 110. However, the query rewrite candidate "CarC manufacturer which is" can generate 80 results. In this example, the scoring and ranking engine 138 can rank the query rewrite candidate "Which is CarC manufacturer" higher relative to the query rewrite candidate "CarC manufacturer Which is".

[0045] Referring to another example presented above, query candidate "Which films has John Doe acted in" may yield audio, video, and text results from search engine 110. However, query candidate "Films acted John DoeDeep Space" may yield only text results. In this example, the scoring and ranking engine 138 may rank query rewrite candidate "Which films has John Doe acted in" higher relative to query rewrite candidate "Films acted John DoeDeep Space" because "Which films has John Doe acted in" has produced a more diverse set of results relative to query rewrite candidate "Films acted John Doe Deep Space". In some implementations, the highest ranked candidate of query rewrite candidates 144 may be determined as: argmax_{x elt of Q)S(x). Argmax represents the argument of the maximum value, that is, the set of points of a given argument at which a given function attains its maximum value. In other words, the arg max of f(x) may be the set of values of x at which f(x) attains its maximum value M.

[0046] As noted above, the scoring and ranking engine 138 can rank query rewrite candidates 144 based on the respective scores of one or more query rewrites being scored. The query rewrite engine 140 can then rewrite a query related to the multimedia content based on the query rewrite candidates ranked in a particular order. In some implementations, the query rewrite candidates ranked in a particular order can be the highest ranked query rewrite candidates. For example, if the query provided by user 176 is "Which films has he acted in?", then the query rewrite engine 140 can automatically rewrite the query to "Which films has John Doe acted in". The query rewrite engine can then provide the rewritten query to the search engine 110. The result manager 142 can then provide a set of results from the search engine 110 for display in response to the query related to the multimedia content, based on the rewritten query. By way of example, the set of results can include the names of movies (e.g., "Deep Space, Galaxy Travel, Mysteries of Jupiter"). The result manager 142 provides the set of results to the user 176 in real time while the user 176 is consuming the multimedia content. The set of results can be provided without interrupting the provision of the multimedia content to the user 176. Thus, the user 176 does not need to perform a search on a separate search engine and then further search through the results obtained from the search engine to identify a web page that includes an answer to the user 176's query regarding the multimedia content. In this way, the attention of the user 176 does not need to be diverted from the multimedia content. This enhances the user experience.

[0047] The set of results can be displayed at a user interface of the client computing device 192. In some implementations, the set of results can be displayed as a pop-up box near the multimedia content. For example, a user can provide a query related to video content by voice input and can view the results of the query in a pop-up interface displayed near the video content. The results can be displayed without interrupting the video or pausing the playback of the video. The results included in the set of results can include relevant responses to the user's query. For example, if the user 176 provides the query "Which films has John Doe acted in?", then the set of results can be displayed as a pop-up box that includes "Deep Space, Galaxy Travel, Mysteries of Jupiter". The set of results can also include audio, video, and any other multimedia content. In some implementations, the set of results can be rendered as speech via a speaker associated with the computing device 190 so that the user can hear the results.

[0048] Figure 2A is used Figure 1An example process 200 for implementing the subject technology is practiced with an example construction. Although described with reference to Figure 1 elements of Figure 2A , the Figure 2A process is not limited thereto and can be applied within other systems.

[0049] Process 200 begins with extracting entities associated with the multimedia content (block 202). Entities include values that characterize one or more objects represented in the multimedia content. As noted above, in some implementations, entity extractor 132 extracts entities associated with the multimedia content. Entities can include values that characterize one or more objects represented in the multimedia content. For example, entities can include the names of people, places, objects that appear in the video, etc. Entities can be extracted from metadata associated with the multimedia content (e.g., file name, media properties, etc.). Entities can also be extracted from other data associated with the multimedia content.

[0050] One or more query rewrite candidates are generated based on one or more terms in the extracted entities and a query related to the multimedia content (block 204). Generation can be performed when a query related to the multimedia content is received from a user. For example, candidate generator 134 generates one or more query rewrite candidates 144 based on one or more terms in the extracted entities and a query provided by user 176. The query provided by user 176 can be related to multimedia content that the user may be consuming via client computing device 190. In some implementations, when a query related to the multimedia content is received from user 176, generation of query rewrite candidates 144 by candidate generator 134 can be performed.

[0051] One or more query rewrite candidates are provided to a search engine (block 206) and can be scored based on characteristics of the corresponding result sets obtained through the provision (block 208). For example, scoring and ranking engine 138 can provide query rewrite candidates 144 to search engine 110. Then, scoring and ranking engine 138 can score one or more query rewrite candidates 144 based on characteristics of the corresponding result sets obtained by providing query rewrite candidates 144 to search engine 110.

[0052] One or more query rewrite candidates can be ranked based on the respective scores of the scored one or more query rewrite candidates (block 210). For example, a query rewrite candidate associated with a higher number of results can have a higher rank compared to another query rewrite candidate having a lower number of results. In another example, a query rewrite candidate associated with a more diverse set of results (e.g., audio, video, and text) relative to the query rewrite candidate can have a higher rank compared to another query rewrite candidate having a less diverse set of results (e.g., only text).

[0053] Queries related to multimedia content can be rewritten based on query rewrite candidates ranked specifically (block 212). For example, the query rewrite engine 140 can then rewrite queries related to multimedia content based on query rewrite candidates ranked specifically. In some implementations, the query rewrite candidates ranked specifically can be the highest ranked query rewrite candidates. For example, if the query provided by the user 176 is "Which films has he acted in?", then the query rewrite engine 140 can automatically rewrite the query as "Which films has John Doe acted in?". Then, the query rewrite engine can provide the rewritten query to the search engine 110.

[0054] In response to a query related to multimedia content, a set of results from the search engine based on the rewritten query can be provided for display (block 214). For example, the results manager 142 can, in response to a query related to multimedia content, provide a set of results from the search engine 110 based on the rewrite request for display. For example, the set of results can include the names of movies (e.g., "Deep Space, Galaxy Travel, Mysteries of Jupiter"). When the user 176 is consuming multimedia content, the results manager 142 provides the set of results to the user 176 in real time. The set of results can be provided without interrupting the provision of multimedia content to the user 176. Thus, the user 176 does not need to perform a search on a separate search engine and then further search through the results from the search engine to identify web pages that include answers to the user 176's query regarding the multimedia content. In this way, the attention of the user 176 does not need to be diverted from the multimedia content. This improves the user experience.

[0055] Figure 2B is a more detailed illustration of an Figure 1 example apparatus for practicing an implementation of the subject technology to Figure 2A obtain the example process of block 204. Although described with reference to Figure 1 the elements of Figure 2B , Figure 2B the process of

[0056] One or more of the extracted entities are scored based on one or more of the following: the time at which the extracted entity is annotated in the multimedia content or co-occurrence of n-grams in the query repository (block 224). For example, the scoring and ranking engine 138 can score the extracted entities based on one or more of the following: the time at which the extracted entity is annotated in the multimedia content or co-occurrence of n-grams in the query repository 192 or any other feature of the queries stored in the query repository 192. Then, the scored entities can be ranked based on the scores (block 226). The scored extracted entities can be combined with one or more query terms to generate one or more query rewrite candidates (block 228).

[0057] Figure 3 is an example process 300 of an implementation of practicing the subject technology using an example apparatus of Figure 1 Although described with reference to the elements of Figure 1 the process of Figure 3 is not limited thereto and can be applied within other systems. Figure 3 Process 300 begins with receiving a query related to multimedia content (block 302). For example, the query can be received at the server 180 from the client computing device 190. The user 176 can provide the query to the client computing device 190 via a user interface (e.g., a web browser interface) displayed at the client computing device 190. While the user is consuming (e.g., watching) multimedia content, the user 176 can provide the query without interrupting the playback or streaming of the multimedia content. In other words, the user does not need to pause or stop the multimedia content or navigate away from the content area where the multimedia content is being displayed.

[0058] One or more terms can be identified from the query (block 304). For example, the entity extractor 132 can identify one or more terms in the query received from the client computing device 190.

[0059]

[0060] ​One or more query rewrite candidates can be generated based on one or more terms in an entity and a query associated with the multimedia content (block 306). The entity can include values characterizing one or more objects represented in the multimedia content. As noted above, in some implementations, the entity extractor 132 extracts entities associated with the multimedia content. The entity can include values characterizing one or more objects represented in the multimedia content. For example, the entity can include the names of people, places, objects that appear in the video, etc. Entities can be extracted from metadata associated with the multimedia content (e.g., file name, timestamp, media nature, etc.). Entities can also be extracted from other data associated with the multimedia content. Generation can be performed when a query related to the multimedia content is received from a user. For example, the candidate generator 134 generates one or more query rewrite candidates 144 based on the extracted entities and one or more terms in the query provided by the user 176. The query provided by the user 176 can be related to the multimedia content that the user may be consuming via the client computing device 190. In some implementations, when a query related to the multimedia content is received from the user 176, the generation of the query rewrite candidates 144 performed by the candidate generator 134 can be executed.

[0061] One or more query rewrite candidates are provided to a search engine (block 308) and can be scored based on characteristics of the corresponding result set obtained through this provision (block 310). For example, the scoring and ranking engine 138 can provide one or more query rewrite candidates 144 to the search engine 110. Then, the scoring and ranking engine 138 can score one or more query rewrite candidates 144 based on characteristics of the corresponding result set obtained by providing the query rewrite candidates 144 to the search engine 110.

[0062] One or more query rewrite candidates can be ranked based on the respective scores of the scored one or more query rewrite candidates (block 312). For example, a query rewrite candidate associated with a higher number of results can have a higher rank compared to another query rewrite candidate having a lower number of results. In another example, a query rewrite candidate associated with a more diverse set of results (e.g., audio, video, and text) relative to the query rewrite candidate can have a higher rank compared to another query rewrite candidate having a less diverse set of results (e.g., only text).

[0063] Queries related to multimedia content can be rewritten based on query rewrite candidates ranked specifically (block 314). For example, the query rewrite engine 140 can then rewrite a query related to multimedia content based on query rewrite candidates ranked specifically. In some implementations, the query rewrite candidates ranked specifically can be the top-ranked query rewrite candidates. For example, if the query provided by the user 176 is "Which films has he acted in?", the query rewrite engine 140 can automatically rewrite the query to "Which films has John Doe acted in?". Then, the query rewrite engine can provide the rewritten query to the search engine 110.

[0064] In response to a query related to multimedia content, a set of results from the search engine based on the rewritten query can be provided for display (block 316). For example, the result manager 142 can, in response to a query related to multimedia content, provide a set of results from the search engine 110 based on the rewrite request for display. For example, the set of results can include the names of movies (e.g., "Deep Space, Galaxy Travel, Mysteries of Jupiter"). When the user 176 is consuming multimedia content, the result manager 142 provides the set of results to the user 176 in real time. The set of results can be provided without interrupting the provision of multimedia content to the user 176. Thus, the user 176 does not need to perform a search on a separate search engine and then further search through the results from the search engine to identify a web page that includes an answer to the user 176's query regarding the multimedia content. In this way, the attention of the user 176 does not need to be diverted from the multimedia content. This enhances the user experience.

[0065] Figure 4 is an example process 400 of an example implementation of practicing the subject technology using an example apparatus. Although described with reference to Figure 1 the elements of Figure 1 the process is not limited thereto and can be applied within other systems. Figure 4 but Figure 4 the process is not limited thereto and can be applied within other systems.

[0066] Process 400 begins with receiving a query related to multimedia content (block 402). For example, the query may be received at server 180 from client computing device 190. User 176 may provide the query to client computing device 190 via a user interface (e.g., a web browser interface) displayed at client computing device 190. The query may include one or more query terms. While the user is consuming (e.g., watching, listening to, interacting with, etc.) multimedia content, user 176 may provide the query without interrupting the playback or streaming of the multimedia content. In other words, the user does not need to pause or stop the multimedia content or navigate away from the content area where the multimedia content is being displayed.

[0067] One or more query rewrite candidates may be generated based on the entities associated with the multimedia content and one or more terms in the query (block 404). Entities may include values that characterize one or more objects represented in the multimedia content. As noted above, in some implementations, entity extractor 132 extracts entities associated with the multimedia content. Entities may include values that characterize one or more objects represented in the multimedia content. For example, entities may include the names of people, places, objects that appear in the video, etc. Entities may be extracted from metadata associated with the multimedia content (e.g., file name, timestamp, media type, etc.). Entities may also be extracted from other data associated with the multimedia content. Generation may be performed when a query related to the multimedia content is received from the user. For example, candidate generator 134 generates one or more query rewrite candidates 144 based on the extracted entities and one or more terms in the query provided by user 176. The query provided by user 176 may be related to multimedia content that the user may be consuming via client computing device 190. In some implementations, when a query related to the multimedia content is received from user 176, the generation of query rewrite candidates 144 by candidate generator 134 may be performed.

[0068] One or more query rewrite candidates are provided to a search engine (block 406) and may be scored based on characteristics of the corresponding result sets obtained by the providing (block 408). For example, scoring and ranking engine 138 may provide one or more query rewrite candidates 144 to search engine 110. Then, scoring and ranking engine 138 may score one or more query rewrite candidates 144 based on characteristics of the corresponding result sets obtained by search engine 110 providing the query rewrite candidates 144.

[0069] The scores of one or more query rewrite candidates can be based on the scored candidates, and one or more query rewrite candidates can be ranked (block 410). For example, a query rewrite candidate associated with a higher number of results can have a higher rank compared to another query rewrite candidate having a lower number of results. In another example, a query rewrite candidate associated with a more diverse set of results (e.g., audio, video, and text) relative to the query rewrite candidate can have a higher rank compared to another query rewrite candidate having less diverse results (e.g., only text).

[0070] A query related to multimedia content can be rewritten based on a query rewrite candidate having a specific rank (block 412). For example, the query rewrite engine 140 can then rewrite a query related to multimedia content based on a query rewrite candidate having a specific rank. In some implementations, the query rewrite candidate having a specific rank can be the highest-ranked query rewrite candidate. For example, if the query provided by the user 176 is "Which films has he acted in?", the query rewrite engine 140 can automatically rewrite the query as "Which films has John Doe acted in?". Then, the query rewrite engine can provide the rewritten query to the search engine 110.

[0071] A set of results from the search engine based on the rewritten query can be provided for display without interrupting the consumption of the multimedia content (block 414). For example, the result manager 142 can provide a set of results from the search engine 110 based on the rewrite request in response to a query related to multimedia content for display. For example, the set of results can include the names of movies (e.g., "Deep Space, Galaxy Travel, Mysteries of Jupiter"). When the user 176 is consuming multimedia content, the result manager 142 provides the set of results to the user 176 in real time. When the user is consuming multimedia content, the set of results can be overlaid on the multimedia content or displayed adjacent to the multimedia content. The set of results can be provided without interrupting the provision (e.g., streaming) of the multimedia content to the user 176. The user 176 can continue to consume the multimedia content and naturally pose questions at any time during consumption. Thus, the user 176 does not need to perform a search on a separate search engine and then further search through the results from the search engine to identify a web page that includes an answer to the user 176's query regarding the multimedia content. In this way, the attention of the user 176 does not need to be diverted from the multimedia content. This improves the user experience.

[0072] Return Figure 1, in some aspects, Server 180 can be implemented using hardware or a combination of hardware and software in a dedicated server, integrated into another entity, or distributed among multiple entities.

[0073] Server 180 includes a bus 124 or other communication means for conveying information and a processor 112 coupled to the bus 124 for processing information. The processor 112 can be a general-purpose microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), programmable logic device (PLD), controller, state machine, gated logic, discrete hardware components, or any other suitable entity capable of performing calculations and other manipulations on information.

[0074] In addition to the hardware, Server 180 may also include code that creates an execution environment for the computer programs under consideration, for example, code that constitutes processor firmware, protocol stacks, database management systems, operating systems, or a combination of one or more of them stored in the memory 120. The memory 120 may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), registers, hard disk, removable disk, CD-ROM, DVD, or any other suitable storage device coupled to the bus 124 for storing information and instructions to be executed by the processor 112. The processor 112 and the memory 120 can be supplemented with or incorporated into dedicated logic circuitry.

[0075] Instructions can be stored in the memory 120 and implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, which are executed by the server 180 or control the operation of the server 180, and according to any method well-known to those skilled in the art, including but not limited to the following computer languages, such as data-oriented languages (e.g., SQL, dBase), system languages (e.g., C, Objective-C, C++, assembly), architecture languages (e.g., Java,.NET), and application languages (e.g., PHP, Ruby, Perl, Python). It can also be in languages such as array languages, aspect-oriented languages, assembly languages, authoring languages, command-line interface languages, compiled languages, concurrent languages, curly-brace languages, dataflow languages, data structure languages, declarative languages, esoteric languages, extension languages, fourth-generation languages, functional languages, interactive mode languages, translation languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, meta-programming languages, multi-paradigm languages, numerical analysis, non-English-based languages, object-oriented class-based languages, object-oriented prototype-based languages, off-side rule languages, procedural languages, reflective languages, rule-based languages, script languages, stack-based languages, synchronous languages, syntax manipulation languages, visual languages, Wirth languages, embeddable languages, and XML-based languages. The memory 120 can also be used to store temporary variables or other intermediate information during the execution of instructions for the processor 112.

[0076] The computer programs discussed herein do not necessarily correspond to files in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program being considered, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logical flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output.

[0077] The server 180 also includes a storage 126, such as a magnetic disk or an optical disk, coupled to the bus 124 for storing information and instructions. The server 180 may be coupled to various devices via the input / output module 128. The input / output module 128 can be any input / output module. Example input / output modules 128 include data ports such as USB ports. The input / output module 128 is configured to be connected to the communication module 118. Example communication modules 118 (e.g., communication modules 118 and 238) include network interface cards such as Ethernet cards and modems. In some aspects, the input / output module 128 is configured to be connected to a plurality of devices such as the input device 116 and / or the output device 114. Example input devices 116 include a keyboard and a pointing device (e.g., a mouse or a trackball), and a user can provide input to the server 180 through the input device 116. Other types of input devices 116 (such as a tactile input device, a visual input device, an audio input device, or a brain-computer interface device) can also be used to interact with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input from the user including acoustic, speech, tactile, or brainwave input can be received. Example output devices 114 include display devices such as LED (light-emitting diode), CRT (cathode ray tube), or LCD (liquid crystal display) screens for displaying information to the user.

[0078] According to one aspect of the present disclosure, the server 180 can be implemented using the server 180 in response to one or more sequences of one or more instructions contained in the memory 120 being executed by the processor 112. These instructions can be read into the memory 120 from another machine-readable medium such as the storage 126. Executing the instruction sequence contained in the main memory 120 causes the processor 112 to perform the processing blocks described herein. One or more processors in a multiprocessing arrangement can also be employed to execute the instruction sequence contained in the memory 120. In alternative aspects, hardwired circuitry can be used to replace software instructions or combined with software instructions to implement various aspects of the present disclosure. Thus, these aspects of the present disclosure are not limited to any particular combination of hardware circuitry and software.

[0079] Aspects of the subject matter described in this specification can be implemented in a computing system that includes a backend component such as, for example, a data server, or a middleware component such as, for example, an application server, or a frontend component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). The communication network (e.g., network 170) can include any one or more of, for example, a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. Additionally, the communication network can include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, etc. The communication module can be, for example, a modem or an Ethernet card.

[0080] Server 180 can be, for example but not limited to, a desktop computer, a laptop computer, or a tablet computer. Server 180 can also be embedded in another device such as, for example but not limited to, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, a video game console, and / or a television set-top box.

[0081] As used herein, the term "machine-readable storage medium" or "computer-readable medium" refers to any one or more media that participate in providing instructions or data for execution to processor 112. Such media can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or flash memory such as storage 126. Volatile media includes dynamic memory such as memory 120. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wiring that includes bus 124. Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH EPROM, any other memory chip or cartridge, or any other medium from which a computer can read. A machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter implementing a machine-readable propagated signal, or a combination of one or more of them.

[0082] As used herein, the phrase "at least one" in front of a series of items separated by the term "and" or "or" modifies the entire list, rather than each member of the list (i.e., each item). The phrase "at least one" does not necessarily require selection of at least one item; rather, the phrase allows for the meaning of at least one in any one of the items, and / or at least one in any combination of the items, and / or at least one in each of the items. By way of example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" each mean either only A, only B, or only C; any combination of A, B, and C; and / or at least one in each of A, B, and C.

[0083] Furthermore, with respect to the use of the terms "comprising", "having", etc. in the specification or claims, such terms are intended to be inclusive in a manner similar to the term "including", since "including" is understood to be a transitional term in claims when employed.

[0084] Unless specifically stated otherwise, an element recited in the singular is not intended to mean "one and only one" but rather "one or more". All structural and functional equivalents of the elements of the various configurations described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claimed subject matter. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is expressly recited in the above specification.

[0085] Although this specification contains many details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of particular implementations of the subject matter. Certain features described in this specification in the context of separate aspects may also be implemented in combination in a single aspect. Conversely, the various features described in the context of a single aspect may also be implemented separately in multiple aspects or in any suitable sub-combination. Additionally, although features may be described above as acting in certain combinations and even so initially claimed, in some cases, one or more features of the claimed combination may be deleted from the combination, and the claimed combination may cover a sub-combination or a variation of a sub-combination.

[0086] Similarly, although operations are depicted in the figures in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above aspects should not be construed as requiring separation in all aspects, but rather should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0087] The subject matter of this specification is described in accordance with particular aspects, but other aspects are achievable and are within the scope of the appended claims. For example, the acts recited in the claims may be performed in a different order and still achieve the desired result. As one example, the processes depicted in the figures need not be in the particular order shown, or sequential order, to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous. Other variations are within the scope of the appended claims.

[0088] These and other implementations are within the scope of the appended claims.

Claims

1. A method for performing a context search on multimedia content implemented by one or more processors, the method comprises: Receiving a voice query of the user during streaming of multimedia content at the user's client device, wherein the voice query is provided by voice input and wherein the voice query includes a plurality of terms; In response to receiving the voice query: Extracting an entity associated with the multimedia content, wherein the entity includes at least one value representing at least one object represented in the multimedia content when the voice query is received; Automatically generating a rewrite of the voice query based on the entity extracted from the multimedia content and based on one or more terms of the voice query, wherein generating the rewrite of the voice query is performed based on the voice query received from the user during streaming of the multimedia content; Providing the rewrite of the voice query to a search engine; and Providing results from the search engine in response to the rewrite of the voice query for presentation in response to the voice query without interrupting consumption of the multimedia content during streaming of the multimedia content.

2. The method according to claim 1, wherein, the multimedia content is a video.

3. The method according to claim 1, wherein, providing results from the search engine in response to the rewrite of the voice query includes providing the results for sounding via a speaker of the client device.

4. The method according to claim 1, wherein, extracting an entity associated with the multimedia content includes: Extracting entities from metadata associated with the multimedia content.

5. The method according to claim 1, wherein, the at least one value representing the at least one object represented in the multimedia content is the name of a person represented in the multimedia content.

6. The method according to claim 1, further comprises: Automatically generating an additional rewrite of the voice query based on the entity extracted from the multimedia content and based on one or more terms of the voice query, wherein the additional rewrite of the voice query is different from the rewrite of the voice query; Scoring the rewrite of the voice query and the additional rewrite of the voice query; and Based on the scoring, providing results in response to the rewrite of the voice query.

7. The method according to claim 1, further comprises: Extracting additional entities associated with the multimedia content, wherein the additional entities are additions to the entity; wherein automatically generating the rewrite of the voice query is further based on the additional entities associated with the multimedia content.

8. The method according to claim 1, further comprises: Extracting additional entities associated with the multimedia content, wherein the additional entities are additions to the entity; Automatically generating an additional rewrite of the voice query based on the additional entities extracted from the multimedia content and based on one or more terms of the voice query, wherein the additional rewrite of the voice query is different from the rewrite of the voice query; Score the rewrite of the voice query and additional rewrites of the voice query; and Based on the scoring, provide a result in response to the rewrite of the voice query.

9. The method according to claim 1,[[]]END]] wherein,[[]]END]] Automatically generating a rewrite of the voice query includes: Automatically generating a first rewrite of the voice query based on the entity extracted from the multimedia content and based on one or more terms of the voice query; Automatically generating at least a second rewrite of the voice query based on the entity extracted from the multimedia content and based on one or more terms of the voice query; Score the first rewrite of the voice query and the second rewrite of the voice query; and Based on the scoring, select a rewrite of the voice query from the first rewrite of the voice query and at least the second rewrite of the voice query to provide to the search engine.

10. A system for performing a context search on multimedia content,[[]]END]] comprising: A memory including instructions; and At least one processor configured to execute the instructions to: Receive a voice query during streaming of a video at a client device, wherein the voice query is provided by voice input and wherein the voice query includes a plurality of terms; In response to receiving the voice query: Extract an entity associated with the video, wherein the entity includes at least one value characterizing at least one object represented in the video, and wherein, when executing the instructions to extract the entity associated with the video, the at least one processor extracts the entity based on the entities represented in the video when the voice query is received; Automatically generate a rewrite of the voice query based on the entity extracted from the video and based on one or more terms of the voice query, wherein, when executing the instructions to automatically generate a rewrite of the voice query, the at least one processor automatically generates a rewrite of the voice query based on the voice query received from the user during streaming of the video; Provide the rewrite of the voice query to a search engine; and Provide a result from the search engine in response to the rewrite of the voice query for presentation in response to the voice query without interrupting consumption of the video during streaming of the video.

11. The system according to claim 10,[[]]END]] wherein,[[]]END]] The result is provided for sounding via a speaker of the client device.

12. The system according to claim 10,[[]]END]] wherein,[[]]END]] The at least one value characterizing the at least one object represented in the video is the name of a person represented in the video.

13. The system according to claim 10,[[]]END]] wherein,[[]]END]] When executing the instructions, the at least one processor further: Automatically generate additional rewrites of the voice query based on the entity extracted from the video and based on one or more terms of the voice query, wherein the additional rewrites of the voice query are different from the rewrite of the voice query; Score the rewrite of the voice query and the additional rewrites of the voice query; and Based on the scoring, provide a rewritten result in response to the voice query.

14. The system according to claim 10, wherein, when executing the instructions, the at least one processor further: extract additional entities associated with the video, wherein the additional entities are additions to the entities; wherein, when executing the instructions for automatically generating a rewrite of the voice query, the at least one processor further generates the rewrite of the voice query based on the additional entities associated with the video.

15. The system according to claim 10, wherein, when executing the instructions, the at least one processor further: extract additional entities associated with the video, wherein the additional entities are additions to the entities; automatically generate an additional rewrite of the voice query based on the additional entities extracted from the video and based on one or more terms of the voice query, wherein the additional rewrite of the voice query is different from the rewrite of the voice query; score the rewrite of the voice query and the additional rewrite of the voice query; and based on the scoring, provide a result in response to the rewrite of the voice query.

16. A non-transitory machine-readable medium, the non-transitory machine-readable medium including instructions stored therein, the instructions when executed by a processor cause the processor to perform operations including the following: receive a voice query of the user during streaming of multimedia content at the user's client device, wherein, the voice query is provided by voice input, and wherein the voice query includes a plurality of terms; in response to receiving the voice query: extract entities associated with the multimedia content, wherein the entities include at least one value characterizing at least one object represented in the multimedia content when the voice query is received; automatically generate a rewrite of the voice query based on the entities extracted from the multimedia content and based on one or more terms of the voice query, wherein generating the rewrite of the voice query automatically is performed based on the voice query received from the user during streaming of the multimedia content; provide the rewrite of the voice query to a search engine; and provide a result from the search engine in response to the rewrite of the voice query for presentation in response to the voice query and without interrupting consumption of the multimedia content during streaming of the multimedia content.

Images