Structured Video Documentation

Abstract

The method (500) includes receiving a content feed (120) including audio data (122) corresponding to a voice utterance and processing the content feed to generate a semantically rich structured document (300). The structured document includes a transcript (310) of the voice utterance (123) and further includes a plurality of words (123), each of the plurality of words (123) being aligned with a corresponding audio segment (222) indicating a time at which the word was recognized in the audio data. During playback of the content feed, the method also includes receiving a query (112) from a user requesting information included in the content feed and processing the query and the structured document with a large-scale language model (180) to generate a response (182) to the query. The response conveys the requested information included in the content feed. The method also includes providing the response to the query in an output from a user device (102) associated with the user.

Description

[Technical field]

[0001] This disclosure relates to structured video documents. [Background technology]

[0002] Video is a common environment in which users consume entertainment, news, and educational content. However, video remains a challenge for users who use it as an information medium due to limited ability to search and recall video content. For information-based tasks, users typically interact with a timeline-based video player user interface to scrub forward / backward through a video to find specific content of interest. The ability to generate transcripts / captions for dialogue in a video has improved the ability to search for content in a video to some extent by allowing users to input keyword searches to find related content in the transcript / caption. However, these user interfaces that use transcripts / captions to search content do not have the ability to understand the meaning of a spoken (or typed) query for specific content in a video, much less execute queries with semantic required information. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving a content feed including audio data corresponding to a voice utterance and processing the content feed to generate a semantically-rich structured document. The structured document includes a transcript of the voice utterance and includes a plurality of words, each of the plurality of words being aligned with a corresponding audio segment of the audio data indicating a time at which the word was recognized in the audio data. During playback of the content feed, the operations also include receiving a query from a user requesting information included in the content feed and processing the query and the structured document with a large-scale language model to generate a response to the query, where the response conveys the requested information included in the content feed. The operations also include providing the response to the query in an output from a user device associated with the user.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include extracting a segment of the transcript containing the requested information conveyed by the response to the query, the segment of the transcript being bounded by a start word and an end word; identifying a start audio segment of the audio data as a corresponding audio segment of the audio data aligned with the start word bounding the segment of the transcript; and identifying an end audio segment of the audio data as a corresponding audio segment of the audio data aligned with the end word bounding the segment of the transcript. In those implementations, providing the response to the query includes replaying, from a user device associated with the user, the audio data from the start audio segment of the audio data to the end audio segment of the audio data. The content feed may further include image data including a plurality of image frames, and the operations further include pausing playback of the plurality of image frames of the image data while replaying the audio data from the start audio segment of the audio data to the end audio segment of the audio data.

[0005] In some examples, the content feed further includes image data including a plurality of image frames, and the semantically rich structured document further includes author-provided text recognized in one or more of the plurality of image frames, where the author-provided text is aligned with corresponding audio segments of the audio data to indicate times at which the author-provided text was recognized in the one or more image frames. In these examples, processing the content feed to generate the semantically rich structured document may further include annotating a transcript of the speech utterance with the author-provided text by inserting the author-provided text between adjacent pairs of words in the transcript based on the corresponding audio segments of the audio data aligned with the author-provided text recognized in the one or more image frames.

[0006] The response to the query may include a text response that conveys the requested information as a coherent, focused response to the query. In some implementations, the operations also include performing text-to-speech conversion on the text response to generate a synthetic voice representation of the response to the query, and providing the response to the query in an output from the user device includes audibly outputting the synthetic voice representation of the response to the query from the user device. In these implementations, the operations may further include pausing playback of the content feed while audibly outputting the synthetic voice representation of the response to the query from the user device. Additionally, the text response to the query may further include one or more references to source material related to the requested information.

[0007] In some examples, the large-scale language model includes a pre-trained large-scale language model, and the large-scale language model performs few-shot learning using the structured documents as context for the query to generate a response to the query. The query may include a natural language question, and the response to the query may include a natural language response to the question.

[0008] In some implementations, processing the content feed to generate a semantically rich structured document includes splitting the audio data into a plurality of audio segments, performing speaker diarization on the plurality of audio segments to predict a diarization result including a corresponding speaker label assigned to each audio segment, and indexing a transcript of the speech utterance using the corresponding speaker label assigned to each audio segment split from the audio data.

[0009] Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving a content feed including audio data corresponding to a voice utterance and processing the content feed to generate a semantically rich structured document. The structured document includes a transcript of the voice utterance and includes a plurality of words, each of the plurality of words being aligned with a corresponding audio segment of the audio data indicating a time at which the word was recognized in the audio data. During playback of the content feed, the operations also include receiving a query from a user requesting information included in the content feed and processing the query and the structured document with a large-scale language model to generate a response to the query, where the response conveys the requested information included in the content feed. The operations also include providing the response to the query in an output from a user device associated with the user.

[0010] This aspect may include one or more of the following optional features: In some implementations, the operations also include extracting a segment of the transcript that includes the requested information conveyed by the response to the query, the segment of the transcript being bounded by a start word and an end word; identifying a start audio segment of the audio data as a corresponding audio segment of the audio data aligned with the start word bounding the segment of the transcript; and identifying an end audio segment of the audio data as a corresponding audio segment of the audio data aligned with the end word bounding the segment of the transcript. In those implementations, providing the response to the query includes replaying, from a user device associated with the user, the audio data from the start audio segment of the audio data to the end audio segment of the audio data. The content feed may further include image data including a plurality of image frames, and the operations further include pausing playback of the plurality of image frames of the image data while replaying the audio data from the start audio segment of the audio data to the end audio segment of the audio data.

[0011] In some examples, the content feed further includes image data including a plurality of image frames, and the semantically rich structured document further includes author-provided text recognized in one or more of the plurality of image frames, where the author-provided text is aligned with corresponding audio segments of the audio data to indicate times at which the author-provided text was recognized in the one or more image frames. In these examples, processing the content feed to generate the semantically rich structured document may further include annotating a transcript of the speech utterance with the author-provided text by inserting the author-provided text between adjacent pairs of words in the transcript based on the corresponding audio segments of the audio data aligned with the author-provided text recognized in the one or more image frames.

[0012] The response to the query may include a text response that conveys the requested information as a coherent, focused response to the query. In some implementations, the operations also include performing text-to-speech conversion on the text response to generate a synthetic voice representation of the response to the query, and providing the response to the query in an output from the user device includes audibly outputting the synthetic voice representation of the response to the query from the user device. In these implementations, the operations may further include pausing playback of the content feed while audibly outputting the synthetic voice representation of the response to the query from the user device. Additionally, the text response to the query may further include one or more references to source material related to the requested information.

[0013] In some examples, the large-scale language model includes a pre-trained large-scale language model, and the large-scale language model performs short-shot learning using the structured documents as context for the query to generate a response to the query. The query may include a natural language question, and the response to the query may include a natural language response to the question.

[0014] In some implementations, processing the content feed to generate a semantically rich structured document includes splitting the audio data into a plurality of audio segments, performing speaker diarization on the plurality of audio segments to predict a diarization result including a corresponding speaker label assigned to each audio segment, and indexing a transcript of the speech utterance using the corresponding speaker label assigned to each audio segment split from the audio data.

[0015] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic diagram of an example system that allows a user to interact with a video using a structured document generated for the video. [Diagram 2] 2 is a schematic diagram of an example document structurator for generating structured documents from audio and image data of an audiovisual feed. [Diagram 3] FIG. 2 is a schematic diagram of an exemplary structured document including a transcript, author-provided text, and an annotated transcript. [Figure 4] 1 is a schematic diagram of an exemplary video interface that presents information from a structured document generated for a video while the video is playing. [Diagram 5] 5 is a flowchart illustrating an example sequence of operations for a method of interacting with a content feed using a structured document during playback of the content feed. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference numbers in the various drawings indicate like elements.

[0018] Video players employed by media playback applications and web browsers allow users to issue commands that control the playback of a video. For example, users can play / pause / stop a video as well as scan forward / backward via dedicated buttons / commands or by scrubbing along the video timeline. Recent advances in automatic speech recognition (ASR) have enabled users to issue these video playback commands via voice. Although the video timeline feature allows users to preview the visual content frame by frame, to find the content of interest, the user must scrub back and forth along the timeline multiple times. For example, if a user is watching a video that explains how to prepare a particular dish, where the performer / instructor / participant (also called the "speaker") speaks the list of ingredients and their respective proportions, if the user has not had time to figure out what proportion one of the ingredients requires, the user must manually scrub backwards through the video to replay the part where the instructor speaks the list of ingredients. Clearly, scrubbing along the timeline is an inefficient and time-consuming process for users trying to find the content of interest. Furthermore, interactive timeline search is limited to the frame-by-frame visual content of the video and does not surface audio content or higher level content such as plot / scene descriptions or topics.

[0019] Some video player user interfaces support user-initiated keyword searches that leverage text transcripts and captions of the audio content in the video. In the above example where a user is watching a cooking instruction video, if the speaker mentions the percentage of cumin, the user may be able to input (spoken or text) a keyword search of "how much cumin?" to find out the percentage of cumin from the transcript of the audio content. However, if the teacher / presenter only mentions the list of ingredients by name and does not specify the percentage of each ingredient needed for the dish, and instead the creator of the cooking instruction video presents / adds a creator-provided chart that visually conveys the percentage of each ingredient, the user will not be able to find the information they are looking for through a keyword search, because that information is not in the transcript.

[0020] Furthermore, information extracted from audio transcripts and captions in response to keyword searches is often time-consuming to read and difficult to find the desired content because audio transcripts and captions tend to contain the halting and redundancies typical of speech. Transcripts simply present long blocks of text, and captions simply contain sequences of short phrases. Thus, the lack of structured organization limits users' ability to browse for specific topics in a video or get any kind of overview of the video content.

[0021] Some of the deficiencies inherent in existing techniques for searching desired content in videos can be addressed by allowing video creators to embed structured text to create a navigable representation of a video and enable users / viewers to search for the content they query. In addition to transcripts and captions, creators can embed structured text documents in videos that convey important topics, chapter titles, plot summaries, and summaries of different segments of the video. While using creator-provided structured text documents can be somewhat effective in helping users / viewers find the desired content in a video, the resources and costs required to create and embed the necessary structured text documents in the video discourage most creators from undertaking it. Furthermore, even if creators are willing to create structured text documents for their video content, the content picked up in response to a user's search is only as good as the structured text documents the creators choose to embed in the video. That is, it is simply impossible for creators to anticipate all kinds of content that may be the subject of a user search for inclusion in a structured text document. By the same token, when entering a query to find content of interest in a video, embedding an author-provided structured text document in the video can never provide a truly interactive experience for the user / viewer, because the meaning of the query cannot be interpreted in an integrated manner by the author-provided structured text document.

[0022] Embodiments herein relate to automatically generating semantically rich structured documents for content feeds (i.e., videos) to enable semantic interpretation of queries requesting information contained in the content feed. With reference to FIG. 1, a system 100 includes a user 2 viewing a content feed 120 played on a computing / user device 10 via a media player application 150. The media player application 150 may be a standalone application running on the user device 10 or a web application accessed via a web browser. In the illustrated example, the content feed 120 includes a recorded recipe video that is played on the computing device 10 for the user 2 to view and interact with. Although the examples herein show the content feed 120 as an audiovisual (AV) feed (e.g., video) that includes both audio data 122 (e.g., audio content, audio signals, or audio streams) and image data 124 (e.g., image content or video content), the content feed 120 may be an audio-only feed that includes only audio data 122, such as, without limitation, a podcast episode or an audiobook. For simplicity, content feed 120 may be referred to interchangeably herein as video, an AV signal, an AV feed, or simply AV data, unless otherwise specified.

[0023] The system 100 also includes a remote system 130 that communicates with the computing device 10 via the network 120. The remote system 130 may be a distributed system (e.g., a cloud computing environment or a storage abstraction) with scalable / elastic resources. The resources include computing resources 134 (e.g., data processing hardware) and / or storage resources 136 (e.g., memory hardware). In some implementations, the remote system 130 hosts (e.g., at the computing resources) a media player application 150, orchestrates playback of the content feed 120 on the computing device, generates a semantically rich structured document 300 about the content feed 120, and uses the structured document 300 to enable a user 2 to interact with the content feed 120 via the computing device 10 by issuing queries 112 requesting information contained in the content feed 120 during its playback. For example, the data processing hardware 134 of the remote system 130 may execute instructions stored in the memory hardware 136 of the remote system 130 to execute the application 150. Additionally or alternatively, media player application 150 may be executed on computing device 10 associated with user 2. For example, data processing hardware 12 of computing device 10 may execute instructions stored in memory hardware 14 of computing device 10 to execute application 150. Some examples of data processing hardware 12 include a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

[0024] The computing device 10 includes or communicates with a display 11 capable of displaying a video interface 400 for presenting image data 124, and a speaker 18 for audible output of audio data 122. The audio data 122 may correspond to audio utterances 123 spoken by a performer(s), lecturer(s), narrator, gathering participant, host, or other individual recorded in the video 120. Some examples of computing devices 10 include computers, laptops, mobile computing devices, smart televisions, monitors, smart devices (e.g., smart speakers, smart displays, smart appliances), wearable devices, etc. In the illustrated example, the content feed 120 includes a recorded recipe instructional video that is played on the computing device 10 for the user 2 to view and interact with. The embodiments herein relate to a media player application 150 that provides an interactive experience to a user 2 during playback of a recipe instruction video 120, allowing the user 2 to issue natural language queries 112 requesting information contained in the video 120, such that the application 150 uses a semantically rich structured document 300 generated for the video 120 to retrieve the requested information and provide a response 182 containing the requested information to the user 2.

[0025] User 2 may issue query 112 as a voice query 112 that is captured in streaming audio by microphone 16 in communication with computing device 10. Application 150 (or another application) may then perform speech recognition to convert voice query 112 into a corresponding textual representation of query 112. Alternatively, user 2 may also have the ability to input query 112 via input device 20, which may include a physical keyboard in communication with computing device 10 or a virtual keyboard presented for display on video interface 400. Input device 20 may also include a mouse, stylus, or graphical user interface that allows a user to input query 112 requesting information about an object displayed in video data 124 (e.g., a word in a closed caption, a word / phrase in author-provided text, an entity shown in a scene of the video) by selecting or hovering over the object.

[0026] As an example, the recipe instruction video 120 plays a segment in which the performer mixes a set of ingredients needed to make a popular Thai seafood curry dish called Haw Mok Talay. If the user 2 does not understand the cumin percentage just mentioned during the playback of the video 120, the user 2 can make a query 112 of "How much cumin?" to ascertain the cumin percentage. Rather than the user having to manually scrub back through the video 120 to find the segment in which the cumin percentage was mentioned, the application 150 can retrieve the cumin percentage (i.e., the requested information) from the semantically rich structured document 300. For example, the application 150 can retrieve the cumin percentage from the transcript 310 of the audio data 122 if the performer articulates the cumin percentage in the speech utterance 123. As another type of information, the structured document 300 can also include author-provided text 320 that is recognized in the image data 124. For example, the creator of the video 120 may temporarily display author-provided text 320 in the image data 124 indicating that a recipe calls for half a teaspoon of cumin. Thus, the application 150 may retrieve the percentage of cumin from the author-provided text 320 included in the structured document 300, regardless of whether the actor articulated that percentage via speech utterance 123, and provide a response 182 to the user's query 112 informing them that half a teaspoon of cumin is needed.

[0027] The media player application 150 includes a document structurator 200, a large-scale language model 180, and an output module 190. The document structurator 200 is configured to receive / ingest and process the audiovisual feed 120 to generate a semantically rich structured document 300. In particular, the document structurator 200 can generate a structured document 300 for an ingested audiovisual feed 120 automatically and without requiring the creator of the audiovisual feed 120 to provide any structured text or contribute to the creation of the structured document 300. Thus, the document structurator 200 can ingest any new or existing content feed 120 and generate a structured document 300 on the fly without input from the creator of the feed 120.

[0028] The audiovisual feed 120 captured by the document structurator 200 includes audio data 122 and image data 124. The audio data 122 may characterize an audio utterance 123, and the image data 124 may include multiple image frames 125 (125a-n (FIG. 2)). As described above, the content feed 120 may include an audio-only feed 120 that includes only audio data 122. The structured document 300 includes a transcript 310 of the audio utterance 123. As explained in more detail below with reference to FIG. 2, the transcript 310 includes multiple words, and the structured document 300 matches each word in the transcript 310 with a corresponding audio segment 222 (FIG. 2) of the audio data 122 that indicates the time when the word was recognized in the audio data 122. That is, the structured document 300 includes a timestamp for each word in the transcript 310.

[0029] In some implementations, the document structurizer 200 also processes the image data 124 to determine whether any author-provided text 320 is recognized in the image data 124. In these implementations, the structured document 300 generated by the document structurizer 200 also includes any author-provided text 320 recognized in one or more image frames 125a-n (FIG. 2) of the image data 124. The document structurizer 200 can recognize any author-provided text 320 in each image frame 125 using techniques such as object character recognition. Author-provided text 320, as used herein, may include textual illustrations (any combination of letters, words, or other symbols) that a video author superimposes on a scene depicted in an image frame to convey the desired content to the user / viewer 2. Author-provided text 320 may also include any text recognized in the actual scene depicted in the image frame. The structured document 300 can match any recognized author-provided text 320 with a corresponding audio segment 222 (FIG. 2) in the audio data 122 and indicate the time when the author-provided text 320 was recognized in one or more image frames 125.

[0030] The document structurizer 200 can further perform other processing techniques (such as, but not limited to, speaker diarization, summarization, and formatting) on ​​the ingested audiovisual feed 120 and store the results of those processing techniques in the structured document 300. Speaker diarization answers the question "who is speaking when" and has a variety of applications, including multimedia information retrieval, speaker turn analysis, audio processing, and automatic transcription of conversational speech, to name a few. The document structurizer 200 can utilize text generation models, which ingest the transcript 310 and / or author-provided text 320 and output major topics or corresponding summaries for one or more different segments of the audiovisual feed 120. Formatting can identify different chapters / scenes in the audiovisual feed 120.

[0031] 2 illustrates an example document structuring unit 200 that includes a diarization module 220, an automatic speech recognition (ASR) module 230, an object character recognition (OCR) module 240, and a generator 250. The application 150 executes the ASR module 230 to generate a transcript 310 (also called a transcript) of audio utterances 123 spoken by one or more speakers (e.g., actors / participants) in a content feed 120 (e.g., an audiovisual signal including audio data 122 and video data 124 or an audio-only signal including only audio data 122).

[0032] The diarization module 220 is configured to receive audio data 122 corresponding to utterances 123 from speaker(s) from the content feed 120 (and possibly image data 124 showing the faces of the speaker(s), divide the audio data 122 into a number of segments 222 (222a-n) (e.g., fixed-length segments or variable-length segments), and generate a diarization result 224 including a corresponding speaker label 226 assigned to each segment 222 using a probabilistic model (e.g., a probabilistic generative model) based on the audio data 122 (and possibly the image data 124). That is, the diarization module 220 includes a set of speaker recognition tasks for short utterances (e.g., segments 222) and determines whether two segments 222 of a given conversation are spoken by the same speaker. At the same time, the diarization module 220 can simultaneously run a face tracking routine to identify which participant is speaking in which segment 222 to further optimize speaker recognition. The diarization module 220 is then configured to repeat the above process for all segments 222 of the conversation, where the diarization result 224 assigns time-stamped speaker labels 226 (226a-n) to the received audio data 122. The time-stamped speaker labels 226, 226a-n not only identify who is speaking in a given segment 222, but also when the speaker changes between adjacent segments 222.

[0033] The ASR module 230 is configured to receive audio data 122 corresponding to the utterance 123 (and possibly image data 124 representing the face of the speaker(s) in the utterance 123). The ASR module 230 transcribes the audio data 122 into a corresponding ASR result 232, where the ASR result 232 refers to a text transcript (e.g., transcript 310) or multiple candidate text transcripts of the audio data 122. In some examples, the ASR module 230 communicates with the diarization module 220 to utilize the diarization results 224 associated with the audio data 122 to improve speech recognition based on the utterance 123. For example, the ASR module 230 may apply different speech recognition models (e.g., language models, prosody models) for different speakers identified from the diarization results 224. Additionally or alternatively, the ASR module 230 and / or the diarization module 220 (or some other component of the application 150) may index the transcript 310 of the audio data 122 using the predicted time-stamped speaker labels 226 for each segment 222 obtained from the diarization results 224. As shown in FIG. 2, the transcript 310 for the content feed 120 may be indexed by speaker to associate multiple portions of the transcript 202 with each speaker in order to identify what was said by each speaker.

[0034] In some implementations, the document structurator 200 receives a caption for the audio utterance 123 that was previously generated by another application or provided by the creator of the content feed 120. The caption may be aligned / time-stamped with the audio segments 222 of the audio data 122 to indicate the time the caption for the utterance 123 was spoken. The caption may be used as the transcript 310 without requiring the ASR module 230 to process the audio data 122, or the caption may be used in combination with or to improve the recognition results 232 for the transcript 310 generated by processing the audio data 122. In some examples, if the caption does not include punctuation, the document structurator 200 adds punctuation to the previously generated caption to improve the accuracy of the responses 182 generated by the large-scale language model 180.

[0035] The transcript 310 of the utterance 123 for inclusion in the structured document 300 also includes alignment information 315. The alignment information 315 provides an alignment between each word 312 (FIG. 3) of a plurality of words 312 (312a-n (FIG. 3)) of the transcript 310 and a corresponding audio segment 222 of the audio data 122 that indicates the time when the corresponding word was recognized.

[0036] The OCR module 240 is configured to recognize any author-provided text 320 that may be present in one or more image frames 125a-n of the image data 124. The OCR module 240 may include an OCR machine learning model (e.g., recognition engine) 244 trained to recognize any author-provided text 320 in each image frame 125. As used herein, author-provided text 320 may include textual figures (any combination of letters, words, or other symbols) that a video author superimposes on a scene depicted in an image frame to convey desired content to the user / viewer 2. Author-provided text 320 may also include any text recognized in an actual scene depicted in an image frame 125. In some examples, the OCR module 240 includes an OCR data store 242 that the OCR machine learning model 244 accesses to recognize a particular font, symbol, or text pattern. Author-provided text 320 recognized in one or more image frames 125a-n may further include corresponding alignment information 322. Here, the alignment information 322 provides an alignment between any recognized author-provided text 320 and a corresponding audio segment 222 (FIG. 2) of the audio data 122 indicating the time at which the author-provided text 320 was recognized within one or more image frames 125.

[0037] In some scenarios, the ASR module 230 uses the recognized author-provided text 320 and the corresponding alignment information 322 to improve the accuracy of the transcript 310. Referring to the above example where the content feed 120 includes a recipe instruction video, the ASR module 230 may generate a recognition result 232 that misrecognizes the name of a Thai dish "Haw Mok Talay" as "Hamook Taley". Similarly, a previously generated caption may misrecognize the name of the dish. A portion of the author-provided text 320 recognized in the image data 124 may include the phrase "Haw Mok Talay". In some examples, the correct spelling ("Haw Mok Talay") may be a low-confidence hypothesis in the list of candidate hypotheses included in the speech recognition result 232, so that a match with the phrase "Haw Mok Talay" present in the recognized author-provided text 320 increases the confidence of the candidate hypothesis of "Haw Mok Talay" in the recognition result 232, such that it is ultimately selected for inclusion in the transcript 310.

[0038] In some implementations, the generator 250 receives the transcript 310, the author-provided text 320 recognized in one or more image frames 125, and corresponding alignment information 315, 322, and generates the structured document 300 by annotating the transcript 310 of the audio utterance 123 with the author-provided text 320. In these implementations, the alignment information 315, 322 may indicate a likelihood of which portion of the audio utterance 123 the author-provided text 320 is associated with. For example, the generator 250 may insert the author-provided text 320 between pairs of adjacent words in the transcript 310 based on the corresponding audio segments 222 of the audio data 120 that are aligned with the author-provided text 320 recognized in one or more image frames.

[0039] Figure 3 shows an example of a semantically rich structured document 300 generated by the document structurator 200 of Figures 1 and 2 for an audiovisual feed 120 corresponding to a recipe instruction video. The transcript 310 relates to an utterance 123 of a speaker discussing some of the steps for making satay marinade for the Thai dish "Haw Mok Talay". The transcript 310 includes a number of words 312 (312a-n) and corresponding alignment information 315 provides an alignment between each word 312 of the number of words 312a-n and a corresponding audio segment 222 of the audio data 122 indicating the time when the corresponding word 312 was recognized. Author-provided text 320 includes "1.5 tsp coriander" and "0.5 tsp cumin," indicating the respective proportions of coriander and cumin needed in a satay marinade. Notably, these proportions are not included in transcript 310 because the speaker never made any utterances 123 announcing these proportions. Corresponding alignment information 322 provides an alignment between author-provided text 320 and corresponding audio segments 222 (FIG. 2) of audio data 122 indicating when author-provided text 320 was recognized within one or more image frames 125.

[0040] In the illustrated example, the structured document 300 also includes an annotated transcript 330 that inserts author-provided text 320 between adjacent word pairs (e.g., "anything" and "Coriander") in the transcript 310 based on corresponding audio segments 222 of the audio data 120 aligned with author-provided text 320 recognized in one or more image frames using alignment information 315, 322. Here, the annotated transcript 330 includes the author-provided text 320 inserted into the appropriate locations of the transcript 310.

[0041] Referring back to FIG. 1 , in some implementations, during playback of the content feed (e.g., audiovisual feed) 120, the large-scale language model 180 is configured to receive as input the semantically rich structured document 300 and the query 112 issued by the user 2, and generate as output a response 182 to the query 112 that conveys the requested information contained in the content feed 120. In some examples, the query 112 includes a natural language question, and the response 182 to the query 112 includes a natural language response that provides an answer to the question. For example, the response 182 to the query 112 may include a text response generated by the large-scale language model 180 that conveys the requested information as a coherent, focused response to the query 112. In some examples, the large-scale language model 180 may further augment the coherent / focused response 182 to the query 112 with reference to source material to emphasize the reliability of the information contained in the response 182. That is, a text response 182 to a query 112 may include one or more references to source material for the requested information (e.g., the response 182 may mention a link to an entity that can direct user 2 to further information). In addition to generating text to provide a natural language response / answer 182 to a natural language query 112, the large-scale language model 182 may perform other generation tasks (e.g., generating natural language text that summarizes one or more portions of the structured document 300).

[0042] The large-scale language model 180 may include a pre-trained large-scale language model 180. The pre-trained large-scale language model 180 is pre-trained for general world recognition using one or more generative tasks (i.e., multi-task learning) to learn highly effective context representations. Thus, the large-scale language model 180 may include a multi-task unified model (MUM). The pre-trained large-scale language model 180 may be based on a transformer model or a conformer model, or other encoding / decoding architectures with a multi-head attention mechanism. For example, the pre-trained large-scale language model may include one encoding branch for encoding the structured document 300, another encoding branch for encoding the query 112, and a common decoder that receives both codes to obtain / generate a response that answers the query. Notably, the transformer / conformer model can be efficiently parallelized to train the large-scale language model, and has been proven to generalize better and achieve significantly better performance than language models based on autoregressive neural network architectures such as recurrent neural network models. The pre-trained neural network model 180 may contain over one billion parameters and may exceed the one trillion parameter limit.

[0043] The embodiment herein relates to a pre-trained large-scale language model 180, which performs short-shot learning using a structured document 300 as a context to generate a response 182 to a query 180. That is, short-shot learning fine-tunes the parameters of the pre-trained large-scale language model 180 so that the language model 180 can be applied to a downstream task of searching for required information contained in an audio-video feed in response to a query 112 issued by a user 2. The use of short-shot learning is particularly useful for tasks where there is limited training data available, because the language model 180 can generalize well based on a structured document 300 that provides labeled examples to improve the search for related information (e.g., information contained in the audio-visual feed 120 currently being viewed by a user 2). Short-shot learning provides a query 112 and a structured document 300 as an input pair to the pre-trained large-scale language model 300, which labels the structured document as somehow related to generate a response 182 as an output. The large-scale language model 180 can also perform zero-shot learning tasks, in which the language model 180 can default to its knowledge of the world when generating answers 182 to queries 112.

[0044] After the large-scale language model 180 generates the response 182, the output module 190 is configured to provide the response 182 for output from the user computing device. The output module 190 may include any combination of a playback settings controller 190a, a user interface (UI) generator 190b, and a text-to-speech (TTS) system 190c. Continuing with the example where the audiovisual feed 120 includes a cooking instruction video, if user 2 realizes that he missed a detail the speaker spoke about in the video about how to toast the coriander seeds used in the recipe, he may provide the query 112, "What, how were they toasted?" The response 182 may include the answer, "They were toasted in a dry saute pan." In some examples, the output module 190 receives the response 182 and the structured document 300 as inputs and extracts a segment of the transcript 310 (and / or a segment of the annotated transcript 330 (FIG. 3)) that contains the requested information conveyed by the response to the query 112. For example, a segment extracted from the transcript 310 may include "You want to toast them in a dry saute pan," which is bounded by the start word "You" and the end word "pan." Accordingly, the output module 190 may then identify both the start audio segment 222 (FIG. 2) of the audio data 122 as the corresponding audio segment that is aligned with the start word that bounds the segment of the transcript, and the end audio segment 222 of the audio data 122 that is aligned with the end word that bounds the segment of the transcript 310.Using the identified starting audio segment 222 and ending audio segment 222, the output module 190 can instruct the playback setting controller 190a to replay the audio data 122 from the starting audio segment to the ending audio segment as an audible output from the speaker 18, thereby replaying the utterance 123, "You want to toast them in a dry saute pan," and communicating the response 182 to the query 112. In particular, the playback setting controller 190a can pause the playback of multiple image frames of the image data 125 while replaying the required audio data 120. In some examples, the controller 190a pauses the playback of the audio video feed 120 in response to receiving the query 112.

[0045] In some additional examples, the output module 190 instructs the TTS system 190c to perform a TTS conversion on the text response 182 output from the large-scale language model 180 to generate a synthetic voice representation of the response 182 to the query 112. The output module 190 can use the TTS system 190c in scenarios where the requested information conveyed by the response to the query 112 is not present in the transcript 310 and therefore was not conveyed at all in the speech utterance 123. In such cases, there is no opportunity to replay any portion of the audio data 122 to convey the information requested by the response 182. For example, referring to FIG. 3, the response 182 to the query 112 "How much cumin?" can only be ascertained by the large-scale language model 180 from the author-provided text 320 as evidenced by the annotated transcript 330. In this example, the output module 190 can receive the text response 182 "half a teaspoon of cumin seeds" generated by the large-scale language model 180 as an answer to the query 112 and can instruct the TTS system 190c to perform text-to-speech conversion on the text response 182 to generate a synthetic voice representation of the response 182 to the query 112. Thus, the media player application 150 can audibly output the synthetic voice representation from the speaker 18 of the computing device 10. In particular, the playback settings controller 190a can completely pause playback of the audiovisual feed 120 while the synthetic voice representation conveying the response 182 "half a teaspoon of cumin seeds" is audibly output from the speaker 18 of the computing device 10. In some examples, the controller 190a pauses playback of the audiovisual feed 120 in response to receiving the query 112.

[0046] Further, the output module 190 may instruct the UI generator 190b to generate an image of the text response 182 and present the image of the text response 182 in the video interface 400 displayed on the display 11 of the computing device 10 during playback of the audiovisual feed 120. Here, the user 2 can simply read the image of the text response 182 presented in the video interface 400 while watching the video 120. The text response 182 may include one or more references to source material related to the requested information. For example, the image of the text response 182 presented in the video interface 400 may provide a hyperlink to the reference to the source material related to the requested information. The user 2 may simply hover (e.g., via a mouse) over or touch a word of interest in the text response presented in the video interface 400 to view additional information or navigate to another source (e.g., a web page).

[0047] FIG. 4 illustrates an exemplary video interface 400 that the media player application 150 displays on the display 11 of the computing device 10 during playback of the audiovisual feed 120. In this example, the media player application 150 also displays information from a semantically rich structured document 300 generated for the audiovisual feed 120 in the video interface 400, allowing the user 2 to interact with the structured document 300 during playback of the audiovisual feed 120. For example, a transcript 310 of an utterance spoken by two different speakers may be displayed, along with corresponding speaker labels 204 indicating which parts of the transcript 310 were spoken by each speaker. The structured document 300 may further provide for multimodal interaction, such as providing hyperlinks to specific terms or entities listed in the transcript 310 that may be relevant to the user 2. For example, User 2 may find out more information about the term "cryptocurrency" by selecting the term or hovering the mouse over the term, where video interface 400 may add a definition of cryptocurrency or an excerpt from the Wikipedia page on cryptocurrency.

[0048] The structured document 300 may further provide summaries 410 of relevant chapters / sections / scenes of the audiovisual feed 120 for presentation in the video interface 400, where the summaries 410 may be generated by the large-scale language model 180 of Figure 1 based on information extracted from the transcript 310, the author-provided text 320, and / or the annotated transcript 330. User 2 may select one of the summaries 410 and the video player may advance to that portion of the video.

[0049] The video interface 400 of the media player application 150 also provides playback setting controls 450 that allow the user 2 to control the playback of the audiovisual feed 120. For example, the playback setting controls 450 may include buttons for play, scan forward / reverse, and pause, as well as a video timeline that the user 2 can manipulate to scrub through the video.

[0050] 5 provides a flowchart illustrating an example sequence of operations for a method 500 of interacting with a content feed 120 using a structured document 300 during playback of the content feed 120. At operation 502, the method 500 includes receiving a content feed 120 that includes audio data 122. The audio data 120 corresponds to a voice utterance 123. The content feed 120 may include an audiovisual feed that further includes image data 124. The image data 124 includes a plurality of image frames 125a-n.

[0051] At operation 504, the method 500 includes processing the content feed 120 to generate a semantically rich structured document 300, where the structured document 300 includes a transcript 310 of the speech utterance 123. The transcript 310 may include a number of words 312. Each of the number of words 312 is aligned with a corresponding audio segment 222 of the audio data 122 indicating a time when the word 312 was recognized in the audio data 122.

[0052] At operation 506, during playback of the content feed 120, the method 500 includes receiving a query 112 from a user 2 requesting information contained in the content feed. At operation 508, during playback of the content feed 120, the method 500 includes processing the query 112 and the structured document 300 with a large-scale language model 180 to generate a response 182 to the query 112, where the response 182 conveys the requested information contained in the content feed 120. At operation 510, the method includes providing the response 182 to the query 112 in an output from a user device 10 associated with the user 2.

[0053] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0054] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0055] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are meant to be merely exemplary and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0056] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or may be otherwise mounted as desired. Processor 610 may process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories as desired. Multiple computing devices 600 may also be connected such that each device performs a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0057] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0058] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In further embodiments, the computer program product is tangibly embodied as an information medium. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information medium is a computer-readable medium or a machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0059] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages low-bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 640 is connected to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is connected to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or can connect to a network device such as a switch or router, for example, via a network adapter.

[0060] The computing device 600 can be implemented in many different forms, as shown in the figure: for example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0061] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs. Such one or more computer programs may be executable and / or translatable into machine code on a programmable system including at least one programmable processor (which may be special purpose or general purpose) connected to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0062] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to send machine instructions and / or data to a programmable processor.

[0063] The processes and logic flows described herein may be executed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be executed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as, for example, a magnetic disk, a magneto-optical disk, or an optical disk, or is operatively connected to such one or more mass storage devices for receiving data from such one or more mass storage devices, transferring data to such one or more mass storage devices, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0064] To provide for interaction with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or touch screen) for displaying information to a user, and possibly a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, such as acoustic, spoken language, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0065] Having described several embodiments, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (500) that, when executed on data processing hardware (134), causes the data processing hardware (134) to perform operations, the operations comprising: receiving a content feed (120) including audio data (122), the audio data (122) corresponding to a voice utterance (123); processing the content feed (120) to generate a semantically rich structured document (300), the structured document (300) including a transcript (310) of the speech utterance (123), the transcript (310) including a plurality of words (312), each of the plurality of words (312) being aligned with a corresponding audio segment (222) of the audio data (122) indicating a time in the audio data (122) when the word (312) was recognized; The operations further include, during playback of the content feed (120): receiving a query (112) from a user requesting information contained in the content feed (120); processing the query (112) and the structured document (300) with a large-scale language model (180) to generate a response (182) to the query (112), the response (182) conveying the requested information contained in the content feed (120); The operations further include providing the response (182) to the query (112) in an output from a user device (10) associated with the user. A computer-implemented method (500).

2. The operation further comprises: extracting a segment of the transcript (310) that contains the requested information conveyed by the response (182) to the query (112), the segment of the transcript (310) being bounded by a start word (312) and an end word (312); identifying a starting audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the starting word (312) that bounds the segment of the transcript (310); identifying an end audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the end word (312) that bounds the segment of the transcript (310); Including, providing the response (182) to the query (112) includes replaying, from the user device (10) associated with the user, the audio data (122) from the starting audio segment of the audio data (122) to the ending audio segment of the audio data (122); The computer-implemented method (500) of claim 1.

3. The content feed (120) further includes image data (124) including a plurality of image frames (125); the operations further include pausing playback of the plurality of image frames (125) of the image data (124) while replaying the audio data (122) from the starting audio segment of the audio data (122) to the ending audio segment of the audio data (122). The computer-implemented method (500) of claim 2.

4. The content feed (120) further includes image data (124) including a plurality of image frames (125); the semantically rich structured document (300) further comprises author-provided text recognized in one or more image frames (125) of the plurality of image frames (125), the author-provided text being aligned with corresponding audio segments (222) of the audio data (122) to indicate times at which the author-provided text was recognized in the one or more image frames (125). A computer implemented method (500) according to any one of claims 1 to 3.

5. 5. The computer-implemented method of claim 4, wherein processing the content feed (120) to generate the semantically rich structured document (300) includes annotating the transcript (310) of the audio utterance (123) with the author-provided text by inserting the author-provided text between pairs of adjacent words (312) in the transcript (310) based on the corresponding audio segments (222) of the audio data (122) that are aligned with the author-provided text recognized in the one or more image frames (125).

6. 6. The computer-implemented method (500) of any one of claims 1 to 5, wherein the response (182) to the query (112) comprises a text response (182) that conveys the requested information as a coherent, focused response (182) to the query (112).

7. The operation further comprises: performing text-to-speech conversion on the text response (182) to generate a synthetic speech representation of the response (182) to the query (112); providing the response (182) to the query (112) for output from the user device (10) includes audibly outputting from the user device (10) a synthesized speech representation of the response (182) to the query (112).

7. The computer-implemented method (500) of claim 6.

8. 8. The computer-implemented method (500) of claim 7, wherein the operations further include pausing playback of the content feed (120) while audibly outputting the synthesized speech representation of the response (182) to the query (112) from the user device (10).

9. The computer-implemented method (500) of any one of claims 6 to 8, wherein the text response (182) to the query (112) further includes one or more references to source material related to the requested information.

10. 10. The computer-implemented method of claim 1, wherein the large-scale language model includes a pre-trained large-scale language model, and performs short-shot learning using the structured document as a context for the query to generate the answer.

11. the query (112) comprises a natural language question; the response (182) to the query (112) includes a natural language response (182) to the question; A computer implemented method (500) according to any one of the preceding claims.

12. Processing the content feed (120) to generate the semantically rich structured document (300) comprises: Dividing the audio data (122) into a plurality of audio segments (222); performing speaker diarization on the plurality of audio segments (222) to predict a diarization result (224) including a corresponding speaker label (226) assigned to each audio segment (222); indexing the transcript (310) of the speech utterance (123) using the corresponding speaker label (226) assigned to each audio segment segmented from the audio data (122); The computer-implemented method (500) of any one of claims 1 to 11, comprising:

13. A system (100), comprising: Data processing hardware (134); memory hardware (136) in communication with the data processing hardware (134), the memory hardware (136) storing instructions that, when executed by the data processing hardware (134), cause the data processing hardware (18) to perform operations; and the operation comprises: receiving a content feed (120) including audio data (122), the audio data (122) corresponding to a voice utterance (123); processing the content feed (120) to generate a semantically rich structured document (300), the structured document (300) including a transcript (310) of the speech utterance (123), the transcript (310) including a plurality of words (312), each of the plurality of words (312) being aligned with a corresponding audio segment (222) of the audio data (122) indicating a time in the audio data (122) when the word (312) was recognized; The operations further include, during playback of the content feed (120): receiving a query (112) from a user requesting information contained in the content feed (120); processing the query (112) and the structured document (300) with a large-scale language model (180) to generate a response (182) to the query (112), the response (182) conveying the requested information contained in the content feed (120); The operations further include providing the response (182) to the query (112) in an output from a user device (10) associated with the user. System (100).

14. The operation further comprises: extracting a segment of the transcript (310) that contains the requested information conveyed by the response (182) to the query (112), the segment of the transcript (310) being bounded by a start word (312) and an end word (312); identifying a starting audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the starting word (312) that bounds the segment of the transcript (310); identifying an end audio segment of the audio data (122) as the corresponding audio segment (222) of the audio data (122) that is aligned with the end word (312) that bounds the segment of the transcript (310); Including, providing the response (182) to the query (112) includes replaying, from the user device (10) associated with the user, the audio data (122) from the starting audio segment of the audio data (122) to the ending audio segment of the audio data (122); The system (100) of claim 13.

15. The content feed (120) further includes image data (124) including a plurality of image frames (125); the operations further include pausing playback of the plurality of image frames (125) of the image data (124) while replaying the audio data (122) from the starting audio segment of the audio data (122) to the ending audio segment of the audio data (122). The system (100) of claim 14.

16. The content feed (120) further includes image data (124) including a plurality of image frames (125); the semantically rich structured document (300) further comprises author-provided text recognized in one or more image frames (125) of the plurality of image frames (125), the author-provided text being aligned with corresponding audio segments (222) of the audio data (122) to indicate times at which the author-provided text was recognized in the one or more image frames (125). A system (100) according to any one of claims 13 to 15.

17. 17. The system of claim 16, wherein processing the content feed (120) to generate the semantically rich structured document (300) includes annotating the transcript (310) of the audio utterance (123) with the author-provided text by inserting the author-provided text between pairs of adjacent words (312) in the transcript (310) based on the corresponding audio segments (222) of the audio data (122) that are aligned with the author-provided text recognized in the one or more image frames (125).

18. The system (100) of any one of claims 13 to 17, wherein the response (182) to the query (112) comprises a text response (182) that conveys the requested information as a coherent, focused response (182) to the query (112).

19. The operation further comprises: performing text-to-speech conversion on the text response (182) to generate a synthetic speech representation of the response (182) to the query (112); providing the response (182) to the query (112) for output from the user device (10) includes audibly outputting from the user device (10) a synthesized speech representation of the response (182) to the query (112).

20. The system (100) of claim 18.

20. 20. The system (100) of claim 19, wherein the operations further include pausing playback of the content feed (120) while audibly outputting the synthesized speech representation of the response (182) to the query (112) from the user device (10).

21. The system (100) of any one of claims 18 to 20, wherein the text response (182) to the query (112) further comprises one or more references to source material related to the requested information.

22. The system (100) of any one of claims 13 to 21, wherein the large-scale language model (180) includes a pre-trained large-scale language model (180) and performs few-shot learning using the structured document (300) as a context for the query (112) to generate the answer (182) to the query (112).

23. the query (112) comprises a natural language question; the response (182) to the query (112) includes a natural language response (182) to the question; A system (100) according to any one of claims 13 to 22.

24. Processing the content feed (120) to generate the semantically rich structured document (300) comprises: Dividing the audio data (122) into a plurality of audio segments (222); performing speaker diarization on the plurality of audio segments (222) to predict a diarization result (224) including a corresponding speaker label (226) assigned to each audio segment; indexing the transcript (310) of the speech utterance (123) using the corresponding speaker label (226) assigned to each audio segment segmented from the audio data (122); The system (100) according to any one of claims 13 to 23, comprising:

Images