Multi-modal game video summarization

Abstract

Video (416) and audio (414) from a computer simulation are processed by a machine learning engine (202) to identify (204) candidate segments of the simulation for use in a video summary of the simulation. Text input (410) is then used to reinforce whether a candidate segment should be included in the video summary.

Description

Technical Field

[0001] This application generally relates to multimodal game video summaries in computer simulations and other applications. Background Technology

[0002] Video summaries of computer-simulated videos or other videos generate concise videos for quickly viewing highlights from platforms such as sports streaming or online gaming platforms, enhancing the viewing experience. As understood in this article, automatically generating effective video summaries is challenging, and manually generating summaries is time-consuming. Summary of the Invention

[0003] An apparatus includes: at least one processor programmed with instructions to: receive audio video (AV) data; and provide a video summary of the AV data that is shorter than the AV data by at least partially inputting first modality data and second modality data to a machine learning (ML) engine. The instructions are executable to receive the video summary of the AV data from the ML engine in response to the input of the first modality data and the second modality data.

[0004] In an exemplary implementation, the first modal data includes audio from the AV data, and the second modal data includes computer-simulated video from the AV data. In other implementations, the second modal data may include computer-simulated chat text associated with the AV data.

[0005] In a non-limiting example, the instructions can be executed to execute the ML engine to extract at least a first parameter from the second modality data and provide the first parameter to the Event Relevance Detector (ERD). In these examples, the instructions may be executed to execute the ML engine to extract at least a second parameter from the first modality data and provide the second parameter to the ERD. The instructions may further be executed to execute the ERD to output the video summary at least in part based on the first parameter and the second parameter.

[0006] In another approach, one method includes identifying an audio-video (AV) entity, such as a computer game audio-video stream. The method includes using audio from the AV entity to identify a plurality of first candidate segments of the AV entity for constructing a summary of the entity, and similarly using video from the AV entity to identify a plurality of second candidate segments of the AV entity for constructing a summary of the entity. The method further includes identifying at least one parameter associated with chat related to the AV entity, and selecting at least some of the plurality of first candidate segments and the plurality of second candidate segments based at least in part on the parameter. The method uses at least some of the plurality of first candidate segments and the plurality of second candidate segments to generate a video summary of the AV entity that is shorter than the AV entity itself.

[0007] In an exemplary implementation of the method, the method may include: presenting the video summary on a display. In a non-limiting embodiment, using video from the AV entity to identify a plurality of second candidate segments of the AV entity includes: identifying scene changes within the AV entity. Alternatively or concurrently, using video from the AV entity to identify a plurality of second candidate segments of the AV entity may include: identifying text within the video of the AV entity.

[0008] In some implementations, using audio from the AV entity to identify multiple first candidate segments of the AV entity may include: identifying acoustic events in the audio. Alternatively, using audio from the AV entity to identify multiple first candidate segments of the AV entity may include: identifying the pitch and / or amplitude of at least one speech sound in the audio. Alternatively, using audio from the AV entity to identify multiple first candidate segments of the AV entity may include: identifying emotions in the audio. Alternatively, using audio from the AV entity to identify multiple first candidate segments of the AV entity may include: identifying words in the speech.

[0009] In an exemplary implementation, identifying the parameters associated with the chat related to the AV entity may include: identifying the mood of the chat. Alternatively, identifying the parameters associated with the chat related to the AV entity may include: identifying the sentiment of the chat. Alternatively, identifying the parameters associated with the chat related to the AV entity may include: identifying the topic of the chat. Alternatively, identifying the parameters associated with the chat related to the AV entity may include: identifying at least one grammatical category of at least one word in the chat. Alternatively, identifying the parameters associated with the chat related to the AV entity may include: identifying a summary of the chat.

[0010] In another aspect, a component includes: at least one display device configured to present an audio-visual (AV) computer game. At least one processor is associated with the display device and configured with instructions to execute a machine learning (ML) engine to generate a shorter video summary of the computer game. The ML engine includes: an acoustic event ML model trained to recognize events in the audio of the computer game; a speech pitch and intensity ML model trained to recognize pitch and intensity in the speech of the audio; and a speech emotion ML model trained to recognize emotions in the audio. The ML engine further includes: a scene change detector ML model trained to recognize scene changes in the video of the computer game. Furthermore, the ML engine includes: a text sentiment detector model trained to identify sentiment in text associated with chat related to the computer game; a text emotion detector model trained to identify sentiment in text associated with the chat; and a text topic detector model trained to identify at least one topic in text associated with the chat. An event relevance detector (ERD) module is configured to receive input from the acoustic event ML model, the speech pitch and intensity ML model, the speech emotion ML model, and the scene change detector ML model to identify multiple candidate segments of the computer game, and to select a subset of the multiple candidate segments based at least in part on input from one or more of the text sentiment detector model, the text emotion detector model, and the text topic detector model to construct the video summary.

[0011] The details of both the structure and operation of this application can be best understood with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the drawings: Attached Figure Description

[0012] Figure 1 This is a block diagram of an exemplary system, illustrating computer components, some or all of which may be used in various implementations;

[0013] Figure 2 This shows a video summary generated using a machine learning (ML) engine;

[0014] Figure 3 The overall logic is illustrated in an exemplary flowchart format;

[0015] Figure 4An exemplary architecture for multimodal summarization is shown;

[0016] Figure 5 An exemplary logic for acoustic event detection is illustrated in an exemplary flowchart format;

[0017] Figure 6 Additional exemplary logic for acoustic event detection is illustrated in an exemplary flowchart format;

[0018] Figure 7 Indicates an acoustic event;

[0019] Figure 8 and Figure 9 The acoustic input is shown graphically.

[0020] Figure 10 An exemplary ML engine or deep learning model is shown for outputting speech features;

[0021] Figure 11 This is a block diagram of an exemplary system for processing emotion detection;

[0022] Figure 12 This illustrates the game audio processing used for the summary;

[0023] Figure 13 This demonstrates the text sentiment and topic extraction used for summarizing; and

[0024] Figure 14 This shows the various aspects of metadata usage. Detailed Implementation

[0025] This disclosure generally relates to a computer ecosystem, which includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems described herein may include server components and client components that can be network-connected, enabling data exchange between the client components and the server components. Client components may include one or more computing devices, including game consoles (such as Sony...). This includes game consoles made by Microsoft, Nintendo, or other manufacturers; virtual reality (VR) headsets; augmented reality (AR) headsets; portable televisions (such as smart TVs and internet-enabled TVs); portable computers (such as laptops and tablets); and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some client computers may use operating systems such as Linux, Microsoft operating systems, or Unix operating systems, or operating systems made by Apple, Inc. or Google. These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Furthermore, the operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0026] The server and / or gateway may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be a game console (such as Sony). Instantiation of personal computers, etc.

[0027] Information can be exchanged between clients and servers over a network. For this purpose, and for security reasons, servers and / or clients may include firewalls, load balancers, temporary storage devices, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements methods for providing secure communities (such as online social networking sites) to network members.

[0028] The processor can be a single-chip or multi-chip processor, which can perform logic by means of various lines (such as address lines, data lines and control lines) as well as registers and shift registers.

[0029] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures can be combined, interchanged, or excluded from other embodiments.

[0030] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes: a system having only A; a system having only B; a system having only C; a system having both A and B; a system having both A and C; a system having both B and C; and / or a system having both A, B and C, etc.

[0031] Now for specific reference Figure 1 An exemplary system 10 is illustrated, which may include one or more of the exemplary devices mentioned above and further described below according to the principles of the invention. The first device among the exemplary devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, HMD, wearable computerized device, computerized internet-enabled music player, computerized internet-enabled headset, computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., communicate with other CE devices implementing the principles of the invention, perform the logic described herein, and perform any other functions and / or operations described herein).

[0032] Therefore, in order to implement this principle, AVD 12 can be... Figure 1Some or all of the components shown are constructed. For example, AVD 12 may include one or more displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen and may be touch-enabled for receiving user input signals via touch on the display. AVD 12 may include one or more speakers 16 for outputting audio according to the principles of the invention, and includes at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. A graphics processor 24A may also be included. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12, and other elements described herein including AVD 12, to implement the principles of the invention, such as controlling display 14 to display images thereon and to receive input from said display. Furthermore, it should be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above.

[0033] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as a High Definition Multimedia Interface (HDMI) port or a USB port that physically connects to another CE device and / or a headphone port that connects headphones to the AVD 12 to present audio from the AVD 12 to the user. For example, input port 26 may be wired or wirelessly connected to a wired or satellite source 26a of audio-visual content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disk player containing content. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 44.

[0034] AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. In some cases, the one or more computer memories are embodied as stand-alone devices within the chassis of the AVD, or as personal video recording devices (PVRs) or video disk players for playing back AV programs, either inside or outside the chassis of the AVD, or as removable memory media. Furthermore, in some embodiments, AVD 12 may include a location or positioning receiver, such as, but not limited to, a cellular phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular phone base station and provide said information to processor 24 and / or in conjunction with processor 24 to determine the altitude at which the AVD 12 is set. Component 30 may also be implemented by an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions.

[0035] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into AVD 12 and capable of being controlled by processor 24 to acquire pictures / images and / or videos according to the principles of the present invention. AVD 12 may also include a Bluetooth transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0036] In addition, AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, odometers, or magnetometers), infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, and gesture sensors (e.g., for sensing gesture commands) that provide input to processor 24. AVD 12 may include an over-the-air (OTA) TV broadcast port 38 for receiving OTA TV broadcasts that provide input to processor 24. In addition to the foregoing, it should be noted that AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power AVD 12, such as a kinetic energy harvester that can convert kinetic energy into electrical energy to charge the battery and / or power AVD 12.

[0037] Still referencing Figure 1In addition to AVD 12, System 10 may also include one or more other CE device types. In one example, the first CE device 44 may be a computer game console that can be used to send audio and video of a computer game to AVD 12 via commands sent directly to AVD 12 and / or via a server, while the second CE device 46 may include components similar to the first CE device 44. In the example shown, the second CE device 46 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player 47. In the example shown, only two CE devices 44, 46 are shown; it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.

[0038] Referring now to the aforementioned at least one server 50, said at least one server includes at least one server processor 52, at least one tangible computer-readable storage medium 54 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 56, said at least one network interface, under the control of the server processor 52, allowing communication via network 22 with... Figure 1 It communicates with other devices and can actually facilitate communication between the server and client devices according to the principles of the invention. It should be noted that network interface 56 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).

[0039] Therefore, in some implementations, server 50 may be an internet server or an entire server "farm," and in exemplary implementations for, for example, online gaming applications, the server may include and perform "cloud" functionality, enabling devices of system 10 to access a "cloud" environment via server 50. Alternatively, server 50 may be a combination of one or more game consoles or... Figure 1 The other devices shown are implemented in the same room or on other computers nearby.

[0040] Figure 2The overall logic is illustrated and can be executed by any suitable processor described herein. Starting at box 200, audio-visual (AV) entities, such as complete computer simulations or computer game recordings or streams, are identified and fed into machine learning (ML) engine 202. ML engine 202 may include one or more individual ML models, as further described below, which output at 204 a video summary of the AV entities received at box 200, wherein video summary 204 is shorter than AV entity 200 and contains a series of segments from the AV entities that ML engine 202 has identified as interesting highlights.

[0041] It should be understood that the audio is first extracted from the video of the AV entity, then the audio and video are aligned temporally (e.g., using timestamps) and processed by the corresponding ML model in segments of length, such as five seconds or other periods. These segments are adjacent to each other and together constitute the AV entity. Each ML model outputs the probability of a segment of interest, and segments whose probabilities from the audio or video processing meet a threshold are candidates for inclusion in the video summary 204, which includes the audio and video of the selected segments, plus, if necessary, X seconds of AV content flanking the selected segments. As discussed further below, while both audio and video are used to identify candidate segments for the video summary, to avoid over-inclusion (and therefore excessively long video summaries), text from chat associated with the AV entity can be used to augment the identified segments. This substantially limits the total length of segments included in the video summary to no more than a predefined percentage of the complete AV entity by eliminating candidate segments whose associated text from the chat indicates less interest than other candidate segments.

[0042] ML models can be like Figure 3 The example demonstrates training using an input data training set, which associates the types of data available to an AV entity with desired decisions about that data. In this example, gameplay videos from an online service could be used, with data annotated by experts, enabling the ML model to learn what data is a good indicator of interesting events, allowing the ML model to indicate which segments of the AV entity are suitable for inclusion in the summary "highlights" video.

[0043] Starting at box 300, the training set of data is fed into the ML engine, such as by feeding the training set into various ML models that will process the corresponding type of data in AV entities. As discussed further below, at box 302, the ML engine combines feature vectors from two or more data type patterns to output a video summary of the AV entity at box 304. The predictive validity of the video summary can be annotated and fed back into the ML engine to refine its processing.

[0044] Figure 4 The ML model architecture is illustrated. An Event Relevance Detector (ERD) 400 receives input from an acoustic event detector 402, a pitch and intensity detector 404, and a speech emotion detector 406. The pitch and intensity detector identifies speech pitch and intensity in the audio. ERD 400 may include a set of heuristic rules applied to the input probabilities received from detectors 402, 404, and the detector 406 (which may be implemented by one or more ML models) to generate video summaries. Alternatively, ERD 400 may include an ML model trained to generate video summaries based on its inputs.

[0045] The acoustic event detector 402 is trained to identify audio segments of AV entities that indicate content of interest and thus indicate that a particular segment is an event for inclusion as a candidate in the video summary. The acoustic event detector 402 is further described below and may include a training set based on predefined "interesting" events to identify acoustic events as one or more layers of an interesting convolutional neural network (CNN).

[0046] Similarly, the pitch and intensity detector 404 is an ML model trained to identify pitch and intensity in speech audio that indicate content of interest. Examples include: higher speech pitch indicates more interest than lower pitch, or wider pitch variations indicate more interest than narrower variations, and louder speech indicates more interest than softer speech. Pitch variations differ significantly in the presence of exciting content and when an interesting event occurs, and this can be detected in their speech. Therefore, regions of sound in speech with high intensity and abrupt changes can be classified as candidate regions for exciting part generation.

[0047] The verbal emotion ML model 406 is trained to identify emotions in audio to recognize those of interest. One or both of categorical emotion detection and dimensional emotion detection can be used. Categorical emotion detection can detect multiple (e.g., ten) different categories of emotions, such as, but not limited to, happiness, sadness, anger, anticipation, fear, loneliness, jealousy, and disgust. Dimensional emotion detection has two variables: arousal and valence.

[0048] Figure 4The diagram also shows that the ERD 400 receives input from a text topic extractor model 408, which is trained to identify topics in text associated with chats such as computer game chats related to AV entities. Viewers commonly use emoticons in game chats; therefore, emoticons also contain crucial information for topic detection. This detection can be achieved by converting emoticons into corresponding text. This information can be used as additional information for the topic detection module. Topics can be identified from a predefined vocabulary or annotations of a given AV topic domain. For example, for war games, a first vocabulary or a first set of annotations can be used to identify topics of interest, while for esports, a second vocabulary or a second set of annotations can be used, where the text topic extractor is trained to identify text topics and which topics indicate segments of interest based on the vocabulary or annotations. Topic detection can be achieved using statistical methods such as Latent Dirichlet Distribution (LDA), which classifies text in chats into specific topics. Chats can be processed individually or grouped together to improve performance. Modern deep learning-based Natural Language Processing (NLP) techniques can also be used for topic modeling. Transformer-based bidirectional encoder representations (BERT) can be used to perform downstream tasks in NLP, such as topic detection and sentiment classification. In addition to these techniques, hybrid models using BERT, LDA, and clustering can be employed to detect text fragments that can be considered candidate events.

[0049] ERD 400 may also receive input from a text sentiment analyzer or detector model 410, which is trained to identify parameters such as, but not limited to, emotions and sentiments in the text associated with the chat 412 related to the AV entity. Emotions differ from sentiments because emotions are generally positive or negative, while sentiments are more specific, as discussed further below. For example, positive emotions may be associated with interesting passages, and negative emotions may be associated with less interesting passages.

[0050] The ERD 400 receives probabilities from the ML model described herein to identify multiple candidate segments of an AV entity based on audio or video probabilities that meet a threshold. The ERD 400 selects a subset of the multiple candidate segments based on chat text probabilities to construct a video summary.

[0051] Figure 4Audio 414, which has been separated from the video 416 of the AV entity being generalized, is input to the acoustic event detector 402. The audio is also input to a speech source separation model 418, which uses, for example, speech and / or speech recognition principles to separate speech in the audio from different channels, outputting each individual speech track in the analyzed segment to a speech pitch and intensity detector 404. Similarly, each speech track is sent to a speech emotion detector 406 for individual emotion analysis of each speech.

[0052] Furthermore, each speech track can be input into an Automatic Speech Recognition (ASR) model 420, which converts the speech of each track into words and sends the probabilities of words indicating terms of interest as defined by the model's training set to an ERD 400. The ASR model 420 can also identify segments as uninteresting based on long periods of silence.

[0053] like Figure 4 As shown, the ML engine also includes a scene change detector ML model 422, which receives AV entity video 416 for each segment and is trained to identify scene changes in the video. The video is also fed into a text detector 424, which detects any text in the video, such as closed captions. The video-based ML model sends the probabilities of interesting scene changes / video text to ERD 400, respectively.

[0054] Now let's move on to the chat text section of the ML engine. Chat can be used to enhance summary prediction based on video and audio. For example... Figure 4 As shown, chat user clustering 426 can be used in conjunction with chat transcription 412 as input to various chat-based ML models, including a text sentiment detector 410 and a topic extraction model 408. Additionally, the text sentiment detector model 428 can be trained to detect sentiment in chat text, outputting the probability of interesting sentiments to ERD 400 based on a predefined training set of interesting sentiments and terms associated with those sentiments.

[0055] Named Entity Recognition (NER) and Aspect Detection (NERAD) model 430 can be used to output the probability of detecting an interesting grammatical type in input text based on a training set that associates words with interesting and uninteresting grammatical types. For example, NERAD model 430 can output the probability that a term is a proper noun, which can be predefined as more interesting than adjectives. NERAD model 430 can also output a short summary of the text in a segment indicating the probability of the segment being interesting or uninteresting.

[0056] It should be noted that chat text may include "stickers" or emoticons that users may need to purchase in some cases. This means that attaching such stickers to chat can indicate a greater interest in the corresponding snippets, thereby enhancing learning from other modalities.

[0057] It should be noted that, in addition to receiving text from chat 412, the chat text-based model can also receive terms from automatic speech recognition model 420 to process together with the terms in the chat text.

[0058] Figure 4 It is also shown that game event data 432 from the game console engine 434 can be sent to the ERD 400. This data may include metadata such as game status, audio cues, video cues, and text cues. That is, if the engine 434 has access to game status and other metadata, the game status and other metadata can be provided to the ERD. Such metadata is referenced below. Figure 14 Further discussion is needed.

[0059] Figure 5 Additional logic for the acoustic event detector 402 is shown. Starting at box 500, the input audio signal is divided into training / test sets, and at box 502, the audio signal is compressed into a feature vector. At box 504, the features from box 502 are used to train the neural network (NN) of the acoustic event detector 402. At box 506, the accuracy of the acoustic event detector 402 is determined for feedback during the training process.

[0060] Figure 6 The diagram shows that after training, the acoustic event detector 402 predicts its probability score for sound events in each segment of the AV entity analysis to be generalized at box 600. Regions without sound are detected at box 602. As indicated at 604, these results are generated continuously as audio is continuously fed into the acoustic event detector 402 to deliver probabilities to the ERD 400. As previously indicated and as... Figure 6 As shown, “N” seconds of adjacent segments can be added to the candidate segments of interest in the video summary.

[0061] Figure 7 The acoustic event detector 402 analyzes audio signal 700 to identify events of various types 702, such as laughter, sighing, singing, coughing, cheering, clapping, hissing, and shouting. Based on a training set, some events indicate segments of interest, while others indicate segments of uninteresting interest. Similarly, emojis 704 may accompany the identified events for further classification.

[0062] Figures 8 to 11 This illustrates another aspect of the speech emotion detector model 406. For example... Figure 8 and Figure 9 As shown, the audio from multiple segments 800 of the AV entity can be decomposed into categories and dimensions 902, including rage, cold anger, unemotion, surprise, contempt, sadness, and happiness. These categories are based on their... Figure 9 The position of the graph, where the x-axis represents valence and the y-axis represents arousal.

[0063] Figure 10 An exemplary model architecture with three parallel processing paths is shown: a first path 1000 for valence (passive or negative), a second path 1002 for arousal (active or inactive), and a third path 1004 for categorical sentiment classification. Each path receives speech features 1006 as input and processes the input sequentially through a common bidirectional long short-term memory (BLSTM) 1008, followed by a respective path BLSTM 1010, an attention layer 1012, and a deep neural network (DNN) 1014. Other models in this paper may employ similar neural network components.

[0064] Figure 11 The speech 1100 shown in audio signal segment 1102 is input to speech activity detection (VAD) box 1104 to detect the presence or absence of speech and to distinguish between speech and non-speech. The output of VAD 1104 is sent to Figure 10 The emotion detection architecture is used to output the probability of emotion category, valence, and arousal to decision pipeline 1106. As discussed elsewhere in this paper, decision pipeline 1106 determines whether the probability of any given emotion meets a threshold. If it does, and the emotion is defined as interesting by the training set, the corresponding segment of the AV content from which the test segment is obtained is labeled as a candidate for inclusion in the video summary.

[0065] Figure 12 Another aspect of the speech pitch and intensity detector 404 is shown. The signal power (i.e., amplitude) at 1202 is calculated using an audio segment 1200 derived from a fragment of the AV entity being generalized, to identify regions of interest within the segment, as defined in the model's training set. These regions are shown in a power plot at 1204, where the x-axis represents time and the y-axis represents amplitude.

[0066] Additionally, as indicated at 1206, the fundamental frequency variation (pitch variation) of signal 1200 is identified. These variations are indicated at 1208. The model is trained to identify segments of interest from the shape of these variations. (See above regarding...) Figure 4 The ASR and NER discussed can be used in this training.

[0067] Figure 13The following is a decision pipeline flow for two exemplary audio parameters. In the example shown, the text topic extractor 408 outputs the probability of topic 1300 of the chat text, and the text sentiment analyzer 410 outputs the probability of sentiment 1302 of the chat text. It should be understood that similar decision pipelines can be used to illustrate the probabilities of other parameters and other patterns. If the probability of a topic identified as “interesting” by the text topic extractor 408 at state 1304 meets a first threshold α, the segment from which the topic was extracted is sent to state 1306 as a candidate segment for the video summary. Otherwise, the segment is not labeled as a candidate. Similarly, if the probability of a sentiment identified as “interesting” by the text sentiment analyzer 410 at state 1308 meets a second, possibly different, threshold β, the segment from which the sentiment was extracted is sent to state 1306 as a candidate segment for the video summary. Otherwise, the segment is not labeled as a candidate. As previously discussed, the assumption that the same segment is identified as interesting by both the audio or video modal model and the chat text modal model can ensure that the segment is included in the video summary, while the same segment not being identified as interesting by the chat text modal model can lead to the segment being excluded from the video summary even if it is necessary to keep the length of the summary within the maximum allowed length.

[0068] It should be noted that in the implementation of ERD 400 by an ML model, the ERD model can be trained using a set of audio, video, and chat text probabilities, as well as corresponding video summaries derived from them, such as those generated by human annotators.

[0069] Figure 14 This illustrates aspects of the metadata described above used in conjunction with the principles outlined herein. Metadata may originate from text and / or video and / or audio, as well as... Figure 4 The game metadata mentioned above. It should be understood that in an implementation without metadata, the video summarizing ML engine is platform-independent and only provides video summaries of the input AV entities. Figure 14 This shows additional features that can be used when providing metadata. The metadata is temporally aligned with the audio, video, and chat text of the video summary.

[0070] As indicated at positions 1400 and 1402 respectively, the metadata can be found from... Figure 4The game event data 434 and the ML engine described herein are both received. For example, metadata related to NER topic and aspect detection topics, along with sentiment, audio, and video features extracted as described herein, and game event data, can be used at box 1404 to generate special audio to be overlaid on the audio of the AV clip, thereby establishing a video summary. The audio may include, for example, cheers or hisses from a crowd indicated by metadata features. The audio may include audio messages driven by game metadata, such as the verbal message "The beast was killed here" in response to game metadata indicating an event like "The beast was killed here." In other words, audio metadata can notify metadata events and information as they arrive.

[0071] Box 1406 indicates that the portion of the video that serves as metadata aligned to the current time can be clearly highlighted, for example, by increasing the brightness of said portion, presenting a line around said portion, etc. For example, if the metadata includes proper nouns (character names), then that character can be highlighted in the video summary during the time period covered by the metadata. In other words, any or all of the metadata can be visually indicated by highlighting the relevant portion of the video summary.

[0072] Metadata can also be used at box 1408 to generate text that can be overlaid on the video summary. Accordingly, any or all of the metadata can be presented in text form on a portion of the video summary. This metadata may include people who express liking for certain parts of the AV entities summarized in the video summary, such as topics presented in the video summary originating from aspect detection blocks, emojis representing the emotions indicated in the metadata, etc.

[0073] It should be understood that although the principles of the invention have been described with reference to some exemplary embodiments, these embodiments are not intended to be limiting, and various alternative arrangements may be used to achieve the subject matter claimed herein.

Claims

1. An apparatus for providing video summaries, comprising: At least one processor, said at least one processor being programmed with instructions for the following: Receive audio and video (AV) data; A shorter video summary of the AV data than the AV data is provided at least in part by the following means: Input first modal data, including audio from the AV data, into the machine learning (ML) engine to identify multiple first candidate segments of the AV data; Inputting second modal data, including computer-simulated chat text related to the AV data, into the ML engine to extract at least a first parameter from the second modal data, and providing the first parameter to the Event Relevance Detector (ERD); and At least some of the plurality of first candidate segments are selected at least in part based on the first parameter, wherein a first candidate segment is excluded if it is identified as interesting based on the first modality data but not as interesting based on the second modality data; The video summary of the AV data is received from the ML engine in response to the input of the first modal data and the second modal data, the video summary including a plurality of selected first candidate segments of the AV data.

2. The device of claim 1, wherein the second modal data comprises computer-simulated video from the AV data.

3. The device of claim 1, wherein the instructions are executable to execute the ML engine to extract at least a second parameter from the first modal data and provide the second parameter to the ERD.

4. The device of claim 3, wherein the instructions are executable to perform the ERD to output the video summary at least in part based on the first parameter and the second parameter.

5. A method for providing video summaries, comprising: Receive audio and video (AV) data; Using audio from the AV data, identify multiple first candidate segments of the AV data for use in building a summary of the data using a machine learning (ML) engine; Use the ML engine to identify at least one parameter associated with the chat related to the AV data; At least some of the plurality of first candidate segments are selected based at least in part on the parameters, wherein a first candidate segment is excluded if it is identified as interesting based on the audio but not based on the chat. as well as Using at least some of the plurality of first candidate segments, a video summary of the AV data that is shorter than the AV data is generated.

6. The method of claim 5, further comprising: Using video from the AV data, identify multiple second candidate segments of the AV data for use in building a summary of the data using the ML engine; as well as At least some of the plurality of second candidate segments are selected based at least in part on the parameters, wherein the video summary used to generate the AV data also uses at least some of the plurality of second candidate segments.

7. The method of claim 5, further comprising: The video summary is displayed on the monitor.

8. The method of claim 6, wherein using video from the AV data to identify a plurality of second candidate segments of the AV data comprises: Identify one or more items selected from a list that includes the following: Scene changes in the AV data; as well as The text in the video of the AV data.

9. The method of claim 5, wherein using audio from the AV data to identify a plurality of first candidate segments of the AV data comprises: Identify one or more items selected from a list that includes the following: Acoustic events in the audio; The pitch and / or amplitude of at least one speech sound in the audio; The emotions in the audio; and The words and phrases in the speech in the audio.

10. The method of claim 9, wherein identifying the parameters associated with the chat related to the AV data comprises: Identify one or more items selected from a list that includes the following: The emotions conveyed in the chat; The emotions conveyed in the chat; The topic of the chat; At least one grammatical category of at least one word in the chat; A summary of the chat.

11. A component for providing video summaries, comprising: At least one display device, said at least one display device being configured to present audio-visual (AV) computer games; At least one processor, the at least one processor being associated with the display device and configured with instructions for performing the method as described in any one of claims 5-10.

Images