Synthetic voice commentary generation

The system generates synthetic voice commentary using text and voice synthesis models to enhance user engagement in virtual environments by creating dynamic and realistic commentary in real-time, addressing the limitations of formulaic content generation in computer games.

WO2024248722A1PCT designated stage expired Publication Date: 2024-12-05STRAIGHT4 STUDIOS LTD +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/SG2024/050073
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-02-09
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Virtual environments like computer games often lack user engagement due to their formulaic nature and require significant computational complexity to provide variability in content, such as audio and commentary, which is typically generated by individuals with limited time and resources.

Method used

A system and method for generating synthetic voice commentary using a text generation model to create commentary text based on gaming event stream data and a voice synthesis model to convert this text into voice commentary, allowing for real-time or near-real-time feedback and conversation-like experiences during gameplay.

Benefits of technology

The solution enhances user engagement by providing dynamic and realistic voice commentary that mimics human commentary, improving the overall gaming experience through efficient and scalable generation of varied content without the need for extensive human input.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050073_05122024_PF_FP_ABST
    Figure SG2024050073_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Voice commentary generation by receiving gaming event stream data, the event stream data comprising information relating events occurring in the process of gameplay; generating first text commentary corresponding to at least a subset of the events using a text generation model; and generating first voice commentary based on the generated first text commentary using a voice synthesis model.
Need to check novelty before this filing date? Find Prior Art

Description

Synthetic Voice Commentary GenerationTechnical Field

[0001] This disclosure generally relates to methods and systems for the generation of synthetic commentary including synthetic voice commentary.Background

[0002] This background description is provided for the purpose of generally presenting the context of the disclosure. Contents of this background section are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0003] Virtual environments and experiences such as computer games are often controlled using tightly pre-programmed paths and flows. The formulaic nature of the structure and flow of events in virtual environments allows the creators of such environments to tightly control the experience of users. However, the formulaic nature of such virtual environments hiders deeper engagement of users with such environments. More often than not, content such as images, audio, conversation text etc. used in the virtual environments are generated by individuals with limited time to produce content. There is significant computational complexity in providing experiences in virtual environments that demonstrate variability while at the same time being integrated with the general flow of events in the virtual environment.Summary

[0004] Some embodiments relate to a system for voice commentary generation, the system comprising: at least one processor; a memory comprising instructions executable by the at least one processor to configure the at least one processor to: receive gaming event stream data, the event stream data comprising information relating events occurring in the process of gameplay;generate first text commentary corresponding to at least a subset of the events using a text generation model; and generate first voice commentary based on the generated first text commentary using a voice synthesis model.

[0005] Some embodiments relate to a computer-implemented method for voice commentary generation, the method comprising: receiving event stream data, the event stream data comprising information relating events occurring in the process of gameplay; generating first text commentary corresponding to at least a subset of the events using a text generation model; and generating first voice commentary based on the generated first text commentary using a voice synthesis model.

[0006] Some embodiments relate to a system for response voice commentary generation in parallel with gameplay, the system comprising: at least one processor; a memory comprising instructions executable by the at least one processor to configure the at least one processor to: receive audio data recorded from a human source and gaming event stream data; transform the audio data into input text data; generate response commentary text using a text generation model based on the input text data and the gaming event stream data; generate response voice commentary based on the response commentary text using a voice synthesis model; make available the generated response voice commentary for playback in parallel with gameplay.

[0007] Some embodiments relate to a computer-implemented method for response voice commentary generation in parallel with gameplay, the method comprising:receiving audio data recorded from a human source and gaming event stream data; transforming the audio data into input text data; generating response commentary text using a text generation model based on the input text data and gaming event stream data; generating response voice commentary based on the response commentary text using a voice synthesis model; making available the generated response voice commentary for playback in parallel with gameplay.Brief Description of the Drawings

[0008] Some embodiments of systems and methods for synthetic voice commentary generation, in accordance with the present disclosure, will now be described, by way of non-limiting example only, with reference to the accompanying drawings in which:

[0009] Figure 1 illustrates a block diagram of a system for synthetic commentary generation;

[0010] Figure 2 illustrates a flowchart of a method of synthetic voice commentary generation; and

[0011] Figure 3 illustrates another flowchart of a method of synthetic voice commentary generation.Detailed Description

[0012] This disclosure relates to systems and methods that generate synthetic voice commentary for integration with a flow of events in a virtual environment such as a computer game. Some of the disclosed systems and methods generate synthetic commentary in response to events or a stream of events occurring in a virtual environment. Some of the disclosed systems and methods generate synthetic commentary in response to audio input received from a user of the virtual environment to provide feedback to the user or provide a virtual back-and-forth conversation-like experience to the user.

[0013] Figure 1 illustrates a system for synthetic commentary generation. System 100 comprises one or more processors 110 and a memory 120. System 100 may be a computer gaming system such as a personal computer or a gaming console. Memory 120 comprises program code implementing executable logic and data for the execution of the steps of the method of generation of synthetic commentary. The embodiments leverage two components for the generation of synthetic commentary. A text generation model 122 generates commentary text. A voice synthesis model 124 transforms the generated speech text or commentary text into voice or speech 130 that can be played by a playback system 140. The playback system 140 may be a speaker system or any system that generates audible sound. In some embodiments, the generation of synthetic commentary may be in part driven based on audio input received from a microphone 150.

[0014] System 100 receives receive an event stream data 104. The event stream data comprises information relating to events occurring in the process of gameplay or events occurring in a virtual environment. The event stream data 104 may comprise the telemetry output of a gaming engine or a virtual environment engine. The event stream data 104 may be generated as a game is being played and may reflect events in the game or virtual environment. The event stream data 104 may comprise a series of discrete data packets, each data packet is timestamped with a time of occurrence of an event to designate a sequence of the events. The data packets may comprise further information including text or numeric data designating details of events occurring during gameplay. The format or structure of the data in the event stream may vary according to the game that is the source of the event stream data. For example, an instance of an event stream data may comprise the information "[Player Name] detected as 1st place at the end of the final lap." Information in the event stream data may relate to outcomes associated with players participating in the gameplay, interactions occurring between two or more players in the gameplay, or interactions between a gaming environment and one or more players.

[0015] The system analyses a subset of the received event stream data to generate text commentary. In some embodiments, memory 120 of the system may comprise unique predefined gaming event signatures, associated with specific categories of types of gaming events. The received event stream data or a series of records in the event stream data may be analyzed by the system to potentially map them onto a predefined gaming event signature from a library of gaming event signatures. For example, an event "[Player Name] detected as 1st place at the end of the final lap" may be mapped onto the signature of "New leading player in the final lap." The gaming event signatures providean abstraction over the received event stream data allowing the generation of text commentary based on the predefined signatures. In some embodiments, the text generation model has access to passive gameplay metrics for use in the generation of text. The passive gameplay metrics may be made available to the text generation model by the gaming software or the gameplay metrics may be derived from the received event stream data.

[0016] The system comprises one or more predefined writing prompts associated with each signature. Examples of writing prompts include:• You are a motorsports play-by-play commentator. [Player name] has just won the race at [Track Name], Congratulate them and note that it’s their first win in X number of starts.• You are a motorsports play-by-play commentator. [Player name] has just won the race at [Track Name]. Congratulate them by referencing their home location in relation to the track location.• You are a motorsports play-by-play commentator. [Player name] has just won the race at [Track Name], Congratulate them and note that it’s their Xth win in this car class.• You are an auto racing play-by-play commentator for an American racing series. [DriverName] has just been involved in a huge crash, resulting in his car flipping upside down. Your job is to react to the incident in a shocked manner, then quickly shift your tone to concern, although you’re fairly likely the driver is unhurt. Your commentary booth is located in a place at the track where you can’t physically see the corner where he crashed, so tell the audience you are waiting to see a replay from the monitors. Make sure to tell the audience that the driver IS moving around in the car and is likely okay.

[0017] An example of generated text commentary based on the last prompt:"Oh my goodness, we have just witnessed a massive crash involving [DriverName]! This is not something you want to see, but we're waiting for a replay to give us a better look at the situation. We can see that the car has flipped over, this is a very scary moment for everyone involved. But wait, I've just received word that [DriverName] is moving around in the car! This is a great sign and I'm relieved to hear that. We will continue to monitor the situation and bring you updates as soon as we have more information. Our thoughtsand prayers are with [DriverName] and the safety crew at this time. Let's take a look at the replay and see what exactly happened here. Ah, it looks like [DriverName]'s car made contact with another car, causing it to lose control and flip over. This is just a very unfortunate situation, but it's good to see that the driver is conscious and moving around. We'll continue to keep a close eye on this and provide updates as they come in. I know all of you join me in hoping that [DriverName] is okay and that this is not a serious incident. This is a reminder of just how dangerous this sport can be, but also the measures that are in place to keep the drivers safe."

[0018] The text generation model 122 generates text based on one or more input prompts. The prompts serve as natural language instruction for the text generation model to generate specific commentary text. The text generation model of some embodiments may be implemented using unsupervised machine learning models such as Generative Adversarial Networks (GANs) or Variational Auto-Encoders (VAEs). The unsupervised machine learning models may be trained using the text of commentary obtained from a real-world environment. For example, commentary from real-world motorsport racing may be obtained to specifically train the text generation model to generate text that sounds more realistic in the context of a simulation of a motorsport race. Similarly, commentary from a real-world football game may be used to train the text generation model to generate realistic-sounding commentary for a football simulation computer game. The text generation model attempts to model linguistic structures, expressions and colloquialisms particular to a real-world environment for use in the generation of realistic commentary text. The context-specific training of the text generation model advantageously focuses the model on generation tasks specific to the virtual environment thereby making the text generation model mode compact, scalable and efficient.

[0019] In some embodiments, the text generation model may be initialized using a personality prompt. The personality prompt may define the basic context, background and tone of the generated text. An example of a personality initialization prompt includes: "You are an American racing engineer, working for a professional-level sports car racing team. You have 17 years of experience in the industry, and have a great reputation for being friendly on the radio - you display a great positive attitude and it's why you've been hired. In this high-stress environment, you need the ability to maintain composure on the radio as your driver may get into intense racing situations and need the contrast of both your voice and word selection to calm him down and keep him focused. However, yougrew up in North Carolina, so your dialect isn't super formal on the radio, in the manner F1 pit engineers are. You're a lot more laid back and use a touch of slang."

[0020] The voice synthesis model 124 generates voice commentary based on the text commentary generated by the text generation model. The voice synthesis model not only transforms text into voice or speech that can be played, but it also models one or more synthetic voices so as to generate life-like commentary of human actors. The voice synthesis model may have access to one or more voice identities, the voice identity being data constructs defining an archetype of a synthetic voice commentary allowing the generation of voice commentary that conforms to a synthetic tone or voice aspect and is recognizable as belonging to a specific voice identity. By using more than one voice identity and switching between the use of the voice identities, the voice synthesis model can emulate a back-and-forth commentary or conversation between the more than one voice identity. In some embodiments, the voice identities may be based on the voices of real-world actors or commentators.

[0021] Figures 2 and 3 illustrate flowcharts of methods performed by system 100. At step 210, system 100 receives event stream data. The event stream data may be telemetry data generated by a gaming engine as described above with reference to Figure 1. At step 220, the event stream data is processed by system 100, in particular by the text generation model 122 to generate text commentary. As described with reference to Figure 1 , the analysis of the received event stream data may comprise identifying a gaming event signature based on a subset of the events. The identified gaming event signature is associated with a text generation prompt that is processed by the text generation model to generate a commentary. The text generation prompt may comprise information relating to the nature of the gaming event surmised from the event stream data and a context of the text intended to be generated. The information relating to the nature of the gaming event may include parameters identifying participants or players associated with the gaming event and a class identifier or descriptor for the kind of gaming event signature identified by analysis of the event stream data. The context of the prompt may include a prompt such as "generate motorsports commentary" etc. The analysis of event stream data and the generation of text commentary occurs in a continuous loop, wherein a subset of the received event stream data is analyzed to identify gaming event signatures and commentary text is generated as the gaming event signatures are identified.

[0022] In some embodiments, multiple distinct text generation prompts may be associated with each gaming event signature. The system may randomly select one of the multiple text generation prompts to initiate the generation of text commentary at 220. The use of multiple distinct text generation prompts and random cycling between the distinct prompts advantageously improves the non-repetitiousness of the generated voice commentary.

[0023] At step 230, the text commentary generated at step 220 is processed by the voice synthesis model to generate voice commentary. The generated voice commentary is in the form of playable audio. The voice synthesis model may comprise a plurality of voice identities, each voice identity being associated with the voice of a virtual or real person. As the text commentary is continuously generated, the voice synthesis model may switch between the available voice identities to generate a series of voice commentaries which when played back give the appearance of a back-and-forth conversation between two or more distinct voice identities.

[0024] In some embodiments, a second text commentary is generated at step 220 by providing to the text generation model a previously generated first text commentary. The input in such embodiments may be in the form of a prompt that indicates "Generate a response to <first text commentary: " or other such prompts to similar effect. By generating a second text commentary based on the first text commentary, system 100 generates a series of voice commentaries that provide the impression of a running conversation between two or more synthetic commentators. At step 240, the generated voice commentary is played back by the playback system 140. The generated voice commentary may be played in parallel with the gameplay as it progresses. The steps of the method of Figure 2 are performed at a rate such that when the generated voice commentary is played, to a user it appears that the voice commentary is generated in real-time or near real-time with respect to the occurrence of events during gameplay. The text generation model and voice synthesis model are optimized to provide the appearance of real-time or near real-time voice commentary generation.Priority Driven Generation

[0025] In some embodiments, the event stream data may also comprise an event priority parameter associated with the respective event. The event priority parameter may inclinator a relative priority of the event and is used by the system 100 to prioritize commentary text generation. As receipt of event stream data and the ultimate playbackof the generated text commentary is a continuous rolling process, in some embodiments, receipt of a high priority event data in the event stream may be used to trigger an interruption in the execution of steps 220, 230 or 240 associated with lower priority event stream data. For example, in the context of racing gameplay, the occurrence of a major collision is a significant high-priority event. Voice commentary previously generated with respect to lower-priority events such as a pass by one player is interrupted by voice commentary associated with the high-priority collision event. By providing event priority parameters in the event stream data, the embodiments advantageously provide a more realistic synthetic commentary generation where the virtual commentary prioritises events of greater interest as they occur.

[0026] The system 100 on receiving event stream data with a high priority parameter value initiates the generation of a second text commentary. The high-priority parameter value may be designated using a numeric value or a 'high priority' label. After the generation of the second text commentary, a second voice commentary is generated by the voice synthesis model. Playback of the first voice commentary is interrupted by the system and playback of the second voice commentary is initiated.

[0027] In some embodiments, the voice generation model is provided as input the event priority parameter associated with the generated text commentary. The voice generation model may be configured to modulate the tone of the generated voice based on the event priority. For example, a grandiose tone may be used to generate voice commentary for the high-priority event. For text commentary associated with a low-priority event, the voice generation model may employ a casual chatter tone. For example:• Text commentary that references the track name, car featured in the race, and expository elements such as a driver’s stats is associated with low-priority events and is generated using a casual tone.• Text commentary that references on-track action, such as overtakes, fastest laps and pit stops may be given a medium priority and be associated with a neutral tone.Text commentary that is associated with serious events, such as a driver winning the race, or a large crash is generated with a high priority rating and the associated voice commentary is generated using a grandiose tone.The priority parameter values are used both for the selection of events to prioritize the generation of voice commentary and the tone used for the generated voice commentary. In some embodiments, the priority for the selection of events for voice commentary generation and the tone of the generated voice commentary may be controlled using separate parameters of the event stream data. For example, an event priority parameter may indicate the relative priority of an event and a heat parameter may indicate the revenant tone to use for the generated voice commentary.Responsive Commentary Generation

[0028] Figure 3 illustrates a flowchart of a method or a part of a method of some embodiments or responsive commentary generation. The method of Figure 3 may be illustrated independently of or in conjunction with the method of Figure 2. The method of Figure 3 relates to the generation of synthetic commentary or responses based on voice input. At step 310, system 100 receives audio input from microphone 150. The audio input may be a voice command, question or direction from a user or a game player. The audio input may be received in conjunction with event stream data being generated with the gameplay. At step 315, system 100 transforms the received audio data to input text data. This transformation may be performed using a speech-to-text conversion model 126.

[0029] At step 320, the text obtained at step 315 is provided as input to the text generation model. The input to the text generation model may be provided in the form of a prompt that is structured to obtain a response to the voice / audio input received at 310. The input may be in the form of a question or a query regarding the gameplay and the prompt may designate a request to generate a response to the query. In generating the response, the text generation model is provided to access to gameplay information such as game statistics to allow the text generation model to generate a meaningful and informed response. The gameplay information may comprise the gaming event stream data captured over time and available for analysis to the text generation model. Output from step 320 is in the form of a response text commentary. Step 320 is similar to step 220 of Figure 2 with the difference being that input to the text generation model in step 320 is the transformed audio input as opposed to game event stream data in step 220.

[0030] At step 330, voice commentary is generated based on the response text commentary of step 320. The generated voice commentary is played back at step 340 and serves as an audio response to the audio received at step 310. The various steps of the method of the flowchart of Figure 3 may be repeatedly performed to simulate a back-and-forth conversation with a synthetic entity responding to comments and questions from a user based on gaming event stream data. In some embodiments, the responsive commentary generation may be intended to emulate a conversation with a virtual gameplay assistant that provides guidance or tips during the course of gameplay. For example, in the context of a racing game, the system may operate as an “Al race engineer” that responds based on a combination of the player’s prompts and the event stream data. The Al engineer could be triggered to prompt information of a previous lap time on completion of a lap. A user quest to the Al Race engineer may be "my car isn't handling very well in turn nine." To respond the system may analyze game events associated with turn nine and generate a suggestion that is played back to the user to allow the user to appropriately respond.Spontaneous Commentary Generation

[0031] In some embodiments, the gameplay events (and associated event stream data) may occur at a fast pace, such as for a simulated basketball game. However, other forms of gameplay may involve gameplay events that occur at lower frequencies, such as a rally racing simulation game. In such games where gameplay events occur at a lower frequency, a reliance on the occurrence of gaming events for commentary generation may lead to lower frequency commentary generation resulting in a lot of "dead air" or gameplay with no associated event-driven commentary. In an F1 race simulation game for example, on some occasions, several minutes may transpire with no cars passing each other, no crashes or other events occurring.

[0032] To remedy this, the system of some embodiments determines a value time since the last event based on the event stream data. On the time since the last event value exceeding a predefined threshold, the system triggers spontaneous commentary generation that is not triggered by the occurrence of an event. The spontaneous commentary generation comprises the generation of a third text commentary by the text generation model. As input for the generation of the third text commentary, the text generation model is provided with one or more gameplay statistics. The gameplay statics may be obtained from the previously received event stream data being generated as part of the gameplay. In some embodiments, the system comprises a plurality of predefined spontaneous commentary generation prompts that may be combined with one or more gameplay statistics and provided as input to the text generation model to generate the third text commentary. Generation and playback of third voice commentary based on thethird text commentary follows similarly to steps 230 and 240 of the method described with reference to Figure 2. In some embodiments, the spontaneous commentary generation prompts may include references or directions to content accessible outside of the game, for example, references to content on external websites. The generated voice commentary may include such references to make the users aware of such content or external websites.

[0033] The various steps of the methods of the flowcharts of Figures 2 and 3 are executed as parallel threads or processes that are interdependent and exchange information to allow continuous processing of the received event stream data and generation of voice commentary. System 100 incorporates parallel processing routines to allow interchange of information between the various threads performing the various steps of the method illustrates in the flowcharts of Figure 2 and 3.

[0034] In some embodiments, the rate of voice commentary generation may be controlled by a generation frequency hyper parameter (chatyness parameter). The value of this hyper parameter may be varied to control how frequently the voice commentary is generated according to the preferences of a user.

[0035] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavor to which this specification relates.

[0036] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0037] The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions,operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.

Claims

Claims1 . A system for voice commentary generation, the system comprising: at least one processor; a memory comprising instructions executable by the at least one processor to configure the at least one processor to: receive gaming event stream data, the event stream data comprising information relating events occurring in the process of gameplay; generate first text commentary corresponding to at least a subset of the events using a text generation model; and generate first voice commentary based on the generated first text commentary using a voice synthesis model.

2. The system of claim 1 , wherein the at least one processor is further configured to make available the generated first voice commentary for playback in parallel with the gameplay.

3. The system of claim 1 or claim 2, wherein the first voice commentary is generated in real-time or near real-time with respect to occurrence of the subset of the events.

4. The system of any one of claims 1 to 3, wherein the text commentary and voice commentary is generated continuously in response to the receipt of the event stream data.

5. The system of any one of claims 1 to 4, wherein the subset of the events is analysed by the at least one processor to identify a first gaming event signature, the gaming event signature being related to a first text generation prompt; wherein the first text generation prompt is processed by the text generation model to generate the first text commentary.

6. The system of claim 5, wherein the subset of the events is analysed by the at least one processor to identify at least one first gaming event parameter; andthe at least one first gaming parameter is processed by the text generation model to generate the first text commentary.

7. The system of any one of claims 1 to 6, wherein the at least one processor is further configured to: generate second text commentary by providing input to the text generation model at least one of: the first text commentary, or a second subset of events; and generate second voice commentary based on the generated second text commentary using the voice synthesis model.

8. The system of claim 7, wherein the first and second voice commentaries are generated using distinct voice identities.

9. The system of any one of claims 1 to 8, wherein the text generation model is configured to generate non-repetitious text commentary.

10. The system of claim 2, wherein the event stream data comprises an event priority parameter associated with each event in the event stream; and wherein the at least one processor is further configured to: receive an event with a high-priority parameter value; generate second text commentary corresponding to the event with a high-priority parameter value; generate second voice commentary based on the generated second text commentary using a voice synthesis model; interrupt playback of the first voice commentary and make available the generated second voice commentary for playback in parallel with the gameplay.

11. The system of any one of claims 1 to 10, wherein the text generation model has access to a plurality of passive gameplay metrics for use in the generation of the first text commentary.

12. The system of any one of claims 1 to 11 , wherein the at least one processor is further configured to: receive audio data recorded from a human source; transform the audio data into input text data; generate response commentary text using the text generation model based on the input text data and the gaming event stream data; generate response voice commentary based on the response commentary text using the voice synthesis model; make available the generated response voice commentary for playback in parallel with the gameplay.

13. The system of any one of claims 1 to 12, wherein the at least one processor is further configured to: determine a time since last event value based on the event stream data; on the time since last event value exceeding a predefined threshold, generate third text commentary based on gameplay statistics derived from the event stream data; generate third voice commentary based on the generated third text commentary using the voice synthesis model; and make available the generated third voice commentary for playback in parallel with the gameplay.

14. A computer-implemented method for voice commentary generation, the method comprising: receiving event stream data, the event stream data comprising information relating events occurring in the process of gameplay; generating first text commentary corresponding to at least a subset of the events using a text generation model; andgenerating first voice commentary based on the generated first text commentary using a voice synthesis model.

15. The method of claim 14, wherein the method further comprises making available the generated first voice commentary for playback in parallel with the gameplay.

16. The method of claim 14 or claim 15, wherein the first voice commentary is generated in real-time or near real-time with respect to occurrence of the subset of the events.

17. The method of any one of claims 14 to 16, wherein the text commentary and voice commentary is generated continuously in response to the receipt of the event stream data.

18. The method of any one of claims 14 to 17, wherein the subset of the events is analysed to identify a first gaming event signature, the gaming event signature being related to a first text generation prompt; wherein the first text generation prompt is processed by the text generation model to generate the first text commentary.

19. The method of claim 18, wherein the subset of the events is analysed to identify at least one first gaming event parameter; and the at least one first gaming parameter is processed by the text generation model to generate the first text commentary.

20. The method of any one of claims 14 to 19, wherein the method further comprises: generating second text commentary by providing input to the text generation model at least one of: the first text commentary or a second subset of events; and generating second voice commentary based on the generated second text commentary using the voice synthesis model.

21. The method of claim 20, wherein the first and second voice commentaries are generated using distinct voice identities.

22. The method of any one of claims 14 to 21, wherein the text generation model is configured to generate non-repetitious text commentary.

23. The method of claim 15, wherein the event stream data comprises an event priority parameter associated with each event in the event stream; and the method further comprises: receiving an event with a high priority parameter value; generating second text commentary corresponding to the event with a high priority parameter value; generating second voice commentary based on the generated second text commentary using a voice synthesis model; interrupting playback of the first voice commentary and make available the generated second voice commentary for playback in parallel with the gameplay.

24. The method of any one of claims 14 to 23, wherein the text generation model has access to a plurality of passive gameplay metrics for use in the generation of the first text commentary.

25. The method of any one of claims 14 to 24, wherein the method further comprises: receiving audio data recorded from a human source; transforming the audio data into input text data; generating response commentary text based on the input text data using the text generation model; generating response voice commentary based on the response commentary text using the voice synthesis model; and making available the generated response voice commentary for playback in parallel with the gameplay.

26. The method of any one of claims 14 to 24, wherein the method further comprises: determining a time since last event value based on the event stream data;on the time since last event value exceeding a predefined threshold, generating third text commentary based on gameplay statistics derived from the event stream data; generating third voice commentary based on the generated third text commentary using the voice synthesis model; and making available the generated third voice commentary for playback in parallel with the gameplay.

27. A system for response voice commentary generation in parallel with gameplay, the system comprising: at least one processor; a memory comprising instructions executable by the at least one processor to configure the at least one processor to: receive audio data recorded from a human source and gaming event stream data; transform the audio data into input text data; generate response commentary text using a text generation model based on the input text data and the gaming event stream data; generate response voice commentary based on the response commentary text using a voice synthesis model; make available the generated response voice commentary for playback in parallel with gameplay.

28. A computer-implemented method for response voice commentary generation in parallel with gameplay, the method comprising: receiving audio data recorded from a human source and gaming event stream data; transforming the audio data into input text data; generating response commentary text using a text generation model based on the input text data and gaming event stream data;generating response voice commentary based on the response commentary text using a voice synthesis model; making available the generated response voice commentary for playback in parallel with gameplay.

Citation Information

Patent Citations

  • An artificial intelligence explanation method and a system for a basketball game

    CN109145733A

  • Sports event intelligent narrate method, server and display equipment

    CN112040272A

  • Automatic commentary device and method for football match

    CN113268515A

  • Natural language configuration and operation for tangible games

    US11645947B1

  • Capturing asynchronous commentary to pre-recorded gameplay

    US20160361646A1