Method and system for generating audio-visual content from video game footage

Abstract

A computer implemented method of generating audio-visual content comprising identifying one or more gameplay events during gameplay, recording event data to produce a timestamp, outputting the video d

Description

FIELD OF THE INVENTION The present disclosure relates to a method of generating audio-visual content from video game footage and a system for generating audio-visual content from video game footage. BACKGROUND It is often desirable to generate audio-video content from gameplay or video game footage. For example, at the end of gameplay, it may be desirable to generate a compilation video corresponding to any highlights, and / or key events that occurred within the gameplay. Currently, when capturing and recording gameplay, the captured gameplay contains the music that was present during gameplay. Depending on the gameplay this can sometimes be very little or no music as this is what is required for gameplay, but this is not necessarily an optimum or fun experience for postgameplay viewing. Content generated from video game footage may be enhanced by adding backing music. For example, it may be desirable to have highlights and / or key events in video game footage matched with a particular sound or particular music track that is distinguishable from the in-game music track that was present during gameplay. Being able to distinguish important or interesting gameplay events from general gameplay, for example using different audio often results in a far more engaging video for the viewer. Often consumers who upload captures of their gameplay to online services overlay new non-game related music to make their videos more exciting. Usually, the creation of video content from video game footage is facilitated by postprocessing software. Generally, such software allows users to import and edit video clips, as well as import audio for accompanying the imported video clips to create a final video. Typically, the user will have to move and trim the different video clips to sync any key moments with parts of the imported audio. Whether or not the video game footage is accompanied by the audio in an appropriate manner will normally be at the user's judgement. This can result in slight mismatches between key moments occurring within the video game footage and the parts of the audio that the user had intended to accompany those moments. The present invention seeks to alleviate at least some of these problems. SUMMARY OF INVENTION According to a first aspect there is provided a computer-implemented method of generating audio-visual content from a video gameplay sequence, the method comprising: during gameplay, identifying one or more gameplay events for audio synchronisation; recording event data for each identified gameplay event, where the event data comprises an event timestamp; outputting video data comprising a recorded sequence of gameplay, and outputting the event data for identified events within the sequence of gameplay; retrieving an audio file comprising audio data including one or more audio features; generating a synchronised audio-visual sequence by aligning the audio file with the video data such that an audio feature is synchronised with the event time stamp of an identified event within the video data The method can be used to create a dynamic score for gameplay video playback based on one or more gameplay events which are identified as being key moments of gameplay, for example first kill, first death, capturing a base etc. During active gameplay, key gameplay moments can be identified, and the timestamp of these key gameplay moments can be recorded to create a saved record of key events for the gameplay. Once the gameplay has finished, a player (also referred to as a user) is able to playback a recording of their saved gameplay. During playback of the saved gameplay an audio file is played which dynamically follows the timeline of key events that occurred during gameplay. This is achieved by aligning the audio file with the video data such that an audio feature present in the audio file is synchronised with the event time stamp of the key events in the gameplay. This allows a tightly synchronised audio-visual sequence to be generated which is tailored to the specific gameplay of a player. Identifying one or more gameplay events may comprise automatically identifying events during gameplay according to a pre-determined list of events stored in a database. In this way, the events are identified in real time while the player is playing the game. This avoids the need to analyse the video data of the recorded gameplay to try and identify key moments after the player has finished playing a game. The events may be identified automatically from the gameplay by a gameplay engine. In this way, the video data does not need to be processed. Instead, the gameplay engine knows when an event happens without having to perform any video data analysis to determine this. Thus, the events can be identified quickly and in real time during gameplay. Avoiding the need to process and analyse video data reduces the computational resource requirements for event identification. The gameplay engine may automatically identify events from the gameplay using object detection and / or object tracking. In this way, an object detection algorithm may be used to label / identify objects within the gameplay. In a specific example, image recognition could be used to identify a football and track as it moves across the screen and when it enters a goal. When the football enters the goal, the game engine may automatically identify this as an event, or more specifically, a “goal scoring” event. In some examples, gameplay engine may automatically identify events from the gameplay using simulations within a gameplay engine. The gameplay engine may replicate real world physics to create interactions between objects. For example, the gameplay engine may simulate a collision event wherein an object collides with another object. In this way, by simulating a collision event, two objects are known to be next to each other such that an event can be detected. For example, if two characters collide, it may be known to the gameplay engine that this is a collision event, or more specifically, a “first kill” event. In some examples, gameplay engine may automatically identify events from the gameplay using rule based event detection. The gameplay engine may be preprogrammed with one or more preset rules based on object properties or special relationships. When it is determined by the gameplay engine that a rule has been satisfied and / or completed it may identify that an event has occurred. The gameplay engine may identify the events based on the pre-determined list of events. In some examples, the player may be able to edit the list of events stored in the database. For example, the player may be able to add new events to the list of pre-determined events or the player may be able to remove events from the list of pre-determined events. This allows the player to customise the list of events which are considered as key moments or important events during a particular gameplay. As a result, the generated synchronised audio-visual sequence is tailored and customised to the individual player. The event timestamp may be stored as metadata in the event data. The metadata may be associated with one or more data packets. In this way, the metadata, or data packets, can be analysed to extract the content of the data or data packet so as to extract the timestamp. In some examples, identifying one or more events may comprise receiving a player-command through a game controller during runtime to identify an event. Receiving a player-command refers to receiving an input or action performed by the player. For example, the player may press one or more buttons on the game controller or peripheral device so as to identify an in-game event. The one or more events may be identified by receiving a player-command in addition to or instead of identifying the one or more events automatically during gameplay. In this way, the player may identify events in the gameplay, to which the audio file is later aligned with, which may be additional events to those automatically detected for example by the gameplay engine. This provides further opportunity for the player to customise the generated synchronised audio-visual sequence. The event data preferably comprises event information encoding the type of event. The type of event can refer to a specific action, incident or situation that occurs ingame. The type of events can vary based on the specific game the user is playing and may include one or more of a fight event, a progression event, an achievement event etc. The type of events may be a generic event, such as those mentioned previously or more a specific event, such as a subcategory of the generic event. Examples of a specific event include a first kill, a first death, a win or lose event, a captured base etc. The event information may be used to compare event data and group similar event data together. For example, event data comprising event information relating to a first type of event may form a first subset and event data comprising event information relating to a second type of event may form a second subset. In this example, the first type of event is different from the second type of event. In this way, the same or similar audio can be applied to a given type of event so as to improve customisation to the specific gameplay of a player. The audio file may be automatically selected based on the event information. This may allow events of a similar type to be associated with similar, or the same, audio files so that during playback of the generated a synchronised audio-visual sequence similar events can be identified aurally. Moreover, this reduces the manual input required from the player to generate a synchronised audio-visual sequence as similar events can be automatically synchronised with an appropriate audio file. Preferably, outputting the event data comprises outputting an event timeline data structure. The event timeline data structure preferably comprises the timestamps of the identified events. The timestamps of the identified events may be arranged in chronological order to form a timeline of identified events. The event timeline data structure provides a convenient means of storing the identified events in chronological order, based on the timestamps of the identified events. This help ensure that the identified events remain in the order in which they happened during gameplay. The event timeline data structure may be displayed on a screen. Preferably the event timeline data structure is output and displayed using graphical representation. This provides the player with a visual indication of whereabouts during gameplay the identified events occurred. This may allow trends in identified events during gameplay to be identified easily. For example, a cluster of identified events within a specified time period may be determined. Graphical representation of the event timeline data structure may also allow players to add audio to the identified events or to edit audio associated with the identified events. In some examples, the event timeline data structure may comprise the type of event. In this way, the player can easily understand which identified events occurred and when they occurred, thereby improving the ease at which the player can customise the gameplay. Preferably, the event timeline data structure is used to generate the audio visual sequence by aligning audio features of one or more audio tracks against the timeline of identified events. This may provide a simple and effective mechanism for helping ensure that the audio features are synchronised with the identified events. The video data and event data may be output to a video editing unit comprising a user interface allowing a player to edit the audio visual sequence. The player is therefore able to customise the audio visual sequence. According to another aspect there is provided a computer-implemented system of generating audio-visual content from a video gameplay sequence, the system comprising a processor configured to: during gameplay, identify one or more gameplay events for audio synchronisation; record event data for each identified gameplay event, where the event data comprises an event timestamp; output video data comprising a recorded sequence of gameplay and the event data for identified events within the sequence of gameplay; retrieve an audio file comprising audio data including one or more audio features; generate a synchronised audio-visual sequence by aligning the audio file with the video data such that an audio feature is synchronised with the event timestamp of an identified event within the video data. In some examples, the system comprises one or more processors, wherein each processor is configured to carry out one or more of the method steps according to the present invention. A computer of a gaming console system may comprise the one or more processors. In some examples, the one or more processors may be external to the gaming console system, and may be in communication with the gaming console system. BRIEF DESCRIPTION OF DRAWINGS Figure 1 is a schematic diagram of a system in which a method according to the invention may be implemented; Figure 2 is a flow chart schematically illustrating steps of a method according to the present invention; Figure 3 is a schematic diagram of an exemplary display of a gaming console system according to the present invention; Figure 4 is a schematic diagram of audio sequence according to the present invention; Figure 5 is a schematic diagram of an exemplary display of a gaming console system according to the present invention. DETAILED DESCRIPTION As described herein, video game players will often generate audio-video content comprising any highlights, and / or key events that occurred within the gameplay. Players will typically enhance the video game footage by adding backing music, often matching these highlights and / or key gameplay events to particular sounds or music. The methods and systems described herein allow a tightly synchronised audio-visual sequence to be generated which is tailored to the specific gameplay of a player. In this way, the problems with mismatches between key moments occurring within the video game footage and the parts of the audio that the user had intended to accompany those moments are alleviated. Figure 1 is a schematic diagram of a system 100, and in particular a gaming console system 100, in which a method according to the present invention may be implemented. As shown in Figure 1, the gaming console system 100 may comprise a display screen 102 coupled to a computer 101 via a wired or wireless connection. The computer 101 has at least a processor and is configured to facilitate game play and / or access to applications. The display screen 102 may be, for example, a television, a home cinema system, or a projector. During gameplay, the visuals of the game will be displayed to the user via the display screen 102. The display screen 102 can also be used to display and / or edit the audio-visual sequence once generated using a method described herein. In other examples, the computer 101 may not be a part of a gaming console system 100 and instead may be a personal computer, a laptop, a mobile device etc. In some examples, the display screen 102 and the computer 101 may be combined in a single device. The gaming console system 100 may carry out one or more steps of the present method. In some examples, all the steps of the present method are carried out by the gaming console system 100. In other examples, the user may use the gaming console system 100 to execute the game so as to record event data and video data. The user may then export this data from the gaming console by memory card, Bluetooth etc to a secondary computer or laptop where the remaining steps of the method can be carried out. Figure 2 is a flow chart schematically illustrating a method 200 of generating audio-visual content from a video gameplay sequence according to the present invention. At step S201, the method includes identifying, during gameplay, one or more gameplay events for audio synchronisation. The one or more gameplay events may comprise moments of gameplay, such as specific actions carried out by the user, dialogue occurring in-game that may signify a key moment in the gameplay, or any other significant event. Other specific examples of gameplay events include a player executing a first kill, a first death of the player or an enemy, the player capturing a base or completing a level etc. The nature of the gameplay events will vary across different games such that what is considered a gameplay event in one game may differ from that of another game. Identifying one or more gameplay events may comprise automatically identifying events during gameplay according to a pre-determined list of events stored in a database. The list of events may include any key events that occur in the game, such as those described above. The pre-determined list of events may be specific to the game which the user is / was playing. In some examples, the pre-determined list of events specific to the game may be manually selected before gameplay or may be automatically identified before / during gameplay. Identifying one or more gameplay events for audio synchronisation may comprise automatically identifying said events using one or more machine learning models or algorithms. Identifying one or more events may comprise receiving a player-command through a game controller during gameplay to identify the one or more gameplay events. As such, the system 100 of Figure 1 may further comprise a peripheral device in which the user can input the player-command. The user may input the player-command by pressing a button, moving the joystick etc of the peripheral device. The computer 101 of Figure 1 may therefore also be configured so as to identify the one or more events when the user executes the player-command. In some examples, the computer 101 of Figure 1 may be configured such that when the player-command is received, the event occurring in-game at the instant the player-command is received is identified as an event. In some examples, the user does not have to execute player-command at the exact instant of the event they wish to identify, and instead the computer 101 may be configured to automatically identify an event that occurred within a predetermined time period before or after the player-command is received. In a specific example, the user may kill an enemy during gameplay and may wish to identify this as an event for audio synchronisation. The user can carry out the player-command, for example by pressing a button, after the enemy kill event. If this player-command is received within a predetermined time period after the enemy kill event, the system will look for significant events occurring within the predetermined time period before the player-command is received and will automatically identify the enemy kill as an event. At step S202, the method includes recording event data for each identified gameplay event. The event data comprises an event timestamp. The event timestamp may indicate a start and / or end time of the identified gameplay event. Alternatively, or additionally, the event timestamp may indicate the duration of the identified gameplay event. The event timestamp can be stored as metadata in the event data, which can later be extracted when generating the synchronised audiovisual sequence. In this way, the player can create a saved record of key events for the gameplay. The event data can comprise event information encoding the type of event. The type of event may be a generic category such as a fight event or may be more specific to a subcategory of the generic category such as a first kill, a first death etc. The event information may be used to compare event data and group similar event data together. The event data can be grouped using machine learning algorithms. At step S203, the method includes outputting video data comprising a recorded sequence of gameplay and the event data for identified events within the sequence of gameplay. This video data may be a continuous recording of the gameplay or it may be more than one recording. The video data may be stored in a library on the videogame console system 100 or may be stored in a database that can then be accessed by the player at a later date. Alternatively, the video data may not be of the user’s own gameplay and may be from another user which can be accessed from, for example, the internet. In some examples, the outputted video data and / or the event data may be pre-processed. For example, the video data may be cropped, clipped or enhanced so as to prepare it into the appropriate format. For example, the user may wish to resize a video such that it is of the appropriate scale for uploading to a video streaming service, for example. Moreover, the user may wish to clip the video data such that it is the appropriate length to fit the desired audio data. Figure 3 is a schematic diagram of an exemplary display screen 102 of a gaming console system 100 according to the present invention. As shown in Figure 3, outputting the video data can comprise outputting an event timeline data structure 301. The event timeline data structure 301 comprises the timestamps 302 of the identified events. Figure 3 shows a graphical representation of the event timeline data structure 301 wherein the identified events are arranged in chronological order to form a timeline of identified events. As shown, in this example the timestamps 302 of the identified events are positioned along a timeline axis. The timeline axis may display markers to indicate the time or relative time. The user may be able to modify the position of the events along the timeline axis using, for example, a peripheral device. In some examples, the event timeline data structure 301 additionally comprises the type of event. As shown in Figure 3, the event type is displayed to the user using text labels: “enemy attack”, “enemy kill”, “crash event” and “player death”. In other examples, such as when event type information is not available, the system may display text such as “event 1”, “event 2” etc. In other examples, the event timeline data structure 301 may be output to a database or to another storage location. Referring back to Figure 2, at step S204, the method comprises receiving an audio file comprising audio data including one or more audio features. In some examples, retrieving the audio data may include the user selecting a specific audio track or clip that they wish to sync with the gameplay footage. The audio file may be selected from a personal database or library or may be selected from a host music service. Such storage may be local or on a cloud network. The audio data may be at least a portion of a song or may be one or more other audio clips. Figure 4 shows an exemplary audio signal 402 with audio features 403. Audio features 403 of the audio data may refer to, for example, a crescendo, a tempo change, a change in instrument combination etc. These audio features 403 can then be synced with key events in the video footage to increase tension and / or invoke an emotional response of the user when watching the gameplay footage. The audio file can be automatically selected based on the event information. A specific audio file may include audio features that are often paired with a given type gameplay event. In a specific example, an audio file comprising a crescendo is selected when the event information indicates the event type is an enemy kill since a crescendo can increase the user excitement and engagement with the footage. The audio file can be automatic selected from a pre-stored database where audio files or sounds are associated with specific event types. For example, “win” events may be associated with a specific audio file such as “Audio file 1”. In other examples, events may be associated with more than one audio file and the system 100 can select an audio file from a list of audio file such as “Audio file 1”, “Audio file 2” and “Audio file 3”. In other examples, machine learning algorithms may be used for the automatic selection of audio files. The machine learning model may be trained using a database of event information with associated audio files. The model may then be used on the event information to predict an appropriate audio file for that event type. Referring back to Figure 2, at step S205, the method includes generating a synchronised audio-visual sequence by aligning the audio file with the video data such that an audio feature is synchronised with the event timestamp 302 of an identified event within the video data. In some examples, the synchronised audio-visual sequence is generated by identifying an event, such as a “win”, obtaining a corresponding audio file, such as “Audio file 1”, from a database and synchronising this audio file with the event timestamp. In some examples, the audio file will comprise a number of pre-composed audio features. Each audio feature may be purpose composed and purpose implemented for specific events. Therefore, when a specific event is identified, the corresponding audio feature is obtained from the audio file. Obtaining these audio features one after another creates the full audio track of the audio-visual sequence. In some examples, the system may comprise a purpose built Graphic User Interface (GUI). In this example, a user may choose to customise which ‘audio features’ is associated with a given event, by selecting these from a database of audio features. In other examples, specific audio features in a sound track may need to be extracted. For example, the user may wish to synchronise the chorus of a song in “Audio track 1” with a “win” event. In such an example, the one or more audio features can be automatically or manually detected in the audio file. To automatically detect the audio features, the computer 101 may identify distinctive features in the audio signal of the audio file such as an increase in amplitude, frequency, harmonics etc. Machine learning algorithms may be used to automatically detect and label the audio features. Therefore, when detecting the audio features, each feature may be given a unique identifier or timestamp. The unique identifier may be based on characteristics of the audio feature. For example, the unique identifier may label the audio feature as a “crescendo”. In this way, the audio features can be synchronised with the appropriate identified event. In some examples, the event timeline data structure 301 is used to generate the audio-visual sequence by aligning audio features 403 of one or more audio tracks against the timeline 501. Figure 5 is a schematic diagram of an exemplary display 102 of a gaming console system 100 according to the present invention. Shown is an exemplary display screen 102 wherein the event timeline data structure 501 is displayed along with an audio sequence 502. As shown, the audio features 503 of the audio track are aligned against the timeline 501. The audio features 503 are aligned with the timestamps of the identified events. In this example, one audio track is used and displayed but in other examples, more than one audio track may be displayed against the timeline 501. By displaying the event timeline data structure 501 in graphical form on the display screen 102, the user may add audio to the identified 5 events, or may edit the existing synchronised audio. In some examples, the video data and event data are output to a video editing unit comprising a user interface allowing a user to edit the audio-visual sequence. The user may use the peripheral device to alter the audio sequence so as to improve or customise the alignment of the audio with the video. For example, the audio 10 features may be moved to better align with the identified events or the user may increase the amplitude of the audio signal at given points.

Claims

1. A computer-implemented method of generating audio-visual content from a video gameplay sequence, the method comprising:during gameplay, identifying one or more gameplay events for audio synchronisation;recording event data for each identified gameplay event, where the event data comprises an event timestamp;outputting video data comprising a recorded sequence of gameplay and the event data for identified events within the sequence of gameplay;retrieving an audio file comprising audio data including one or more audio features;generating a synchronised audio-visual sequence by aligning the audio file with the video data such that an audio feature is synchronised with the event timestamp of an identified event within the video data.

2. The method of claim 1 wherein the identifying one or more gameplay events comprises automatically identifying events during gameplay according to a pre-determined list of events stored in a database.

3. The method of claim 1 or 2 wherein the identifying one or more events comprises receiving a player-command through a game controller during gameplay to identify an event.

4. The method of any preceding claim wherein the event data comprises event information encoding the type of event.

5. The method of claim 4 wherein the audio file is automatically selected based on the event information.

6. The method of any preceding claim wherein outputting the video data comprises outputting an event timeline data structure, the event timeline data structure comprising the timestamps of the identified events.

7. The method of claim 6 wherein the timestamps of the identified events are arranged in chronological order to form a timeline of identified events.

8. The method of claim 6 or claim 7 wherein the event timeline data structurecomprises the type of event.

9. The method of claim 7 wherein the event timeline data structure is used to generate the audio visual sequence by aligning audio features of one or more audio tracks against the timeline.

10. The method of any of claims 6 to 9 comprising displaying the event timeline data structure in graphical form allowing players to add audio to the identified events.

11. The method of any preceding claim comprising outputting the video data and event data to a video editing unit comprising a user interface allowing a player to edit the audio visual sequence.

12. A computer-implemented system of generating audio-visual content from a video gameplay sequence, the system comprising:a processor configured to:during gameplay, identify one or more gameplay events for audio synchronisation;record event data for each identified gameplay event, where the event data comprises an event timestamp;output video data comprising a recorded sequence of gameplay and the event data for identified events within the sequence of gameplay;retrieve an audio file comprising audio data including one or more audio features;generate a synchronised audio-visual sequence by aligning the audio file with the video data such that an audio feature is synchronised with the event timestamp of an identified event within the video data.

Images