Method and system for generating audiovisual content from video game footage

Abstract

SOLUTION: A method for generating audio-visual content from video game footage is provided. The method comprises obtaining a user-selected audio track and obtaining video game footage. Statistical analysis is performed on the audio track so as to determine an excitement level associated with respective portions of the audio track. Statistical analysis is performed on the video game footage so as to determine an excitement level associated with respective portions of the video game footage. Portions of the video game footage is matched with portions of the audio track, based on a correspondence in the determined excitement level. Based on the matching, a combined audio-visual content comprising the portions of the video game footage matched to corresponding portions of the audio track is generated. In this way, calm and exciting moments within the video footage is matched to corresponding moments in the audio track. A corresponding system is also provided.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a method for generating audiovisual content from video game footage and a system for generating audiovisual content from video game footage. [Background technology]

[0002] It is often desirable to generate video content from video game footage. For example, at the end of a video game tournament, it may be desirable to generate an edited video corresponding to highlights and / or bloopers that occurred during the video game tournament. In some cases, a competitor may simply want to review their gameplay during a single video game session or even multiple sessions of the same or different video game. Currently, there are numerous online video channels that utilize video game footage, some of which have followers in excess of 10 million.

[0003] The addition of background music can enhance video content generated from video game footage. For example, it may be desirable to match a particularly dramatic moment in the video game footage with a corresponding dramatic moment in the music track. Similarly, it may be desirable to match a calmer moment in the video game footage with a calmer portion of the same music track. The introduction of music often results in a video that is more appealing to viewers. More commonly, competitors may want to generate music videos using video game footage of themselves or others.

[0004] Post-processing software typically facilitates the creation of video content from video game footage. For example, PS4 offers "SHAREFactory" (trademark) that allows players to edit gameplay videos captured on the PS4. Typically, such software allows users to import and edit video clips, as well as import audio to accompany the imported video clips, to create the final video. Often, users must move and trim various video clips in a timeline to synchronize highlight moments with corresponding exciting portions of the imported audio (and similarly, low-light moments with calmer portions of audio). It is typically up to the user to determine whether audio accompanies the video game footage in an appropriate manner. This can result in slight mismatches between highlight moments occurring in the video game footage and portions of the audio the user intended to accompany those moments.

[0005] Some video editing software may provide information about the song that the user imports, such as the timing of the beat. The user can use this information to trim or position the video clips on a timeline to time the video cuts and transitions to the beat of the song. However, creating video content in this manner is still a somewhat labor-intensive process for the user, and in some cases, the beat may not necessarily indicate whether a segment of the song actually corresponds to the "exciting" part of the song.

[0006] The present invention seeks to alleviate these problems. Summary of the Invention

[0007] According to a first aspect disclosed herein, there is provided a method for generating audiovisual content from video game footage as claimed in any one of claims 1 to 11.

[0008] According to a second aspect disclosed herein, there is provided a system for generating audiovisual content from video game footage as claimed in any one of claims 13 to 25. [Brief description of the drawings]

[0009] To aid in understanding the present disclosure and to show how embodiments may be practiced, reference will now be made by way of example to the accompanying drawings in which:

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a video game playing device. [Diagram 2] FIG. 2 is a diagram illustrating a schematic example of a method for generating audiovisual content according to the present disclosure. [Figure 3A] FIG. 2 shows an example of a spectrogram of an audio track. [Figure 3B] FIG. 2 shows an example of two portions of an audio track, each portion having a different fullness. [Figure 4] FIG. 1 illustrates an example of a normal distribution and corresponding z-scores. [Diagram 5] FIG. 2 illustrates an example of successive video frames of a video game and frames representing the pixel intensity differences between those frames. [Figure 6] FIG. 2 is a diagram illustrating an example of an LAB color space. [Figure 7] FIG. 1 shows example frames of a video game video and example scores determined for each frame. [Figure 8] FIG. 1 illustrates a schematic diagram of an example of a system for generating audiovisual content according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] As an example of a video game playing device, Figure 1 illustrates generally the overall system architecture of a Sony® PlayStation 4® entertainment device. It will be appreciated that the device illustrated in Figure 1 is merely exemplary, and that in some embodiments the entertainment device may include a next generation console, such as a Sony® PlayStation 5® device.

[0012] A system unit 10 is provided, and various peripheral devices can be connected to the system unit. The system unit 10 includes an Accelerated Processing Unit (APU) 20, which is a single chip including a Central Processing Unit (CPU) 20A and a Graphics Processing Unit (GPU) 20B. The APU 20 has access to a Random Access Memory (RAM) unit 22.

[0013] APU 20 communicates with a bus 40 and optionally through an I / O bridge 24 , which may be a separate component or part of APU 20 .

[0014] Connected to the bus 40 are data storage components such as a hard disk drive 37, a Blu-ray drive 36 operable to access data on a compatible optical disc 36A, etc. Additionally, a RAM unit 22 may be in communication with the bus 40.

[0015] An optional co-processor 38 is also connected to bus 40. The co-processor 38 may be provided to run or support an operating system.

[0016] The system unit 10 communicates with peripherals, as needed, via an audio / visual input port 31, an Ethernet port 32, a Bluetooth wireless link 33, a Wi-Fi wireless link 34, or one or more Universal Serial Bus (USB) ports 35. Audio and video may be output via an AV output 39, such as an HDMI port.

[0017] The peripherals may include a monoscopic or stereoscopic video camera 41, such as the PlayStation Eye®, a wand-style video game controller 42, such as the PlayStation Move®, and a conventional handheld video game controller 43, such as the DualShock4®, portable entertainment devices 44, such as the PlayStation Portable® and PlayStation Vita®, a keyboard 45 and / or a mouse 46, a media controller, e.g., a remote control 47, and a headset 48. Similarly, other peripherals are contemplated, such as a microphone, speakers, a mobile phone, a printer, or a 3D printer (not shown).

[0018] GPU 20B, optionally in cooperation with CPU 20A, generates video images and audio for output via AV output 39. Optionally, audio may be generated in cooperation with or by an audio processor (not shown).

[0019] The video, and optionally the audio, may be presented to a television 51. If the television supports it, the video may be stereoscopic. The audio may be presented to a home cinema system 52 in one of several formats, such as stereo, 5.1 surround sound, 7.1 surround sound, etc. Similarly, the video and audio may be presented to a head mounted display unit 53 worn by a user 60.

[0020] By default, the entertainment device runs an operating system such as a variant of FreeBSD® 9.0. The operating system may run on CPU 20A, on auxiliary processor 38, or on a combination of the two.

[0021] The entertainment apparatus may comprise a temporary buffer for storing video generated by the entertainment apparatus during the playing of a video game, the temporary buffer may represent a ring buffer configured to periodically record the last X minutes of video game footage, where X may represent, for example, 15 minutes.

[0022] Alternatively or additionally, the entertainment apparatus may be configured to transmit video generated during video game play to a server as a continuous video stream, for example over a Wi-Fi connection. The video may be stored and processed at the server for later retrieval by the entertainment apparatus. As can be seen from the above, storing video outside of the entertainment apparatus allows for significantly longer than 15 minutes of gameplay to be recorded at a time. Processing (local and / or remote) of video game footage is further described in the embodiments described herein.

[0023] The entertainment apparatus may further include or have access to a listening unit (not shown) configured to automatically detect and capture recordings of highlight events occurring in a video game played on or through the entertainment apparatus. The listening unit may, for example, utilize machine learning to detect such highlight moments. An example of such a listening unit is described in GB 1819865.5. Video clips corresponding to the highlight events may be stored in RAM of the entertainment apparatus or retrieved from a server in communication with the entertainment apparatus.

[0024] The entertainment devices described in FIG. 1 are examples of devices that may be used to capture video game footage.

[0025] As mentioned above, a user may wish to create a more appealing video by matching video game footage to music. However, this matching process typically requires at least some manual effort, resulting in a suboptimal final video. Desirably, this matching is performed automatically, matching portions of the video game footage to corresponding portions of the music in terms of excitement level. This means, for example, that exciting moments in the video game footage, such as a player defeating an opponent, winning a match, crashing a car, causing an explosion, etc., can be automatically synchronized to portions of the music that correspond to, for example, a chorus, crescendo, solo, key change, etc. Similarly, less exciting moments in the video game footage (so-called "low lights") can be matched to calmer portions of the same music.

[0026] Furthermore, it would be desirable to be able to do this matching in a manner that does not require prior knowledge of the video game (the video game from which the video game footage originates) or the audio track to which the video game footage is to be matched. That is, it would be desirable to be able to perform the matching process in a manner that is video game and song independent. A method for generating such video content is described with reference to FIG.

[0027] Figure 2 shows an example of a method for generating audiovisual content according to the invention, where the expression audiovisual content is used to denote a video that includes visual content (such as RGB or YUV video frames) and audio (such as music).

[0028] In a first step S201, a user selected audio track is obtained. This may include, for example, selecting music that the user would like to use in creating a music video from the video game footage. The music may be, for example, an audio track selected from one's own collection or a music hosting service such as Spotify™, Apple Music™, Tidal™, etc. In some cases, the user may have uploaded a song or audio that they created themselves. For example, the audio track may be music generated in the video game Dreams™ using sound canvas™. The audio selection may be provided as an option within a post-processing application through which a music video according to the present disclosure may be generated.

[0029] In some examples, the post-processing application may allow a user to select the portion of a song that they want to use in creating a music video, for example using a trimming tool. This may be appropriate, for example, if a user wants to create a relatively short video that corresponds primarily to highlights, and thus it may not be desirable to use, for example, a five-minute track.

[0030] In a further or alternative example, a user may upload multiple different tracks or portions of tracks to fit into respective segments of the video game footage.

[0031] In a second step S202, at least one video generated by at least one video game playing device during the playing of at least one video game is captured. The video may include, for example, a plurality of RGB or YUV frames. Although this is described as a second step, it will be appreciated that the user may import the audio and video into the post-processing application in any order. The at least one video may be captured by a user who has selected one or more videos for import into the post-processing application. The video may be, for example, an MPEG4 file, and audio output during the playing of the video game may be isolated and muted, if desired.

[0032] In some examples, a user may select a single video to match with the imported audio. As previously mentioned, a video game playing device may be configured to capture a single continuous recording of the last X minutes of a player's gameplay, and this video may be used to match with the imported audio track.

[0033] Alternatively or additionally, this single video may not be of the athlete's own gameplay, but rather a video of another athlete's gameplay accessible via a video sharing service such as YouTube™, Twitch™, etc. For example, it may be possible to obtain a relatively long video from one or several different athletes that includes gameplay of one or several different games, which can be imported into a post-processing application.

[0034] In some examples, a user may select multiple recordings of gameplay of a video game to be used in creating a music video with a selected audio track. For example, a user may keep a library of recordings corresponding to moments in gameplay captured manually (e.g., via a "Share" button) or automatically over several video game sessions. The PS4's "Capture Gallery" is an example of a library that can keep these recordings. A user may want to generate a music video using all recordings generated over an extended period of time, such as the last month. In some cases, a contestant may want to generate a music video using video clips from a particular game and may select recordings specific to that game.

[0035] It will be appreciated that in some examples, the selection of the video(s) may be facilitated through some level of automated creation and classification. For example, the videos may be classified by game and date, and it is these videos that the user can select when creating a music video. Furthermore, the selection step itself may include some level of automation, e.g., the user need only select the game and / or time period from which they wish to generate a music video, and the corresponding video clips are automatically imported into the post-processing application.

[0036] In additional or alternative examples, at least some of the plurality of video recordings may not correspond to the user's own video game footage. For example, a user may wish to generate a music video using video clips provided via video hosting platforms such as YouTube™ or Twitch™. In one example, the inventors discovered that a 15-hour "Lets Play" video clip was suitable for interfacing with a 3-minute audio track.

[0037] In some embodiments, a user may wish to generate a music video for an offline or online (i.e., e-sports) video game tournament. In such embodiments, the video imported into the post-processing application may be video game footage captured for each different competitor in the tournament. These video clips may be captured (manually or automatically) during the tournament and uploaded to a user-accessible server, or the like. These video clips may be imported into the post-processing application for use in generating a music video for the tournament.

[0038] In general, the total length of the video game footage used to match with the audio track is preferably relatively long, e.g., 15 minutes or more. With longer videos, the contestant is more likely to encounter more varied environments and scenarios in the video game. This means that there is a richer palette of highlight and lowlight moments to choose from when matching these to the corresponding portions of the audio track. This is especially true when at least one video corresponds to a continuous stream of gameplay. In general, the quality of the output video (matched to a song) will be higher the longer the length of video footage available for use in creating the output video. As a general rule, the minimum length of the at least one video should be equal to the length of the selected audio track (or a trimmed version of it), but can be shorter if repeating clips are allowed.

[0039] In short, access to more videos provides more opportunities to find good combinations with the audio, but for any size video, combinations (or at least portions of the video that have higher scores for a given musical section than other portions, as described below) can be found.

[0040] Returning to FIG. 2, in a third step S203, a statistical analysis is performed on the audio track to identify an excitement level associated with each portion of the audio track. This may include, for example, separating the audio track into respective portions based on beat detection, each portion including at least two beats. An example of such a method for detecting beats in an audio track is described in Brian McFee et al., "librosa: Audio and Music Signal Analysis in Python," pages 18-23. In one example, a 4 / 4 time signature may be assumed for the separated audio track, such that each portion includes four consecutive beats, i.e., each portion corresponds to a measure. In another example, the time signature of the song may be estimated based on beat detection using any suitable method, for example by counting the number of beats in a repeating pattern. The pattern itself may be detected, for example, from autocorrelation in low and optionally high frequency bands over a period of several seconds, further the bands possibly being outside the normal voiced speech frequency range.

[0041] The statistical analysis may include analyzing the energy of the user-selected audio track to classify each portion of the audio track by its associated excitement level. This may include, for example, determining characteristics of the entire audio track by an overall spectrogram and determining whether individual portions of the audio track deviate from this "normal behavior" by more than a threshold. A more detailed example of a technique for performing statistical audio analysis is described in conjunction with Figures 3A-4.

[0042] Audio Statistical Analysis FIG. 3A shows an example of a generated spectrogram of an audio track. In FIG. 3A, the x-axis represents a number of time intervals and the y-axis represents a number of different frequency intervals (i.e., frequency bins). The shades of the spectrogram represent the intensity at a particular time interval and at a particular frequency range. That is, the spectrogram includes a number of buckets, and the shade of each bucket represents the intensity of the audio track at that particular frequency range and at the particular time interval. Lighter shades represent greater intensity and darker shades represent less intensity. In one example, the spectrogram may have 128 vertical buckets corresponding to 128 frequency ranges.

[0043] In the example shown in Figure 3A, it can be seen that the stronger parts of the audio track are concentrated in the low and mid frequency ranges (e.g., 20 to 20,000 Hz), with less strength in the higher frequency ranges. In the art, the degree to which different frequencies contribute to the strength of an audio signal is sometimes described as "fulness." A signal that has a low contribution to strength from some (or all) frequencies is described as having low fullness. A signal that exhibits significantly greater strength at multiple different frequencies is described as full (or fuller).

[0044] FIG. 3B shows an example of two different portions of an audio track having different fullness. In FIG. 3B, the portion of the audio signal on the right side is described as being fuller than the portion of the audio signal on the left side. In general, the full portion of the audio signal corresponds to an exciting portion of the audio track, while the less full portion corresponds to a calm portion of the audio track. By determining the fullness of different portions of the audio track, the excitement level associated with those portions can be identified.

[0045] As can be seen from the above, the variation in fullness is relative to what is considered "normal" for a given audio track. In some cases, the differences are not as dramatic as those shown in Figure 3B. For example, in the case of an audio track that is entirely composed of drumming, it may not be clear enough for a listener to perceive which parts of the audio track are more exciting than other parts.

[0046] To address variability in the audio track, the statistical audio analysis may include determining the average intensity and standard deviation of each of the intensity of the audio track at a number of different frequency intervals. That is, for each frequency interval and over all parts of the audio track (i.e., its entire length), the average intensity and standard deviation may be determined for that frequency interval. In FIG. 3A, this may involve simply adding the intensities over a particular row (each row corresponding to a frequency interval) and dividing by the number of time intervals to determine the average intensity. This may be repeated for each row to determine the average intensity and standard deviation for each frequency interval. The standard deviation may be determined in a conventional manner, for example using the following formula:

number

[0047] where μ represents the average intensity in a particular frequency range, N represents the number of time intervals, and x i represents the strength of that frequency range in each time interval. If your audio track contains a lot of bass drums etc., you can expect to see high average values ​​for the buckets around 20-100 Hz.

[0048] In some examples, it may be assumed that a normal distribution can be used to accurately model each frequency. For example, the mean and standard deviation determined as above for a particular frequency range may be used to model the intensity of the audio track in that particular frequency range. Each portion of the audio track (such as a bar) may then be analyzed against this model to determine whether the intensity of the audio track at that frequency deviates significantly from the expected intensity.

[0049] In some examples, a normal distribution may be used in calculating the z-score to determine whether the intensity at a particular frequency interval of a given portion (e.g., a bar) is typical or atypical for the audio track. This may include, for example, generating a spectrogram of each portion of the audio track and, for each frequency interval across the portion, determining the average intensity of that frequency interval. The average intensity of each portion of the audio track may then be compared to a model of the corresponding frequency range to determine a z-score for that frequency range. The z-score indicates the number of standard deviations that the average intensity of a particular portion of the audio track at a particular frequency range is relative to the average intensity of the entire audio track at that frequency range. The z-scores for each particular frequency range of each portion of the audio track may then be combined (e.g., added together) to determine an overall score indicative of the excitement level associated with that portion of the audio track. This may be repeated for each portion of the audio track to identify an excitement level for each portion.

[0050] It will be appreciated that instead of calculating the average intensity of each frequency interval for each part, the total intensity at each frequency interval for that part can be used to calculate a z-score. That is, the total intensity at each frequency interval can be compared to the average total intensity of the track at that frequency interval and used to calculate a corresponding z-score. The z-scores for each frequency interval for a given part can then be combined to identify an overall arousal level.

[0051] Thus, more generally, a portion of an audio track may be considered to be exciting if it has an intensity level that is above a threshold, as evaluated on an interval-by-interval or sum basis over the relevant intervals, within a predetermined number of frequency intervals (e.g., each generated frequency interval, or a representative sample of intervals across the entire frequency range).

[0052] FIG. 4 shows diagrammatically an example of a normal distribution that can be used to model the intensity of an audio track in a particular frequency range and the corresponding z-score, where different points on the distribution correspond to a z-score.

[0053] A portion of an audio track may be identified as having a high excitement level if the combined z-score of that portion is equal to or greater than a threshold. Conversely, a portion of an audio track may be identified as having a low excitement level if the combined z-score of that portion is less than a threshold. While the threshold may be somewhat arbitrary, the inventors have found that a combined z-score of 0.25 or greater is usually a good indication of whether a portion of an audio track can be considered to have a high excitement level. Each portion of an audio track may be assigned an identifier indicating whether that portion of the audio track has been identified as having a high or low excitement level (i.e., each portion is identified as being one of two classifications, exciting or calm). These identifiers may then be used to match the respective portions of the audio track with corresponding videos in excitement level, as described below.

[0054] As mentioned above, in some examples, the spectrogram may include 128 frequency buckets, and therefore, for each portion of the audio track, the statistical analysis may include determining 128 z-scores, i.e., one z-score for each frequency interval. These scores may then be combined (e.g., added together) to determine how typical or atypical that portion of the audio track is relative to the audio track as a whole. In general, a positive combined z-score indicates that the given portion of the audio track has higher than average fullness, while a negative z-score indicates that the given portion of the audio track has less than average fullness.

[0055] Although the audio analysis has been described above in terms of a normal distribution and corresponding z-scores, it will be appreciated that any statistical method that allows the fullness of different parts of an audio track to be compared to an overall average fullness may be used to determine whether individual parts of an audio track have a high or low level of excitement.

[0056] Returning to FIG. 2, in a third step S203, a statistical analysis is performed on the at least one video to identify an excitement level associated with each portion of the at least one video. This may include separating the video into each portion for subsequent analysis. Typically, the at least one video includes a variety of different scenarios and environments within one or more video games, which may require some separation so that the excitement level identified for a given portion is at least relative to larger portions of the video that correspond to the same video game or moments within a given video game. For example, if the video is a relatively long continuous stream of a player playing for two hours, a global color analysis of the entire video may not be useful in determining when and where exciting moments are occurring within the video game, since a large number of environments and situations may be encountered. In some cases, for example, if the video corresponds to a continuous stream of gameplay, 15 minutes may be selected as an arbitrary division for dividing the video, since it is unlikely that a player will encounter an environment that is significantly different from the environment seen in, for example, the last 15 minutes.

[0057] In instances where multiple videos have been acquired to create a music video, the individual video files may constitute an inherent means for separating the videos into respective portions, and thus there may be no explicit step of dividing the videos into respective portions, however, in some instances it may still be desirable to divide these individual videos into respective portions to be able to identify the excitement levels associated with the frames of these portions.

[0058] A more detailed example of a technique for performing statistical analysis on at least one video will now be described with reference to FIGS.

[0059] Video Statistical Analysis In some embodiments, the statistical analysis performed on the video may include determining, for at least some frames of each portion of the video, a score indicative of the movement associated with those frames of the respective portion. Generally, the more movement that occurs in a portion of the video, the more exciting the action occurring in that portion of the video.

[0060] In some examples, motion may be determined on a frame-by-frame basis. For example, motion associated with a given frame may be determined by taking the absolute difference in intensity value of each pixel of that frame relative to the same pixel in the previous and next frames. Pixels that are identical between successive frames will receive a score of 0, while a pixel that changes from pure white to pure black will receive a score of, for example, 255 (for JPG or PNG images). The scores of each pixel of a given frame may then be summed to determine an overall motion score associated with that frame. If an image is composed of, for example, 1280x720 pixels, motion may be determined by summing the scores of 921,600 pixels. If the overall motion score exceeds a threshold, the frame may be identified as containing a lot of motion.

[0061] Figure 5 shows an example of two consecutive video frames 502A, 502B from the game "Uncharted 4™." Also shown in Figure 5 is a difference image 504 that represents the difference between the individual pixel values ​​between frames 502B and 502A. The difference image can then be converted to a numerical value (by summing the pixel values) to obtain a global assessment of the motion associated with a given frame.

[0062] In an additional or alternative example, at least one selected video may include metadata indicative of motion vectors or the like associated with different regions within frames of the at least one video, which metadata may be used to identify motion associated with one or more frames, and thus a score indicative of motion for at least some of the frames may be determined based on this metadata.

[0063] In some examples, the score indicative of motion may correspond to a z-score. This may include, for example, determining the average and standard deviation of the motion occurring over a portion of frames of a video (such as between consecutive pairs). For a given frame, the z-score may be determined by comparing the motion (calculated as above) associated with that frame to the average and standard deviation of the corresponding video portion. Frames having z-scores above a threshold may be identified as having a high level of arousal.

[0064] Alternatively or additionally, the statistical analysis performed on the video may include determining, for at least some frames of the respective portion of the video, a score indicative of a change in value of the A and / or B channel of the LAB color space for each pixel of those frames relative to one or more other frames of the respective portion of the video. Again, the change in value of the A and / or B color channel for each pixel may be determined on a frame-by-frame basis.

[0065] Looking at pixel value changes in the LAB color space, as opposed to the RGB color space, is advantageous for two reasons. First, the LAB color space is more perceptually uniform than the RGB color space, meaning that, for example, a 10 point change will be relatively uniform in each direction. Thus, a large change in the A and / or B channels of a given pixel indicates an interesting event occurring within the video. Second, video games often have varying lighting conditions that may not necessarily indicate a particularly exciting moment occurring within the video game. Using the LAB color space allows more weight to be given to pure color changes, as opposed to lighting changes, when identifying the excitement level associated with a given frame. Figure 6 shows the LAB color space and examples of the A and B channels, respectively, and their respective values.

[0066] In some examples, the scores indicative of the change in the A and / or B channel values ​​for each pixel may be similar to those previously described for motion. For example, for each frame, the A and / or B color values ​​for each pixel may be compared to the corresponding pixel in the previous and / or subsequent frames, and the difference in the A and / or B values ​​across these frames may be used to determine the score for the current frame. For example, a frame that suddenly changes from yellow to blue may receive a high score for the color change in the B channel, for example.

[0067] In some examples, this analysis may include determining two scores (referred to herein as L(A)B and LA(B) interframe scores, respectively) for one change in each of the A and B color channels of a pixel in a given frame. These scores indicate the color changes between successive frames, which generally indicate the action occurring within a given scene. The higher the score (or scores) for a given frame, the higher the excitement level likely to be associated with that frame.

[0068] In some examples, the L(A)B and LA(B) interframe scores may correspond to z-scores. These may be determined in a similar manner as the motion z-scores. That is, for each portion of the video, the mean and standard deviation of the change in the A and / or B channels across the portion may be determined. For each frame in each portion, a z-score may be determined for the mean and standard deviation of the change in A and / or B across the portion based on the change in the A and / or B channels (calculated as above) for that frame. Frames having L(A)B and / or LA(B) z-scores above a threshold may be identified as having a high excitement level.

[0069] It should be understood that any variation in RGB values ​​for each pixel in a given frame may be used, and this is ultimately at the discretion of the designer.

[0070] In yet an additional or alternative example, the statistical analysis performed on the video may include determining a score indicative of the deviation of values ​​of the A and / or B channels of the LAB color space of pixels in at least some of the frames relative to an "average frame" representing the portion of the video. Calculating this score may include, for example, determining an average value of at least some of the pixels of the A and / or B color channels across multiple frames of the respective portion of the video (i.e., the "average frame"), and then determining, for each individual frame, the deviation of the pixel values ​​of the A or B color channels relative to the corresponding pixels of the "average frame." This deviation may be used to determine an L(A)B and LA(B) score for each frame.

[0071] The portion of the video may be, for example, a portion into which the video has been divided based on an arbitrary division time, or based on different video clips of known length selected for matching with an imported audio track. In general, the greater the deviation of the pixel values ​​of a given frame from the average frame, the more exciting action is occurring within that frame.

[0072] In some examples, the L(A)B and LA(B) scores may correspond to z-scores. This may include, for example, determining an average L(A)B frame and / or an average LA(B) frame that represents the average A and / or B values ​​of pixels across multiple frames of the respective portion of the video. Then, for each frame of the portion of the video, the difference in pixel values ​​of the A and / or B channels may be determined relative to the corresponding pixels of the associated average frame. The average and standard deviation of the differences across the frames of the video portion may then be determined. The z-score for each frame may be calculated by comparing the A and / or B channel differences of the current frame to the overall average and standard deviation of the differences of the respective video portion. A high z-score indicates a high level of color change and therefore indicates a high level of exciting content.

[0073] It will be appreciated that one or more of the above scores may be calculated and combined to determine an overall score for a frame, which is indicative of an excitement level associated with that frame. In some examples, a motion score, an LA(B) interframe score, an L(A)B interframe score, an L(A)B score, and an LA(B) score may be determined for each individual frame. As mentioned above, each of these scores may correspond to a respective z-score. An excitement level may be determined for a given frame, for example, based on a weighted sum of these scores, followed by a smoothing process, for example, using Savitzky-Golay.

[0074] Thus, more generally, a video frame may be considered exciting if it has a parameter level that exceeds a threshold (e.g., an average parameter level or a pre-set parameter level) of a predetermined number of image parameters (e.g., the five z-scores mentioned above) by a predetermined amount, or if the combined parameter level of a predetermined number of image parameters exceeds a predetermined combined threshold (such as a weighted sum of the thresholds) by a predetermined amount.

[0075] In some examples, each frame may be assigned an identifier indicating whether the frame has a high level of excitement. A frame may be assigned an identifier indicating a high level of excitement if the overall score (i.e., the combined smoothed score) exceeds a threshold. Conversely, a frame may be assigned an identifier indicating a low level of excitement if the overall score associated with the frame is less than a threshold score. In other examples, the overall score may be used as an explicit indicator of the excitement level, with a higher overall score corresponding to a higher level of excitement associated with those frames.

[0076] In some examples, identifying a frame as having a low excitement level may include identifying a number of consecutive frames with an overall score below a threshold score, for example 80 frames. Generally, a low overall score indicates little movement and color variation associated with the frame, with very average color.

[0077] FIG. 7 shows an example of a graph corresponding to the z-scores calculated for each frame of a portion of a video corresponding to the video game God of War®. In FIG. 7, a green arrow 702 represents the current time point, corresponding to frame 700, where the character Kratos® is shown (as the background of the graph) performing an attack. As shown in key 704, the scores are represented by different colors. In FIG. 7, it can be seen that at the time point indicated by arrow 702, the L(A)B and LA(B) scores rise, and there are peaks in the inter-frame L(A)B and L(B) scores around this time point. In FIG. 7, the overall score, or combined score, is shown by graph 706. The combined score has been smoothed, which reduces peaks in the graph and prevents one or more large single-frame spikes in the scores that would invalidate a period of excitement.

[0078] Matching segments of video with corresponding segments of audio Returning to Figure 2, in a fifth step S205, at least some portions of at least one video are matched to respective portions of the audio track based on a correspondence in excitement level. At a high level, this corresponds to matching portions of the video that contain frames identified as having high (or low) excitement levels with corresponding portions of the audio that have been identified as having high (or low) excitement levels.

[0079] It will be recalled that after a statistical analysis has been performed on the audio track, the start and end times of each portion of the audio track (e.g., a measure of four beats) may be known, and each of these portions may be assigned an identifier indicative of an excitement level (either "high" or "low"). In some examples, the start and end times of each portion may correspond to an identified beat (e.g., start on beat n, end on beat n+3).

[0080] Following statistical analysis performed on at least one video, at least some of the frames of each portion of the video are assigned an identifier indicative of whether they have a high or low excitement level, and if the excitement level is low, a group of frames may be identified as having such a level.

[0081] It will be appreciated that in some embodiments, the excitement levels of both the portions of the audio track and the frames of the video may be variable (i.e., not binary) and the matching may be based on similarity of values. For example, one or more frames identified as having a maximum excitement level may be used to generate a video clip that is matched to a corresponding portion of the audio that is also identified as having a maximum excitement level.

[0082] To match portions of the audio to corresponding portions of the video, the method may include generating video clips from portions of at least one video and matching the video clips to corresponding portions of the audio track. The generated video clips may correspond to highlights (including one or more frames identified as having a high excitement level) or lowlights (a number of consecutive frames identified as having a low excitement level).

[0083] The highlight video clips may be generated by identifying a portion of at least one video that includes one or more frames identified as having a high excitement level, and generating clips that correspond in length to portions of the audio track that have also been identified as having a high excitement level, until each portion of the audio track with a high excitement level is matched to a corresponding highlight video clip.

[0084] In some examples, the length of the highlight video clip may be a measure (e.g., 4 beats) of the audio track. This may be appropriate, for example, if the song has a 4 / 4 time signature, and therefore each portion (e.g., a measure) is the same length. In other examples, the length of each video clip may be variable depending on (and corresponding to) the portion of the audio track that the video clip fits into.

[0085] The length of the video clip may be adjusted so that at least one of the frames identified as having a high excitement level (or maximum excitement level) is synchronized to one of the beats of the corresponding portion of the audio track (e.g., the beat of the second bar). As mentioned above, there may be a step of identifying beats in the audio track, and if the timing of these beats is known, a given frame in the video clip can be aligned to this beat. This adjustment may be performed automatically as part of the video clip generation process.

[0086] Additionally, if the start and end of the audio portion are determined by different beats and different video clips are timed to different portions of the audio track, this can be used to ensure that the transitions between the different video clips are cut to the beats. For example, the highlight clip may be cut to start on beat 1 and end on beat 4, and the lowlight clip may be cut to start on beat 4 and end on beat 12. If two different video clips are cut to appear on the same beat, appropriate fading may be applied. In general, the transitions between different video clips will sound less harsh if they are made on beat.

[0087] In the case of a low-light video clip, a video clip may be generated from a portion of the video in which multiple consecutive frames have been identified as having a low (or lower) excitement level. The low-light video clip may be generated to correspond in length to the corresponding portion of the audio track identified as having a low excitement level. Since in most audio tracks, low exciting periods last longer than exciting periods, the portion of the audio track identified as having a low excitement level may be longer (or include multiple portions) than the portion of the audio track identified as having a high excitement level. This may be repeated until each portion of the audio track with a low excitement level is fitted into a corresponding low-light video clip. If the audio track has a regular structure, all low-light clips may be generated to have the same length (corresponding to calm moments in the audio track with the same or similar length).

[0088] It will be appreciated that in some examples, it may be desirable to impose a lower limit on the length of each video clip to avoid an excessive amount of jumping between different video clips. Thus, in some examples, the video clips may be generated according to a length distribution, thereby defining a minimum length for each of the highlight and lowlight video clips. The minimum length may, for example, correspond to a multiple of the length of one or more audio portions. For example, a highlight clip may be restricted to have a length of at least 4 beats, while a lowlight clip may be restricted to have a length of at least 8 beats.

[0089] The order in which different video clips are generated and matched to corresponding portions of the audio track may be random, predetermined, or dynamically determined.

[0090] In examples where multiple different videos are acquired and analyzed in steps S202 and S204, a predefined order may be defined such that each subsequent portion of the audio track is matched to a video clip generated from a different video. It will be appreciated that this is only possible if a sufficient number of highlight and lowlight moments occur across the different videos. In some examples, separation of video clips generated from the same video by at least two different video clips generated from different videos may be required. By controlling the order in which content from different videos is displayed, a more diverse and appealing final video can be created.

[0091] In some examples, the predetermined order may be chronological. For example, the video clips may be generated based on a chronological order, and an earlier portion of the at least one video may be used prior to a later portion of the at least one video to generate the video clip. The clips generated from the earlier portion of the at least one video may be matched to the corresponding earlier portion of the audio track. In this manner, the final video displays highlight and lowlight moments in the order in which they were revealed while the athlete was playing one or more video games. This may be desirable when the athlete wants to obtain an overview of previous plays, for example, upon completing a story mode in a single athlete game or, for example, upon winning a sports tournament.

[0092] In a further or alternative example, the selection of video clips to match corresponding portions of the audio track may be based on information related to the game to which the at least one video pertains. This metadata may be encoded with the at least one video (or each portion of the video). The metadata may define the game to which the at least one video pertains, and this information may be used to ensure that footage of a sufficient number of different games is used to create the final video.

[0093] In some examples, the metadata may define state information within a game, such as the level, the match, whether a player won or lost, made a kill, scored a goal, etc., and this information may be used to define the order in which different video clips are generated and timed to corresponding portions of the audio track. For example, a video clip corresponding to a player winning the final match or game may be purposely timed to an exciting portion of the audio track that occurs later in the audio track (e.g., a key change that occurs during the final chorus of a song).

[0094] In a sixth step S206, a composite video (i.e., a combined audiovisual content) is generated that includes at least some portions of the video timed to respective portions of the audio track. This video corresponds to a music video where various portions of the video game footage are timed to an audio track (e.g., a song) selected by the contestant. Once assembled, the contestant may be notified that the music video is ready to play. This may be in the form of, for example, a system notification displayed on the home screen of the home menu of the video game console or a notification sent to the user's phone via the video game application. The user may be given the option to share this video on social media or a video hosting service.

[0095] In some examples, the method may further include an additional step (not shown) of presenting the composite video to a user. For example, the composite video may be automatically displayed to a contestant at the end of a video game session, e.g., when the video from that session has been used to create a music video. In such a case, 15 minutes of video cached in a temporary buffer may be used to create, e.g., a 30-second highlight video that is matched to a selected audio track. In some examples, the music video may be automatically created after an in-game event (e.g., a contestant completes the final level in a story mode of a video game). Displaying the composite video in any of these ways requires the user to select a song before finishing one or more video game sessions. Alternatively, a contestant may select a song after completing one or more video game sessions, which means that the contestant must wait for the music video to be assembled by the system.

[0096] It will be appreciated that in some cases, there may be an insufficient number of video clips to fit the corresponding portions of the audio track, such as when there are too few frames corresponding to highlight moments throughout at least one video to fit the corresponding number of exciting moments identified in the audio track. In such cases, the method may include generating a truncated music video that includes a shortened version of the audio track that is fitted to the generated video clips. The truncation may be based, for example, on the first X minutes of the song, a set number of consecutive low and high excitement portions of the song, or a portion of the song centered around a portion identified as having a high (or maximum) excitement level.

[0097] It will be appreciated that any of the foregoing method steps may be performed by a computer system including a computer-readable medium having computer-executable instructions adapted to cause the system to perform any of the method steps.

[0098] 8 illustrates generally one example of a system 800 for generating audiovisual content from video game footage according to the present disclosure. The system 800 includes an input unit operable to receive an audio track including music. The audio track may correspond to any of the audio tracks described above in relation to FIG. 2. Typically, the audio track will have been selected by a user.

[0099] The input unit may also be operable to receive at least one video. In some embodiments, the system 800 may comprise separate units for each of the selected audio tracks and the at least one video. An example of such an embodiment is shown in Figure 8, where the system 800 is shown as comprising an audio input unit 802 and a video input unit 804.

[0100] The at least one video may have been acquired in any one of the ways described above. For example, it may have been automatically or manually captured on a video game playing device. Or it may have been imported from a video hosting website. In the latter case, the video hosting website may provide videos (or video clips) categorized according to games or in-game events. A user may select one or more of these videos to use in assembling a composite video. This may, for example, help YouTubers in creating intros and outros, allowing them to use content not yet encountered while playing a given video game.

[0101] The system 800 further includes an audio analyzer 806 configured to receive input from the input unit (or audio input unit) and perform statistical analysis on the audio track to identify an excitement level associated with at least some portions of the audio track.

[0102] In some examples, the system 800 may comprise a beat detection unit (not shown) operable to detect beats in the audio track and to divide the audio track into portions based on the beat detection. The portions may be defined such that each portion includes at least two consecutive identified beats. In a preferred example, each portion of the audio track corresponds to a measure. The start and end of each portion may be defined by a respective beat. The beat detection unit may be part of the audio analyzer 806 or a separate component (not shown) preceding the audio analyzer 806. The audio analyzer 806 may be configured to perform a statistical analysis on the audio track using the audio track divided based on the beat analysis.

[0103] In some examples, the audio analyzer 806 may be configured to generate a spectrogram for each portion of the audio track and determine, for each portion, whether the intensity of the audio track at a number of different frequency intervals deviates from the average intensity of the audio track at those frequencies by more than a threshold. As described above (see "Audio Statistical Analysis"), this may include determining z-scores for a number of frequency intervals of each portion of the audio track to determine an overall score for that portion that represents fullness. The audio analyzer 806 may be configured to determine such scores for each portion of the audio track and, based thereon, identify excitement levels associated with different portions of the audio track. In general, the audio analyzer 806 may be configured to perform any of the statistical audio analyses described above.

[0104] The audio analyzer 806 may be configured to assign to respective portions of the audio track an identifier indicating whether those portions have a high or low arousal level. A portion of the audio track may be assigned an identifier indicating a high arousal level if the intensity of that portion at a number of different frequency intervals deviates from the average intensity of the audio track at those frequencies by more than a threshold. More simply, the audio analyzer 806 may determine whether an overall score associated with a portion of the audio track exceeds a threshold, and if so, may assign that portion an identifier indicating that the portion has a high arousal level. If this is not the case for a given portion of the audio track, the audio analyzer 806 may assign that portion an identifier indicating that the portion has a low arousal level.

[0105] The system 800 further comprises a video analyzer 808 configured to receive input from the input unit (or a dedicated video input unit) and perform a statistical analysis on the at least one video to determine an excitement level associated with at least a portion of the at least one video.

[0106] In some examples, the system 800 may include a video splitting unit (not shown) operable to split the at least one video into respective portions. As previously mentioned, this may include splitting the at least one video into, for example, 12 micro-portions based on any time interval. Additionally or alternatively, the splitting may be based on a marker, such as a file name or identifier, that indicates that a given portion of the video is in fact a video relating to a different video game or a moment within a given video game. The video splitting unit may, for example, constitute part of the video analyzer 808 or a separate component preceding the video analyzer 808.

[0107] The video analyzer 808 may be configured to determine a score associated with at least some of the frames in each portion of the video, the score being: i. the motion relative to different frames of each portion of the video, and associated with that frame; ii. the change in values ​​of the A and / or B channels of the LAB color space of that frame relative to different frames of the respective portion of the video; iii. the deviation of the values ​​of the A and / or B channels of the LAB color space for that frame relative to an average frame representing that portion of the video; At least one of the following is shown.

[0108] The video analyzer 808 may be configured to determine this score in any of the ways previously described (see "Video Statistical Analysis"). In one example, this may include determining one or more individual scores representing motion, LA(B) frame-to-frame change, L(A)B frame-to-frame change, change in LA(B) relative to the average frame, and change in L(A)B relative to the average frame. As previously described, these one or more individual scores may be combined (e.g., using a weighted sum) to determine an overall score from which an excitement level associated with a given frame may be identified.

[0109] In some examples, the video analyzer 808 may be operable to identify an excitement level associated with each frame of a respective portion of the video based on whether the score associated with that frame is greater than or equal to a threshold score. A frame may be identified as having a high excitement level if its associated score is greater than or equal to the threshold score and a low excitement level otherwise. The video analyzer 808 may be configured to assign an identifier to at least some of the frames indicative of whether they have a high or low excitement level.

[0110] In FIG. 8, it can be seen that the system comprises a combiner 810 operable to receive inputs from an audio analyzer 806 and a video analyzer 808. The combiner 810 is configured to match at least some portions of at least one video to at least some portions of the audio track based on the identified correspondence in excitement levels. Based on this matching, the combiner 810 is configured to generate a composite video including at least some portions of the video matched to at least some portions of the audio track. The term "composite video" is used interchangeably with the term "music video" and corresponds to a final video composed of portions of one or more videos, the relative positions of the portions in the final video being based on the correspondence in excitement levels between the portions and the portions of the audio track.

[0111] In some examples, the system 800 may include a video clip generator (not shown) operable to generate video clips from at least one of the respective portions of the at least one video for fitting to at least one of the portions of the audio track. The video clip generator may be configured to determine the length of at least one of the portions of the audio track identified as having a corresponding excitement level and generate video clips having at least that length, such that each video clip is as long as the respective portion of the audio track to which it is fitted (e.g., a bar containing four beats). The combiner 810 may then be configured to fit these generated video clips to the corresponding portions of the audio track.

[0112] The video clip generator may be configured to generate video clips that include frames identified as having a high excitement level (so-called highlight clips) and align these clips with corresponding portions of audio tracks identified as having a high excitement level. In some examples, the video clips may be generated such that the frames identified as having a high excitement level are synchronized to at least one of the beats of the corresponding portions of the audio tracks to which the clips are aligned. The highlight video clips may be generated in any of the ways described above in connection with Figures 2-7.

[0113] The video clip generator may also be configured to generate video clips that include a number of consecutive frames that are identified as having a low excitement level (so-called lowlight clips). The combiner may then be configured to match these clips with corresponding portions of the audio track that are identified (e.g., by an identifier assigned thereto) as having a low excitement level, such that less exciting moments in the video game footage are output simultaneously with calmer moments in the audio track.

[0114] In general, the video clip may be generated in any of the ways described above in connection with FIGS.

[0115] In some examples, the combiner 810 may be configured to combine portions of at least one video with respective portions of the audio track such that each video clip begins on a different beat of the audio track. For example, the combiner 810 may combine clips with portions of the audio track such that, for example, clip 1 begins on beat 1 and ends on beat 4, clip 2 begins on beat 4 and ends on beat 8, etc. This causes the different videos to be cut to the beats. In some examples, this is a natural consequence of how the different video clips were generated (as their lengths depend on the length of the portions of the audio track to which they were matched).

[0116] System 800 may further include a display device (not shown) for outputting composite video for display.

[0117] It will be appreciated that one or more components of the system 800 described above may be implemented on the same device or different devices. In some examples, one or more of these components may be implemented on a video game playing device. For example, an input unit may be installed on the video game playing device and receive video(s) captured by the video game playing device. Audio tracks may also be stored on or downloaded to the video game playing device and received by the input unit in this manner.

[0118] Once acquired, the audio and video analysis and combining may be performed, for example, on a server in communication with the video game playing device. For example, video and audio analysis is a somewhat intensive process in terms of required computing resources and may often benefit from being performed using the cloud or the like as opposed to the video game playing device (which may already be in use by the contestant to render the video content). Once the video game footage has been assembled into a music video, the music video may be provided to the user. This may include, for example, sending the assembled video to the contestant's video game playing device or making the video available through an application (e.g., the "PS app") or web page that the contestant can access via one of their devices. In a preferred example, the video and audio analysis is performed automatically, eliminating the need for user intervention with respect to creating various video clips and matching them to appropriate music.

[0119] It will be understood that the method(s) described herein can be implemented in suitable conventional hardware adapted to be applicable by software instructions or by incorporation into or replacing dedicated hardware. Thus, the necessary adaptations to existing parts of conventional equivalent devices may be implemented in the form of a computer program product including processor-implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory, any combination of these or other storage media, or may be realized in hardware as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other configurable circuit suitable for use in adapting conventional equivalent devices. Alternatively, such computer programs may be transmitted via data signals over a network such as Ethernet, a wireless network, the Internet, any combination of these or other networks.

Claims

1. 1. A method for generating audiovisual content from video game footage, comprising the steps of: Obtaining a user selected audio track including music; acquiring at least one video, the video comprising a plurality of frames, generated by at least one video game playing device during playing of at least one video game; performing a statistical analysis on the audio track to identify an excitement level associated with at least some portions of the audio track; performing a statistical analysis on the at least one video to identify an excitement level associated with at least some portions of the at least one video; matching at least some portions of the at least one video with at least some portions of the audio track based on the identified correspondence in excitement levels; generating a combined audiovisual content based on the alignment, the combined audiovisual content including the at least some portions of the video aligned with respective portions of the audio track; based on beat detection, separating the audio track into respective portions, each portion including at least two beats; performing a statistical analysis on the audio track, generating a spectrogram of each portion of said audio track; determining a mean intensity and a standard deviation of each of the intensity of the audio track at a plurality of different frequency intervals; determining, for at least some of the portions of the audio track, a deviation of the intensity at the plurality of different frequency intervals from an average intensity of the audio track at the plurality of frequency intervals;

2. 2. The method of claim 1, wherein identifying an excitement level associated with each portion of the audio track comprises determining a z-score associated with the intensity of that portion at the plurality of different frequency intervals and determining whether a combination of z-scores for that portion exceeds a threshold.

3. 3. The method of claim 2, wherein each portion of the audio track is assigned an identifier indicating whether the portion has a high or low excitement level, and the excitement level is determined to be high if the combined z-scores for that portion are equal to or greater than a threshold, and low if the combined z-scores are less than the threshold.

4. Performing a statistical analysis on the at least one video includes, for each frame of a respective portion of the video: i. the motion associated with the frame relative to different frames of the respective portions of the video; ii. the change in value of the A and / or B channels of the LAB color space of the frames relative to different frames of the respective portions of the video; iii. The deviation of the values ​​of the A and / or B channels of the LAB color space of that frame relative to an average frame representing the respective portion of the video; The method of claim 1 , further comprising determining a respective score indicative of at least one of:

5. determining, for each portion of the video, whether any of the frames in that portion have a score equal to or greater than a threshold score; The method of claim 4 , wherein frames having a score equal to or greater than the threshold score are assigned an identifier indicating that they have a high excitement level.

6. determining, for each portion of the video, whether a number of consecutive frames of that portion have a score less than the threshold score; The method of claim 5 , wherein the plurality of consecutive frames having a score below the threshold score are assigned an identifier indicating that they have a low excitement level.

7. Fitting at least some portions of the at least one video to respective portions of the audio track comprises: aligning portions of the at least one video including frames identified as having a high excitement level with respective portions of the audio track identified as having a high excitement level; 7. The method of claim 5 or 6, comprising: matching a portion of the at least one video comprising a plurality of consecutive frames identified as having a low excitement level with a respective portion of the audio track identified as having a low excitement level.

8. 10. The method of claim 1 or 7, comprising adjusting a length of at least some portions of the video that include frames identified as having a high excitement level such that at least one of the frames in that portion of the video identified as having a high excitement level is synchronized with one of the beats of a corresponding portion of the audio track.

9. acquiring a plurality of videos, each video corresponding to the videos generated by one or more video game playing devices during a different video game playing session; 9. A method according to claim 1, comprising: matching at least some portions of the video with respective portions of the audio track according to a predetermined order, the predetermined order defining the order in which portions of different videos appear in the combined audiovisual content.

10. A computer readable medium having computer executable instructions adapted to cause a computer system to perform the method of any one of claims 1 to 9.

11. 1. A system for generating audiovisual content from video game footage, comprising: an input unit operable to receive a user selected audio track including music and at least one video generated by the video game playing device during play of the at least one video game; an audio analyzer operable to perform a statistical analysis on at least some portions of the audio track to identify an excitement level associated with at least some portions of the audio track; a video analyzer operable to perform a statistical analysis on the at least one video to identify an excitement level associated with at least some portions of the at least one video; a combiner configured to match at least some portions of the at least one video to respective portions of the audio track based on a correspondence in excitement level, and to generate a combined audiovisual content including the at least some portions of the video matched to respective portions of the audio track based on the matching, further comprising a beat detection unit operable to detect beats in the audio track and, based on the beat detection, to divide the audio track into portions, each portion comprising at least two beats; The system, wherein the audio analyzer is configured to generate a spectrogram for each portion of the audio track and determine, for each portion, whether the intensity of the audio track at a plurality of different frequency intervals deviates from an average intensity of the audio track at those frequencies by more than a threshold value.

12. the audio analyzer is configured to assign identifiers to respective portions of the audio track indicative of whether those portions are of a high or low arousal level; 12. The system of claim 11, wherein the audio analyzer is configured to assign an identifier indicative of a high excitement level to a respective portion of the audio track if the intensity of that portion at the plurality of different frequency intervals deviates from an average intensity of the audio track at those frequencies by more than a threshold value, and to assign an identifier indicative of a low excitement level otherwise.

13. a video splitting unit operable to split said at least one video into said respective portions; The video analyzer is configured to determine a score associated with each frame of a respective portion of the video, the score comprising: i. the motion associated with the frame relative to different frames of the respective portions of the video; ii. the change in value of the A and / or B channels of the LAB color space of the frames relative to different frames of the respective portions of the video; iii. The deviation of the values ​​of the A and / or B channels of the LAB color space of that frame relative to an average frame representing that portion of the video; 13. The system according to claim 11 or 12, further comprising at least one of:

14. the video analyzer is operable to determine an excitement level associated with each frame of a respective portion of the video based on whether the score associated with each frame is greater than or equal to a threshold score; The system of claim 13 , wherein the video analyzer is configured to assign to each frame an identifier indicating whether the frame is of a high arousal level or a low arousal level.

15. the video analyzer is configured to assign an identifier to each frame indicating a high arousal level if the score associated with that frame is equal to or greater than the threshold score; The system of claim 14 , wherein the video analyzer is configured to assign an identifier indicating a low arousal level to each frame if the score associated with that frame is less than the threshold score.

16. a video clip generator operable to generate a video clip from at least one of the respective portions of the video for matching with at least one of the respective portions of the audio track; 16. A system according to any one of claims 13 to 15, wherein the video clip generator is configured to identify a duration of at least one of the respective portions of the audio track and to generate a video clip having at least that duration.

17. the video clip generator is configured to generate a video clip including frames identified as having a high excitement level; 17. The system of claim 16, wherein the combiner is configured to match video clips that include frames identified as having a high excitement level with corresponding portions of the audio track that are identified as having a high excitement level.

18. the video clip generator is configured to identify beat-related timing information for each portion of an audio track; 20. The system of claim 17, wherein the video clip generator is configured to generate a video clip including at least one frame identified as having a high excitement level that is synchronized with at least one of the beats of the respective portion of the audio track.

19. the video clip generator is configured to generate a video clip comprising a plurality of consecutive frames identified as having a low arousal level; 19. The system of claim 17 or 18, wherein the combiner is configured to match a video clip comprising consecutive frames identified as having a low excitement level with a corresponding portion of the audio track identified as having a low excitement level.

20. 20. The system of claim 11, wherein the input unit is operable to acquire a plurality of videos, each video corresponding to video generated during a different video game play session by the same player or a different player.

21. 21. The system of claim 20, wherein the combiner is configured to determine whether at least one portion of the video has already been fitted to a respective portion of the audio track, and, if a portion from that video has already been fitted to a respective portion of the audio track, select a portion of a different video to fit to a subsequent portion of the audio track.

22. the start and end points of each portion of the audio track are defined by respective beats; 22. The system of claim 11, wherein the combiner is configured to combine portions of the at least one video with respective portions of the audio track such that each portion of the at least one video begins with a first beat of a corresponding portion of the audio track.

Images