Video editing device and method

Abstract

A video evaluation method for a video comprising consecutive video frames, comprising the steps of generating optical flow vectors for at least a subset of corresponding pixels between pairs of frames, calculating differences between the corresponding optical flow vectors of consecutive pairs of frames, and evaluating the resulting differences with respect to a predetermined criterion, to identify a corresponding pair of frames for further processing.

Description

Technical Field

[0001] The present application relates to a video editing device and method. Background Art

[0002] Playing video games has become an increasingly social activity, with users wanting to post their experiences to social media, or share their gaming stories with friends. However, when the user's primary focus will be on reacting to the game, it can be difficult to simultaneously play a video game and selectively record footage to share, especially for exciting or surprising content.

[0003] Embodiments of the present application seek to solve or alleviate this problem. Summary of the invention

[0004] Various aspects and features of the present invention are defined in the accompanying claims and within the text of the accompanying description.

[0005] In a first aspect, there is provided a video assessment method according to the claims.

[0006] In another aspect, there is provided an entertainment device adapted to perform a video assessment method according to the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more complete understanding of the present disclosure and many of its attendant advantages will be readily obtained as the present disclosure and many of its attendant advantages become better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0008] - Figure 1 is a schematic diagram of a device according to an embodiment of the present specification.

[0009] - Figure 2 is a flow chart of a video evaluation method according to an embodiment of the present specification.

[0010] - Figure 3 is a schematic diagram of image frame region selection according to an embodiment of the present specification.

[0011] - Figure 4 is a schematic diagram of image frame region selection according to an embodiment of the present specification.

[0012] - Figure 5 is a schematic diagram of vector component selection according to an embodiment of the present specification.

[0013] - Figure 6 is a flow chart of a video evaluation method according to an embodiment of the present specification.

[0014] - 7A to 7D is a schematic diagram of comparison between hashes of consecutive video frames according to an embodiment of the present specification.

[0015] - Figure 8 is a flow chart of a video evaluation method according to an embodiment of the present specification. DETAILED DESCRIPTION

[0016] A video editing apparatus and method are disclosed. In the following description, many specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. However, it is apparent to those skilled in the art that these specific details need not be employed to practice the present invention. Instead, for the sake of clarity, specific details known to those skilled in the art have been omitted where appropriate.

[0017] Referring now to the drawings, in which like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 An example device according to an embodiment of the present specification is shown. As a non-limiting example, the device takes the form of an entertainment system 10, such as (PS5). Other example devices may include other entertainment systems or video game consoles, personal computers, phones or tablets, or any device capable of simultaneously playing a video game and recording footage thereof.

[0018] The entertainment system 10 includes a central processor 20. This can be a single core or multi-core processor, for example including eight cores as in the PS5. The entertainment system also includes a graphics processing unit or GPU 30. The GPU can be physically separate from the CPU, or integrated with the CPU as a system on a chip (SoC), as in the PS5.

[0019] The entertainment device also includes RAM 40, and may have separate RAM for each of the CPU and GPU, or shared RAM as in the PS5. The or each RAM may be physically separate, or integrated as part of the SoC as in the PS5. Further storage is provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or as an internal solid state drive as in the PS5.

[0020] The entertainment device may be suitably connected via one or more data ports 60 (such as a USB port, port, port, It can also optionally receive data via an optical drive 70.

[0021] Audio / video output from the entertainment device is typically provided through one or more A / V ports 90 or through one or more of the wired or wireless data ports 60 .

[0022] An example of a device for displaying images output by the entertainment system is a head mounted display “HMD” 120 worn by the user 1 , such as a PlayStation VR 2 “PSVR2”.

[0023] Where components are not integrated, they may be connected by dedicated data links or via bus 100 as appropriate.

[0024] One or more handheld controllers (130, 130A) are typically used to provide interaction with the system, such as in the case of the PS5. Controller (130), and / or one or more VR controllers (130A-L, R) in the case of an HMD.

[0025] Video Assessment - Excitement

[0026] In an embodiment of the present specification, the apparatus is configured (e.g., by suitable software instructions) to perform a method for picking the most exciting moments in a video sequence. While generating highlights for video game plots is a particularly valuable use case, the method is not only game-agnostic, but can also work on non-game footage, and can therefore be used to extract exciting moments in other video feeds, such as feeds from security cameras, wildlife monitoring cameras, cameras that accumulate image information for machine learning training purposes, etc.; as well as more traditional video content such as TV shows and movies.

[0027] Advantageously, embodiments of the present description provide a technique with particularly low computational cost compared to techniques based on, for example, machine learning or semantic analysis of image information. These methods are not only computationally intensive, but they also typically require training or tuning to specific types of content.

[0028] The method involves statistical analysis of a set of metrics for each image frame.

[0029] These include two or more selected from the list consisting of:

[0030] i. The inter-frame grayscale delta sum of the absolute differences between the current frame and the immediately preceding frame. This captures the overall difference in brightness between frames.

[0031] ii. a first inter-frame color delta sum of absolute differences for at least a first color channel between a current frame and an immediately preceding frame;

[0032] iii. a second inter-frame color delta sum for absolute differences of at least a second color channel different from the first color channel between the current frame and the immediately preceding frame;

[0033] iv. for a first color channel, a difference between a mean value of a property of the first color channel and a current measurement value, the mean value being derived over a predetermined time period; and

[0034] v. For a second color channel, the difference between a mean value of the property of the second color channel and the current measurement value, the mean value being derived over a similar predetermined time period.

[0035] For metrics ii.-v., the color channels can be, for example, RGB, but preferably the channels correspond to the LAB color space, because differences in LAB are more representative of how people perceive changes in images. So in this case, the first color channel would be A (usually red / green values), and the second color channel (usually blue / yellow values) would be B. Grayscale is a suitable representation of the L channel (brightness values).

[0036] Optionally, for metrics i.-iii., the incremental sum of absolute differences can be between the current frame and a short rolling average of the previous frame (e.g., instead of just the immediately preceding image frame, it can be the average of the previous two, three, four, or five frames).

[0037] For metrics iv. and v., the characteristic measured for each color channel is typically the average value of the color channel over the entire image over a predetermined period of time, but may alternatively or additionally be a standard deviation.

[0038] Alternatively, instead of the or each attribute being on the whole image, the attribute may be on a sub-portion of the image; for example, equal subdivisions of the image such as the top and bottom halves of the image, the left and right halves of the image, a quarter, a sixth, an eighth, a sixteenth or a thirty-second of the image.

[0039] It is worth noting that the measurements do not proceed down to a per-pixel level or even close to a per-pixel level; typically a subdivision will represent at least 10,000 pixels (32nds of a 1920×1080 image will each consist of approximately 64,000 pixels), so the mean or standard deviation of such a sub-region is based on at least this many pixels over a predetermined period of time.

[0040] One reason for this is that a particular benefit of using advanced statistics is that these statistics can be efficiently stored over a considerable period of time, thereby enabling the assessment of visual excitement within a wide context of the content. Thus, for example, for video game content generated at a typical frame rate of 60 frames per second, the predetermined time period described above can be between one minute and five minutes, and is typically three minutes. It should be understood that 3 minutes multiplied by 60 seconds multiplied by 60 frames results in approximately 10,000 frames; therefore, maintaining an average or standard deviation of segments as low as approximately 10,000 pixels will result in no more storage requirements than a single normal video frame, and can therefore be efficiently managed by the CPU or GPU of the device. It should also be understood that approximations of rolling averages and standard deviations can be generated that do not require the storage of all data contributing to the sample window, thereby further reducing storage overhead.

[0041] For metrics iv. and v., optionally, when the generation of the edit is on live content, or on content where video frames are not stored in the buffer long enough to access them after a predetermined period of time, or where the computing bandwidth is insufficient to evaluate past video frames at a higher rate than the current frame rate in order to catch up with the current image generation, a bootstrap process can be used. In these cases, the values ​​of previous instances of the content can be stored (e.g., when a user stops playing a game on one day and resumes on another day, the average and / or standard deviation values ​​of the last three minutes of playing the game the previous day can be used to guide the evaluation of playing the game one day later). Optionally, this information can be stored with a specific game save so that the metric is faithful to the restored game state. This information can also be generated by other users or game testers at least in the first approximation, and thus the bootstrap data can be included in the game, or can be accessed from the server based on a query indicating the current progress. Such information itself can be averaged over multiple such other users, or a mean or median user within multiple users can be identified for this purpose.

[0042] As mentioned above, two or more of these metrics may be used. Optionally but preferably, at least one used metric is based on an inter-frame short-term measurement (e.g. one of metrics i.-iii.) and one metric is based on a long-term measurement (e.g. one of metrics iv.-v.).

[0043] Inter-frame or short-term measurements are good at detecting events in a scene, while longer-term measurements are better at detecting scene changes, so one of each complements these capabilities.

[0044] The selected measurement is converted to a z-score relative to the recent history of these measurements. It should be understood that a z-score is a statistical measure that describes the relationship of a value to the mean of a set of values, usually in terms of standard deviations from the mean. Thus, the z-score of a measurement (e.g., one of the above-mentioned measurements i.-v.) for the current frame is derived based on its recent history (e.g., the average of the historical measurements).

[0045] The history period for the z-score may be short (e.g. between 2 and 10 frames), may be of medium duration (e.g. measured for frames 1 to 10 seconds prior), or may be of long duration (e.g. measured for frames 30 to 300 seconds prior), or values ​​between these rough durations. Similarly, the history periods used to derive the z-scores for the different measurements may themselves be different, with the history period for measurements i.-iii. being shorter than the history period for measurements iv. or v., e.g. having a duration similar to the period involved in the compilation of the measurements themselves, but alternatively longer than this. It will be appreciated that shorter history periods aid in detecting events in a scene, while longer history periods aid in detecting scene changes.

[0046] A weighted sum of the z-scores is then performed to generate a single value for the frame. Typically, z-scores derived from inter-frame measurements (measurements i.-iii.) benefit from higher weights than z-scores derived from longer term measurements (measurements iv. and v.).

[0047] Thus, as a non-limiting example, measurements i.-iii. may each have a weight of 1, while measurements iv. and v. may each have a weight of 0.25.

[0048] More generally, the z-scores for measurements i.-iii. can each have a weight in the range of 0.3-1, while the z-scores for measurements iv. and v. can each have a weight in the range of 0.05-0.5, with an optional further limitation that the weights of the z-scores for measurements i.-iii. are each at least a predetermined interval value higher than the weights of the z-scores for measurements iv. and v., the predetermined interval value itself being in the range of 0.1 to 0.9.

[0049] A rolling average of the weighted sum is then generated. Again, the period over which the average is derived may be of short, medium, or long duration as previously defined. Again, it should be appreciated that shorter historical periods aid in detecting events in a scene, while longer historical periods aid in detecting changes in a scene.

[0050] Then, if the weighted sum of consecutive frames remains above the rolling average of that sum over a minimum number of frames, a frame selection window is established. This frame selection window typically lasts from when the weighted sum moves above the average until it returns below the average. The minimum number of frames may again be selected to alter the sensitivity of the system, but it will be appreciated that the minimum number should be significantly shorter than the period of the rolling average of the weighted sum, and typically less than 50% of that period, preferably less than 10% of that period, more preferably less than 1% of that period, subject to the practical limitation that the minimum number is at least one, and typically of the order of 2-10 frames.

[0051] It will therefore be appreciated that if the weighted sum exceeds the rolling average of this minimum number of frames, then at least this minimum number of image frames must be maintained in the buffer so that they can be recorded to a separate recording memory (or marked for storage within the buffer).

[0052] It will be appreciated that because the system selects "exciting" images based solely on statistics of previous frames of the current content (typically a mix of short-term and long-term statistics), and based on the relative difference (or minimum number thereof) of the scores of the current frame and the rolling history of those scores, the system automatically calibrates what counts as exciting based on (relative to) the nature of the content being displayed. It is therefore able to extract the most exciting elements of any given content, where the most exciting elements are optionally adjusted by adjusting one or more of the following elements:

[0053] - the number of previous frames in measurement i.-iii. (e.g., between 1 and 10), as discussed elsewhere herein;

[0054] - measuring the length of the predetermined time periods in iv. and v. (e.g., between 30 seconds and 5 minutes), as discussed elsewhere herein;

[0055] - the number of previous corresponding measurements used to calculate the z-score for a selected measure (e.g., short, medium, or long duration as discussed elsewhere herein);

[0056] - the relative weights used to generate the weighted sum of z-scores, as discussed elsewhere herein;

[0057] - the number of prior weights and z-scores used to generate the rolling average (e.g., short, medium, or long duration as discussed elsewhere herein); and

[0058] - The minimum number of frames for which the current weighted sum of the scores should exceed the rolling average, as discussed elsewhere in this document.

[0059] Optionally, because the system adapts to the relative excitement of the content, if the footage is very quiet (such as may be the case for security camera footage, wildlife monitoring camera footage, etc.), the system can become very sensitive, so the system can become very sensitive to small changes (such as garbage moving in the breeze).

[0060] Thus, optionally, the system may additionally check the excitement of the selected clips, e.g., against absolute values, e.g., against some or all of the absolute value or threshold comparison measures i.-v., or run an alternative excitement measurement scheme against the selected clips. Alternatively or additionally, particularly where immediate output of the clips is not required, the selected clips may be ranked for excitement (e.g., based on a weighted sum of the z-scores of the clips), such that the most exciting clips are ranked higher than less exciting clips. The clips may optionally still be kept in chronological order, but the ordering may be used as a filter that may be used to subsequently exclude selected clips that are relatively less exciting than other clips.

[0061] Modified Embodiments

[0062] In addition to the above-mentioned metrics i.-v., optionally a sixth metric is the audio energy, for example smoothed over a short time period such as 0.1 to 1.0 seconds.

[0063] Optionally, a set of audio metrics from which one or more audio metrics may be selected may be considered in a manner similar to the set of image metrics previously discussed herein. These may include the inter-frame audio sum of each audio channel, optionally normalized to the average of all channels for the current frame. Optionally, alternatively or additionally, the sum of absolute differences between corresponding audio channel pairs may be similarly calculated to determine whether the audio transitions from balanced to strong left or right channels or front or rear channels, etc. (i.e., assessing spatial dynamics, while the audio level of each channel alone helps assess volume dynamics); similarly, the audio energy of certain channels (such as a subwoofer channel) may be assessed in a manner similar to the inter-frame grayscale deltas discussed previously and in a similar manner. Similarly, in a manner similar to the color channels previously discussed, different frequency bands may also be assessed; for example, one or more of the base range (e.g., 15-100 Hz), the vocal range (100-10,000 Hz), and the treble range (10,000-20,000 Hz). These ranges may vary or overlap based on empirical assessments.

[0064] It should also be understood that it is desirable that the content of the clips may be interrupted from time to time by other content; in particular, within a video game, gameplay footage may be interrupted by non-game gameplay footage (such as pause menus and system screens), and may include footage that is part of the game but does not represent more general game progression, such as interactions with inventory menus and skill trees. In addition, some games use cut scenes, which will be standard for all users of the game. Optionally, the methods herein pause during some or all of these outputs so that first they are not selected and second they do not adversely change the statistics characterizing the game script. For example, it should be understood that after interacting with the inventory menu for one minute, it is possible that any subsequent actual gameplay may be considered exciting unless the inventory menu is excluded from the statistics.

[0065] A similar approach can be used for advertisements, logos, and credits in traditional TV and movie content.

[0066] Finally, optionally, some elements of the image frame may be discounted / ignored for any assessment of excitement; for example, an area of ​​a heads-up display showing a weapon charging may be relatively dynamic but not contribute to the game; similarly, a logo that appears to indicate that the game is being saved may cause a delta value to increase but is not part of the game itself. These elements / areas of the image frame may be excluded using a mask or other area indication provided by the game developer or set by the user. In some applications, a user interface may be provided to achieve this; for example, in the case of a security camera whose field of view is in a traffic light, it may be desirable to discount the area occupied by the traffic light because this will change regularly and perhaps more than the rest of the scene.

[0067] Finally, it should be understood that the above technology can be implemented on a variety of platforms, including within a graphics processing unit, a central processing unit, or a combination of both, and these can be located on a server or in the cloud, or on an entertainment device such as the PS5. The technology can run on live (generated or captured) or pre-recorded content.

[0068] Now go to Figure 2 In the embodiment of the invention of this specification, the method of video evaluation includes the following steps:

[0069] - For the current video frame, two or more metrics are generated in a first step s210. As described elsewhere herein, typically these are from a list consisting of:

[0070] i. the inter-frame grayscale increment sum of the absolute difference between the current frame and the immediately preceding frame;

[0071] ii. a first inter-frame color delta sum of absolute differences for at least a first color channel between a current frame and an immediately preceding frame;

[0072] iii. For at least

[0073] a second inter-frame color delta sum of absolute differences of a second color channel;

[0074] iv. for a first color channel of the current video frame, a delta between a mean value of a property of the first color channel and a current measured value, the mean value being derived over a predetermined period of time; and

[0075] v. For a second color channel of the current video frame, a delta between a mean value of a property of the second color channel and a current measured value, the mean value being derived over a similar predetermined time period.

[0076] - In a second step s220, a corresponding z-score of the generated metric is generated, as described elsewhere herein.

[0077] - In a third step s230, the z-scores are summed to generate a frame score for the frame, as described elsewhere herein.

[0078] - In a fourth step s240, a rolling average of the frame scores over a plurality of frames is generated, as described elsewhere herein.

[0079] - and in a fifth step s250, if the frame score of the current frame exceeds a rolling average of a predetermined number of previous frames, identifying (e.g., marking, updating metadata, flipping a suitable flag in the header or elsewhere, etc.) the current video frame for further processing, as described elsewhere in this document.

[0080] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the apparatus described and claimed herein are considered to be within the scope of the present invention, including but not limited to:

[0081] - In an example of an embodiment of the invention, the method comprises at least one of the metrics i.-iii. and at least one of the metrics iv.-v., as described elsewhere herein;

[0082] - In an example of an embodiment of the present invention, the step of summing the z-scores comprises generating a weighted sum, the weight of any one of the metrics i.-iii. exceeding the weight of any one of the metrics iv.-v., as described elsewhere herein;

[0083] - in an example of an embodiment of the Summary of the Invention, for any of the metrics i.-iii., the incremental sum of the absolute differences between the current frame and a rolling average of a predetermined number of previous frames, as described elsewhere herein;

[0084] - in an example of an embodiment of the present invention, for any of the metrics iv.-v., the attributes are one or more selected from the list consisting of the mean of the color channel and the standard deviation of the color channel, as described elsewhere herein;

[0085] - In this case, optionally, the or each metric is for two or more of the image frames

[0086] Many more subdivisions, as described elsewhere in this article;

[0087] - in an example of an embodiment of the present invention, for any of the metrics iv.-v., a previously recorded average value is accessed at the start of the video evaluation, as described elsewhere herein;

[0088] - In an example of an embodiment of the present disclosure, further processing includes the steps of: establishing a frame selection window, and selecting frames including the current frame until a frame score of another frame no longer exceeds a rolling average of a predetermined number of frames, thereby generating or identifying a video clip, as described elsewhere herein;

[0089] - More generally, further processing may include one or more of the following: recording, transmission / sharing, encoding, watermarking, combining, post-processing, ranking or scoring on an absolute and / or relative basis, or labeling for classification or training purposes;

[0090] - In an example of an embodiment of the invention, the method comprises the following steps: adjusting the recognition sensitivity by adjusting one or more elements selected from a list consisting of: the number of previous frames in measurements i.-iii., the length of the predetermined period in measurements iv. and v., the number of previous corresponding measurements used to calculate the z-score of the selected measurement, the relative weights used to generate the weighted sum of z-scores, the number of previous weighted sum z-scores used to generate the rolling average, and the minimum number of frames for which the current frame score should exceed the rolling average, as described elsewhere herein;

[0091] - in an example of an embodiment of the Summary of the Invention, wherein the value provided for the identifying step as the rolling average of the frame scores is the greater of a predetermined minimum value and an actual rolling average of the frame scores, as described elsewhere herein;

[0092] - In an example of an embodiment of the invention, the method comprises the steps of: evaluating frame scores of frame sequences identified for further processing and ranking the frame sequences based on the evaluation, as described elsewhere herein;

[0093] - In an example of an embodiment of the invention, the method further comprises at least a first measure of audio energy based on one or more selected from a list consisting of: a sum of audio per audio channel or frequency range thereof, a sum of absolute differences between corresponding pairs of audio channels or frequency ranges thereof, and an inter-frame delta sum of absolute differences between audio associated with a current frame and a previous frame of one or more audio channels or frequency ranges thereof, as described elsewhere herein;

[0094] - In an example of an embodiment of the invention, the method includes the steps of pausing the steps of generating metrics and z-scores, summing z-scores, generating a rolling average of frame scores, and identifying video frames while interrupted by a predetermined content type (e.g., when gameplay is interrupted by a non-playable element such as a cut scene, or a non-environmental element such as a menu, or a predetermined subset of such interruptions), as described elsewhere herein; and

[0095] - In an example of an embodiment of the invention, one or more portions of an image in consecutive frames are consistently discounted for use in generating the or each metric (eg, the area containing traffic lights in safety footage), as described elsewhere herein.

[0096] It should be understood that the above methods may be performed by software instructions or by appropriately adjusting the hardware by including or replacing dedicated hardware.

[0097] Thus, the required adaptation of existing parts of equivalent devices may be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (application-specific integrated circuit) or FPGA (field programmable gate array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program may be transmitted via a data signal over a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0098] Therefore, in another embodiment of the present invention, the entertainment device 10 (such as a PlayStation 5 or other console) is adapted (such as through suitable software instructions) to perform a method of video evaluation, the method comprising:

[0099] A video evaluation processor (e.g., CPU 20 and / or GPU 30) configured (e.g., via suitable software instructions) to perform the following steps:

[0100] For the current video frame, generate two or more metrics from a list consisting of:

[0101] i. the inter-frame grayscale increment sum of the absolute difference between the current frame and the immediately preceding frame;

[0102] ii. a first inter-frame color delta sum of absolute differences for at least a first color channel between a current frame and an immediately preceding frame;

[0103] iii. a second inter-frame color delta sum for absolute differences of at least a second color channel different from the first color channel between the current frame and the immediately preceding frame;

[0104] iv. for a first color channel of the current video frame, a delta between a mean value of a property of the first color channel and a current measured value, the mean value being derived over a predetermined period of time; and

[0105] v. for a second color channel of the current video frame, a delta between a mean measurement of a property of the second color channel and a current measurement, the mean being derived over a similar predetermined time period,

[0106] Generate the corresponding z-scores for the generated metrics;

[0107] summing the z-scores to generate a frame score for the frame;

[0108] generating a rolling average of the frame scores over a plurality of frames; and

[0109] If the frame score of the current frame exceeds a rolling average of a predetermined number of previous frames, the current video frame is identified for further processing.

[0110] Examples of embodiments of the present disclosure that implement the methods and techniques described herein (eg, by using appropriate software instructions) are contemplated to be within the scope of the present application outlined and described elsewhere herein.

[0111] Video Evaluation - Smoothness

[0112] As a corollary to the evaluation of video to identify excitement as described elsewhere herein, there are also times when it is useful to identify smoothness (as opposed to just static images such as system menus, inventory, etc.) Examples may be panning shots from cut scenes (pre-rendered or in-engine), smooth driving (e.g., when leading a race with no opponents on screen, the audience may be excited or satisfied even if there is limited visual excitement), or other establishing shots that provide context for subsequent action.

[0113] In the same way that assessments for identifying exciting videos are advantageously content-agnostic, the techniques disclosed herein are also content-agnostic, and further identify the smoothness of movement regardless of the type of movement depicted - whether lateral (panning in any direction), zooming (e.g., flying into the screen), or curved (e.g., rotating, or an arbitrary path such as a river or stream).

[0114] The image frames are usually from a video game, but can be anything else, such as a movie or television show, or a video monitor for a production line, such as a conveyor belt or other continuously operating machine.

[0115] Therefore, in the embodiment of the present specification, dense optical flow is calculated between consecutive image frames.

[0116] The optical flow vector may be calculated using any known and suitable optical flow means.

[0117] In principle, a vector can be calculated for each corresponding pixel between a pair of frames.

[0118] In practice, subsets can be used to reduce computational overhead and still generate a good approximation of the general flow in the scene. Thus, for example, one pixel out of 4, 9, 16, 25, 36, 49, 64, etc. pixels may be used (corresponding to one pixel in square pixel blocks of varying sizes, although non-square blocks may also be used). Alternatively or additionally, some or all pixels in a subset of blocks in the entire frame may be used to detect local divergence of flow within the image.

[0119] In cases where the displayed content is variable (e.g., in games or movies), then the interaction between the regular sampling pattern and the unconstrained image sequence will still provide a reasonable Monte Carlo approximation of the dense optical flow measure.

[0120] For fixed views of a scene, such as monitoring machinery, such a regular sampling pattern may also work, but in principle may always miss key features of the scene. In this case, the overall sample density can be adjusted, or the sample offset / positioning can be adjusted to capture the key elements. Similarly, a user interface can be used to identify key areas of a fixed view for relatively more intensive analysis, and the sample density can be adjusted in those key areas.

[0121] Alternatively, in either case, a random distribution of samples can be used. Likewise, the density of samples can be adjusted to capture relevant areas of a fixed scene.

[0122] In contrast, in such fixed scenes or even in unconstrained image sequences from a game or television show, there may be parts of the image that are preferably ignored, such as a heads-up display or game statistics such as a health bar, a channel logo, subtitles, or letterbox boundaries that do not represent a stream in the actual depicted scene. Samples from these areas can be discounted (e.g., by not sampling from these areas). The areas themselves can be identified by metadata associated with the game / content, and / or by rules or heuristics, such as detecting black boundaries, and / or building a long period rolling average of the image or a portion thereof to identify permanent features, such as a logo (which will be the only element that retains high spatial frequency features and / or may be a different color distribution from the rest of the image in such an average).

[0123] Thus, more generally, optical flow vectors may be calculated / generated for at least a subset of corresponding pixels between a pair of frames, where the subset samples at least a predetermined region of interest in each pair of frames and is sampled to at least a predetermined sampling density.

[0124] Thus, in any case, a set of representative optical flow vectors is established for corresponding pixels of consecutive frame pairs (whether for all pixels, 1 / 2, 1 / 4 or less in a regular or random distribution of regular or variable density, optionally ignoring one or more regions that do not represent flow in the image).

[0125] Typically this will be between shared / overlapping pairs of frames, e.g., pairs 1-2, 2-3, 3-4, etc., but in principle could be adjacent / non-overlapping frames, e.g., pairs 1-2, 3-4, 5-6, etc.; this would cut the number of comparisons that need to be made in half, but assumes that for smooth motion, non-overlapping pairs of frames are a sufficient metric (i.e., it is unlikely that there are isolated action frames within a series of smooth frames). In this document, "consecutive pairs of frames" may be taken to mean either form unless specified.

[0126] Next, the increment (difference) between corresponding optical flow vectors (i.e., consecutive frame pairs) is calculated. In other words, the value representing the change in the optical flow vector value between consecutive frame pairs.

[0127] If the motion is smooth, these increments will be small. If the motion is variable, these increments (or at least some of them) will be large.

[0128] The differences may be evaluated against various criteria to identify corresponding pairs of frames for further processing.

[0129] As a first example, the delta sum of a frame pair characterizes the overall smoothness of the video in that frame pair.

[0130] Therefore, the first criterion is to detect smooth overall motion by setting the increments of the frame pairs and a first threshold that should not be exceeded. Optionally, the first threshold can be different depending on whether overlapping or non-overlapping frame pairs are used; the increments of overlapping frames are continuous, while the increments of non-overlapping frames are not continuous and in principle can show slightly larger differences for similar smooth overall motion.

[0131] False positives can also be avoided by summing the vector itself - low deltas due to little or no movement (such as in a game menu or pause scene) can be identified because the actual vector will also be close to zero.

[0132] Therefore, the second criterion is to detect smooth and significant motion by setting a first threshold that the delta sum of a frame pair should not exceed and a second threshold that the optical flow vector sum should exceed. Likewise, for deltas based on non-overlapping frame pairs, the second threshold can be slightly larger.

[0133] It will be appreciated that the threshold may be normalized according to the number of samples per image used.

[0134] Subsequently, frame pairs showing smooth motion, or alternatively smooth and significant motion, are identified (eg, by tags, updated metadata, appropriate flags in flip headers or elsewhere, etc.) for further processing.

[0135] Optionally, the first step here (before identification or as the first step of further processing) is to identify a continuous sequence of frame pairs of length M, all of which show smooth (and optionally significant) motion, and thus display a scene (of duration M frames, where M is a predetermined number) exhibiting such motion. This step is optional, since, for example, for production line monitoring, smooth motion may be the default state.

[0136] Such scenes may then be selected or identified for recording, for example in a highlight clip.

[0137] More generally, further processing may include one or more of: recording, transmitting / sharing, encoding, watermarking, combining, post-processing, ranking or scoring on an absolute and / or relative basis, or labeling for classification or training purposes.

[0138] Optionally, where the video assessment system implements assessments of excitement as well as smoothness, scores for identifying both qualities (e.g., as described herein) may then be tracked in parallel, and optionally the score for one may be used to inform decisions regarding further processing of the other. For example, a scene with smooth motion within a P frame preceding an exciting scene may be specifically selected for a highlight shot (or, e.g., a security alert, depending on the purpose) in conjunction with the exciting scene to provide context and contrast (and therefore interest) to the viewer. In this case, typically where scenes are within each other's P frames, intermediate P frames are also included to create a continuous combined scene.

[0139] Modified Embodiments

[0140] Smooth motion can be caused by movement of the viewpoint (e.g., in most games, movies, and TV content) or movement of the scene (e.g., in monitoring applications such as production lines or traffic flow, but also in games, movies, and TV content). As mentioned elsewhere herein, the pixels selected to generate the optical flow vectors can be any representative distribution within the scene, and need not be all of the scene, or the sampling density need not be the same across all of the scene (or equivalently but potentially less efficiently, the weighting of the samples within the sum need not be the same across all of the scene).

[0141] Therefore, now refer to Figure 3 Optionally, where the motion of one or more objects within the scene is an important aspect, the sampled pixels may be concentrated (primarily or exclusively) within the area of ​​the captured image that the one or more objects will or should occupy. Figure 3 In , an example is the flow of traffic along a road (but could similarly be a river, or the movement of recyclable waste along a conveyor belt). In general, this will therefore apply to situations where the content of the image can be anticipated, such as in surveillance scenarios. In this case, areas of the image where the object of interest will not (or should not) go are indicated by the shaded triangles, and will have fewer or no samples. In this case, this can improve the accuracy of the smooth motion estimate of interest by reducing or excluding unwanted variability (e.g. from roadside trees swaying in the wind). Thus, more generally, in this case, a subset of corresponding pixels between a pair of frames conforms (primarily or exclusively) to a region to which the object or objects of interest should generally be constrained.

[0142] In contrast, where the motion of one or more objects within the scene can be considered as unwanted noise when identifying smooth motion of the viewpoint, the sampled pixels can be concentrated (primarily or exclusively) outside the area of ​​the captured image that the one or more objects will occupy. Figure 4, for example in a game with a third person character view that attempts to keep the player character within the center third of the screen, then the center third of the screen (indicated by the center shaded area in the figure) can be essentially ignored by reducing the number of samples in that area (perhaps to zero). In this case, this can improve the accuracy of smooth motion estimation for general scenes by essentially not considering the animation of the main character, and typically also not considering the animation of any protagonist / antagonist directly in front of it. Thus, more generally, in this case, a subset of corresponding pixels between pairs of frames conforms to a region outside of which one or more unwanted objects should generally be constrained.

[0143] Similarly, where only a specific type of movement (either for an object or for a viewpoint) is of interest, then flows (and corresponding deltas) of that type or movement may be detected.

[0144] For example, zooming of a scene (e.g. in a driving game) should generate optical flow vectors in the directions indicated by the arrows at the corresponding positions of these arrows. Movements of this type of zooming are selectively detected using only the vertical or horizontal component (as appropriate) of the flow vectors indicated by the arrows. Thus, smooth zooming movements can be detected even if other motion occurs (e.g. the activities of other users in other cars). In this way, the smoothness of the user (if they are controlling the viewpoint directly or indirectly) can be detected regardless of what else is happening on the screen. Therefore, more generally, the calculation of the difference between optical flow vectors is based only on a subset of the directional components of the optical flow (whether only these components are derived when obtaining the vectors, or they are extracted during the calculation). According to Figure 5 , the subsets may include different directional components in different regions of the image.

[0145] Other net motions such as horizontal (or in any particular direction) translation can be filtered by obtaining relevant vector components (as described above) in the image or in a portion thereof. Similarly, a particular motion path can be specified by selecting relevant vector components in corresponding regions of the image.

[0146] Additionally, detection of exclusive specific motions may be detected by measuring optical flow in directions other than a specified direction (e.g., + / - a predetermined angular tolerance).Alternatively, these motions may only be considered problematic if they are not themselves smooth, as determined using the techniques herein.

[0147] Thus, smooth and optionally substantial motion of a viewpoint and / or an object within a view may be detected for any motion (default) or for a particular kind of motion (as defined by an associated selection of thresholds), optionally detecting whether the motion is exclusive to the scene, and / or whether other motion in the scene is also smooth. In response, qualifying frames may be identified, and a clip (e.g., of M consecutive frames) may be stored, identified for storage, or otherwise further processed.

[0148] Reference now Figure 6 In an embodiment of the invention of the present specification, a video assessment method for a video including consecutive video frames includes the following steps.

[0149] - in a first step 610, generating optical flow vectors for at least a subset of corresponding pixels between a pair of frames, as described elsewhere herein;

[0150] - in a second step s620, calculating the difference between corresponding optical flow vectors of consecutive pairs of frames, as described elsewhere herein; and

[0151] - In a third step s630, the obtained differences are evaluated against a predetermined criterion to identify corresponding pairs of frames for further processing, as described elsewhere herein.

[0152] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the apparatus described and claimed herein are considered to be within the scope of the present invention, including but not limited to:

[0153] - at least one subset of pixels samples at least a predetermined region of interest in each frame pair, as described elsewhere herein;

[0154] - at least a subset of pixels is sampled at least at a predetermined sample density (and optionally in a regular or random pattern with uniform or non-uniform density, further optionally based on user input), as described elsewhere herein;

[0155] - the predetermined criterion comprises that the sum of the differences between corresponding optical flow vectors of consecutive pairs of frames does not exceed a first threshold (the first threshold optionally depends on the overlapping or non-overlapping nature of the pair of frames, and / or optionally on any selectivity of the position within the image of the pair of frames), as described elsewhere herein;

[0156] - In this case, optionally, the predetermined criterion includes that the sum of the vectors does exceed a second threshold

[0157] values, as described elsewhere in this article;

[0158] - The method comprises the steps of: identifying a consecutive sequence of M identified pairs of frames for recording, as described elsewhere herein;

[0159] -The method comprises the steps of: identifying video frames having a threshold activity level and, if within a predetermined distance of each other within a sequence of frames, associating those video frames with the identified frame pairs for recording as described elsewhere herein;

[0160] - In this case, optionally, the activity is detected according to a video evaluation method which

[0161] The method includes the following steps:

[0162] For the current video frame, generate two or more metrics from a list consisting of:

[0163] i. the inter-frame grayscale increment sum of the absolute difference between the current frame and the immediately preceding frame;

[0164] ii. a first inter-frame color delta sum of absolute differences for at least a first color channel between a current frame and an immediately preceding frame;

[0165] iii. a second inter-frame color delta sum for absolute differences of at least a second color channel different from the first color channel between the current frame and the immediately preceding frame;

[0166] iv. for a first color channel of the current video frame, a delta between a mean value of a property of the first color channel and a current measured value, the mean value being derived over a predetermined period of time;

[0167] as well as

[0168] v. for a second color channel of the current video frame, a delta between a mean measurement of a property of the second color channel and a current measurement, the mean being derived over a similar predetermined time period,

[0169] Generate the corresponding z-scores for the generated metrics;

[0170] summing the z-scores to generate a frame score for the frame;

[0171] generating a rolling average of the frame scores over a plurality of frames; and

[0172] identifying the current video frame for further processing if the frame score of the current frame exceeds a rolling average of a predetermined number of previous frames,

[0173] As mentioned elsewhere in this article;

[0174] - a subset of corresponding pixels between a pair of frames conforms to a region to which one or more objects of interest should generally be constrained, as described elsewhere herein; or,

[0175] - a subset of corresponding pixels between a pair of frames conforms to a region outside of which one or more unwanted objects should generally be constrained, as described elsewhere herein;

[0176] - the calculation step is based only on a subset of the directional components of the generated optical flow vectors, as described elsewhere herein; and

[0177] - In this case, the subset of directional components may include different directional components in different regions of the frame, as described elsewhere herein.

[0178] It should be understood that the above-described methods may be performed on hardware suitably adapted to the application by software instructions or by inclusion or replacement of dedicated hardware.

[0179] Thus, the required adaptation may be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program may be transmitted via a data signal over a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0180] Therefore, and again referring to Figure 1 In an embodiment of the invention of the present specification, the entertainment system includes a video evaluation processor (e.g., CPU 20 and / or GPU 30), which is adapted (e.g., through suitable software instructions) to perform the following steps for a video including consecutive video frames in operation:

[0181] - generating optical flow vectors for at least a subset of corresponding pixels between a pair of frames, as described elsewhere herein;

[0182] - computing the difference between corresponding optical flow vectors for consecutive pairs of frames, as described elsewhere herein; and

[0183] - The resulting differences are evaluated against a predetermined criterion to identify corresponding pairs of frames for further processing as described elsewhere herein.

[0184] Examples of embodiments of the present disclosure that implement the methods and techniques described herein (e.g., by using appropriate software instructions) are contemplated within the scope of this application and include, but are not limited to:

[0185] - at least a subset of pixels samples at least a predetermined region of interest in each frame pair, as described elsewhere herein; and

[0186] - At least a subset of the pixels is sampled at least at a predetermined sample density, as described elsewhere herein.

[0187] Video Evaluation - Scene Switching

[0188] Besides exciting and smoothing events within a video, another important event is scene change, i.e. switching between views (and potentially between entire locations and contents).

[0189] Scene cuts can be a regular event within TV and movies, but are less common in video games, where they typically signal the start of a so-called "cut scene", or a different display mode (e.g. when interacting with a non-player character), or a change in game mode (e.g. switching to an inventory or other menu, or switching to a system menu).

[0190] It is desirable to identify scene cuts in a computationally inexpensive manner.

[0191] If the stimulation scheme described elsewhere in this document has been used, then optionally it can also be used to detect scene changes with little additional computational cost. This will be based on measuring one or more of i, ii or iii, and will involve detecting a single inter-frame spike in the corresponding delta sum in one or more of these (usually any of all those measured simultaneously) surrounded by lower inter-frame delta sum values. Typically, exciting events (even explosions) will not end in a single frame, and the switch is virtually instantaneous, so will result in an isolated value spike.

[0192] However, this approach still leaves room for error (e.g., camera flashes, lightning, and similar global changes in lighting or color may be instantaneous and therefore may be detected as false positives).

[0193] Therefore, as an alternative or supplement to the above solution, in an embodiment of the present specification, the method of switching detection includes generating a perceptual hash of the current image frame and then comparing it with a corresponding hash of at least a previous frame.

[0194] Perceptual hashing is a method of generating a consistent and small representation of an image. The resulting hash (unlike a cryptographic hash) is similar for similar images.

[0195] A simple example is as follows:

[0196] i. Take the original image and reduce it to a predetermined size; for example, 8×8 pixels (although this is a non-limiting example, and the reduced image can, for example, maintain a known aspect ratio - for example, as a 16×9 pixel image). This reduction removes high-frequency image features and retains only low-frequency structure.

[0197] ii. The image can optionally be made grayscale. Using the 8×8 pixel example, this reduces the representation of the image to just 64 values ​​(the color version would be 3×64).

[0198] iii. In either case, calculate the grayscale or average value for each color channel.

[0199] iv. Then generate a bit sequence by setting the bit of each gray value according to whether it is above or below the average. In effect, this reduces the image to 64 bits representing whether a portion of the image is darker or lighter than the average. The color version will have 3×64 bits.

[0200] v. Finally, optionally construct a hash by converting the 64 bits into a 64-bit integer using any predetermined and consistent bit order. As a non-limiting example from an 8×8 image, the bits can be read from left to right, top to bottom, generating an integer using the big-endian convention.

[0201] So, in summary, hashing involves - removing high frequencies / keeping low frequencies, usually by reducing the image size. Optionally but preferably, reducing the colors to, for example, grayscale, and then encoding the changes in the grayscale image (e.g. as a binary threshold based on a global average). The resulting encoding can then be stored as is, either as a binary string or a hashed integer.

[0202] A similar perceptual hashing aims to achieve the same goal, but using the frequency domain, as follows:

[0203] i. Again, the original image is reduced in size, but to a different extent than in the first example; for example to a 32x32 image (ie, approximately 4 times larger than in the first example). This removes some high frequency elements, but to a different extent than in the first example.

[0204] ii. Optionally, reduce the color to grayscale again.

[0205] iii. Compute the discrete cosine transform (DCT) of the image (e.g., to generate a 32×32 DCT that represents the image in frequencies). Note that the image is reduced to 32×32 pixels primarily to simplify the DCT computation; larger images (e.g., 64×64 or 128×128 or indeed at native resolution) could similarly be used, but would impose an unnecessary computational burden.

[0206] iv. Only the top left 8x8 values ​​are retained; these represent the lowest frequencies in the picture. Thus, at this point, the frequencies retained by both this technique and the first example are roughly the same, but in this technique the frequencies are more accurately represented.

[0207] v. Calculate the average of these 8x8 DCT values ​​again, this time excluding the first (DC) term, which may be very different from the others.

[0208] vi. Again a bit sequence is produced by setting one bit for each 8x8 DCT value depending on whether it is above or below the mean. Thus, this again results in a 64-bit representation, this time indicating the relative presence or absence of low frequency image components.

[0209] vii. Finally, the hash is again optionally constructed by converting the 64 bits into a 64-bit integer using any predetermined and consistent bit order. In this case, the bits may optionally be read using a big-endian convention emanating from the corner (DC) positions, such as (1,1) (2,1) (2,2) (1,2) (3,1) (3,2) (3,3) (2,3) (1,3) etc.; this will mean that the lowest frequency elements within the image have the greatest influence on the value of the resulting hash number.

[0210] So, the two methods are basically the same, but one method computes the hash in the spatial domain (using a downscaled picture) and one method computes the hash in the frequency domain (using a downscaled DCT).

[0211] A similar approach in the frequency domain can be done by replacing the DCT with a wavelet transform.

[0212] As mentioned above, 8x8 and 32x32 images are merely non-limiting examples, but are sufficient to characterize images in order to detect relative similarities of consecutive images.

[0213] For consecutive images, corresponding hashes are generated using one of the spatial or frequency based methods described above (that is, one of these methods is used consistently). Each hash is then compared to at least the hash of the immediately preceding hash.

[0214] This can be done simply by subtracting the value of one hash from the other, e.g. hash_t2 – hash_t1 = 0 for the same image.

[0215] Typically, the hashes will be different for consecutive images in a video. Thus, in one example, hash_t2 - hash_t1 = 20, meaning the difference in the 64-bit value of the hash is 20.

[0216] However, because the value of the bits has different weights within a 64-bit image, the weight is greater in some bits than in others. The bit ordering scheme for frequency domain hashing disclosed elsewhere herein is one example in which the bits associated with the lowest frequency features have the highest weights (i.e., the lowest frequency bits become the most significant bits), so that changes in the hash value roughly correlate with structural changes in the image.

[0217] In principle, a naive threshold when consecutive images correspond to switches between scenes is when 50% of the bits have changed (since for unrelated images one can assume that each feature has even a chance to be above or below the average threshold, so a 50 / 50 change between two images is the same).

[0218] Thus, for a 64-bit sequence, the threshold may be a Hamming distance of 32 (ie, 32 bits of difference between the hashes of the two images).

[0219] Using the Hamming distance means that converting the bits into an actual hash number is not necessary (although it may be a convenient way to store the bits), so in practice the 64-bit sequence may not need to be handled and manipulated as a number.

[0220] For some forms of content, there may be certain features that tend to persist between scenes, such as, for example, long horizontal features of ground / sky or floor / wall. Therefore, optionally, a weighted Hamming distance may be used, which varies the contribution of certain bits in the spatial or frequency based hash. These bits may be determined empirically. However, such weighting makes the evaluation of the hash more complex, and for switches between scenes, typically an unweighted comparison may be sufficient. However, in practice a threshold Hamming distance other than 32 / 50% may be determined empirically, for example based on the type of content or the type of game or even a specific title.

[0221] In any case, perceptual hashing is a fast, robust, and low-overhead means for detecting significant differences between consecutive images.

[0222] As previously described, switching scenarios can be detected based on comparing the current hash to the immediately preceding hash, but if measurements i, ii or iii are used this can lead to similar problems to those mentioned previously, i.e., near-instantaneous events such as camera flashes or lightning may be misinterpreted as lightning strikes.

[0223] Thus, for any of the switching detection techniques described herein, a comparison of a sequence of images may optionally be used, where the sequence may include numbers suitable for distinguishing transient events in a scene, such as a camera flash or optionally an explosion.

[0224] Reference now FIG. 7A to FIG. 7C , in each case, the x-axis refers to time (e.g., in terms of image frames counting backwards in time from the current frame 't'), and the y-axis refers to the Hamming difference between adjacent image hashes (or equivalently, the sum of inter-frame deltas measuring one or more of i.-iii.).

[0225] Fig. 7AThe difference between the hash of image frame t and the hash of the previous image frame within a single scene is shown. As the scene evolves, a gradual increase in the difference can be seen. At the same time, Figure 7B An event in the scene (e.g., an explosion) is shown occurring at frame t-2. In this case, there is a significant difference from the hash of this frame, but the difference from the hash of earlier frames again shows a gradual increase in the difference (here with a nominal offset reflecting the persistent difference in the visible scene caused by the explosion in frame t-2). Finally, Figure 7C A scene cut is shown occurring at frame t-2. In reality, frame t-2 is uncorrelated with frame t, so the Hamming distance immediately jumps to a value close to 50%, and since the earlier images in this other scene are similarly uncorrelated, the distances of these images (with some noise) are also close to a Hamming distance of 50% (or whatever empirical threshold is determined). Therefore, frame t-2 can be identified as the boundary of a scene cut.

[0226] Thus, a scene change can be identified by detecting a step function in the Hamming distance between the perceptual hashes of the current image frame and N earlier image frames, where N is 2 or greater (i.e., a scene change is captured at time t-1). Longer values ​​of N give greater certainty of a scene change, up to values ​​of N where the scene change was long enough ago that the change in the current scene has accumulated to the point where the hash from the beginning of the current scene is now similarly close to 50% of the Hamming distance - so, for example, at FIG. 7A to FIG. 7C In the purely exemplary scenario shown, once the current scene is 6 frames old, it cannot be distinguished from a switch (e.g., Figure 7C In the Fig. 7A The Hamming distance of the old scene represented by the dotted line is compared with the value from the scene before the switch represented by the solid line).

[0227] This upper limit for N can be determined empirically, or alternatively, a formula comprising Figure 7B The value of N for transient events of the illustrated content. Thus, for example, if a typical explosion is determined to last 10 frames + / - 2 frames, a value of N=15 may optionally be used.

[0228] Finally, it should be noted that when Figure 7B is the current frame, then all previous frames look very different, and this could be mistaken for a scene cut; this is Fig.7D (here, the explosion in frame t makes all earlier frames in the scene look different, although there is still some correlation.) Therefore, it is again preferred that the value of N contains Figure 7B A transient event of the instantiated content such that the current frame and preferably two or more older frames are on either side of the transient event.

[0229] Alternatively or additionally, optionally, scene boundaries represented by a step function should only be identified at frame N or earlier (e.g., at least t-2 or earlier), since this excludes sharp boundaries caused by the current frame as transient events and requires that at least the previous frame t-1 is also different from t-2 (or more generally, frame t-(N-1) is different from frame tN).

[0230] Modified Embodiments

[0231] As an alternative or in addition to the entire image frame, the image frame can be divided into sub-parts, such as, for example, a left half and a right half. This can then be used to detect common switches between viewpoints in a conversation; the overall background may be similar between viewpoints, but the person framed in the shot will switch from left to right; for a half-image perspective, this would be similar to a scene cut.

[0232] Return to reference Figure 4 Similarly, the center third of the image can be ignored (e.g., replaced with a median gray in a reduced 8×8 or 32×32 image), so that significant changes (such as explosions and jump scares) are generally ignored. At the same time, changes in the entire scene, such as switching from exterior to interior or from a game environment to a menu, can still be detected.

[0233] It should be understood that one or more of the inter-frame delta sums of measurements i, ii and iii mentioned elsewhere herein, or differences in perceptual hashing (spatial or frequency based) as mentioned elsewhere herein, may be used, optionally over N>=2 previous frames, and optionally for one or more subsets or sub-regions of the image, to detect switches between scenes.

[0234] This switch can be used, for example, to limit the number of frames used before (or after) an excitement cut to within the same scene. Similarly, where a single divergent frame value can be ignored for evaluating a smooth scene, the switch detection can detect whether this is actually due to a scene change and reset the estimate of smoothness. More generally, it can be used to establish the start of a sequence of frames whose metrics are then used to detect excitement and / or smoothness.

[0235] Reference now Figure 8 In an embodiment of the invention of the present specification, a video assessment method for a video including consecutive video frames comprises the following steps:

[0236] - in a first step s810, for each video frame image, generating a perceptual hash of at least a first portion of the corresponding image, as described elsewhere herein;

[0237] - in a second step s820, comparing the generated perceptual hash of the current video frame image with a corresponding perceptual hash of at least one previous video frame image (e.g., an immediately preceding video frame), as described elsewhere herein; and

[0238] - In a third step s830, if the difference between the compared perceptual hashes exceeds a predetermined threshold, a previous video frame is identified as a boundary of a scene cut within the video, as described elsewhere herein.

[0239] It should be understood that the various video assessment methods and systems disclosed herein may be used in any suitable combination, including those described herein. Thus, for example, where a clip has been identified as exciting, the clip may be extended to the beginning and / or end of a scene identified by a scene cut detector, for example based on how long the intervening gap between the exciting clip and the scene boundary is.

[0240] The foregoing discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, the scope of the present invention and other claims. The present disclosure, including any readily discernible variations taught herein, defines in part the scope of the preceding claim terms so that no inventive subject matter is dedicated to the public.

Claims

1. A video assessment method for a video comprising consecutive video frames, the method comprising the following steps: generating an optical flow vector for at least a subset of corresponding pixels between the pair of frames; Calculate the difference between corresponding optical flow vectors of consecutive frame pairs; as well as The resulting differences are evaluated against a predetermined criterion to identify corresponding pairs of frames for further processing.

2. The method according to claim 1, wherein: At least a subset of the pixels samples at least one predetermined region of interest in each frame pair.

3. The method according to claim 1 or 2, wherein: At least a subset of the pixels is sampled at least at a predetermined sampling density.

4. A method according to any one of the preceding claims, wherein: The predetermined criterion includes that the sum of the differences between corresponding optical flow vectors of consecutive frame pairs does not exceed a first threshold.

5. The method according to claim 4, wherein: The predetermined criterion includes that the sum of the vectors does exceed a second threshold.

6. The method according to any one of the preceding claims, comprising the steps of: A contiguous sequence of M identified frame pairs is identified for recording.

7. The method according to any one of the preceding claims, comprising the steps of: Video frames having a threshold activity level are identified and, if within a predetermined distance of each other within a sequence of frames, associated with the identified frame pair for recording.

8. A method according to any one of the preceding claims, wherein: The subset of corresponding pixels between the frame pair corresponds to the region to which the object or objects of interest should generally be constrained.

9. The method according to any one of claims 1 to 7, wherein: The subset of corresponding pixels between the frame pair conforms to a region outside of which one or more unwanted objects should generally be constrained.

10. A method according to any one of the preceding claims, wherein: The calculation step is based only on a subset of the directional components of the generated optical flow vectors.

11. The method according to claim 10, wherein: The subset of directional components can include different directional components in different regions of the frame.

12. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any preceding claim.

13. An entertainment system comprising: The video evaluation processor is adapted to perform the following steps in operation on a video comprising successive video frames: generating an optical flow vector for at least a subset of corresponding pixels between the pair of frames; Calculating the difference between corresponding optical flow vectors for consecutive pairs of frames; and The resulting differences are evaluated against a predetermined criterion to identify corresponding pairs of frames for further processing.

14. The entertainment system of claim 13, wherein: At least a subset of the pixels samples at least one predetermined region of interest in each frame pair.

15. An entertainment system according to claim 13 or claim 14, wherein: At least a subset of the pixels is sampled at least at a predetermined sampling density.

Images