Color grading content based on similarity to examples

Abstract

Systems and methods are provided for color grading images and videos based on similarities to examples. In preparation for color grading of new content, example frames related to expected new content may be obtained, and color grading parameters for the example frames may be obtained. For color grading of new content when the new content is created or received, a similarity between a frame of the new content and an example frame may be determined. A similarity between a frame of the new content and the example frame may be determined, and the similarity may be combined with the obtained color rating parameters from the example frame to determine suitable color rating parameters for application to the new content. New content may then be color graded using the determined color grading parameters.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 391,528, filed on July 22, 2022, and European Patent Application No. 22197590.7, filed on September 23, 2022, which are incorporated herein by reference in their entirety. Technical Field

[0003] The present application generally relates to systems and methods for image processing, image display, and image reproduction. Embodiments of the present invention provide methods and apparatus for processing image data to transform color and / or hue for display or reproduction locally or at a downstream device. Some example embodiments are displays configured to receive and process image data and display the processed image data for viewing. Summary of the invention

[0004] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments of the invention. The embodiments described herein relate to systems and methods for color grading images and videos based on similarity to examples. In preparation for color grading new content, example frames related to the expected new content can be obtained, and color grading parameters for the example frames can be obtained. In order to color grade the new content as it is created or received, the similarity between the frames of the new content and the example frames can be determined. The similarity between the frames of the new content and the example frames can be combined with the color grading parameters obtained from the example frames to determine appropriate color grading parameters to apply to the new content. The new content can then be color graded using the determined color grading parameters.

[0005] In an exemplary aspect of the present disclosure, a method for color grading a source image using multiple color grading operations is provided. The method includes: obtaining a source image for color grading; obtaining multiple color grading parameter settings associated with each example frame from multiple example frames, wherein each color grading parameter setting is used as an input parameter for an associated color grading operation in the multiple color grading operations; obtaining multiple similarity measurement results, each similarity measurement result indicating a similarity level between the source image and a different corresponding example frame from the multiple example frames; obtaining multiple weighted averages based on the similarity measurement results and the multiple color grading parameters, wherein each weighted average is associated with a different corresponding color grading operation in the multiple color grading operations; applying multiple color grading operations to the source image using each of the weighted averages as an input parameter for the color grading operation associated with each weighted average to produce a color graded image; and providing the color graded image.

[0006] In another exemplary aspect of the present disclosure, a color grading system is provided, the color grading system comprising at least one controller configured to obtain a source image for color grading; obtain a plurality of color grading parameter settings, wherein each color grading parameter setting is used as an input to a color grading operation, and wherein each color grading parameter setting is associated with a different corresponding example frame from a plurality of example frames; calculate a plurality of similarity measurements, each similarity measurement indicating a similarity level between the source image and a different corresponding example frame from the plurality of example frames; calculate a weighted average based on the plurality of similarity measurements and the plurality of color grading parameter settings; apply the color grading operation to the source image using the weighted average as an input parameter of the color grading operation to produce a color graded image; and provide the color graded image.

[0007] In another exemplary aspect of the present disclosure, a method is provided, the method comprising: obtaining a sequence of image frames; obtaining multiple similarity measurement results, each similarity measurement result indicating a similarity level between a pair of different corresponding image frames from the sequence of image frames; and selecting a group of example frames from the sequence of image frames by: (i) adding a first image frame from the sequence of image frames to the group of example frames; (ii) based on the similarity measurement results, identifying which image frame among the image frames that have not yet been added to the group of example frames has the least similarity to (multiple) image frames in the group of example frames; (iii) adding the image frame identified in (ii) to the group of example frames; and (iv) repeating (ii) and (iii) until a completion condition is met.

[0008] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0009] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0010] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] These and other more detailed and specific features of various embodiments are more fully disclosed in the following description with reference to the accompanying drawings, in which:

[0012] Figure 1 Depicted is an example process for an image encoding and decoding pipeline.

[0013] Figure 2 Depicted is an example process for an image encoding pipeline.

[0014] Figure 3 An example process for identifying an example frame is depicted. DETAILED DESCRIPTION

[0015] The present disclosure and its various aspects may be embodied in various forms, including: hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​the various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0016] In the following description, many details such as optical device configuration, timing, operation, etc. are set forth to provide an understanding of one or more aspects of the present disclosure. It is obvious to those skilled in the art that these specific details are only exemplary and are not intended to limit the scope of the present application.

[0017] Figure 1 An example process of an image transmission pipeline (100) is depicted, showing various stages from image capture to display of image content. An image (102) that may include a sequence of video frames (102) is captured or generated using an image generation block (105). The image (102) may be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide image data (107). Alternatively, the image (102) may be captured on film by a film camera. The film is converted to a digital format to provide image data (107). In a production stage (110), the image data (107) is edited to provide an image production stream (112).

[0018] The image data of the production stream (112) is then provided to a processor (or one or more processors, such as a central processing unit (CPU)) at block (115) for example-based color grading (e.g., real-time production editing). Example-based color grading at block (115) may include adjusting or modifying the color or brightness of the image to enhance the image quality or achieve a specific look of the image in accordance with the creative intent of the image creator. This is sometimes referred to as "color timing" or "color grading." The methods described herein may be performed by a processor at block (115). A final version of the production (117) is generated from the image data resulting from the example-based color grading (115) for distribution.

[0019] Figure 1 The example-based color grading (115) and pipeline (100) can be configured for transmitting live content where dynamic conditions produce unpredictable images, such as in live broadcasts, gaming content, virtual reality content, etc. When transmitting such live content, it is generally not possible for a creative artist or other user to manually color grade each image frame (to enhance image quality or achieve a specific look for the image in accordance with the creative intent of the image creator). For example, while a creative artist may provide a single set of color grading parameters or adjustments to apply to all image frames, the fixed color grading adjustments will not be suitable for the changing conditions of the live content.

[0020] Example-based color grading (115) uses predetermined color grading parameters (126) of example frames for a range of unknown future views and conditions, thereby enabling color grading of live content. As an example, an image creator may obtain past live content (e.g., a previous live broadcast, a game content clip, a virtual reality content clip, etc.) as part of example color grading (124), identify example frames from the past live content, and use a color grading station (125) to provide the image creator's artistic input to manually color grade the example frames. Then, when new live content is processed through example-based color grading (115) and pipeline (100), frames of the live content can be compared for similarity to the manually color graded example frames. The new frames can then be manually color graded based on the similarity between the new frames and the color graded example frames by transferring the image creator's color grading of the example frames to the new frames. In this manner, Figure 1 The example-based color grading (115) and pipeline (100) can color grade live content in real time or near real time despite the somewhat unpredictable nature of live content.

[0021] In some embodiments, the color grading operations applied in the example-based color grading (115) and the example-based color grading (124) may include operations that make the effect on the image output approximately monotonic. A color grading operation may be approximately monotonic when a change in a particular direction of a setting for the operation also causes the output to move in a particular direction without changing in the opposite direction. As a specific example, a brightness control with a setting range of 0 to 1 should not make the image brightest at a setting of 0.5, but should always increase brightness as the setting increases from 0 to 1. Examples of color grading operations that can be applied in the example-based color grading (115) and the example-based color grading (124) include, but are not limited to, adjustments to the following: ambient color temperature, contrast, brightness, hue, color saturation, highlights, shadows, white clipping, black clipping, slope (multiplying brightness and / or one or more chroma channels by a factor), offset (adding an offset to brightness and / or one or more chroma channels), etc.

[0022] After example-based color grading (115), image data of the final work (117) may be transmitted to an encoding block (120) for downstream transmission to decoding and playback devices such as computer monitors, televisions, set-top boxes, movie theaters, etc. In some embodiments, the encoding block (120) may include an audio encoder and a video encoder such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats to generate an encoded bitstream (122). In the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) representing an identical or nearly similar version of the signal (117). The receiver may be attached to a target display (140) which may have unique characteristics. In this case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137). The additional methods described herein may be performed by the decoding unit (130) or the display management block (135). Both the decoding unit (130) and the display management block (135) may include their own processors, or may be integrated into a single processing unit. Although the present disclosure refers to a target display (140), it should be understood that this is merely an example. It should be further understood that the target display (140) may include any device configured to display or project light; for example, a computer monitor, a television, an OLED display, an LCD display, a quantum dot display, a movie theater, consumer and other commercial projection systems, a head-up display, a virtual reality display, etc.

[0023] Figure 2A method (200) is provided that allows color grading based on similarity to an example frame. The method (200) may be performed by, for example, a processor as part of block (115) and / or block (120) for encoding.

[0024] At step (202), original video (or image) data is obtained. As an example, the original video (or image) data may be live content, such as live broadcast, game content, virtual reality content, etc.

[0025] At step (204), an example frame and predetermined color grading parameters are obtained. The example frame may be a frame from historical live content. In some embodiments, according to Figure 3 In some embodiments, these predetermined color grading parameters are obtained by Figure 1 The color grading station (125) is Figure 3 The color grading parameters obtained in step (204) may include any color grading parameters that are adjustable via the color grading station (125) or another color grading device.

[0026] At step (206), the method may involve determining a similarity between a frame of the original video (or image) obtained in step (202) and an example frame obtained in step (204). In some embodiments, a feature vector may be calculated for each image frame obtained in step (202) and for each example frame obtained in step (204). The feature vectors for the original video data may be calculated in real time or, in the case of processing recorded content rather than live content, at any time prior to step (206). As an example, the feature vectors may be calculated during step (204) or as part of step (206). Figure 3 As part of the operation of the method (300), a feature vector for an example frame is calculated. The feature vector can be used to determine the similarity between the frame of the original video (or image) and the example frame obtained in step (204). As an example, the feature vector for each frame may include a count of non-zero bins in a 3-D histogram bin count in a preferred color space, plus a mean and standard deviation of brightness in a predetermined number of horizontal regions divided from top to bottom in the frame.

[0027] Step (206) may include calculating a 3-D histogram for some or each of the image frames obtained from steps (204 and 206). Each 3-D histogram may include bars that divide the initial luminance range into downsampled ranges and divide the chrominance parameters (e.g., u' and v' parameters of the Commission Internationale de l'Eclairage CIE) into downsampled ranges. As an example, an image frame encoded using a perceptual quantizer based on Society of Motion Picture and Television Engineers ST.2084 (SMPTE 2084) may initially have a luminance range of 0 to 10,000 cd / m2 encoded using 10 to 16 bits. 2 The image frame of the present invention may be a luminance range, and may have a pair of chrominance parameters, each chrominance parameter being encoded using 10 to 16 bits. As part of calculating the characteristic vector, the initial luminance range may be divided into a finite number of ranges, such as 16 luminance bars (e.g., 4-bit depth), and each chrominance parameter may be similarly divided into a finite number of ranges, such as 8 chrominance parameter ranges (e.g., 3-bit depth). The divided luminance ranges may each be sized so that each luminance range spans a relatively similar perceptual difference. In other words, if the perceived brightness of the brightest value in a given divided luminance range is approximately twice the perceived brightness of the darkest value in the same luminance range, then the perceived brightness of the brightest value of the remaining luminance range may be expected to be approximately twice the darkest value of the luminance range. As an example, step (206) may include populating a 3-D histogram of some or all of the image frames obtained from steps (204 and 206), wherein a first dimension of the histogram relates to luminance, a second dimension relates to a first chrominance parameter (e.g., u'), and a third dimension relates to a second chrominance parameter (e.g., v'). Each dimension may include any desired number of bars. As an example, there may be 16 bars in the luminance dimension (e.g., 16 different luminance ranges), there may be 8 bars in the first chrominance dimension (e.g., 8 different ranges of u'), and there may be 8 bars in the second chrominance dimension (e.g., 8 different ranges of v').

[0028] Calculating a 3-D histogram for a given image frame (e.g., one of the image frames obtained from steps (204 and 206)) may include counting the number of pixels in the given image that are associated with each bin of the 3-D histogram. In other words, if a given bin in the 3-D histogram covers a luminance range, a chrominance range in the first chrominance dimension, and a given chrominance range in the second chrominance dimension, and the image frame in question includes X pixels that fall within all three separate ranges of the given bin, then the given bin may be assigned a value of X.

[0029] In some embodiments, the resolution of the image frame can be downsampled as part of calculating the characteristic vector of the image frame, which can reduce the processing load of the characteristic vector calculation. As an example, the image frame can be downsampled to a resolution of 100 times 100 pixels, 150 times 150 pixels, 180 times 180 pixels, 200 times 200 pixels, 250 times 250 pixels or another desired resolution. The downsampling of the image frame can be completed while maintaining or not maintaining the initial aspect ratio of the image frame. In such an embodiment, the pixel count in the 3-D histogram can be based on a downsampled version of the image frame.

[0030] Step (206) may also include calculating the average and standard deviation of brightness in a predetermined number of horizontal regions divided from top to bottom in the object frame. As an example, calculating the feature vector of a given image frame may involve dividing the given image frame into a predetermined number of horizontal regions, and then calculating the average brightness of each horizontal region and the standard deviation of the brightness of each horizontal region. For example, each of the horizontal regions may span the entire width of the object frame while spanning a corresponding portion of the vertical dimension. If desired, the horizontal regions may have similar or identical sizes (e.g., each horizontal region may span a similar or identical portion of the vertical dimension), or may have different sizes. As an example, the vertical span of a central horizontal region may be larger or smaller than the vertical span of another horizontal region below or above the central horizontal region. The horizontal regions may partially overlap, may surround each other without overlap or vertical gaps, or may be separated by vertical gaps. As an example, the predetermined number of horizontal regions may be three horizontal regions, four horizontal regions, five horizontal regions, six horizontal regions, seven horizontal regions, eight horizontal regions, nine horizontal regions, ten horizontal regions, or more than ten horizontal regions. If desired, the calculation of the mean and standard deviation of luminance in a predetermined number of horizontal regions may be performed after downsampling the original image (as discussed above).

[0031] Step (206) may also include comparing the feature vector of the original video frame with the feature vector of the example frame. As an example, the feature vector of some or each original video frame may be compared with the feature vector of the example frame. Determining the similarity between the feature vector of the original video frame and the feature vector of the example frame may include comparing the number of 3-D histogram bars of the two image frames and comparing the mean and standard deviation of the brightness of the horizontal region, and then combining the two comparisons.

[0032] As an example, comparing the number of 3-D histogram bars of two image frames can involve calculating the sum of the overlap of the number of 3-D histogram bars of the two image frames. As a specific example, if a given 3-D histogram bar has a non-zero value X for a given original video frame and a non-zero value Y for a given example frame, the smaller value of X and Y can be added to the sum of the similarities between the indicating frames. This calculation can be repeated for some or all of the 3-D histogram bars until the final sum of the overlaps in the 3-D histograms is obtained. If the two 3-D histograms are the same, the total will be equal to 100% (1.0). If there is no overlap in the two 3-D histograms, the total will be 0% (0.0). Most frames will have some common pixels and will therefore be between these two extremes and will therefore have a final normalized similarity of the number of 3-D histogram bars, ranging from 0.0 to 1.0.

[0033] As an example, comparing the mean and standard deviation of the brightness of the above-mentioned horizontal areas may involve calculating the normal distribution overlap between the mean and standard deviation of each horizontal area in the horizontal areas of the two image frames being compared. The normal distribution overlap between the matching horizontal areas of the two image frames may be a real value in the range of 0.0 to 1.0. In some embodiments, the normal distribution overlaps of different horizontal areas may be averaged together to produce an overall matching value also in the range of 0.0 to 1.0. If desired, the lowest horizontal area may have a first weight (W1), the second lowest horizontal area may have a second weight (W2), the central horizontal area may have a second weight (W3), the second highest horizontal area may have a fourth weight (W4), and the highest horizontal area may have a fifth weight (W5), where some or all weights may be unique. In some embodiments, the weight (W3) of the central horizontal area may be greater than the weights of other horizontal areas. As an example, calculating the normal distribution overlap of the horizontal areas of the two image frames may involve calculating the weighted average of the normal distribution overlap of each horizontal area in the horizontal area.

[0034] The output of the 3-D histogram bar comparison and the normal distribution overlap of the horizontal area can be combined using any desired weighting factor. As an example, the two metrics can be combined with an equal 1 / 2 weighting. In some embodiments, the two metrics can be combined with unequal weighting (e.g., the weighting of the 3-D histogram bar comparison can be higher or lower than the weighting of the normal distribution overlap). In some embodiments, the output of the 3-D histogram bar comparison includes a similarity value in the range of 0.0 to 1.0, and the normal distribution overlap of the horizontal area also includes a similarity value in the range of 0.0 to 1.0. When these two comparison metrics are aggregated with an equal 1 / 2 weighting, the final similarity value between a given original frame and a given example frame can also have a range of 0.0 to 1.0.

[0035] At step (208), the similarity values ​​between one or more original frames and the example frames and the predetermined color grading parameters associated with the example frames obtained in step (204) can be combined to determine the color grading parameters of the original frames. In some embodiments, the similarity values ​​obtained in step (206) can be used to assign weights to the predetermined color grading parameters associated with the example frames. If the original frame is exactly the same as the example frame (e.g., having a similarity value of 1.0), the set of predetermined color grading parameters for the example frame can have the maximum weight. Conversely, if the similarity value between the original frame and the example frame is 0.0, the set of predetermined color grading parameters for the example frame can have a zero weight. In some embodiments, the weights of the predetermined color grading parameters applied to the original frame i from the example frame, can be calculated using equation (1).

[0036] w(i,j)=s(i,j) / [Ks-s(i,j)] (1)

[0037] In equation (1), w(i, j) may represent the weight of the original frame i relative to the example frame j, s(i, j) may represent the similarity between the original frame i and the example frame, and K s It can be a predetermined constant value, and the predetermined constant value can be greater than 1.0. In some embodiments, K s may have a value of 1.05, which yields a maximum weight of 20. Other formulas for converting similarity values ​​into weights may be used instead of formula (1).

[0038] As discussed above, there may be M characteristic vectors (one for each of the M example image frames), and each characteristic vector may have an associated set of between 1 and N tone mapping parameter settings. Some characteristic vectors (e.g., some example images) may be associated with tone mapping parameter settings for all N possible tone mapping operations, while other characteristic vectors (e.g., other example images) may be associated with tone mapping parameter settings for less than N possible tone mapping operations, and may be associated with as few as a single tone mapping parameter setting for a single tone mapping operation of the N possible tone mapping operations. When a new raw image frame arrives (e.g., in step (302)), the new raw image frame may optionally be downsampled and its characteristic vector calculated. The characteristic vector of the new raw image frame is then compared with each of the M characteristic vectors of the example frames to obtain M similarity values. If desired, the M similarity values ​​may be converted into M weights (e.g., using formula (1) above). Then, as part of step (208), the M weights (or M similarity values) may be used to determine N color grading parameters for the N tone mapping operations of the new original image frame. In some embodiments, each of the N tone mapping parameters of the new original image frame may be calculated using a weighted average. As an example, the N tone mapping parameters of the new original image frame may be calculated using a weighted average of equation (2).

[0039] p(k,i)=[∑ j p(k,j)w(i,j)] / ∑ j w(i,j) (2)

[0040] In equation (2), p(k, i) represents the parameter settings for tone mapping operation k applied to new frame i, p(k, i) represents the parameter settings for tone mapping operation k applied to example frame i, and w(i, j) represents the weight of original frame i relative to example frame i (as discussed in conjunction with equation (1)). In some embodiments, only examples with settings provided or associated for the kth tone mapping operation are used in the weighted average of equation (2). In the event that there is no non-zero weight for a given tone mapping operation (e.g., there is no example that has both the provided settings for the kth tone mapping operation and a non-zero weight relative to the new frame), the method may instead assign a global default value for the kth tone mapping operation. The content creator may provide global default values ​​for any desired number of possible tone mapping operations among the N possible tone mapping operations.

[0041] At step (210), each of the tone mapping operations N can be applied to the raw video frame, wherein each tone mapping operation is applied using the associated parameter settings for each tone mapping operation determined in step (208). In this manner, the tone mapping operations can be applied to the raw video frame in real time without requiring manual review or input from a color grader. At step (212), the color graded video frame can be distributed for use and / or storage.

[0042] Figure 3 A method for identifying example frames and obtaining these example frames for use in Figure 2 Method (300) of color grading parameters used in method (200) of . Method (300) may be performed, for example, by a processor as part of block (124) and / or block (126). A content creator may have access to past live broadcasts or sample game footage (e.g., a set of possible example frames), but the content creator may not desire to individually color grade every single possible example frame. Method (300) may be an example of a method for reducing a set of possible example frames to a more limited and still appropriate set of example frames for individual color grading. As an example, it is desirable that the number of example frames that are individually color graded may be a manageable number while also being as different as possible to cover the widest possible range of raw video inputs in the future.

[0043] At step (302), raw sample video data may be obtained, wherein the raw sample video contains X possible sample frames. The raw sample video data may be associated with a future raw video to be color graded. As an example, if the raw video to be color graded is a future broadcast of a sporting event, step (302) may involve obtaining past broadcasts of similar sporting events (e.g., previous games at the same location, at a similar time of day, etc.). As another example, if the raw video to be color graded is game content from a particular game, step (302) may involve obtaining sample content from the particular game.

[0044] At step (304), characteristics vectors may be calculated for some or all of the original example video frames, wherein the characteristics vectors may be calculated as described in step (206) of method (300).

[0045] At step (306), similarity values ​​between the feature vectors and their associated original example video frames may be calculated (e.g., as described in step (206)). If desired, step (306) may involve filling a symmetric matrix with a 1.0 diagonal (e.g., the similarity of each frame to itself), with the remainder of the symmetric matrix indicating similarity values ​​between the feature vectors and their associated original example video frames.

[0046] At step (308), a style example frame may be identified and selected as the first example frame in a set of M final example frames. As an example, step (308) may involve finding which row of the symmetric matrix has the largest sum indicating the most average frame (e.g., style) of the original example video frames.

[0047] At step (310), a frame having the smallest maximum similarity to the example frame(s) in the set of final example frames may be identified, and then the frame may be added to the set of final example frames. As an example, step (310) may involve identifying which row in the symmetric matrix has the smallest maximum value (excluding the 1.0 diagonal), and the example frame associated with that row may be added to the set of final example frames.

[0048] As indicated by loop (312), step (310) may be repeated until an end condition is reached. Examples of potential end conditions include, but are not limited to, reaching a predetermined number (M) of final example frames and a candidate identified in step (310) having a similarity with a previous entry in the set of final example frames greater than a predetermined threshold. As an example, if the frame identified in step (310) as having the smallest maximum similarity to an example frame(s) in the set of final example frames has a maximum similarity of at least 0.7 to a frame already present in the set of final example frames, then method (300) may stop loop (312) and the final identified frame may or may not be added to the set of final example frames.

[0049] At step (314), color grading parameters for the example frames identified in step (310) may be obtained. As an example, the color grading parameters may be obtained by a color grader using a color grading station (125). In particular, the color grader may provide input via a control interface of the color grading station. If desired, step (310) and / or step (314) may also involve receiving user input to remove one or more example frames from the set of final example frames identified in step (310). In particular, method (300) may identify some example frames that are not particularly useful for color grading, such as frames during a gradient transition, completely black frames, etc. In general, excluding such frames from the smaller set of example frames identified in step (310) should be easier than excluding such frames from the original raw example frames obtained in step (302).

[0050] The above systems and methods can enable color grading of images and videos based on similarity to examples.Systems, methods and devices according to the present disclosure can adopt any one or more of the following configurations.

[0051] (1) A method for color grading a source image using multiple color grading operations, the method comprising: obtaining a source image for color grading; obtaining multiple color grading parameter settings associated with each example frame from a plurality of example frames, wherein each color grading parameter setting is used as an input parameter for an associated color grading operation in the plurality of color grading operations; obtaining multiple similarity measurements, each similarity measurement indicating a level of similarity between the source image and a different corresponding example frame from the plurality of example frames; obtaining multiple weighted averages based on the similarity measurements and the multiple color grading parameters, wherein each weighted average is associated with a different corresponding color grading operation in the plurality of color grading operations; applying the multiple color grading operations to the source image using each of the weighted averages as an input parameter for the color grading operation associated with each weighted average to produce a color graded image; and providing the color graded image.

[0052] (2) The method of (1), wherein the plurality of color grading operations include a brightness adjustment operation, a hue adjustment operation, and a saturation adjustment operation.

[0053] (3) The method of (1) or (2), wherein the plurality of color grading operations include a global brightness adjustment operation, a global hue adjustment operation, and a global saturation adjustment operation.

[0054] (4) The method of any one of (1) to (3), wherein the plurality of color grading operations include at least one regional brightness adjustment operation, at least one regional hue adjustment operation, and at least one regional saturation adjustment operation.

[0055] (5) A method according to any one of (1) to (4), wherein obtaining the multiple similarity measurement results includes: calculating multiple data structures, wherein each data structure is associated with a different one of the source image and the example frame from the multiple example frames, and wherein each data structure has multiple bars, each bar is associated with a unique chromaticity value range and a unique luminance value range, and each bar includes a count of the number of pixels in the image associated with the data structure that have a chromaticity value within the chromaticity value range associated with the bar and have a luminance value that is also within the luminance value range associated with the bar.

[0056] (6) A method according to (5), wherein each example frame from the plurality of example frames and the source image are encoded using a transfer function that covers a luminance range of X nits at a Y bit depth, and wherein the bars of the data structure cover the luminance range of X nits at a Z bit depth, wherein Z is no more than half of Y.

[0057] (7) A method according to (5) or (6), wherein each example frame from the multiple example frames and the source image are encoded using a transfer function having chrominance values, wherein the chrominance values ​​cover a given color space at a Y bit depth, and wherein the vertical bars of the data structure cover the given color space at a Z bit depth, wherein Z is greater than half of Y.

[0058] (8) A method according to any one of (5) to (7), wherein calculating the multiple data structures further includes: downsampling the source image before calculating the data structure associated with the source image, and downsampling the example frame from the multiple example frames before calculating the data structure associated with the source image.

[0059] (9) The method of any one of (5) to (8), wherein obtaining the plurality of similarity measurements further comprises identifying overlap between the data structure associated with the source image and the data structure associated with the example frame.

[0060] (10) A method according to any one of (1) to (9), wherein obtaining the multiple similarity measurement results further includes: dividing each example frame from the multiple example frames and the source image into multiple horizontal bands; obtaining the average value and standard deviation of the brightness within each of the horizontal bands; and calculating the overlap of the normal distribution of the average value and standard deviation of the brightness obtained within each of the horizontal bands.

[0061] (11) A method according to any one of (5) to (9), wherein obtaining the multiple similarity measurement results further includes: identifying the overlap between the data structure associated with the source image and the data structure associated with the example frame; dividing each example frame from the multiple example frames and the source image into multiple horizontal bands; obtaining the mean value and standard deviation of the brightness within each of the horizontal bands; calculating the overlap of the normal distribution of the obtained mean value and standard deviation of the brightness within each of the horizontal bands; and calculating the similarity measurement results by combining a first metric with a second metric, wherein the first metric is based on the identified overlap between the data structure associated with the source image and the data structure associated with the example frame, and the second metric is based on the calculated overlap of the normal distribution.

[0062] (12) A color grading system comprising: at least one controller, the at least one controller being configured to: obtain a source image for color grading; obtain a plurality of color grading parameter settings, wherein each color grading parameter setting is used as an input to a color grading operation, and wherein each color grading parameter setting is associated with a different corresponding example frame from a plurality of example frames; calculate a plurality of similarity measurements, each similarity measurement indicating a level of similarity between the source image and a different corresponding example frame from the plurality of example frames; calculate a weighted average based on the plurality of similarity measurements and the plurality of color grading parameter settings; apply the color grading operation to the source image using the weighted average as an input parameter of the color grading operation to produce a color graded image; and provide the color graded image.

[0063] (13) The system of (12), further comprising a display, wherein the controller is configured to provide the color graded image to the display, and wherein the display is configured to display the color graded image.

[0064] (14) The system of (13), wherein the source image comprises live content, and wherein the at least one controller is configured to provide the color graded image to the display within 200 milliseconds of obtaining the source image.

[0065] (15) The system of any one of (12) to (14), wherein the at least one controller is configured to obtain the plurality of color grading parameter settings prior to creating the source image.

[0066] (16) A method comprising: obtaining a sequence of image frames; obtaining multiple similarity measurement results, each similarity measurement result indicating a similarity level between a pair of different corresponding image frames from the sequence of image frames; and selecting a group of example frames from the sequence of image frames by: (i) adding a first image frame from the sequence of image frames to the group of example frames; (ii) based on the similarity measurement results, identifying which image frame among the image frames that have not yet been added to the group of example frames has the least similarity to the (multiple) image frames in the group of example frames; (iii) adding the image frame identified in (ii) to the group of example frames; and (iv) repeating (ii) and (iii) until a completion condition is satisfied.

[0067] (17) The method of (16), wherein repeating (ii) and (iii) until the completion condition is satisfied comprises repeating (ii) and (iii) until at least a predetermined number of example frames are added to the set of example frames.

[0068] (18) A method according to (16) or (17), wherein repeating (ii) and (iii) until the completion condition is met includes: repeating (ii) and (iii) until the image frame identified in (ii) has a similarity measurement result greater than a predetermined threshold with an image frame that has been added to the group of exemplary frames.

[0069] (19) A method according to any one of (16) to (18), wherein adding the first image frame from the image frame sequence to the group of example frames includes: adding the most average frame from the image frame sequence to the group of example frames.

[0070] (20) A method according to any one of (16) to (19), wherein adding the first image frame from the image frame sequence to the group of example frames includes: adding a randomly selected frame from the image frame sequence to the group of example frames.

[0071] (21) A method according to any one of (16) to (20), wherein (ii) based on the similarity measurement result, identifying which image frame among the image frames that have not yet been added to the group of example frames has the smallest similarity to the (multiple) image frames in the group of example frames includes: identifying which image frame among the image frames that have not yet been added to the group of example frames has the smallest maximum similarity with respect to any image frame among the (multiple) image frames in the group of example frames.

[0072] (22) The method according to any one of (16) to (21), wherein obtaining the multiple similarity measurement results includes: calculating a feature vector for each image frame in the image frame sequence.

[0073] (23) A method according to (22), wherein calculating the characteristic vector of a given image frame includes: downsampling the given image to produce a downsampled image having a lower resolution than the given image; and calculating the average brightness of multiple horizontal regions of the downsampled image and the standard deviation of the brightness.

[0074] (24) A method according to (22), wherein the image frame is encoded using a transfer function, the transfer function having a luminance range encoded using X bits and having at least two chrominance parameters each encoded using Y bits, and wherein calculating the characteristic vector for a given image frame includes: downsampling the given image to produce a downsampled image having a lower resolution than the given image; calculating an average brightness and a standard deviation of the brightness for multiple horizontal regions of the downsampled image; and calculating a data structure from the downsampled image, wherein the data structure includes a plurality of bars, wherein the luminance range of the transfer function is encoded using A bits in the data structure, wherein the at least two chrominance parameters are encoded using B bits in the data structure, wherein A is less than X and B is less than Y, and wherein calculating the data structure includes: counting the number of pixels associated with each bar from the downsampled image.

[0075] (25) The method according to any one of (16) to (24), further comprising: after the completion condition is met, obtaining at least one color grading parameter setting for each example frame in the group of example frames, wherein each of the color grading parameter settings is used as an input for a color grading operation.

[0076] (26) The method according to (25) further includes: obtaining a source image for color grading; calculating multiple additional similarity measurement results, each additional similarity measurement result indicating a similarity level between the source image and a different corresponding example frame from the group of example frames; calculating a weighted average based on the multiple similarity measurement results and the color grading parameter settings associated with the example frames in the group of example frames; applying the color grading operation to the source image using the weighted average as an input parameter of the color grading operation to produce a color graded image; and providing the color graded image.

[0077] (27) A computer program product having instructions, which, when executed by a computing system, cause the computing system to perform the method described in any one of (1) to (11) or (16) to (26).

[0078] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a particular ordered order, these processes can be practiced with the described steps performed in an order different from that described herein. It should further be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0079] Accordingly, it should be understood that the above description is intended to be illustrative rather than restrictive. By reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the full range of equivalents to which the appended claims and these claims are eligible for consideration. It is expected and desired that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In a word, it should be understood that the present application is capable of modification and change.

[0080] All terms used in the claims are intended to be given their broadest reasonable interpretations and ordinary meanings as understood by those skilled in the art in view of the description herein, unless an explicit indication to the contrary appears herein. In particular, the use of singular articles such as "a", "an", "the", "said", etc. should be understood to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0081] The abstract of the present disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. This abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together into various embodiments for the purpose of connecting the present disclosure as a whole. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments incorporate more features than those explicitly recited in each claim. On the contrary, as reflected in the attached claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Therefore, the attached claims are hereby incorporated into the detailed description, and each claim stands alone as a separately claimed subject matter.

Claims

1. A computer-implemented method for color grading a source image using a plurality of color grading operations, the method comprising: Obtaining source images for color grading; obtaining, in association with each example frame from the plurality of example frames, a plurality of color grading parameter settings, wherein each color grading parameter setting is used as an input parameter for an associated color grading operation of the plurality of color grading operations; obtaining a plurality of similarity measurements, each similarity measurement indicating a level of similarity between the source image and a different corresponding example frame from the plurality of example frames; obtaining a plurality of weighted averages based on the similarity measurement and the plurality of color grading parameters, wherein each weighted average is associated with a different corresponding color grading operation of the plurality of color grading operations; applying the plurality of color grading operations to the source image using each of the weighted averages as an input parameter to its associated color grading operation to produce a color graded image; and The color graded image is provided.

2. The method of claim 0, wherein: The plurality of color grading operations include a brightness adjustment operation, a hue adjustment operation, and a saturation adjustment operation.

3. The method according to claim 0 or 2, wherein: The plurality of color grading operations include a global brightness adjustment operation, a global hue adjustment operation, and a global saturation adjustment operation.

4. The method according to any one of claims 0 to 3, wherein: The plurality of color grading operations include at least one regional light brightness adjustment operation, at least one regional hue adjustment operation, and at least one regional saturation adjustment operation.

5. The method according to any one of claims 0 to 4, wherein: Obtaining the plurality of similarity measurement results comprises: A plurality of data structures are calculated, wherein each data structure is associated with a different one of the source image and an example frame from the plurality of example frames, and wherein each data structure has a plurality of bins, each bin being associated with a unique range of chrominance values ​​and a unique range of luminance values, and each bin comprising a count of the number of pixels in the image associated with the data structure that have chrominance values ​​within the range of chrominance values ​​associated with the bin and that have luminance values ​​that are also within the range of luminance values ​​associated with the bin.

6. The method of claim 0, wherein: Each example frame from the plurality of example frames and the source image are encoded using a transfer function that covers a luminance range of X nits at a Y bit depth, and wherein the bars of the data structure cover the luminance range of X nits at a Z bit depth, wherein Z is no more than half of Y.

7. The method according to claim 0 or 6, wherein: Each example frame from the plurality of example frames and the source image are encoded using a transfer function having chrominance values ​​covering a given color space at a Y bit depth, and wherein the bars of the data structure cover the given color space at a Z bit depth, wherein Z is greater than half of Y.

8. The method according to any one of claims 0 to 7, wherein: Computing the plurality of data structures further comprises downsampling the source image prior to computing the data structure associated with the source image, and downsampling the example frame from the plurality of example frames prior to computing the data structure associated with the source image.

9. The method according to any one of claims 0 to 8, wherein: Obtaining the plurality of similarity measurements further includes identifying overlap between the data structure associated with the source image and the data structure associated with the example frame.

10. The method according to any one of claims 1 to 9, wherein: Obtaining the plurality of similarity measurement results further comprises: dividing each example frame from the plurality of example frames and the source image into a plurality of horizontal bands; Obtaining a mean value and a standard deviation of the light brightness within each of the horizontal bands; and The overlap of the normal distribution of the mean and standard deviation of the obtained light intensity within each of the horizontal bands is calculated.

11. The method according to any one of claims 0 to 9, wherein: Obtaining the plurality of similarity measurement results further comprises: identifying an overlap between the data structure associated with the source image and the data structure associated with the example frame; dividing each example frame from the plurality of example frames and the source image into a plurality of horizontal bands; Obtaining a mean value and a standard deviation of the light brightness in each of the horizontal bands; calculating an overlap of normal distributions of the mean and standard deviation of the obtained light intensity within each of the horizontal bands; and The similarity measure is calculated by combining a first metric based on the identified overlap between the data structure associated with the source image and the data structure associated with the example frame with a second metric based on the calculated overlap of the normal distribution.

12. A color grading system comprising: At least one controller, the at least one controller being configured to: Obtaining source images for color grading; obtaining a plurality of color grading parameter settings, wherein each color grading parameter setting is used as an input to a color grading operation, and wherein each color grading parameter setting is associated with a different corresponding example frame from the plurality of example frames; calculating a plurality of similarity measurements, each similarity measurement indicating a level of similarity between the source image and a different corresponding example frame of the plurality of example frames; calculating a weighted average based on the plurality of similarity measurements and the plurality of color grading parameter settings; applying the color grading operation to the source image using the weighted average as an input parameter of the color grading operation to produce a color graded image; and The color graded image is provided.

13. The color grading system of claim 0, further comprising a display, wherein: The controller is configured to provide the color graded image to the display, and wherein the display is configured to display the color graded image.

14. The color grading system of claim 0, wherein: The source image comprises live content, and wherein the at least one controller is configured to provide the color graded image to the display within 200 milliseconds of obtaining the source image.

15. A color grading system according to any one of claims 0 to 14, wherein: The at least one controller is configured to obtain the plurality of color grading parameter settings prior to creating the source image.

16. A method comprising: Obtaining an image frame sequence; obtaining a plurality of similarity measurements, each similarity measurement indicating a level of similarity between a pair of different corresponding image frames from the sequence of image frames; as well as A set of example frames are selected from the sequence of image frames by: (i) adding a first image frame from the sequence of image frames to the set of example frames; (ii) identifying, based on the similarity measurement result, which image frame among the image frames that have not yet been added to the set of example frames has the smallest similarity to the image frame(s) in the set of example frames; (iii) adding the image frame identified in (ii) to the set of example frames; as well as (iv) Repeat (ii) and (iii) until the completion condition is met.

17. The method of claim 0, wherein: Repeating (ii) and (iii) until the completion condition is satisfied includes repeating (ii) and (iii) until at least a predetermined number of example frames are added to the set of example frames.

18. The method of claim 0 or 17, wherein: Repeating (ii) and (iii) until the completion condition is met includes repeating (ii) and (iii) until the image frame identified in (ii) has a similarity measurement result greater than a predetermined threshold with an image frame that has been added to the set of exemplary frames.

19. The method according to any one of claims 0 to 18, wherein: Adding the first image frame from the sequence of image frames to the set of example frames includes adding a most average frame from the sequence of image frames to the set of example frames.

20. The method of any one of claims 0 to 19, wherein: Adding the first image frame from the sequence of image frames to the set of example frames includes adding a randomly selected frame from the sequence of image frames to the set of example frames.

21. The method of any one of claims 0 to 20, wherein: (ii) based on the similarity measurement result, identifying which image frame among the image frames that have not yet been added to the group of example frames has the smallest similarity to the (multiple) image frames in the group of example frames, including: identifying which image frame among the image frames that have not yet been added to the group of example frames has the smallest maximum similarity with respect to any image frame among the (multiple) image frames in the group of example frames.

22. The method of any one of claims 0 to 21, wherein: Obtaining the plurality of similarity measurement results includes: calculating a characteristic vector of each image frame in the image frame sequence.

23. The method of claim 0, wherein: Calculating the characteristic vector for a given image frame includes: downsampling the given image to generate a downsampled image having a lower resolution than the given image; and The average brightness of a plurality of horizontal regions of the downsampled image and the standard deviation of the brightness are calculated.

24. The method of claim 0, wherein: The image frame is encoded using a transfer function having a luminance range encoded using X bits and having at least two chrominance parameters each encoded using Y bits, and wherein calculating the characteristics vector for a given image frame comprises: downsampling the given image to generate a downsampled image having a lower resolution than the given image; calculating average brightness of a plurality of horizontal regions of the downsampled image and a standard deviation of the brightness; and A data structure is calculated from the downsampled image, wherein the data structure includes a plurality of bars, wherein a luminance range of the transfer function is encoded in the data structure using A bits, wherein the at least two chrominance parameters are encoded in the data structure using B bits, wherein A is less than X and B is less than Y, and wherein calculating the data structure includes counting a number of pixels associated with each bar from the downsampled image.

25. The method according to any one of claims 0 to 24, further comprising: After the completion condition is met, at least one color grading parameter setting is obtained for each example frame in the set of example frames, wherein each of the color grading parameter settings is used as an input to a color grading operation.

26. The method of claim 0, further comprising: Obtaining source images for color grading; calculating a plurality of additional similarity measures, each additional similarity measure indicating a level of similarity between the source image and a different corresponding example frame from the set of example frames; calculating a weighted average based on the plurality of similarity measurements and the color grading parameter settings associated with the example frames in the set of example frames; applying the color grading operation to the source image using the weighted average as an input parameter of the color grading operation to produce a color graded image; and providing the color graded image.

27. A computer program product having instructions which, when executed by a computing system, cause the computing system to perform the method of any one of claims 1 to 11 or 16 to 26.

Images